Layer 3 Switch Troubleshooting

Posted on April 2, 2010 by Judson Bishop

The big topics of this chapter include HSRP, VRRP, GLBP, TCAM and the port error table. I broke out my notes from the BCMSN and I’ll summarize each topic from them, then move into the troubleshooting of each protocol.

Hot Standby Router Protocol (HSRP)
Cisco proprietary protocol to allow multiple routers to appear as a single gateway IP address. EAch router is assigned to a common HSRP group one active router and one standby router, the others, up to 16 group members, just listen. There is no preemption without configuration.

  • Election of one active router — Based on priority that is assigned when the router is configured for the group. Default priority is 100, range is from 0 – 255, highest priority for election is 255. Highest IP address on HSRP interface breaks ties.

  • Election of one standby router — Second highest priority. Listens to ehloo messages, by default every 3 seconds with hold time of 10 seconds.

  • Virtual router — IP and MAC address pair that end devices use as a gateway. The active router processes all packets and frames to this address, the virtual router processes no physical frames.

    • Troubleshoot
    sh standby …
    sh standby delay

    sh standby
    [code]
    DSW1#sh standby
    Vlan10 – Group 10
    State is Active
    Virtual IP address is 10.2.1.254
    Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
    Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.704 secs
    Preemption enabled
    Active router is local
    Standby router is 10.2.1.2, priority 100 (expires in 10.480 sec)
    Priority 200 (configured 200)
    Group name is "hsrp-Vl10-10" (default)
    [/code]

    sh standby br
    [code]
    DSW1#sh standby br
    P indicates configured to preempt.
    |
    Interface Grp Pri P State Active Standby Virtual IP
    Vl10 10 200 P Active local 10.2.1.2 10.2.1.254
    [/code]

    sh standby delay
    [code]
    DSW1#sh standby delay
    Interface Minimum Reload
    Vlan10 1 5
    [/code]

    Virtual Router Redundancy Protocol (VRRP)
    VRRP is on open protocol that provides redundancy for the real IP address of a router or the virtual address of a group. By default all VRRP routers are configured to preempt the current master if their priority is greater.

  • Election of active router — If a real IP address is used, the master is the router with that IP. If a virtual one is used the master becomes the one with the highest priority, all others are in backup state. VRRP group numbers are from 0 to 255, router priority is from 1 to 254 with 254 being the highest and default priority is 100.

    • Troubleshoot
    sh vrrp
    sh vrrp brief

    sh vrrp
    [code]
    R7#sh vrrp
    FastEthernet0/0 – Group 1
    "Just playing"
    State is Master
    Virtual IP address is 10.2.1.253
    Virtual MAC address is 0000.5e00.0101
    Advertisement interval is 1.000 sec
    Preemption enabled
    Priority is 200
    Master Router is 10.2.1.247 (local), priority is 200
    Master Advertisement interval is 1.000 sec
    Master Down interval is 3.218 sec
    [/code]

    sh vrrp brie
    [code]
    R8#sh vrrp br
    Interface Grp Pri Time Own Pre State Master addr Group addr
    Fa0/0 1 150 3414 Y Backup 10.2.1.247 10.2.1.253
    [/code]

    Gateway Load Balancing Protocol (GLBP)
    Cisco proprietary protocol that is more robust than other redundancy protocols. When a client sends an ARP request looking for the router address, GLBP send back a reply with the virtual MAC supported by one of the routers in the GLBP group.

  • GLBP Active Virtual Gateway (GLBP-AVG) — Group members elect the AVG which replies to all host ARP requests. It has either the highest priority value or the highest IP address in the group is the tie breaker. Priority ranges from 0 to 255 with 255 being highest, 100 is the default. AVG assigns the virtual router address and assigns the secondary roles. Up to four virtual MAC address can be used in any group.
  • GLPB Active Virtual Forwarder (GLBP-AVF) — Each gateway assumes responsibility for forwarding packets that are sent to the virtual MAC address assigned to that gateway. GLBP load balances across the AVFs in round robin by default. AVFs are expected to send hellos to every other GLBP peer. If the hold time on the AVG expires without receiving a hello packet from an AVF the virtual MAC is assigned to another router.

    • Troubleshoot:
    sh glbp [brief]

    sh glbp
    [code]
    R8(config-if)#do sh glbp
    FastEthernet0/0 – Group 1
    State is Active
    2 state changes, last state change 00:03:58
    Virtual IP address is 10.2.1.252
    Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.776 secs
    Redirect time 600 sec, forwarder time-out 14400 sec
    Preemption enabled, min delay 0 sec
    Active is local
    Standby is 10.2.1.247, priority 150 (expires in 7.744 sec)
    Priority 200 (configured)
    Weighting 100 (default 100), thresholds: lower 1, upper 100
    Load balancing: round-robin
    Group members:
    0004.dd69.fd01 (10.2.1.248) local
    00d0.bbef.64c1 (10.2.1.247)
    There are 2 forwarders (1 active)
    Forwarder 1
    State is Active
    1 state change, last state change 00:03:48
    MAC address is 0007.b400.0101 (default)
    Owner ID is 0004.dd69.fd01
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
    Forwarder 2
    State is Listen
    MAC address is 0007.b400.0102 (learnt)
    Owner ID is 00d0.bbef.64c1
    Redirection enabled, 597.584 sec remaining (maximum 600 sec)
    Time to live: 14397.584 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 10.2.1.247 (primary), weighting 100 (expires in 7.584 sec)
    [/code]

    Ternary Content Addressable Memory (TCAM):
    TCAM works with CEF to make extremely fast forwarding decisions. TCAM entries are composed of Value, Mask and Result combinations.

  • Values — Always 134 bits consisting of source and destination addresses along with relevant protocol infromation.
  • Masks — Also 134 bits set to match value bits, or not match value bits that do not matter. This is so that a switch can do a bitwise and with access lists or access control entities (ACE).
  • Results — Numerical values that represent what action to take after the TCAM lookup.

    • When the TCAM cannot process the packet it is punted to the CPU for processing. Here are some reasons why a packet is punted:

  • Protocols that send multicast or broadcast packets.
  • Administrative access to a switch such as telnet or ssh.
  • Packets that use features not supported in hardware.
  • A switch’s TCAM has reached capacity. TCAM might reach capacity with large routing tables or a large number of ACLs.

    Troubleshoot TCAM:
    3550:
    show tcam
    3560:
    show platform tcam
    show platform ip unicast counts

    3550#sh tcam inacl 1 statistics
    [code]
    3550#sh tcam inacl 1 statistics
    Ingress ACL TCAM#1: Number of active labels: 3
    Ingress ACL TCAM#1: Number of masks allocated: 14, available: 194
    Ingress ACL TCAM#1: Number of entries allocated: 17, available: 1647
    [/code]

    3560#sh platform tcam utilization
    [code]
    3560#sh platform tcam utilization

    CAM Utilization for ASIC# 0 Max Used
    Masks/Values Masks/values

    Unicast mac addresses: 544/4352 25/76
    IPv4 IGMP groups + multicast routes: 144/1152 6/26
    IPv4 unicast directly-connected routes: 544/4352 25/76
    IPv4 unicast indirectly-connected routes: 176/1408 15/69
    IPv6 Multicast groups: 544/4352 25/76
    IPv6 unicast directly-connected routes: 544/4352 25/76
    IPv6 unicast indirectly-connected routes: 262/2096 11/42
    IPv4 policy based routing aces: 256/256 2/2
    IPv4 qos aces: 768/768 324/324
    IPv4 security aces: 512/512 27/27
    IPv6 policy based routing aces: 0/0 0/0
    IPv6 qos aces: 0/0 0/0
    IPv6 security aces: 204/510 5/5

    Note: Allocation of TCAM entries per feature uses
    a complex algorithm. The above information is meant
    to provide an abstract view of the current TCAM utilization
    [/code]

    Port Error Table:

    Error

    		Description

    		Cause
    Xmit-Err

    		Transmit buffer overlflow

    		Check for speed mismatch.
    Rcv-Err

    		Receive buffer overflow

    		Check for duplex mismatch.
    UnderSize

    		Has valid checksum but less than 64 bytes.

    		Host is sending invalid frames.
    Single-Col

    		A single collision occurred before the port transmitted the frame.

    		High bandwidth utilization or duplex mismatch.
    Multi-Col

    		Multiple collisions occur before the port transmitted the frame.

    		High bandwidth utilization or duplex mismatch.
    Late-Col

    		Collision detected further into frame being forwarded.

    		Cable too long or duplex mismatch.
    Excess-Col

    		Frame has experienced sixteen successive collisions, resulting in the frame being dropped.

    		High bandwidth, duplex mismatch or too many devices on a segment.
    Carri-Sen

    		Normal operation on a half-duplex link.

    		This is not a problem.
    Runts

    		Frames smaller than 64 bytes with a bad CRC.

    		Duplex mismatch or layer 1 problem.
    Giants

    		Frame size greater than 1518 bytes with a bad FCS.

    		Bad host NIC.

    Source: TSHOOT p.125

    • This entry was posted in Routing. Bookmark the permalink.

    Leave a Reply