There are three general causes of router performance issues:
1. High CPU load
2. Router packet switching mode
3. Excessive memory use
Processes that can be a cause of high CPU load:
• ARP Input process — Heavy traffic load can cause a the ARP Input process to spike.
• Net Background process — If an interface has full buffers but still needs to use a globally available buffer, the Net Background process handles it. There might also be a corresponding rise in throttles, ignored and overrun parameters of the sh int command.
• IP Background process — When an interface changes state this process does the work.
Commands to troubleshoot the different processes:
sh process cpu | inc [process name] — Is there an inordinate amount of CPU usage?
sh arp — Are there too many entries in the ARP table that would cause the ARP Input process to spike?
sh int fa0/0 — Are the throttles, ignored and overrun parameters climbing?
sh ctp stat — A high connection load can result in the TCP Timer process spiking.
R4#sh arp Protocol Address Age (min) Hardware Addr Type Interface Internet 10.1.4.10 53 0018.1825.2543 ARPA FastEthernet0/1 Internet 10.1.4.9 - 001b.d421.480b ARPA FastEthernet0/1 Internet 10.1.4.6 68 0012.d9a5.1542 ARPA FastEthernet0/0 Internet 10.1.4.5 - 001b.d421.480a ARPA FastEthernet0/0
sh int fa0/0 | inc thrott|ignor|over
R4#sh int fa0/0 | inc thrott|ignor|over Received 851934 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
sh tcp stat
R4#sh tcp stat Rcvd: 432 Total, 0 no port 0 checksum error, 0 bad offset, 0 too short 267 packets (8194 bytes) in sequence 1 dup packets (172 bytes) 0 partially dup packets (0 bytes) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) with data after window 0 packets after close 0 window probe packets, 0 window update packets 0 dup ack packets, 0 ack packets with unsend data 304 ack packets (10167 bytes) Sent: 440 Total, 0 urgent packets 51 control packets (including 0 retransmitted) 262 data packets (10120 bytes) 0 data packets (0 bytes) retransmitted 0 data packets (0 bytes) fastretransmitted 127 ack only packets (65 delayed) 0 window probe packets, 0 window update packets 27 Connections initiated, 0 connections accepted, 24 connections established 30 Connections closed (including 1 dropped, 3 embryonic dropped) 0 Total rxmt timeout, 0 connections dropped in rxmt timeout 0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive
Packet Switching Modes
• Process switching — The CPU is used to make packet switching decisions, the entire data flow is processed in the control plane. To turn on process switching issue the command no ip route-cache.
• Fast Switching — The CPU processes the first packet of a data flow, the rest are handled by the fast cache, reducing processor load. Turn on fast switching with the command ip route-cache.
• Cisco Express Forwarding (CEF) — CEF maintains the Forwarding Information Base (FIB) for layer 3 forwarding and the Adjacency Table for layer 2 next hops. The entire flow is processed in the data plane.
Packet Switching commands that look interesting to me:
sh ip int fa0/0 — Displays the packet switching mode.
sh ip cache — If fast switching is enabled it displays the fast cache.
sh ip cef — Displays the FIB contents.
sh adjacency det — Displays the adjacency table of a router if CEF is enabled.
sh ip int fa0/0 | inc IP
R4#sh ip int fa0/0 | inc IP IP fast switching is enabled IP fast switching on the same interface is disabled IP Flow switching is enabled IP CEF switching is enabled IP CEF Flow Fast switching turbo vector IP multicast fast switching is enabled IP multicast distributed fast switching is disabled IP route-cache flags are Fast, Flow cache, CEF, Subint Flow IP output packet accounting is disabled IP access violation accounting is disabled TCP/IP header compression is disabled RTP/IP header compression is disabled
sh ip cache
R4#sh ip cache IP routing cache 0 entries, 0 bytes 0 adds, 0 invalidates, 0 refcounts Minimum invalidation interval 2 seconds, maximum interval 5 seconds, quiet interval 3 seconds, threshold 0 requests Invalidation rate 0 in last second, 0 in last 3 seconds Last full cache invalidation occurred 5w1d ago Prefix/Length Age Interface Next Hop
sh ip cef
R4#sh ip cef Prefix Next Hop Interface 0.0.0.0/0 0.0.0.0 Null0 0.0.0.0/8 drop 0.0.0.0/32 receive 10.1.1.0/30 10.1.1.9 Serial0/0/0.34 10.1.1.4/30 10.1.1.9 Serial0/0/0.34 10.1.1.8/30 attached Serial0/0/0.34 10.1.1.8/32 receive 10.1.1.10/32 receive
sh adj det
R4#sh adj det Protocol Interface Address IP FastEthernet0/1 10.1.4.10(16) 0 packets, 0 bytes 001818252543001BD421480B0800 ARP 01:41:49 Epoch: 0 IP FastEthernet0/0 10.1.4.6(16) 17 packets, 1818 bytes 0012D9A51542001BD421480A0800 ARP 02:47:29 Epoch: 0 IP Serial0/0/0.34 point2point(13) 80697 packets, 102706013 bytes 64310800 CEF expires: 00:02:53 refresh: 00:00:53 Epoch: 0
Troubleshoot Memory Usage
• Memory leak — When not all memory used by a process is returned to the memory pool.
• Memory allocation failure — Shows up as a MALLOCFAIL error message.
• Bufffer leak — Similar to a memory leak, a process does not return a buffer after use.
• Runaway process — A process is consuming an inordinate amount of memory.
Commands to troubleshoot memory problems
sh buffers — Check the number of free in the list.
sh processes memory sorted — What are the largest memory users?
R4#sh buffers Buffer elements: 1119 in free list (1119 max allowed) 7519248 hits, 0 misses, 619 created Public buffer pools: Small buffers, 104 bytes (total 50, permanent 50): 49 in free list (20 min, 150 max allowed) 5531838 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Middle buffers, 600 bytes (total 25, permanent 25, peak 64 @ 3w4d): 25 in free list (10 min, 150 max allowed) 362047 hits, 14 misses, 42 trims, 42 created 0 failures (0 no memory)
sh processes memory sorted
R4#sh processes memory sorted Processor Pool Total: 307697040 Used: 14563424 Free: 293133616 I/O Pool Total: 41943040 Used: 4264864 Free: 37678176 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 27799336 6726984 14720188 0 0 *Init* 39 0 654476 1272 635204 0 0 USB Startup 26 0 67553324 66736804 605916 46720 0 Exec 0 0 0 0 393528 0 0 *MallocLite*