Troubleshoot Router Performance

There are three general causes of router performance issues:
1. High CPU load
2. Router packet switching mode
3. Excessive memory use

Processes that can be a cause of high CPU load:
• ARP Input process — Heavy traffic load can cause a the ARP Input process to spike.
• Net Background process — If an interface has full buffers but still needs to use a globally available buffer, the Net Background process handles it. There might also be a corresponding rise in throttles, ignored and overrun parameters of the sh int command.
• IP Background process — When an interface changes state this process does the work.

Commands to troubleshoot the different processes:
sh process cpu | inc [process name] — Is there an inordinate amount of CPU usage?
sh arp — Are there too many entries in the ARP table that would cause the ARP Input process to spike?
sh int fa0/0 — Are the throttles, ignored and overrun parameters climbing?
sh ctp stat — A high connection load can result in the TCP Timer process spiking.

sh arp

R4#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.1.4.10              53   0018.1825.2543  ARPA   FastEthernet0/1
Internet  10.1.4.9                -   001b.d421.480b  ARPA   FastEthernet0/1
Internet  10.1.4.6               68   0012.d9a5.1542  ARPA   FastEthernet0/0
Internet  10.1.4.5                -   001b.d421.480a  ARPA   FastEthernet0/0

sh int fa0/0 | inc thrott|ignor|over

R4#sh int fa0/0 | inc thrott|ignor|over
     Received 851934 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

sh tcp stat

R4#sh tcp stat 
Rcvd: 432 Total, 0 no port
      0 checksum error, 0 bad offset, 0 too short
      267 packets (8194 bytes) in sequence
      1 dup packets (172 bytes)
      0 partially dup packets (0 bytes)
      0 out-of-order packets (0 bytes)
      0 packets (0 bytes) with data after window
      0 packets after close
      0 window probe packets, 0 window update packets
      0 dup ack packets, 0 ack packets with unsend data
      304 ack packets (10167 bytes)
Sent: 440 Total, 0 urgent packets
      51 control packets (including 0 retransmitted)
      262 data packets (10120 bytes)
      0 data packets (0 bytes) retransmitted
      0 data packets (0 bytes) fastretransmitted
      127 ack only packets (65 delayed)
      0 window probe packets, 0 window update packets
27 Connections initiated, 0 connections accepted, 24 connections established
30 Connections closed (including 1 dropped, 3 embryonic dropped)
0 Total rxmt timeout, 0 connections dropped in rxmt timeout
0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive

Packet Switching Modes
• Process switching — The CPU is used to make packet switching decisions, the entire data flow is processed in the control plane. To turn on process switching issue the command no ip route-cache.
• Fast Switching — The CPU processes the first packet of a data flow, the rest are handled by the fast cache, reducing processor load. Turn on fast switching with the command ip route-cache.
• Cisco Express Forwarding (CEF) — CEF maintains the Forwarding Information Base (FIB) for layer 3 forwarding and the Adjacency Table for layer 2 next hops. The entire flow is processed in the data plane.

Packet Switching commands that look interesting to me:
sh ip int fa0/0 — Displays the packet switching mode.
sh ip cache — If fast switching is enabled it displays the fast cache.
sh ip cef — Displays the FIB contents.
sh adjacency det — Displays the adjacency table of a router if CEF is enabled.

sh ip int fa0/0 | inc IP

R4#sh ip int fa0/0 | inc IP
  IP fast switching is enabled
  IP fast switching on the same interface is disabled
  IP Flow switching is enabled
  IP CEF switching is enabled
  IP CEF Flow Fast switching turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, Flow cache, CEF, Subint Flow
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled

sh ip cache

R4#sh ip cache 
IP routing cache 0 entries, 0 bytes
   0 adds, 0 invalidates, 0 refcounts
Minimum invalidation interval 2 seconds, maximum interval 5 seconds,
   quiet interval 3 seconds, threshold 0 requests
Invalidation rate 0 in last second, 0 in last 3 seconds
Last full cache invalidation occurred 5w1d ago

Prefix/Length           Age       Interface       Next Hop

sh ip cef

R4#sh ip cef
Prefix              Next Hop             Interface
0.0.0.0/0           0.0.0.0              Null0
0.0.0.0/8           drop
0.0.0.0/32          receive
10.1.1.0/30         10.1.1.9             Serial0/0/0.34
10.1.1.4/30         10.1.1.9             Serial0/0/0.34
10.1.1.8/30         attached             Serial0/0/0.34
10.1.1.8/32         receive
10.1.1.10/32        receive

sh adj det

R4#sh adj det
Protocol Interface                 Address
IP       FastEthernet0/1           10.1.4.10(16)
                                   0 packets, 0 bytes
                                   001818252543001BD421480B0800
                                   ARP        01:41:49  
                                   Epoch: 0
IP       FastEthernet0/0           10.1.4.6(16)
                                   17 packets, 1818 bytes
                                   0012D9A51542001BD421480A0800
                                   ARP        02:47:29  
                                   Epoch: 0
IP       Serial0/0/0.34            point2point(13)
                                   80697 packets, 102706013 bytes
                                   64310800
                                   CEF   expires: 00:02:53
                                   	 refresh: 00:00:53
                                   Epoch: 0

Troubleshoot Memory Usage
• Memory leak — When not all memory used by a process is returned to the memory pool.
• Memory allocation failure — Shows up as a MALLOCFAIL error message.
• Bufffer leak — Similar to a memory leak, a process does not return a buffer after use.
• Runaway process — A process is consuming an inordinate amount of memory.

Commands to troubleshoot memory problems
sh buffers — Check the number of free in the list.
sh processes memory sorted — What are the largest memory users?

sh buffers

R4#sh buffers
Buffer elements:
     1119 in free list (1119 max allowed)
     7519248 hits, 0 misses, 619 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50):
     49 in free list (20 min, 150 max allowed)
     5531838 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 64 @ 3w4d):
     25 in free list (10 min, 150 max allowed)
     362047 hits, 14 misses, 42 trims, 42 created
     0 failures (0 no memory)

sh processes memory sorted

R4#sh processes memory sorted 
Processor Pool Total:  307697040 Used:   14563424 Free:  293133616
      I/O Pool Total:   41943040 Used:    4264864 Free:   37678176

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   27799336    6726984   14720188          0          0 *Init*          
  39   0     654476       1272     635204          0          0 USB Startup     
  26   0   67553324   66736804     605916      46720          0 Exec            
   0   0          0          0     393528          0          0 *MallocLite*  
This entry was posted in Routing. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s