There are three general causes of router performance issues:
1. High CPU load
2. Router packet switching mode
3. Excessive memory use

Processes that can be a cause of high CPU load:
• ARP Input process — Heavy traffic load can cause a the ARP Input process to spike.
• Net Background process — If an interface has full buffers but still needs to use a globally available buffer, the Net Background process handles it. There might also be a corresponding rise in throttles, ignored and overrun parameters of the sh int command.
• IP Background process — When an interface changes state this process does the work.

Commands to troubleshoot the different processes:
sh process cpu | inc [process name] — Is there an inordinate amount of CPU usage?
sh arp — Are there too many entries in the ARP table that would cause the ARP Input process to spike?
sh int fa0/0 — Are the throttles, ignored and overrun parameters climbing?
sh ctp stat — A high connection load can result in the TCP Timer process spiking.

sh arp
[code]
R4#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.4.10 53 0018.1825.2543 ARPA FastEthernet0/1
Internet 10.1.4.9 – 001b.d421.480b ARPA FastEthernet0/1
Internet 10.1.4.6 68 0012.d9a5.1542 ARPA FastEthernet0/0
Internet 10.1.4.5 – 001b.d421.480a ARPA FastEthernet0/0
[/code]

sh int fa0/0 | inc thrott|ignor|over
[code]
R4#sh int fa0/0 | inc thrott|ignor|over
Received 851934 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
[/code]

sh tcp stat
[code]
R4#sh tcp stat
Rcvd: 432 Total, 0 no port
0 checksum error, 0 bad offset, 0 too short
267 packets (8194 bytes) in sequence
1 dup packets (172 bytes)
0 partially dup packets (0 bytes)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) with data after window
0 packets after close
0 window probe packets, 0 window update packets
0 dup ack packets, 0 ack packets with unsend data
304 ack packets (10167 bytes)
Sent: 440 Total, 0 urgent packets
51 control packets (including 0 retransmitted)
262 data packets (10120 bytes)
0 data packets (0 bytes) retransmitted
0 data packets (0 bytes) fastretransmitted
127 ack only packets (65 delayed)
0 window probe packets, 0 window update packets
27 Connections initiated, 0 connections accepted, 24 connections established
30 Connections closed (including 1 dropped, 3 embryonic dropped)
0 Total rxmt timeout, 0 connections dropped in rxmt timeout
0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive
[/code]

Packet Switching Modes
• Process switching — The CPU is used to make packet switching decisions, the entire data flow is processed in the control plane. To turn on process switching issue the command no ip route-cache.
• Fast Switching — The CPU processes the first packet of a data flow, the rest are handled by the fast cache, reducing processor load. Turn on fast switching with the command ip route-cache.
• Cisco Express Forwarding (CEF) — CEF maintains the Forwarding Information Base (FIB) for layer 3 forwarding and the Adjacency Table for layer 2 next hops. The entire flow is processed in the data plane.

Packet Switching commands that look interesting to me:
sh ip int fa0/0 — Displays the packet switching mode.
sh ip cache — If fast switching is enabled it displays the fast cache.
sh ip cef — Displays the FIB contents.
sh adjacency det — Displays the adjacency table of a router if CEF is enabled.

sh ip int fa0/0 | inc IP
[code]
R4#sh ip int fa0/0 | inc IP
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is enabled
IP CEF switching is enabled
IP CEF Flow Fast switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow cache, CEF, Subint Flow
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
[/code]

sh ip cache
[code]
R4#sh ip cache
IP routing cache 0 entries, 0 bytes
0 adds, 0 invalidates, 0 refcounts
Minimum invalidation interval 2 seconds, maximum interval 5 seconds,
quiet interval 3 seconds, threshold 0 requests
Invalidation rate 0 in last second, 0 in last 3 seconds
Last full cache invalidation occurred 5w1d ago

Prefix/Length Age Interface Next Hop
[/code]

sh ip cef
[code]
R4#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 0.0.0.0 Null0
0.0.0.0/8 drop
0.0.0.0/32 receive
10.1.1.0/30 10.1.1.9 Serial0/0/0.34
10.1.1.4/30 10.1.1.9 Serial0/0/0.34
10.1.1.8/30 attached Serial0/0/0.34
10.1.1.8/32 receive
10.1.1.10/32 receive
[/code]

sh adj det
[code]
R4#sh adj det
Protocol Interface Address
IP FastEthernet0/1 10.1.4.10(16)
0 packets, 0 bytes
001818252543001BD421480B0800
ARP 01:41:49
Epoch: 0
IP FastEthernet0/0 10.1.4.6(16)
17 packets, 1818 bytes
0012D9A51542001BD421480A0800
ARP 02:47:29
Epoch: 0
IP Serial0/0/0.34 point2point(13)
80697 packets, 102706013 bytes
64310800
CEF expires: 00:02:53
refresh: 00:00:53
Epoch: 0
[/code]

Troubleshoot Memory Usage
• Memory leak — When not all memory used by a process is returned to the memory pool.
• Memory allocation failure — Shows up as a MALLOCFAIL error message.
• Bufffer leak — Similar to a memory leak, a process does not return a buffer after use.
• Runaway process — A process is consuming an inordinate amount of memory.

Commands to troubleshoot memory problems
sh buffers — Check the number of free in the list.
sh processes memory sorted — What are the largest memory users?

sh buffers
[code]
R4#sh buffers
Buffer elements:
1119 in free list (1119 max allowed)
7519248 hits, 0 misses, 619 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50):
49 in free list (20 min, 150 max allowed)
5531838 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 64 @ 3w4d):
25 in free list (10 min, 150 max allowed)
362047 hits, 14 misses, 42 trims, 42 created
0 failures (0 no memory)
[/code]

sh processes memory sorted
[code]
R4#sh processes memory sorted
Processor Pool Total: 307697040 Used: 14563424 Free: 293133616
I/O Pool Total: 41943040 Used: 4264864 Free: 37678176

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 27799336 6726984 14720188 0 0 *Init*
39 0 654476 1272 635204 0 0 USB Startup
26 0 67553324 66736804 605916 46720 0 Exec
0 0 0 0 393528 0 0 *MallocLite*
[/code]