Let me start by what this chapter does not include, a nice set of filters for common show commands that will help you find the most pertinent information quickly. I think every networker has some of their favorite commands such as this one for BGP:
R1# sh ip bgp neigh | inc BGP BGP neighbor is 188.8.131.52, remote AS 65002, external link BGP version 4, remote router ID 184.108.40.206 BGP state = Established, up for 1w3d BGP table version 16, neighbor version 16/0 Last reset 1w3d, due to BGP Notification sent, hold time expired
Or one of my favorite sh run commands:
R1#sh run | sect int|router interface FastEthernet0/0 no ip address shutdown duplex auto speed auto ... output omitted for brevity ... ipv6 router ospf 6 router-id 10.1.1.1 log-adjacency-changes
Filtering the show command
R4#sh ip int br | inc 10.1 FastEthernet0/0 10.1.4.5 YES NVRAM up up FastEthernet0/1 10.1.4.9 YES NVRAM up up Serial0/0/0.34 10.1.1.10 YES NVRAM up up
Using exclude to show the same information:
R4#sh ip int br | exc unass Interface IP-Address OK? Method Status Protocol FastEthernet0/0 10.1.4.5 YES NVRAM up up FastEthernet0/1 10.1.4.9 YES NVRAM up up Serial0/0/0.34 10.1.1.10 YES NVRAM up up
I had never seen the redirect and append commands in IOS, and while I have used tee and >> or 2>&1 on UNIX servers, I had never used redirection in IOS. Redirecting with append:
R4#sh ip int br | redirect tftp://10.2.2.10/test.txt ! R4#sh ip route | append tftp://10.2.2.10/test.txt % Appending is not supported in this file system
Interesting, I’m not able to append to tftp while they can in the book. Let’s troubleshoot 🙂
R4#sh ip route | ? append Append redirected output to URL (URLs supporting append operation only) begin Begin with the line that matches exclude Exclude lines that match include Include lines that match redirect Redirect output to URL section Filter a section of output tee Copy output to URL
So where can I append?
R4#sh ip route | append ? flash: Uniform Resource Locator ftp: Uniform Resource Locator nvram: Uniform Resource Locator R4#sh ip route | tee ftp://10.2.2.10/test-ftp.txt Writing test-ftp.txt Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP ... output omitted for brevity ... O IA 10.1.1.0/30 [110/192] via 10.1.1.9, 1w3d, Serial0/0/0.34 C 10.1.4.4/30 is directly connected, FastEthernet0/0 O IA 10.1.1.4/30 [110/128] via 10.1.1.9, 1w3d, Serial0/0/0.34 D* 0.0.0.0/0 is a summary, 1w0d, Null0 R4#sh ipv route | append ftp://10.2.2.10/test-ftp.txt Writing test-ftp.txt
Lesson learned. I can append to ftp while the TSHOOT author, Kevin Wallace can append to tftp. I wonder what IOS version he is using? Just for clarity here is my version:
R4#sh ver Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(25a), RELEASE SOFTWARE (fc2)
Before I go into what all ping can do on IOS I want to point out a link about Mike Muus, the man who wrote ping, it’s an interesting read for some historical perspective.
What the different characters mean in the ping response field:
! — Each exclamation point indicates receipt of a reply.
. — Each period indicates the network server timed out while waiting for a reply.
U — A destination unreachable error PDU was received.
Q — Source quench (destination too busy).
M — Could not fragment.
? — Unknown packet type.
& — Packet lifetime exceeded.
Some of the ping options from the TSHOOT book:
size — Then number of bytes per datagram.
repeat — The number of ICMP Echo messages sent.
timeout — Seconds to wait for an ECMP Echo Reply.
source — Source IP of the datagrams.
df-bit — Set the do not fragment bit.
If you’re going to turn on ip packet debugging on a router, you better set up an ACL. This is me playing with debugging ICMP, remember access lists are almost always set for inbound traffic.
R4(config)#ip access-list extended 100 R4(config-ext-nacl)#permit icmp host 10.2.1.1 any R4(config-ext-nacl)#^Z R4#debug ip packet 100 IP packet debugging is on for access list 100 R4#ping 10.2.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.2.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms R4# Mar 22 21:05:53.381: IP: tableid=0, s=10.2.1.1 (FastEthernet0/0), d=10.1.4.5 (FastEthernet0/0), routed via RIB Mar 22 21:05:53.381: IP: s=10.2.1.1 (FastEthernet0/0), d=10.1.4.5 (FastEthernet0/0), len 100, rcvd 3 Mar 22 21:05:53.381: IP: tableid=0, s=10.2.1.1 (FastEthernet0/0), d=10.1.4.5 (FastEthernet0/0), routed via RIB ... output omitted for brevity ...
I also did not know about the ping sweep capability of IOS:
R4#ping Protocol [ip]: Target IP address: 10.2.1.1 Repeat count : Datagram size : Timeout in seconds : Extended commands [n]: y Source address or interface: Type of service : Set DF bit in IP header? [no]: Validate reply data? [no]: Data pattern [0xABCD]: Loose, Strict, Record, Timestamp, Verbose[none]: Sweep range of sizes [n]: y Sweep min size : 1450 Sweep max size : 1550 Sweep interval : Type escape sequence to abort. Sending 505, [1450..1550]-byte ICMP Echos to 10.2.1.1, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
So you would use ping to test layer 3 and telnet to test layer 4. Let’s test ftp in our testlab. This will come back and bite me some day, why remember port numbers when they are always at your disposal.
R4#sh ip nbar port-map | inc ftp port-map ftp tcp 21 port-map secure-ftp tcp 990 port-map tftp udp 69 R4#telnet 10.2.2.10 21 Trying 10.2.2.10, 21 ... Open 220 (vsFTPd 2.0.5) quit [Connection to 10.2.2.10 closed by foreign host]
This portion of the book is frustrating to me. I am an interactive learner and this portion is not as fun. It also feels like I am just reiterating the Key Topic points in the book. That is frustrating as well. I guess in time Cisco will have some more interesting documentation concerning troubleshooting as the pendulum swings toward documentation for the new test.
Hardware Debugging Commands:
show processes cpu — Is the switch/router able to handle the traffic?
show memory — Memory usage.
show interfaces — If needed use the clear counters command.
input queue drops — Receiving packets faster than it can process.
output queue drops — Could not send fast enough, i/o speed mismatch?
input errors — Frames were not received correctly, cabling problem?
output errors — Frames were net sent correctly, duplex mismatch?
I added another ethernet card to the server and connected it to gi0/12 on DSW2. That way I can run wireshark and capture traffic. You can download my libpcap file here.
DSW2(config)#monitor sess 1 source int g0/14 DSW2(config)#monitor sess 1 dest int gi0/12
RSPAN allows you capture traffic on switch that is sent from the port on another. That way you don’t have to have a collector in every closet.
First you need to configure the vlan as a remote-span vlan:
DSW2(config)#vlan 21 DSW2(config-vlan)#name SPAN DSW2(config-vlan)#remote-span DSW2(config-vlan)#do sh vlan VLAN Name Status Ports ---- -------------------------------- --------- ------------------------------- 1 default active Gi0/2, Gi0/3, Gi0/4, Gi0/5 Gi0/6, Gi0/7, Gi0/8, Gi0/9 ... output omitted for brevity ... 20 20Testing active 21 SPAN active ... output omitted for brevity ... Remote SPAN VLANs ------------------------------------------------------------------------------ 21 ... output omitted for brevity ...
The switches are set up with port-channels between them, however, rspan does not give an option for a portchannel interface as the reflector port. I just pointed it at one interface of the port-channel and it worked.
ASW1(config)#monitor session 1 destination remote vlan 21 reflector-port ? FastEthernet FastEthernet IEEE 802.3 GigabitEthernet GigabitEthernet IEEE 802.3z LongReachEthernet Long-Reach Ethernet interface
Here is the complete setup:
ASW1(config)#monitor session 1 source int fa0/2 ASW1(config)#monit sess 1 des remote vlan 21 reflector-port fa0/21 ASW1(config)#do sh run | inc mon monitor session 1 source interface Fa0/2 monitor session 1 destination remote vlan 21 reflector-port Fa0/21 ASW1(config)#do sh mon Session 1 --------- Type : Remote Source Session Source Ports : Both : Fa0/2 Reflector Port : Fa0/21 Dest RSPAN VLAN : 21
And on the destination switch:
DSW2(config)#monitor session 1 source vlan 21 both DSW2(config)#monitor session 1 destination interface gi 0/12 DSW2(config-if-range)#do sh mon Session 1 --------- Type : Local Session Source VLANs : Both : 21 Destination Ports : Gi0/12 Encapsulation : Native Ingress : Disabled
Uses a pull model to collect device statistics. The command ifindex persist ensures the interface index stays consistent across reboots.
R1(config)#snmp-server ? chassis-id String to uniquely identify this chassis community Enable SNMP; set community string and access privs contact Text for mib object sysContact context Create/Delete a context apart from default drop Silently drop SNMP packets enable Enable SNMP Traps engineID Configure a local or remote SNMPv3 engineID file-transfer File transfer related commands group Define a User Security Model group host Specify hosts to receive SNMP notifications ifindex Enable ifindex persistence inform Configure SNMP Informs options ip IP ToS configuration for SNMP traffic location Text for mib object sysLocation manager Modify SNMP manager parameters packetsize Largest SNMP packet size queue-length Message queue length for each TRAP host source-interface Assign an source interface system-shutdown Enable use of the SNMP reload command tftp-server-list Limit TFTP servers used via SNMP trap SNMP trap options trap-source Assign an interface for the source address of all traps trap-timeout Set timeout for TRAP message retransmissions user Define a user who can access the SNMP engine view Define an SNMP MIB view R1(config)#snmp-server community collection ro R1(config)#snmp-ser comm changes rw R1(config)#snmp-serv conta x9995 R1(config)#snmp-ser ifindex persist
Uses a push model to collect detailed traffic statistics.
R4(config-if)#int fa0/1 R4(config-if)#ip flq R4(config-if)#ip fl R4(config-if)#ip flow ingr R4(config-if)#ip flow ingress R4(config-if)#int s0/0/0 R4(config-if)#ip fl R4(config-if)#ip flow ingr R4(config-if)#ip flow ingress R4(config-if)#do sh ip cach flo IP packet size distribution (219750 total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .003 .622 .015 .022 .001 .001 .001 .001 .001 .001 .001 .001 .001 .001 .001 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .001 .001 .001 .019 .296 .000 .000 .000 .000 .000 .000 IP Flow Switching Cache, 278544 bytes 3 active, 4093 inactive, 3591 added 721344 ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds IP Sub Flow Cache, 21640 bytes 1 active, 1023 inactive, 3211 added, 3211 added to flow 0 alloc failures, 0 force free 1 chunk, 1 chunk added last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) -------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-FTP 12 0.0 9 55 0.0 1.8 3.8 TCP-WWW 2 0.0 12 65 0.0 9.7 1.4 TCP-other 13 0.0 2 40 0.0 0.0 7.8 UDP-NTP 3073 0.0 1 76 0.0 0.0 15.8 UDP-other 108 0.0 5 38 0.0 0.0 15.5 ICMP 29 0.0 2742 1285 0.0 1363.3 4.9 IP-other 351 0.0 388 60 0.1 1791.7 2.4 Total: 3588 0.0 61 504 0.2 186.3 14.3 SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa0/0 10.1.4.6 Null 220.127.116.11 58 0000 0000 19 Fa0/1 10.1.4.10 Null 18.104.22.168 58 0000 0000 97 Se0/0/0.34 10.1.1.9 Local 10.1.1.10 29 0000 0000 6
Embedded Event Manager:
EEM monitors events through event detectors which then trigger an action based upon defined policies.
Event detectors can be an CLI command, interface counter, SNMP event or syslog event.
Actions occur in response to an event, examples of action include generating an SNMP trap, reloading IOS or generating a syslog message.
Policies are either an applet or a script.
As I was playing around in the EEM I came across this situation, it would be a mean way to play a game on someone.
R4(config)#event manager applet JUD R4(config-applet)#event cli pattern "show run" skip yes sync no occurs 1
On a more serious note:
R4(config)#event manager applet JUD R4(config-applet)#event cli pattern "clear counters" sync no occurs 1 skip no R4(config-applet)#action JUD syslog msg "Why clear counters" priority 0 R4(config-applet)#^Z R4#clear counters Clear "show interface" counters on all interfaces [confirm] Mar 26 21:18:51.107: %HA_EM-0-LOG: JUD: Why clear counters [confirm]y R4#sh log Syslog logging: enabled (1 messages dropped, 1 messages rate-limited, ... removed for brevity ... Mar 26 21:18:47.979: %SYS-5-CONFIG_I: Configured from console by console Mar 26 21:18:51.107: %HA_EM-0-LOG: JUD: Why clear counters
I am going to end with some things I would like to see in the IOS CLI toolbox:
&& I should be able to see the output of this command:
R2#sh ip int br | exc unass && sh run | sect int
Or another command:
R2#conf t && int fa0/1 && ip add 10.1.1.1 255.255.255.0 && do sh run int fa0/1