Let me start by what this chapter does not include, a nice set of filters for common show commands that will help you find the most pertinent information quickly. I think every networker has some of their favorite commands such as this one for BGP:

R1# sh ip bgp neigh | inc BGP
BGP neighbor is 209.65.200.226,  remote AS 65002, external link
  BGP version 4, remote router ID 209.65.200.242
  BGP state = Established, up for 1w3d
  BGP table version 16, neighbor version 16/0
  Last reset 1w3d, due to BGP Notification sent, hold time expired

Or one of my favorite sh run commands:

R1#sh run | sect int|router
interface FastEthernet0/0
 no ip address
 shutdown
 duplex auto
 speed auto
... output omitted for brevity ...
ipv6 router ospf 6
 router-id 10.1.1.1
 log-adjacency-changes

Filtering the show command
Using include:

R4#sh ip int br | inc 10.1
FastEthernet0/0            10.1.4.5        YES NVRAM  up                    up      
FastEthernet0/1            10.1.4.9        YES NVRAM  up                    up      
Serial0/0/0.34             10.1.1.10       YES NVRAM  up                    up      

Using exclude to show the same information:

R4#sh ip int br | exc unass
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            10.1.4.5        YES NVRAM  up                    up      
FastEthernet0/1            10.1.4.9        YES NVRAM  up                    up      
Serial0/0/0.34             10.1.1.10       YES NVRAM  up                    up  

Redirecting output
I had never seen the redirect and append commands in IOS, and while I have used tee and >> or 2>&1 on UNIX servers, I had never used redirection in IOS. Redirecting with append:

R4#sh ip int br | redirect tftp://10.2.2.10/test.txt
!
R4#sh ip route | append tftp://10.2.2.10/test.txt   
% Appending is not supported in this file system

Interesting, I’m not able to append to tftp while they can in the book. Let’s troubleshoot 🙂

R4#sh ip route | ?                               
  append    Append redirected output to URL (URLs supporting append operation
            only)
  begin     Begin with the line that matches
  exclude   Exclude lines that match
  include   Include lines that match
  redirect  Redirect output to URL
  section   Filter a section of output
  tee       Copy output to URL

So where can I append?

R4#sh ip route | append ?
  flash:  Uniform Resource Locator
  ftp:    Uniform Resource Locator
  nvram:  Uniform Resource Locator

R4#sh ip route | tee ftp://10.2.2.10/test-ftp.txt
Writing test-ftp.txt 
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
... output omitted for brevity ...
O IA    10.1.1.0/30 [110/192] via 10.1.1.9, 1w3d, Serial0/0/0.34
C       10.1.4.4/30 is directly connected, FastEthernet0/0
O IA    10.1.1.4/30 [110/128] via 10.1.1.9, 1w3d, Serial0/0/0.34
D*   0.0.0.0/0 is a summary, 1w0d, Null0
R4#sh ipv route | append ftp://10.2.2.10/test-ftp.txt
Writing test-ftp.txt 

Lesson learned. I can append to ftp while the TSHOOT author, Kevin Wallace can append to tftp. I wonder what IOS version he is using? Just for clarity here is my version:

R4#sh ver
Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(25a), RELEASE SOFTWARE (fc2)

Ping:

Before I go into what all ping can do on IOS I want to point out a link about Mike Muus, the man who wrote ping, it’s an interesting read for some historical perspective.

What the different characters mean in the ping response field:
! — Each exclamation point indicates receipt of a reply.
. — Each period indicates the network server timed out while waiting for a reply.
U — A destination unreachable error PDU was received.
Q — Source quench (destination too busy).
M — Could not fragment.
? — Unknown packet type.
& — Packet lifetime exceeded.

Some of the ping options from the TSHOOT book:
size — Then number of bytes per datagram.
repeat — The number of ICMP Echo messages sent.
timeout — Seconds to wait for an ECMP Echo Reply.
source — Source IP of the datagrams.
df-bit — Set the do not fragment bit.

If you’re going to turn on ip packet debugging on a router, you better set up an ACL. This is me playing with debugging ICMP, remember access lists are almost always set for inbound traffic.

R4(config)#ip access-list extended 100 
R4(config-ext-nacl)#permit icmp host 10.2.1.1 any 
R4(config-ext-nacl)#^Z
R4#debug ip packet 100 
IP packet debugging is on for access list 100
R4#ping 10.2.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
R4#
Mar 22 21:05:53.381: IP: tableid=0, s=10.2.1.1 (FastEthernet0/0), d=10.1.4.5 (FastEthernet0/0), routed via RIB
Mar 22 21:05:53.381: IP: s=10.2.1.1 (FastEthernet0/0), d=10.1.4.5 (FastEthernet0/0), len 100, rcvd 3
Mar 22 21:05:53.381: IP: tableid=0, s=10.2.1.1 (FastEthernet0/0), d=10.1.4.5 (FastEthernet0/0), routed via RIB
... output omitted for brevity ...    

I also did not know about the ping sweep capability of IOS:

R4#ping 
Protocol [ip]: 
Target IP address: 10.2.1.1
Repeat count [5]: 
Datagram size [100]: 
Timeout in seconds [2]: 
Extended commands [n]: y
Source address or interface: 
Type of service [0]: 
Set DF bit in IP header? [no]: 
Validate reply data? [no]: 
Data pattern [0xABCD]: 
Loose, Strict, Record, Timestamp, Verbose[none]: 
Sweep range of sizes [n]: y
Sweep min size [36]: 1450
Sweep max size [18024]: 1550
Sweep interval [1]: 
Type escape sequence to abort.
Sending 505, [1450..1550]-byte ICMP Echos to 10.2.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

So you would use ping to test layer 3 and telnet to test layer 4. Let’s test ftp in our testlab. This will come back and bite me some day, why remember port numbers when they are always at your disposal.

R4#sh ip nbar port-map | inc ftp
port-map ftp                      tcp 21 
port-map secure-ftp               tcp 990 
port-map tftp                     udp 69
R4#telnet 10.2.2.10 21
Trying 10.2.2.10, 21 ... Open
220 (vsFTPd 2.0.5)
quit

[Connection to 10.2.2.10 closed by foreign host]

This portion of the book is frustrating to me. I am an interactive learner and this portion is not as fun. It also feels like I am just reiterating the Key Topic points in the book. That is frustrating as well. I guess in time Cisco will have some more interesting documentation concerning troubleshooting as the pendulum swings toward documentation for the new test.

Hardware Debugging Commands:

show processes cpu — Is the switch/router able to handle the traffic?
show memory — Memory usage.
show interfaces — If needed use the clear counters command.
input queue drops — Receiving packets faster than it can process.
output queue drops — Could not send fast enough, i/o speed mismatch?
input errors — Frames were not received correctly, cabling problem?
output errors — Frames were net sent correctly, duplex mismatch?

Packet Captures:
I added another ethernet card to the server and connected it to gi0/12 on DSW2. That way I can run wireshark and capture traffic. You can download my libpcap file here.

DSW2(config)#monitor sess 1 source  int g0/14
DSW2(config)#monitor sess 1 dest int gi0/12

RSPAN:
RSPAN allows you capture traffic on switch that is sent from the port on another. That way you don’t have to have a collector in every closet.

First you need to configure the vlan as a remote-span vlan:

DSW2(config)#vlan 21
DSW2(config-vlan)#name SPAN
DSW2(config-vlan)#remote-span
DSW2(config-vlan)#do sh vlan 

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi0/2, Gi0/3, Gi0/4, Gi0/5
                                                Gi0/6, Gi0/7, Gi0/8, Gi0/9
... output omitted for brevity ...
20   20Testing                        active    
21   SPAN                             active    
... output omitted for brevity ...
Remote SPAN VLANs
------------------------------------------------------------------------------
21
... output omitted for brevity ...

The switches are set up with port-channels between them, however, rspan does not give an option for a portchannel interface as the reflector port. I just pointed it at one interface of the port-channel and it worked.

ASW1(config)#monitor session 1 destination remote vlan 21 reflector-port ?
  FastEthernet       FastEthernet IEEE 802.3
  GigabitEthernet    GigabitEthernet IEEE 802.3z
  LongReachEthernet  Long-Reach Ethernet interface

Here is the complete setup:

ASW1(config)#monitor session 1 source int fa0/2
ASW1(config)#monit sess 1 des remote vlan 21 reflector-port fa0/21
ASW1(config)#do sh run | inc mon
monitor session 1 source interface Fa0/2
monitor session 1 destination remote vlan 21 reflector-port Fa0/21
ASW1(config)#do sh mon
Session 1
---------
Type              : Remote Source Session
Source Ports      :
    Both          : Fa0/2
Reflector Port    : Fa0/21
Dest RSPAN VLAN   : 21

And on the destination switch:

DSW2(config)#monitor session 1 source vlan 21 both
DSW2(config)#monitor session 1 destination interface gi 0/12
DSW2(config-if-range)#do sh mon
Session 1
---------
Type                   : Local Session
Source VLANs           :
    Both               : 21
Destination Ports      : Gi0/12
    Encapsulation      : Native
          Ingress      : Disabled

SNMP:
Uses a pull model to collect device statistics. The command ifindex persist ensures the interface index stays consistent across reboots.

R1(config)#snmp-server ?
  chassis-id        String to uniquely identify this chassis
  community         Enable SNMP; set community string and access privs
  contact           Text for mib object sysContact
  context           Create/Delete a context apart from default
  drop              Silently drop SNMP packets
  enable            Enable SNMP Traps
  engineID          Configure a local or remote SNMPv3 engineID
  file-transfer     File transfer related commands
  group             Define a User Security Model group
  host              Specify hosts to receive SNMP notifications
  ifindex           Enable ifindex persistence
  inform            Configure SNMP Informs options
  ip                IP ToS configuration for SNMP traffic
  location          Text for mib object sysLocation
  manager           Modify SNMP manager parameters
  packetsize        Largest SNMP packet size
  queue-length      Message queue length for each TRAP host
  source-interface  Assign an source interface
  system-shutdown   Enable use of the SNMP reload command
  tftp-server-list  Limit TFTP servers used via SNMP
  trap              SNMP trap options
  trap-source       Assign an interface for the source address of all traps
  trap-timeout      Set timeout for TRAP message retransmissions
  user              Define a user who can access the SNMP engine
  view              Define an SNMP MIB view

R1(config)#snmp-server community collection ro
R1(config)#snmp-ser comm changes rw
R1(config)#snmp-serv conta x9995
R1(config)#snmp-ser ifindex persist

NetFlow:
Uses a push model to collect detailed traffic statistics.

R4(config-if)#int fa0/1
R4(config-if)#ip flq
R4(config-if)#ip fl 
R4(config-if)#ip flow ingr
R4(config-if)#ip flow ingress 
R4(config-if)#int s0/0/0
R4(config-if)#ip fl
R4(config-if)#ip flow ingr
R4(config-if)#ip flow ingress
R4(config-if)#do sh ip cach flo
IP packet size distribution (219750 total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .003 .622 .015 .022 .001 .001 .001 .001 .001 .001 .001 .001 .001 .001 .001

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .001 .001 .001 .019 .296 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
  3 active, 4093 inactive, 3591 added
  721344 ager polls, 0 flow alloc failures
  Active flows timeout in 30 minutes
  Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 21640 bytes
  1 active, 1023 inactive, 3211 added, 3211 added to flow
  0 alloc failures, 0 force free
  1 chunk, 1 chunk added
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-FTP             12      0.0         9    55      0.0       1.8       3.8
TCP-WWW              2      0.0        12    65      0.0       9.7       1.4
TCP-other           13      0.0         2    40      0.0       0.0       7.8
UDP-NTP           3073      0.0         1    76      0.0       0.0      15.8
UDP-other          108      0.0         5    38      0.0       0.0      15.5
ICMP                29      0.0      2742  1285      0.0    1363.3       4.9
IP-other           351      0.0       388    60      0.1    1791.7       2.4
Total:            3588      0.0        61   504      0.2     186.3      14.3

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Fa0/0         10.1.4.6        Null          224.0.0.10      58 0000 0000    19 
Fa0/1         10.1.4.10       Null          224.0.0.10      58 0000 0000    97 
Se0/0/0.34    10.1.1.9        Local         10.1.1.10       29 0000 0000     6 

Embedded Event Manager:

EEM monitors events through event detectors which then trigger an action based upon defined policies.
Event detectors can be an CLI command, interface counter, SNMP event or syslog event.
Actions occur in response to an event, examples of action include generating an SNMP trap, reloading IOS or generating a syslog message.
Policies are either an applet or a script.

As I was playing around in the EEM I came across this situation, it would be a mean way to play a game on someone.

R4(config)#event manager applet JUD
R4(config-applet)#event cli pattern "show run" skip yes sync no occurs 1

On a more serious note:

R4(config)#event manager applet JUD
R4(config-applet)#event cli pattern "clear counters" sync no occurs 1 skip no 
R4(config-applet)#action JUD syslog msg "Why clear counters" priority 0 
R4(config-applet)#^Z
R4#clear counters
Clear "show interface" counters on all interfaces [confirm]
Mar 26 21:18:51.107: %HA_EM-0-LOG: JUD: Why clear counters

[confirm]y
R4#sh log
Syslog logging: enabled (1 messages dropped, 1 messages rate-limited,
... removed for brevity ...
Mar 26 21:18:47.979: %SYS-5-CONFIG_I: Configured from console by console
Mar 26 21:18:51.107: %HA_EM-0-LOG: JUD: Why clear counters

I am going to end with some things I would like to see in the IOS CLI toolbox:
&& I should be able to see the output of this command:

R2#sh ip int br | exc unass && sh run | sect int

Or another command:

R2#conf t && int fa0/1 && ip add 10.1.1.1 255.255.255.0 && do sh run int fa0/1