Virtual Switching System (VSS)

Posted on March 18, 2012 by Judson Bishop

This post was tedious in the formatting and was one of the reasons I put off posting it. The notes were taken months ago put I was weary of posting it because of the time involved in formatting it. As a result this post could have been better, I just dreaded working on it. Either I have to find a new editor or this will be the last post of this variety with unordered lists and line items to make bulleted points.

A few months ago we replaced our cores with a pair of 6509Es. The night of go live we had trouble because of some decisions we made at the last minute, and these notes saved us. I hope you find them as useful as we did. These are my notes from the design guide.

Because this post turned out so long, I put what most people will want to see at the top, the configuration. My notes from the configuration guide follow.

Configuration

1. Define the domain ID.
[code]
VSS(config)# switch virtual domain 100
[/code]

2. Set the switch number:
[code]
VSS(config-vs-domain)# switch 1
[/code]

2a. Have the switches use virtual MAC addresses:
[code]VSS(config-vs-domain)# mac-address use-virtual[/code]

2b. Check to make sure OOB is active and set to 480 seconds.
[code]VSS# sh mac-address-table synchonize statistics !sh stats for OOB[/code]

3. Configure VSL port-channel
[code]VSS (config-vs-domain)# exit[/code]

3a.
Standalone SW1:
[code]
no hw-module 1 oversubscription
no hw-module 2 oversubscription
int po1
switch virtual link 1
int rnage t1/1, t2/1
channel-gr 1 mode on
[/code]

Standalone SW2:
[code]
no hw-module 1 oversubscription
no hw-module 2 oversubscription
int po2
switch virtual link 2
int rnage t1/1, t2/1
channel-gr 2 mode on
[/code]

4. Convert to VSS mode:
[code]
VSS# switch convert mode virtual
(Switch will ask to reload)
(Reload)
[/code]

5. Only the first time conversion is this needed, this merges only VSL-related configurations, they say you MUST execute this command:
[code]
VSS# switch accept mode virtual
[/code]

6. Configure fast-hello for dual-active detection. (p.4-29)
[code]
! Enable fast-hello under VSS global config.
VSS(config)# switch virtual domain 100
VSS(config-vs-domain)# dual-active detection fast-hello

!Enable fast-hello at the interface level
VSS(config)# int gi1/5/3
VSS(config-if)# dual-active fast-hello

VSS(config)# int gi2/5/3
VSS(config-if)# dual-active fast-hello

! Confirm fast-hello
VSS# sh switch virtual dual-active fast-hello
VSS# remote command standby-rp show switch virtual dual-active fast-hello
[/code]

Commands
These are some commands that I kept for handy reference.

sh vslp lmp neighbor
[code]
VSS#sh vsl lmp nei

Instance #2:

LMP neighbors

Peer Group info: # Groups: 1 (* => Preferred PG)

PG # MAC Switch Ctrl Interface Interfaces
—————————————————————
*1 9999.aaaa.0000 1 Te2/5/4 Te2/5/4, Te2/5/5
[/code]

sh switch virtual role
[code]
VSS#sh switch virtual role

Switch Switch Status Priority Role Session ID
Number Oper(Conf) Local Remote
——————————————————————
LOCAL 2 UP 100(100) ACTIVE 0 0
REMOTE 1 UP 100(100) STANDBY 9111 9273
[/code]

sh int vsl
[code]
VSS#sh int vsl

VSL Port-channel: Po1
Port: Te1/5/4
Port: Te1/5/5

VSL Port-channel: Po2
Port: Te2/5/4
Port: Te2/5/5
[/code]

sh switch virtual
[code]
VSS#sh switch virtual
Switch mode : Virtual Switch
Virtual switch domain number : 100
Local switch number : 2
Local switch operational role: Virtual Switch Active
Peer switch number : 1
Peer switch operational role : Virtual Switch Standby

sh switch virtual redundancy
VSS#sh switch virtual redundancy
My Switch Id = 2
Peer Switch Id = 1
Last switchover reason = none
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso

Switch 2 Slot 5 Processor Information :
———————————————–
Current Software state = ACTIVE
Uptime in current state = 14 weeks, 4 days, 14 hours, 34 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXJ1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 22-Jun-11 18:03 by prod_rel_team
BOOT = sup-bootdisk:s72033-ipservicesk9_wan-mz.122-33.SXJ1.bin,12;
Configuration register = 0x2102
Fabric State = ACTIVE
Control Plane State = ACTIVE

Switch 1 Slot 5 Processor Information :
———————————————–
Current Software state = STANDBY HOT (switchover target)
Uptime in current state = 14 weeks, 4 days, 14 hours, 31 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXJ1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 22-Jun-11 18:03 by prod_rel_team
BOOT = sup-bootdisk:s72033-ipservicesk9_wan-mz.122-33.SXJ1.bin,12;
Configuration register = 0x2102
Fabric State = ACTIVE
Control Plane State = STANDBY

[/code]

sh vsl rrp summ
[code]
VSS#sh vsl rrp summ
RRP Summary:
————————————————————————
RRP information for Instance 2

——————————————————————–
Valid Flags Peer Preferred Reserved
Count Peer Peer

——————————————————————–
TRUE V 1 1 1

Peer Valid Switch Status Priority Role Local Remote
Switch Group Number Oper(Conf) SID SID
———————————————————————
Local 0 TRUE 2 UP 100(100) ACTIVE 0 0
Remote 1 TRUE 1 UP 100(100) STANDBY 9111 9273

Peer 0 represents the local switch

Flags : V – Valid
[/code]

sh mls cef
[code]
sh mls cef
Codes: decap – Decapsulation, + – Push Label
Index Prefix Adjacency
64 0.0.0.0/32 receive
! Removed for brevity
[/code]

sh mac-address-table synchronize statistics
[code]
VSS#sh mac-address-table synchronize statistics

MAC Entry Out-of-band Synchronization Feature Statistics:
———————————————————

Switch [1] Module [1]
———————–

Module Status:
Statistics collected from Switch/Module : 1/1
Number of L2 asics in this module : 1

! Removed for brevity.
[/code]

sh switch virtual redundancy mismatch
[code]
VSS#sh switch virtual redundancy mismatch

No Config Mismatch between Active and Standby switches
[/code]

redundancy reload peer
[code]
! Reload a switch from RPR mode to hot-standby
VSS#redundancy reload peer
! Did not get output.
[/code]

Configuration Guide Notes Below

Virtual Switch Member Boot-up Behavior

  • Diagnostics
  • VSL Link Initialization
  • LMP Establishment
  • Role negotiation through RRP

Link Management Protocol (LMP)

  • Establishes and verifies bidirectional communication during startup and normal operation
  • Exchange switch ID
  • Sends hello packets to monitor health of VSL and peer

Role Resolution Protocol (RRP)

  • Determines the operational status of each switch member.

Virtual Switch Link (VSL)
Each member link must be configured configured in unconditional EtherChannel mode:
channel-group 12 mode on

Stateful Switch Over (SSO)

  • Enables supervisor redundnacy in a standalone 6000, keeping the backup
    supervisor up to date.
  • State 13-Active If in active state the supervisor is responsible for forwarding
    and managing the control plane. Manage control plane functions,
    synchornizes the configuration and the protocols.
  • State 8-Standby Supervisor is synchronized with with the active. This is the
    final state hot-standby supervisor.
  • SSO is the core of VSS, however VSS is a dual forwarding solution while the
    control plane is managed by one supervisor.

Virtual Switch Prioroity

  • The first to boot will become active.
  • If simultaneous boot, lowest switch ID becomes active.
  • Highest priority wins, except highest priority switch will not become active unless preemption is enabled.
  • Default priority is 100.
  • Switch preemption should not be taken lightly.
    • It forces multiple reboots of the VSS member.
    • Cisco recommends _not_ configuring preemption.

Multi-chassis Etherchannel (MEC)

  • Preferred connectivity method using VSS.
  • Extends etherchannel to from multiple ports on one switch to multiple ports on two chassis.
  • Access-layer switches are configured with traditional etherchannel.
  • VSS with MEC is loop-free.

MEC Configuration

  • Do not explicitly create layer-2 MEC from the CLI, allow IOS to generate the interface.
  • Create a layer-3 MEC explicitly and associate the port-channel group under each member interface.
  • This syslog configuration command is recommended in VSS with MEC interfaces.

    • [code]
    int po20
    logging event link-status
    logging event spanning-tree status
    [/code]

  • These hidden commands are now available in 12.2(33)SXH1

    • [code]
    remote command switch test EtherChannel load-balance interface po 1 ip 1.1.1.1 2.2.2.2
    show EtherChannel load-balance hash-result interface port-channel 2 205 ip 10.120.7.65 vlan 5 10.121.100.49
    [/code]

MAC Addresses

  • MAC address allocation is derived from the back plane EEPROM on each chassis, therefore a VSS instance has two pools. The VSS MAC address pool is determined by RRP. MAC address allocation does not change during a switch over event, however, MAC addresses will change in the event both switches reboot without the mac-address use-virtaul command. This avoids gratuitous ARPs.
  • When upgrading the change of MAC address for the default gateway can cause problems for hosts not capable of updating the default gateway ARP entry. It is typically cached for four hours.
  • MAC Out-of-Band Sync (OOB)
  • MAC addresses normally age out age out in a single chassis environment.
  • Depending upon the etherchannel hash MAC addresses have the chance to age out because they are not updated.
  • MAC OOB is designed to synchronize MAC addresses in all line cards of the VSS over the VSL.
  • In VSS trunk mode of a port-channel interfaces being desirable or undesirable does not act the same as in standalone mode. When a link member is brought on line it is not a separate negotiation, it is an addition to MEC. p.3-25

PAgP

  • The active switch is responsible for origination and termination of PAgP control plane traffic.
  • The same device ID is sent by both VSS switches so the end device assumes a single logical device.
  • Cisco recommends PAgP neighbors to be in desirable-desirable mode with the silent sub option.

LACP p. 2-37

  • In VSS it works for both layer-2 and layer-3 interfaces.
  • The recommended mode for LACP neighbors is Active-Active
  • During the EtherChannel bundling process LACP performs a configuration consistency check on each link trying to become a port-channel member.
  • If a port does not pass it is placed in a “lettered” system bundle.
  • The first etherchannel bundle contains the ports that passed the configuration check.
  • The second “lettered” bundle includes the ports that did not pass the configuration check.
  • Avoid using the min-links LACP command
  • Avoid LACP fast-hello in VSS
    • During failover and recovery the VSS might not be able to recover before the remote end declares VSS down. False positive.
    • Fast-hello as sent per link which can overrun a switch CPU in large deployments.

[code]
6500-VSS# show etherchannel 20 summary | inc Gi
Po20(SU) LACP Gi2/1(P)
Po20B(SU) LACP Gi2/2(P) ! Bundled in separate system-generated
! port-channel interface
[/code]

Implementation Notes

Recommended to have one port from the supervisor and one from a line card, however, the have different queue structures and the etherchannel bundle would fail. To fix this turn on:
[code]
no mls channel-consistency
[/code]

The Sup720-10G uplink port can be configured in one of two modes:

  • Default, Non-10g-only mode:
    • All supervisor ports have the same CoS queuing mode if any 10G port is used for VSL. VSL only allows CoS-based queuing.
  • Non-blocking, 10g-only mode:
    • All 1G ports are disabled, the entire module operates in non-blocking mode. 12.2(33)SXI allows non-VSL 10G ports to be DSCP based.

Resilient VSL Design Options (p2-18 thru 2-20)

  • Use the two 10G ports on the Sup720-10G supervisor.
    • Most common, does not provide optimal hardware diversity.
  • Use on 10G port on the Sup720-10G and another from a VSL capable line card.
    • Best for balancing cost and redundancy.
  • Use 10G ports on two separate VSL capable line cards.
    • Best option for flexibility but not as cost effective.

EtherChannel
Etherchannel is the fundamental building block of VSS. Traditionally load
sharing and failure are governed by STP, FHRP and topology (looped and
non-looped). In VSS Etherchannel replaces all three.

  • The etherchannel hash algorithm becomes more important to get right in VSS.
  • Layer-4 hashing is more random than layer-3 hashing.
  • Layer-2 hashing is not as efficient when all hosts are sending to a default
    gateway.

There are a variety of etherchannel options in VSS.
[code]
VSS(config-if)# port-channel port hash-distribution X
[/code]
By default the load-sharing hash method on all non-VSL etherchannel is fixed.

VLAN ID

Traffic optimized when:

  • With VSS it is possible to have more VLANs per closet.
  • Traffic might not be fairly hashed due to similarities such as default gateway or multicast traffic.

[code]
VSS# sh platform hardware pfc mode
VSS# sh etherchannel load-balance
[/code]

  • Layer 3 and 4 Hash Tuning
    • dst-mixed-ip-port
    • src-dst-mixed-ip-port
    • rc-mixed-ip-port
  • For lower end switches:
    • Cisco Catalyst 4500
      • src-dst-ip
    • Cisco Catalyst 36xx, 37xx Stack, 29xx
      • src-dst-ip

Failures

Convergence

  • FHRP recovery default is 10 seconds, with tuning 900msec.
  • VSS 200msec convergence.

    • VSS Member Failures

  • Recovery is based on etherchannel, it detects the failure then rehashes the flow.

    • Core to VSS Failure

  • If all links fail from one VSS member to the core traffic will traverse the VSL.

    • Access Layer to VSS Failure

  • Traffic will flow over the VSL.

    • STP Loops and VSS

    • These issues can introduce a loop that STP might not block
      • Faulty hardware causes a missed BPDU
      • Faulty software cause high CPU load, preventing BPDU processing.
      • Configuration mistake
      • Non-standard switch implementation
    • VSS over comes these issues
      • Creates a loop free topology using MEC.
      • No FHRP needed, replaced by one logical node.

    Unidirection Link Detection (UDLD)

    • Aggressive UDLD should _not_ be used as link-integrity check. VSS is by definition a loop-free topology.
    • STP protocols (RPVST+ and MST) converge faster than UDLD detects.

    Spanning Tree Configuration with VSS

    • The root of the STP should always be VSS.
    • Loop guard is not needed.
    • The active switch is responsible for generating the BPDU.

        • Routing with VSS
        Layer-3 MEC is the recommended design rather than ECMP.

        Routing Protocols, Topology and Interaction

        • Two ways to connect VSS to the core:
          • Equal Cost Multipath (ECMP)
          • Layer-3 MEC

        Link Failure Convergence

        • The higher the number of routes the longer ECMP takes to recover.
        • Because MEC failer detection is hardware based, it does not matter
          the number of routes, the hardware will detect failure and adjust
          traffic to the healthy link.
        • Advantage MEC.

        Path availability during link failure

      • A single link failure in ECMP will result in path reprogramming.

        • Routing Protocol Interaction During Active Failure

        • When the active supervisore fails, the standby supervisor must reinitialize the routing protocol.
        • This can be mitigated with Non-Stop Forwarding (NSF) and the neighboring router must be NSF aware.
        • NSF must be enable on both the VSS and adjacent nodes.

          • [code]
          VSS(config)# router ospf 100
          VSS(config-router)# nsf cisco
          VSS# sh ip osfp nsf
          [/code]

        • NSF configuration http://www.cisco.com/en/US/docs/ios/ha/configuration/guide/ha-nonstp_fwdg.html#wp1056927

        Dual Active Detection (p. 4-29)

        • PAgP
        • Fast-Hello
        • BFD

        Fast-Hello

        • Requires a dedicated physical port between the VSS nodes.
        • The dedicated link is not capable of carrying control-plan or user-data traffic.
        • During dual-active the that is configured to carry fast-hello is operational and continues
          to exchange hellos. If the old-active continues to see hellos during
          what it believes to be a failover state, then it knows dual-active has occurred.
        • The Sup720-10G 1Gb uplink ports can be used if the supervisor is not
          configured in 10Gb on mode.

        Configure fast-hello for dual-active detection. (p.4-29)
        [code]
        ! Enable fast-hello under VSS global config.
        VSS(config)# switch virtual domain 100
        VSS(config-vs-domain)# dual-active detection fast-hello

        !Enable fast-hello at the interface level
        VSS(config)# int gi1/5/1
        VSS(config-if)# no shut
        VSS(config-if)# dual-active fast-hello

        VSS(config)# int gi2/5/1
        VSS(config-if)# no shut
        VSS(config-if)# dual-active fast-hello

        ! Confirm fast-hello
        VSS# sh switch virtual dual-active fast-hello
        VSS# remote command standby-rp show switch virtual dual-active fast-hello
        [/code]

        Using Bidirectional Forwarding Detection

        • BFD session establishment is the indication of dual-active condition.
        • Normally VSS would not be able to establish BFD with itself because it is one logical node.
        • BFD takes 20-25 seconds for detection.
          • Requires IP connectivity.
          • Needs IP processes and static route.

        Configure BFD for dual-active Detection
        [code]
        VSS(config)# switch virtual domain 10
        VSS(config)# dual-active pair interface gi1/5/1 int gi2/5/1 bfd
        !
        ! Enable unique IP subnet and BFD interval on interfaces.
        VSS(config)# int gi1/5/1
        VSS(config-if)# ip add 192.168.1.1 255.255.255.0
        VSS(config-if)# bfd interval 50 min_rx 50 multiplier 3
        !
        VSS(config)# int gi2/5/1
        VSS(config-if)# ip add 192.168.2.1 255.255.255.0
        VSS(config-if)# bfd interval 50 min_rx 50 multiplier 3
        !
        ! The static route is automatically added.
        ! Confirm and monitor BFD.
        VSS# sh switch virtual dual-active bfd
        VSS# sh switch virtual dual-active summary
        [/code]

        Dual-Active Recovery

        • Once the VSL connectivity is established RRP handles the negotiation.

        OSPF Tuning
        [code]
        VSS(config)# router ospf 100
        VSS(config-router)# nsf
        VSS(config-router)# auto-cost reference bandwidth 20000
        ! Confirm OSPF
        VSS# sh ip ospf neighbor detail
        VSS# sh ip protocol
        [/code]

    Posted in Routing | Leave a comment

    TSHOOT Tickets

    Posted on November 15, 2011 by Judson Bishop

    I took and did not pass my TSHOOT exam last week. Bummer. I had not studied specifically for the test very hard. In my mind I had already started thinking about the CCIE written and even started doing the “easy” CCIE labs. I have a long way to go as the best I could do was a 2 hour CCIE lab in 5 hours. That was the best. There were a few that took a couple of days to complete. So when I went to take my test I felt I had a good chance at passing but was by no means assured of a passing grade.

    Without breaking the NDA here are some of my thoughts.

  • I was not prepared mentally.
  • I did not have a game plan for troubleshooting the lab.
  • I did not go over the basic troubleshooting commands before I took the test.
  • I was more prepared for this specific test in May than I was last week.

    • How I prepared for the last test:
    I went and reviewed this blog. Reading my own writing and doing the commands on the lab routers refreshed some of the old memories.

    I configured the TSHOOT topology from scratch again. Then I had my coworker go through and randomly break things. He would log in and break it, give me a hint and then I would go in and fix the configuration. This was easy, possibly too easy, and it was slow. We would only do this once a day and I wouldn’t worry if it took 15 or 20 minutes, I was just playing around in the lab before I went home. That all changed during the test. Because I had configured my lab I knew the configs cold, I had configured it all and I could spot typos or simple errors quickly by just viewing the running configuration. That is not the case during the lab.

    How I am preparing now:
    I am not concerned about theory. I have been studying TCP/IP Volume I with the CCIE written in mind and am on the last chapter.

    Over the weekend I made 15 different configurations with errors. I don’t even remember what each ticket does, just that it creates an error somewhere in the lab. I saved them to the flash of each switch or router and wrote the following script. It randomly selects a trouble ticket to load, then calls the testlab update expect script, tlue, and loads the bad configuration on the device.

    At this point I am not concerned with the actual error. I did not have a process to follow last week during the test and was not consistent in my troubleshooting process. By being able to load a number of errors in quick succession I will be able to troubleshoot a number of errors using a consistent process. My goal is to bombard myself with random errors like I saw in the test and get my troubleshooting process down pat.

    The other key here is to rely on mastering a few commands that tell me the most about the situation on a router or switch and not on reading router configurations to troubleshoot. I feel that was my weakness in my previous studies and became my weakness during the lab.

    Below is the script that loads random configurations with errors in the lab.

    [code lang=”bash”]
    #!/bin/bash
    # 2010-11-15 Jud Bishop
    # tl-ticket
    # This script randomly picks a ticket and loads the configuration from flash.

    I=$((RANDOM%15+1))

    #1: DSW1
    #2: ASW1
    #3: ASW1
    #4: ASW1
    #5: R4
    #6: R4
    #7: R2
    #8: R1
    #9: R1
    #10: R1
    #11: R2 — IPv6
    #12: DSW1
    #13: R4
    #14: R4 — IPv6
    #15: R4

    case $I in
    1 )
    tlue DSW1 replace ticket1.cfg
    ;;
    2 )
    tlue ASW1 replace ticket2.cfg
    ;;
    3 )
    tlue ASW1 replace ticket3.cfg
    ;;
    4 )
    tlue DSW2 replace ticket4.cfg
    ;;
    5 )
    tlue R3 replace ticket5.cfg
    ;;
    6 )
    tlue R4 replace ticket6.cfg
    ;;
    7 )
    tlue R2 replace ticket7.cfg
    ;;
    8 )
    tlue R1 replace ticket8.cfg
    ;;
    9 )
    tlue R1 replace ticket9.cfg
    ;;
    10 )
    tlue R2 replace ticket10.cfg
    ;;
    11 )
    tlue R2 replace ticket11.cfg
    ;;
    12 )
    tlue DSW1 replace ticket12.cfg
    ;;
    13 )
    tlue R4 replace ticket13.cfg
    ;;
    14 )
    tlue R4 replace ticket14.cfg
    ;;
    15 )
    tlue R4 replace ticket15.cfg
    ;;
    esac
    [/code]

    Posted in Code, Routing | Leave a comment

    Does Google Hurt Efficiency?

    Posted on August 2, 2011 by Judson Bishop

    The other night we were doing a hardware upgrade on a cluster and testing. We were working with the command clusvcadm to relocate a service from one host in the cluster to another but the originating server kept getting power fenced. We assumed it was the command switches we were running so I went straight to the man page, my coworker went straight to google. Just for reference there is a 10 year difference in our ages, I grew up with man pages and it is a pet peeve of mine when either no man page exists or it is a terrible placeholder. I digress, through his search he came upon a webified man page while I was reading the man page. When I needled him about it his answer was, “But mine is nicely formatted and I can search the web page.” I was surprised, I can search the man page too, right in the pager and can even change man page viewers by changing the PAGER variable.

    Three weeks ago I needed to bring up an https server on Ubuntu and spent 45 minutes googling around reading old, outdated or completely wrong howtos before finally going to help.ubuntu.com and 20 minutes later it was done.

    The same thing happened over the past couple of weeks working with Xen and VirtualBox. I’ve toiled away looking at poorly written documentation and even mentioned it in my last RedHat class. The instructor worked for Red Hat and took umbrage with my statement. He was amazed that I did not think Red Hat had great documentation, I was even more shocked that he considered their documentation more than rudimentary. Have a look for your self at the Red Hat documentation.

    Just this week I was helping a friend who is the server and network administrator for a small school system configure the proper etherchannel load balancing for a server and he was frustrated at the Cisco documentation. I was astonished. It seemed that he was overwhelmed. He was stuck googling around trying find the “right” documentation rather than learning the layout of the Cisco documentation website.

    The point of this post is that lately it seems I waste more time trying to find good information through searching on the web than trying to find the best source of information.

    Posted in Linux, Thoughts | Leave a comment

    Mac OSX TFTP Server

    Posted on July 6, 2011 by Judson Bishop

    At the Circus we have a network management server that runs all of the normal services needed to manage a small network and so I rarely need to fire up the tftp server on my laptop. Today was one of those days I needed a quick tftp server and I spent too much time figuring it out. This is my attempt to remedy that shortcoming.

    What is ironic is that after I googled around I found that I had “self documented” in the /private/tftpboot directory, unfortunately I expected the tftp directory for the tftp server to be in /tftpboot. I realize I can just put a symlink from /tftpboot to /private/tfpboot but I learned when working on AIX it is better to understand the file system layout of a UNIX vendor that it is to make it like another OS. It will bite you eventually.

    Here is a listing of the /private/tftboot directory, you will notice the very last file is tftp.txt. That is where I told myself how to do this in the past. It also appears most of the IOS images were for the testlab.

    [code]
    asa821-k8.bin
    c1841-adventerprisek9-mz.150-1.M.bin
    c2500-is-l.123-26.bin
    c2600-adventerprisek9-mz.124-25c.bin
    c3560e-ipbasek9npe-mz.122-55.SE1.bin
    c3620-j1s3-mz.123-26.bin
    c3640-a3js-mz.124-25b.bin
    c3640.txt
    tftp.txt
    [/code]

    This is what I had listed in the tftp.txt file. It tells how to start and stop a service in Mac OSX using launchctl.

    [code lang=”bash”]
    sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist
    sudo launchctl unload -F /System/Library/LaunchDaemons/tftp.plist
    [/code]

    For thoroughness I am including the tftp.plist file below. If I wanted the tftp daemon to start every time I turned on my laptop I would change Disabled to EnableTransactions.

    [code]
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;
    <plist version="1.0">
    <dict>
    <key>Disabled</key>
    <true/>
    <key>Label</key>
    <string>com.apple.tftpd</string>
    <key>ProgramArguments</key>
    <array>
    <string>/usr/libexec/tftpd</string>
    <string>-s</string>
    <string>/private/tftpboot</string>
    </array>
    <key>inetdCompatibility</key>
    <dict>
    <key>Wait</key>
    <true/>
    </dict>
    <key>InitGroups</key>
    <true/>
    <key>Sockets</key>
    <dict>
    <key>Listeners</key>
    <dict>
    <key>SockServiceName</key>
    <string>tftp</string>
    <key>SockType</key>
    <string>dgram</string>
    </dict>
    </dict>
    </dict>
    </plist>
    [/code]

     

    Posted in Linux | Leave a comment

    EIGRP MPLS VPN PE-CE SOO

    Posted on June 6, 2011 by Judson Bishop

    I couldn’t resist using all of those acronyms.
    EIGRP – Enhanced Interior Gateway Routing Protocol
    MPLS – Multiprotocol Label Switching
    VPN – Virtual Private Networking
    PE-CE – Provider Equipment – Customer Equipment
    SOO – Site Of Origin

    MPLS SOO
    MPLS Fundamentals pp. 220-226

    BGP->EIGRP and EIGRP->BGP

    Advertisement of the SOO BGP extended community attribute is used to identify routes that have originated from a site so that they are not re-advertised back into the same site. Each SOO uniquely identifies the site and allows for the routes to be filtered. SOO filtering is configured at the interface level. It is commonly used when a site contains both VPN and back door links.

    From the Cisco document:
    The configuration of the SOO extended community allows MPLS VPN traffic to be filtered on a per-site basis. The SoO extended community is configured in an inbound BGP route map on the PE router and is applied to the interface with the ip vrf sitemap command. The SOO extended community can be applied to all exit points at the customer site for more specific filtering but must be configured on all interfaces of PE routers that provide VPN services to CE routers.

    Posted in Routing | Leave a comment

    Routing Mnemonics

    Posted on June 3, 2011 by Judson Bishop

    I’ve been keeping track of some the mnemonics that I have come across or have figured out for myself. For instance in general, in layer 2 elections the lower priority usually wins, however, in layer 3 elections the higher priority usually wins.

    Layer 2

    LACP System Priority
    2-bytes priority values followed by a 6-byte MAC address. Lowest system priority makes decisions about the etherchannel setup.

    LACP Port Priority
    LACP port priority is a 2-byte priority followed by a 2-byte port number. Lowest port priority is used to decide which ports are put in standby mode when not all ports can be put in etherchannel.

    STP
    Root bridge election, lowest bridge ID wins. Bridge ID consists of:
    –2-byte bridge priority from 0-65,535 with a default of 32,768.
    –6-byte MAC address
    If the bridge priorities are equal, lowest MAC wins.

    Root port, lowest root path cost.

    Designated port — lowest root path cost or if equal use ties breakers:
    1. Lowest root bridge ID
    2. Lowest root path cost to root bridge
    3. Lowest sender bridge ID
    4. Lowest sender port ID

    Frame Relay
    DCE requires the clock rate, DCE, DTE, clock rate starts with a c and DCE is the one with a c in it.

    Layer 3

    HSRP
    Active router election is based upon priority, highest priority wins. Default priority of 100 and a range of 0-255. Highest IP address on HSRP interface breaks ties.
    Standby router is the second highest priority.

    VRRP
    Election of master is the router with the gateway IP address or if not a “real” IP address, the router with the highest priority. Priority ranges from 1 to 254 with 254 being highest, 100 is the default.

    GLBP
    Active virtual gateway (AVG) is elected by the highest priority value, tie breaker is the highest IP address in the group. Router priority is 1-255 with 255 being highest, 100 is the default.

    OSPF DR/BDR Election
    1. Highest priority wins.
    2. Highest router ID breaks ties.
    Priority range is 0-255 with 255 being highest, 1 is the default and 0 means the router will not participate in the election.

    OSPF RID
    1. router-id command wins.
    2. If no router-id is set, the highest loopback address wins, even if it is not advertised and it is not advertised by default.
    3. Highest physical address wins.

    OSPF summary-address command or the range command.
    The summary-address command is used on an ASBR and has an “S” in it, whereas the area range command is used on an ABR and does not have an “S” in it. Both commands are used to summarize routes.

    OSPF ExStart
    During ExStart of the OSPF packet exchange the neighbor with the highest RID will become the master and sets the DD sequence number.

    DVMRP
    An exception to the rule of Layer 2 lower takes priority and Layer 3 higher takes the priority. If two routers are the same distance from the source, the router with the numerically lower IP address becomes the designated forwarder for the network.

    BGP best path mnemonic
    We love oranges as oranges mean pure refreshment.

    We — Weight (highest)
    Love — LOCAL_PREF (highest)
    Oranges — Originate (local)
    AS — AS_PATH (shortest)
    Oranges — Origin Code (IGP > EGP > Incomplete)
    Mean — Med (lowest)
    Pure — Paths (External > Internal)
    Refreshment — RID (lowest)

    Redistribution
    RIP and any other protocol that has the letters R-I-P in it requires a seed metric, RIP, IGRP, EIGRP.

    Posted in Routing | Leave a comment

    DRBD and Heartbeat

    Posted on June 1, 2011 by Judson Bishop

    2011-06-01 10:31:23
    DRBD and Heartbeat
    I spent a considerable amount of time over the last couple of days working with DRBD and Heartbeat.

    Below are the links I used to get things running:
    http://wiki.centos.org/HowTos/Ha-Drbd
    http://www.howtoforge.com/vm_replication_failover_vmware_debian_etch_p3
    http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/s-intro-pacemaker.html
    http://www.drbd.org/users-guide/s-heartbeat-r1.html
    http://www.drbd.org/users-guide/s-heartbeat-config.html
    http://www.drbd.org/users-guide/s-heartbeat-crm.html

    Part of my problem was not understanding the difference between R1 and DRM style clusters and their accompanying daemons; heartbeat, pacemaker and the different protocol versions. Pacemaker is a more advanced cluster resource manager that can work with both Corosync and Heartbeat. Heartbeat uses an older protocol whereas pacemaker uses OpenAIS to be compatible with RedHat cluster services.

    Regardless here are my notes for configuration, and just for completeness my notes are a mix of doing this first on VMWare and then on a Xen cluster so any inconsistencies are a result of doing this multiple times in different environments. Regardless the errors are mine and I would recommend reading the documentation linked above.

    The basics behind the setup is that DRBD replicates data between two servers. DRBD is the network block device that mirrors the data. The heartbeat daemon keeps track of the shared IP, the daemons that are in HA and runs the init scripts appropriately.

    DRBD Initialization

    Format the disk:
    [code]
    fdisk /dev/xvdb
    Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
    Building a new DOS disklabel. Changes will remain in memory only,
    until you decide to write them. After that, of course, the previous
    content won’t be recoverable.

    The number of cylinders for this disk is set to 10443.
    There is nothing wrong with that, but this is larger than 1024,
    and could in certain setups cause problems with:
    1) software that runs at boot time (e.g., old versions of LILO)
    2) booting and partitioning software from other OSs
    (e.g., DOS FDISK, OS/2 FDISK)
    Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

    Command (m for help): p
    Disk /dev/xvdb: 85.8 GB, 85899345920 bytes
    255 heads, 63 sectors/track, 10443 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System

    Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    p
    Partition number (1-4): 1
    First cylinder (1-10443, default 1):
    Using default value 1
    Last cylinder or +size or +sizeM or +sizeK (1-10443, default 10443):
    Using default value 10443

    Command (m for help): t
    Selected partition 1
    Hex code (type L to list codes): 83

    Command (m for help): p
    Disk /dev/xvdb: 85.8 GB, 85899345920 bytes
    255 heads, 63 sectors/track, 10443 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/xvdb1 1 10443 83883366 83 Linux

    Command (m for help): w
    The partition table has been altered!

    Calling ioctl() to re-read partition table.
    Syncing disks.
    [/code]

    Make sure that the names names are consistent throughout all of these configuration files. This may mean ensuring they are correct in DNS and /etc/hosts.

    Locally configure name for this server:
    [code]
    uname -n
    drbd01.chainringcircus.org

    uname -n
    drbd02.chainringcircus.org
    [/code]

    DNS name for this server:
    [code]
    dig +short drbd01.chainringcircus.org
    192.168.1.191
    dig +short drbd02.chainringcircus.org
    192.168.1.192
    [/code]

    The /etc/drbd.conf file was designed to allow a verbatim copy on both nodes of the cluster.
    [code]
    cat /etc/drbd.conf
    #
    # please have a a look at the example configuration file in
    # /usr/share/doc/drbd83/drbd.conf
    #

    global {
    usage-count no;
    }

    common {
    protocol C;
    handlers {
    pri-on-incon-degr "echo ‘!DRBD! pri on incon-degr’ | wall ; sleep 60 ; halt -f";
    #pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
    #pri-on-incon-degr This handler is called if the node is primary, degraded and the local
    #copy of the data is inconsistent. It broadcasts an error, sleeps for 60 seconds and then halts.
    }

    startup {
    wfc-timeout 10; # Wait for connection timeout. The init script blocks the boot process
    # until the DRBD resources are connected. We wait for 10 seconds.
    degr-wfc-timeout 30; # Wait for connection timeout if this node was a degraded cluster.
    }

    disk {
    on-io-error detach;
    } # or panic, …

    net {
    cram-hmac-alg "sha1";
    shared-secret "CHANGEME"; # Don’t forget to choose a secret for auth
    max-buffers 20000; # Play with this setting to achieve highest possible performance
    unplug-watermark 12000; # Play with this setting to achieve highest possible performance
    max-epoch-size 20000; # Should be the same as max-buffers
    }
    syncer {
    rate 100M;
    }
    }

    resource sites {
    device /dev/drbd0;
    disk /dev/sdb;
    meta-disk internal; # Internal means that the last part of the backing device is used to store the metadata.
    on drbd01.chainringcircus.org { #on hostname as seen in uname -n and the DNS lookup.
    address 192.168.1.191:7788;
    }
    on drbd02.chainringcircus.org {
    address 192.168.1.192:7788;
    }
    }
    [/code]

    Copy the configuration file:
    [code]
    scp /etc/drbd.conf root@drbd02.chainringcircus.org:/etc/
    [/code]

    Tried to start DRBD but got an error:
    [code]
    service drbd start
    Starting DRBD resources: [
    sites
    no suitable meta data found 🙁
    Command ‘/sbin/drbdmeta 0 v08 /dev/sdb internal check-resize’ terminated with exit code 255
    drbdadm check-resize sites: exited with code 255
    d(sites) 0: Failure: (119) No valid meta-data signature found.

    ==> Use ‘drbdadm create-md res’ to initialize meta-data area. <==

    [sites] cmd /sbin/drbdsetup 0 disk /dev/sdb /dev/sdb internal –set-defaults –create-device –on-io-error=detach failed – continuing!

    s(sites) n(sites) ]……….
    /etc/init.d/drbd status
    drbd driver loaded OK; device status:
    version: 8.3.8 (api:88/proto:86-94)
    GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
    m:res cs ro ds p mounted fstype
    0:sites WFConnection Secondary/Unknown Diskless/DUnknown C

    /etc/init.d/drbd stop
    Stopping all DRBD resources: .
    [/code]

    I did not initialize the meta data storage and this needs to be done before a DRBD resource can be brought online. The DRBD resource needs to be down or detached from its backing storage.

    [code]
    drbdadm create-md sites
    md_offset 1073737728
    al_offset 1073704960
    bm_offset 1073672192

    Found some data

    ==> This might destroy existing data! <==

    Do you want to proceed?
    [need to type ‘yes’ to confirm] yes

    Writing meta data…
    initializing activity log
    NOT initialized bitmap
    New drbd meta data block successfully created.

    service drbd start
    Starting DRBD resources: [
    sites
    Found valid meta data in the expected location, 1073737728 bytes into /dev/sdb.
    d(sites) s(sites) n(sites) ]……….
    [/code]

    Check the status:
    [code]
    cat /proc/drbd
    version: 8.3.8 (api:88/proto:86-94)
    GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
    0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r—-
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1048508
    [/code]

    Make it primary:
    [code]
    drbdadm — –overwrite-data-of-peer primary sites
    cat /proc/drbd
    version: 8.3.8 (api:88/proto:86-94)
    GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
    0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r—-
    ns:67584 nr:0 dw:0 dr:67584 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:980924
    [>……………….] sync’ed: 6.7% (980924/1048508)K delay_probe: 10
    finish: 0:01:27 speed: 11,264 (11,264) K/sec
    [root@localhost etc]# cat /proc/drbd
    version: 8.3.8 (api:88/proto:86-94)
    GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
    0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r—-
    ns:1019904 nr:0 dw:0 dr:1019904 al:0 bm:62 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:28604
    [==================>.] sync’ed: 97.7% (28604/1048508)K delay_probe: 195
    finish: 0:00:02 speed: 11,132 (10,404) K/sec
    [root@localhost etc]# cat /proc/drbd
    version: 8.3.8 (api:88/proto:86-94)
    GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
    0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r—-
    ns:1048508 nr:0 dw:0 dr:1048508 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
    You have new mail in /var/spool/mail/root
    [/code]

    Make a file system:
    [code]
    mkfs.ext3 /dev/drbd0
    mke2fs 1.39 (29-May-2006)
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    131072 inodes, 262127 blocks
    13106 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=268435456
    8 block groups
    32768 blocks per group, 32768 fragments per group
    16384 inodes per group
    Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

    Writing inode tables: done
    Creating journal (4096 blocks): done
    Writing superblocks and filesystem accounting information: done

    This filesystem will be automatically checked every 24 mounts or
    180 days, whichever comes first. Use tune2fs -c or -i to override.
    [/code]

    Testing the filesystem:
    [code]
    mount /dev/drbd0 /sites

    mount
    /dev/sda2 on / type ext3 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    /dev/sda5 on /home type ext3 (rw)
    /dev/sda1 on /boot type ext3 (rw)
    tmpfs on /dev/shm type tmpfs (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    .host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)
    none on /proc/fs/vmblock/mountPoint type vmblock (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    /dev/drbd0 on /sites type ext3 (rw)

    touch /sites/test.txt

    ls /sites
    lost+found test.txt

    umount /sites

    drbdadm secondary sites
    [/code]

    On the second server:
    [code]
    drbdadm primary sites

    mount /dev/drbd0 /sites/

    mount
    /dev/sda2 on / type ext3 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    /dev/sda5 on /home type ext3 (rw)
    /dev/sda1 on /boot type ext3 (rw)
    tmpfs on /dev/shm type tmpfs (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    .host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)
    none on /proc/fs/vmblock/mountPoint type vmblock (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    /dev/drbd0 on /sites type ext3 (rw)

    ls /sites
    lost+found test.txt
    [/code]

    Heartbeat R1-style

    Heartbeat in R1 configuration uses 3 files that must be configured if you are using the heartbeat protocol.
    /etc/ha.d/ha.cf
    /etc/ha.d/haresources
    /etc/ha.d/authkeys

    [code]
    cat /etc/ha.d/authkeys
    auth 1 # A numerical identifier between 1 and 15 inclusive
    # must be unique within the file.
    1 sha1 CHANGEME # Methods can be md5 sha1 or crc.
    # The password is just a string.
    [/code]

    [code]
    chmod 600 /etc/ha.d/authkeys
    [/code]

    Before we take care of the ha.cf file we need to set up the ha_logd configuration file.
    [code]
    cp /usr/share/doc/heartbeat-2.1.3/logd.cf /etc/
    [/code]

    And make changes to the logd.cf file accordingly. Be sure to copy /etc/logd.cf to both servers. Also note that I had to completely stop and then restart the heartbeat daemon for my logging changes to take affect.
    [code]
    cat /etc/logd.cf
    # File to write debug messages to
    # Default: /var/log/ha-debug
    debugfile /var/log/ha-debug.log

    #
    #
    # File to write other messages to
    # Default: /var/log/ha-log
    logfile /var/log/ha.log

    #
    #
    # Facility to use for syslog()/logger
    # Default: daemon
    #logfacility daemon

    # Entity to be shown at beginning of a message
    # for logging daemon
    # Default: "logd"
    entity logd

    # Do we register to apphbd
    # Default: no
    #useapphbd no

    # There are two processes running for logging daemon
    # 1. parent process which reads messages from all client channels
    # and writes them to the child process
    #
    # 2. the child process which reads messages from the parent process through IPC
    # and writes them to syslog/disk

    # set the send queue length from the parent process to the child process
    #
    #sendqlen 256

    # set the recv queue length in child process
    #
    #recvqlen 256
    [/code]

    [code]
    cat /etc/ha.d/ha.cf
    # The recommendation is to use logd.
    use_logd yes
    # Default option is 0, values are 0-255 with 1-3 being the most useful.
    debug 0
    # Timing according to the FAQ at http://www.linux-ha.org/wiki/FAQ
    # warntime should be at least 2 * keepalive
    # warntime should be 1/2 to 1/4 deadtime
    # The interval between heartbeat packets.
    keepalive 1
    # How quickly Heartbeat should issue a "late heartbeat" warning. Warntime is
    # important for tuning deadtime.
    warntime 5
    # How long to decide a cluster node is dead. Too low will flasely declare
    # a death and too high will hinder takeover during a failure.
    # Can be specified as a floating point number followed by a untis-specifier.
    # If units are omitted it defaults to seconds.
    # deadtime 1
    # deadtime 100ms 100 milliseconds
    # deadtime 1000us 1000 microseconds
    deadtime 10
    # 694 is the default but can be changed if multiple clusters are in use.
    udpport 694
    # Which interfaces send UDP broadcast traffic, more than one can be specified.
    bcast eth0
    # auto_failback can be "on" "off" or "legacy"
    auto_failback off
    # Set the nodes in the cluster.
    node in1.eamc.org
    node in2.eamc.org
    # Make sure this IP address is pingable from the bcast network above.
    ping 192.168.1.1
    respawn hacluster /usr/lib/heartbeat/ipfail
    [/code]

    [code]
    cat /etc/ha.d/haresources
    drbd01 192.168.1.190 drbddisk::sites Filesystem::/dev/drbd0::/sites::ext3 httpd
    # Explanation:
    # Primary Server name –> virtual IP address to be used –> DRBD resource as configurd in /etc/drbd.conf
    # –> where to mount the DRBD resource and the filesystem type –> resource to start/stop in case of failover
    [/code]

    Cluster Management
    To take over cluster management from a primary server:
    [code]
    /usr/lib/heartbeat/hb_takeover
    [/code]

    Relinquishing cluster management to a secondary server:
    [code]
    /usr/lib/hearbeat/hb_standby
    /etc/init.d/heartbeat stop
    [/code]

    The order of operations as set by the init scripts:
    [code]
    ls -al /etc/rc3.d/ | egrep "hear|drb"
    lrwxrwxrwx 1 root root 14 Apr 1 11:40 S70drbd -> ../init.d/drbd
    lrwxrwxrwx 1 root root 19 Jun 1 08:58 S75heartbeat -> ../init.d/heartbeat
    [/code]

    Note for Xen users:
    [code]
    # cat /etc/modprobe.d/drbd.conf
    options debd disable_sendpage=1
    [/code]

    Posted in Linux | Leave a comment

    Script to Edit GNS3 Topology Files

    Posted on May 29, 2011 by Judson Bishop

    I have recently been doing more of the GNS3Vault labs and have gotten tired of changing the image name each time I do a lab. The script below is the result. I just change into the directory where the topology.net file is located and run this command.

    The are a couple of tricks to notice about the sed line. First, I replaced the sed delimiter from “/” to “:” to make it easier to work with directory names. Second, the single quotes around $PWD expands the variable before the command is run.

    I hope this helps someone else.

    [code lang=”bash”]
    #!/bin/bash
    # gns3-vault-fix
    # 2011-05-29 Jud Bishop
    # Change image file for GNS3Vault topology.net configuration files.

    PWD=`pwd`
    SAVE=$PWD/topology.net.0
    CONF=$PWD/topology.net

    # Back up the original topology.net file.
    mv $CONF $SAVE

    # A couple of things to notice on this line.
    # 1. I replaced the sed delimiter from "/" to ":" to
    # make it easier to work with directory names.
    # 2. The single quotes around $PWD expands the variable before
    # sed goes to work.
    sed -e ‘s:image .*:image = ‘$PWD’/c3640-a3js-mz.124-25b.bin:g’ &lt;$SAVE &gt;$CONF
    [/code]

    Posted in Routing | Leave a comment

    Veritas/Symantec Baremetal Restore

    Posted on April 12, 2011 by Judson Bishop

    I spent a considerable amount of time over the last couple of months testing different restore processes. This is my documentation for restoring Veritas/Symantec backups to a Linux server.

    The general outline is this:
    1. Create a LiveUSB drive to boot CentOS with a persistent overlay.
    2. Install Symantec backupexec on the LiveUSB drive.
    3. Recreate the drive layout on the new server.
    4. Restore to the new server.

    Create LiveUSB
    CentOS makes a LiveCD toolset for CentOS. They also have directions for how to create a LiveUSB drive with persistent overlay. Please follow those links for more in depth directions.

    You must install CentOS LiveUSB on an ext2/3/4 formatted USB drive in order for Symantec to work. If you leave the VFat partition Symantec will nowork properly and you will get the error “An unknown error occurred within the NDMP subsystem.” Once I reformatted the USB drive as ext3 and installed a new LiveUSB with persistent overlay Symantec worked. My guess is it has to do with permission bits but that is only a guess.

    I downloaded the LiveCD tools for Centos here.

    Here is some of my history from that server:

    [code]
    umount /mnt
    fdisk /dev/sdb
    mkfs -t ext3 /dev/sdb1
    mkfs -t ext3 /dev/sdb2
    livecd-iso-to-disk –overlay-size-mb 1500 CentOS-5.5-i386-LiveCD.iso /dev/sdb1
    mount /dev/sdb1 /mnt
    ls /mnt
    [/code]

    LiveUSB Setup
    I wanted to give it a persistent name and IP address for use in our data center. For some of this I was also shooting in the dark in order to get Symantec working, for thoroughness I include it here.

    [code]
    vi /etc/sysconfig/network
    HOSTNAME=recovery.chainringcircus.org
    [/code]

    [code]
    vi /etc/sysconfig/networking/devices/ifcfg-eth0
    DEVICE=eth0
    BOOTPROTO=none
    ONBOOT=yes
    NETMASK=255.255.255.0
    IPADDR=192.168.1.200
    GATEWAY=192.168.1.1
    TYPE=Ethernet
    USERCTL=no
    IPV6INIT=no
    PEERDNS=yes
    DNS1=192.168.1.201
    DNS2=192.168.1.202
    DOMAIN=chainringcircus.org
    [/code]

    After I configured the hostname and network settings I rebooted to make sure that the persistent overlay worked. I also turned on sshd and set it to runlevel 3 in /etc/inittab because I did not want to mess with a gui, but that is your choice. When everything came up properly I installed Symantec and we did a test restore.

    Install Symantec
    I cover installing Symantec on Linux in another post here. You need to install an older package for compatibility:

    [code]
    yum install compat-libstdc++-296-2.96-138.i386
    [/code]

    The specific Symantec rpms I installed are listed below. I did try a newer package from Symantec but it did not allow us to restore erroring with a different message. I will also say that was when we were on a VFat partition. Once I got everything working on an ext3 partition I quit testing.

    [code]
    VRTSvxmsa-4.2.1-211.i386.rpm
    VRTSralus-10.00.5629-0.i386.rpm
    [/code]

    Recreate Drive Layout
    For thoroughness I am going to cover creating the logical volumes that are default for CentOS and RHEL.

    First I need to lay out the drive mappings. This is from the old server which I am cloning onto a similar server. In this section I am just going to show the output of a number of commands that confirm the file system layout of the server.

    File layout on the old server
    From the file /etc/fstab:

    [code]
    LABEL=/boot /boot ext3 defaults 1 2
    /dev/VolGroup00/LogVol00 / ext3 defaults 1 1
    /dev/VolGroup00/LogVol01 swap swap defaults 0 0
    [/code]

    From the mount command:

    [code]
    /dev/sda1 on /boot type ext3 (rw)
    /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
    [/code]

    From the fdisk command:

    [code]
    Disk /dev/sda: 219.8 GB, 219823472640 bytes
    255 heads, 63 sectors/track, 26725 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 13 104391 83 Linux
    /dev/sda2 14 26725 214564140 8e Linux LVM
    [/code]

    Working my way up from the bottom of the LVM stack with the physical volume, the volume group and finally the logical volume.

    From pvdisplay:

    [code]
    — Physical volume —
    PV Name /dev/sda2
    VG Name VolGroup00
    PV Size 204.62 GB / not usable 31.29 MB
    Allocatable yes
    PE Size (KByte) 32768
    Total PE 6547
    Free PE 4
    Allocated PE 6543
    PV UUID jAuzGO-3Zpz-4T3K-mqcI-Ql6D-1dqf-wj917q
    [/code]

    From vgdisplay:

    [code]
    — Volume group —
    VG Name VolGroup00
    System ID
    Format lvm2
    Metadata Areas 1
    Metadata Sequence No 3
    VG Access read/write
    VG Status resizable
    MAX LV 0
    Cur LV 2
    Open LV 2
    Max PV 0
    Cur PV 1
    Act PV 1
    VG Size 204.59 GB
    PE Size 32.00 MB
    Total PE 6547
    Alloc PE / Size 6543 / 204.47 GB
    Free PE / Size 4 / 128.00 MB
    VG UUID LJc2HJ-D7Gr-ketA-5TSe-ppQM-m5di-4YMEgZ
    [/code]

    From lvdisplay:

    [code]
    — Logical volume —
    LV Name /dev/VolGroup00/LogVol00
    VG Name VolGroup00
    LV UUID HcyaVT-DOEs-1Rdy-h7af-7i0t-P0EF-K2cCxy
    LV Write Access read/write
    LV Status available
    # open 1
    LV Size 202.53 GB
    Current LE 6481
    Segments 1
    Allocation inherit
    Read ahead sectors auto
    – currently set to 256
    Block device 253:0

    — Logical volume —
    LV Name /dev/VolGroup00/LogVol01
    VG Name VolGroup00
    LV UUID ZpAnvu-Of5D-PoEO-HaDN-2krv-zIXp-1fF5av
    LV Write Access read/write
    LV Status available
    # open 1
    LV Size 1.94 GB
    Current LE 62
    Segments 1
    Allocation inherit
    Read ahead sectors auto
    – currently set to 256
    Block device 253:1
    [/code]

    On the old server the drive is broken into two partitions, sda1 and sda2:
    sda1 /boot 100MB
    sda2 Volume Group ~200GB

    The volume group on the old server on the sda2 partition is broken into two logical volumes:
    LogVol00 / ~200GB
    LogVol01 swap ~2GB

    It is important to remember that the drive mappings on the old server will not necessarily match the mappings on the new one. For instance on the old server the raid was on /dev/sda and on the new server the raid drive is mapped on /dev/sdb. That is only because I am booting from /dev/sda on the LiveUSB, under normal circumstance it will come back up as /dev/sda.

    Working on the new server recreate the partitions

    [code]
    fdisk /dev/sdb
    Command (m for help): p

    Disk /dev/sdb: 1199.9 GB, 1199906488320 bytes
    255 heads, 63 sectors/track, 145880 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System

    Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    p
    Partition number (1-4): 1
    First cylinder (1-145880, default 1): 1
    Last cylinder or +size or +sizeM or +sizeK (1-145880, default 145880): +200M

    Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    p
    Partition number (1-4): 2
    First cylinder (26-145880, default 26):
    Using default value 26
    Last cylinder or +size or +sizeM or +sizeK (26-145880, default 145880):
    Using default value 145880

    Command (m for help): a
    Partition number (1-4): 1

    Command (m for help): t
    Partition number (1-4): 2
    Hex code (type L to list codes): 8e
    Changed system type of partition 2 to 8e (Linux LVM)

    Command (m for help): p

    Disk /dev/sdb: 1199.9 GB, 1199906488320 bytes
    255 heads, 63 sectors/track, 145880 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 1 25 200781 83 Linux
    /dev/sdb2 26 145880 1171580287+ 8e Linux LVM

    Command (m for help): w
    The partition table has been altered!

    Calling ioctl() to re-read partition table.

    WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
    The kernel still uses the old table.
    The new table will be used at the next reboot.
    Syncing disks.
    [/code]

    [code]
    /usr/sbin/lvmdiskscan
    /dev/ramdisk [ 16.00 MB]
    /dev/live-squashed [ 669.80 MB]
    /dev/root [ 4.00 GB]
    /dev/ram [ 16.00 MB]
    /dev/live-osimg [ 4.00 GB]
    /dev/live [ 7.45 GB]
    /dev/ram2 [ 16.00 MB]
    /dev/live-overlay [ 1.46 GB]
    /dev/sda2 [ 7.47 GB] LVM physical volume
    /dev/ram3 [ 16.00 MB]
    /dev/ram4 [ 16.00 MB]
    /dev/ram5 [ 16.00 MB]
    /dev/ram6 [ 16.00 MB]
    /dev/ram7 [ 16.00 MB]
    /dev/ram8 [ 16.00 MB]
    /dev/ram9 [ 16.00 MB]
    /dev/ram10 [ 16.00 MB]
    /dev/ram11 [ 16.00 MB]
    /dev/ram12 [ 16.00 MB]
    /dev/ram13 [ 16.00 MB]
    /dev/ram14 [ 16.00 MB]
    /dev/ram15 [ 16.00 MB]
    /dev/sdb1 [ 196.08 MB]
    /dev/sdb2 [ 1.09 TB]
    7 disks
    16 partitions
    0 LVM physical volume whole disks
    1 LVM physical volume
    [/code]

    Turn off the the LVM in order to make changes, this is just a precautionary step if you have repartitioned your drive.

    [code]
    lvm vgchange -an
    [/code]

    Create the LVM.

    [code]
    vgscan
    Reading all physical volumes. This may take a while…

    pvcreate -ff /dev/sdb2
    Physical volume "/dev/sdb2" successfully created
    [/code]

    Create and activate the volume groups.

    [code]
    vgcreate VolGroup00 -l 0 -p 0 -s 32m /dev/sdb2
    Volume group "VolGroup00" successfully created

    vgchange -ay VolGroup00
    0 logical volume(s) in volume group "VolGroup00" now active
    [/code]

    Finally, create the logical volumes. Even though I have 1.1T I decided to start using 800G, leaving myself room if I want to add another mount point.

    [code]
    lvcreate -L 800000m -r auto -n LogVol00 VolGroup00
    Logical volume "LogVol00" created

    lvcreate -L 4096m -r auto -n LogVol01 VolGroup00
    Logical volume "LogVol01" created
    [/code]

    Read in the new volume groups.

    [code]
    vgscan
    Reading all physical volumes. This may take a while…
    Found volume group "VolGroup00" using metadata type lvm2
    [/code]

    Format all of the partitions:

    [code]
    mkfs -t ext3 /dev/sdb1
    mke2fs 1.39 (29-May-2006)
    Filesystem label=
    OS type: Linux
    Block size=1024 (log=0)
    [output removed for brevity]
    [/code]

    [code]
    mkfs -t ext3 /dev/VolGroup00/LogVol00
    mke2fs 1.39 (29-May-2006)
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    [output removed for brevity]
    [/code]

    [code]
    mkswap /dev/VolGroup00/LogVol01
    Setting up swapspace version 1, size = 4294963 kB
    [/code]

    Here are some quick commands if you mess up and need to delete any of the LVM stack.

    [code]
    lvremove -f /dev/VolGroup00/LogVol00
    lvm lvremove -f /dev/VolGroup00/LogVol01
    lvm vgchange -a n VolGroup00
    lvm vgremove VolGroup00
    [/code]

    Restore
    Mount the root under /mnt and then have Veritas restore to that mount point.

    [code]
    mount -t ext3 /dev/VolGroup00/LogVol00 /mnt
    [/code]

    If you have made it this far then you must really need the information. Here are a couple of screenshots from our backup guru to help in the restore process.

    “Preserve Tree”, by default it is selected.
    restore1

    Select “Restore over existing files”, “Restore all information for files and directories” and “Preserve tree”.
    restore2

    Install grub on the new machine
    During the restore we restored all of the files and directories to /mnt, including /boot. In order to get everything working again we need to setup the boot directory and then grub. But Red Hat and CentOS 4.X uses legacy grub.

    Copy all of /mnt/boot to the real /boot directory.

    [code]
    mkdir /mnt/newboot
    mount /dev/sdb1 /mnt/newboot
    cp -r /mnt/boot/* /mnt/newboot/

    umount /mnt/newboot

    mount /dev/sdb1 /boot

    grub> root (hd1,0)
    Filesystem type is ext2fs, partition type 0x83

    grub> find /grub/stage1
    (hd1,0)

    grub> setup (hd1)
    Checking if "/boot/grub/stage1" exists… no
    Checking if "/grub/stage1" exists… yes
    Checking if "/grub/stage2" exists… yes
    Checking if "/grub/e2fs_stage1_5" exists… yes
    Running "embed /grub/e2fs_stage1_5 (hd1)"… 16 sectors are embedded.
    succeeded
    Running "install /grub/stage1 (hd1) (hd1)1+16 p (hd1,0)/grub/stage2 /grub/grub.conf"… succeeded
    Done.

    grub > quit
    [/code]

    Restore /dev and /tmp
    Depending upon your backup options you may need to restore the /dev directory and create a tmp directory. You need to set the sticky bit on /tmp.

    [code]
    cp devices.tar /mnt/lvm/VolGroup00-LogVol00/
    cd /mnt/lvm/VolGroup00-LogVol00/
    tar -tvf devices.tar
    tar -xvf devices.tar

    chroot /mnt/lvm/VolGroup00-LogVol00/
    mkdir /tmp
    ls -al /
    chmod a+rwx /tmp
    chmod +t /tmp
    exit
    [/code]

    Finally you need to set up your ethernet interfaces by editing the file,
    /etc/sysconfig/networking/devices/ifcfg-eth0

    Posted in Linux | Leave a comment

    Xenix

    Posted on March 16, 2011 by Judson Bishop

    Today on Slashdot someone had a question about getting data off of an old Xenix server. A few years ago I did a consulting job for a customer who had an old Xenix server with no ethernet card that needed to get some data for the State Police. The server ran an old database that kept track of training records and they needed that information in order to “webify” the process.

    I wish I could remember the ins and outs of the problem. I gave the customer a write up of how I did it and the problems I had getting the usr directory off of the system. Unfortunately I can’t find the report.

    Regardless, it was an interesting project, here is the script that eventually worked:
    [code lang=”bash”]
    #!/bin/sh

    for I in bin boot dos etc lib oa.files once shlib tmp u unit57 unit58 unit59 usr xenix
    do
    echo "tar -cf – $I|uuencode -"
    tar -cf – $I|uuencode –
    done

    tar -cf – `find /usr -type f -print` >2/usr.err|uuencode –

    for I in `cat usr.txt`
    do
    tar -cf – $I |uuencode –
    done
    [/code]

    Posted in Linux | Leave a comment