Beware the config register…

Okay, I had a really embarrassing experience at a client site recently. While deploying a set of VSS switches, I ran into a sup that continued to boot into ROMMON mode. Typically this can be caused by:

• Missing or invalid IOS image
• Incorrect boot statement(s) in the configuration
• Configuration register configured to boot into ROMMON mode
• Corrupt memory (can happen)

Well, the sup would boot up just fine when the boot ROMMON command was entered, so I doubted that the memory was corrupt.  Issuing a dir sup-bootdisk: (from within the IOS) confirmed that an IOS image was present, as was validated by the fact that I was able to boot up into the IOS image without any difficulty (after manually issuing the boot ROMMON command).

The next thing I did was check the configuration register setting.  Typically 0×2102 is used, as was the case when I looked at the show version and show bootvar commands.  Also, within ROMMON mode, looking at the output from the set command, it indicated that the conf-reg was 0×2102 (correct).  I was perplexed.

I did some digging around on the Internet and found the following article which helped solve the problem: IOS Catalyst 6500/6000 Resets with Error “System returned to ROM by power-on (SP by abort)”.

Remember that a 6500 sup has a two-step bootup process:

1. Initialize the Switch Processor (SP)
2. Initialize the Route Processor (RP)

This bootup process can be observed a console cable is attached to a 6500 sup during the startup sequence.  A copyright notice along with some mention of the SP (_sp) is shown.  This is the part of the bootup that deals with the SP.  Next, the console is handed over to the RP (once the SP is initialized), at which point another copyright notice typically appears, with mention of the RP (_rp).

The SP and RP have completely separate and indepedent configuration registers.  This can lead to some confusion, which is what I was experiencing at the customer site.  The show version, show bootvar and set (ROMMON) commands I was examing were all displaying the output of the RP config-register.  It wasn’t until I used the remote command switch show bootvar command (which I found in the above CCO URL) that I was able to see the SP config-register.

What is confusing to me is that the SP is what was triggering the ROMMON mode, yet the ROMMON appeared to be the RP ROMMON, not the SP ROMMON (the config-register shown in ROMMON was correct, yet wasn’t the actual value configured on the SP).

To have a short, happy ending, I verified that the SP config-register was not 0×2102 by using the remote command switch show bootvar command.  To fix the problem, I simply went into configuration mode and used the boot config-register 0×2102 command, after which I verified that the SP was going to change upon the next reload (again with the remote command switch show bootvar command).

It’s good to learn about things and reinforce what you already know (or should know/remember).  Sometimes these experiences come at embarassing times (such as at a customer site), but can be useful if properly handled.