Catalyst 4500 VSS upgrade failed: invalid input detected at '^' marker

[JohnCKirk]

I recently tried to upgrade a VSS pair of Catalyst 4500 switches (with the WS-X45-SUP7-E supervisor), from version 3.4.0 to 3.11.4.

I copied over the new firmware file (to bootflash and slavebootflash), changed the boot variable, then rebooted the standby switch:

redundancy reload peer

When the switch rebooted, it displayed the new version:

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.11.04.E RELEASE SOFTWARE (fc2)

However, it then got an error:

Initializing as Virtual Switch STANDBY processor
 ^
% Invalid input detected at '^' marker.

This was followed by another message which repeated every 5 minutes (300 seconds):

The communication with the active supervisor cannot be established.
Possible reasons:
-Active sup. not completely initialized.
-Faulty active supervisor.
-Error on the communication channel.
Ipc packets sent so far: 300 (1 packet/sec).

I found a bug report which suggests that the "invalid input" message is just a cosmetic issue:
CSCuw85722 : Bug Search Tool (cisco.com)

However, it only appears with version 3.11.4, i.e. it didn't appear when I reverted to version 3.4.0.

I'm guessing that there's something in the running-config which has been deprecated in the new version. However, I'd normally expect the "invalid input" message to actually indicate what's wrong, with the caret symbol pointing to the relevant place.

I didn't see anything relevant in the release notes, but I'm checking again. I'm guessing that I'll need to go in smaller hops, e.g. 3.4 to 3.6, then 3.6 to 3.8, etc.

In the meantime, is there any way to get more details on what's gone wrong?

[Leo Laohoo]

---

@JohnCKirk wrote:

-Faulty active supervisor.

---

What is the ROMMON version?

[JohnCKirk]

The ROMMON version is 15.0(1r)SG10.

I believe that should be ok, based on the release notes:
Release Notes for the Cisco Catalyst 4500E Series Switch, Cisco IOS XE 3.11.xE - Cisco

"If you are upgrading to Cisco IOS XE Release 3.9.xE and using Supervisor Engine 7-E or 7L-E, we recommend that you use ROMMON version 15.0(1r)SG10 or a higher version (if available)."

[Reza Sharifi]

On the 4500, you usually don't need to do a step upgrade and can go directly to 3.11.4.

After loading the IOS to both sups and setting the boot variable, and saving the config, in a maintenance window reboot both switches by using

redundancy reload shelf

HTH

[Leo Laohoo]

Console into the Primary Sup Engine and reboot. 

Post the entire bootup process.  

[JohnCKirk]

Thanks both. Reza, I'd prefer not to reboot both switches at once if I can avoid it, because that would cause problems for the rest of the network (i.e. they're set up with redundancy at the moment).

Leo, I've attached the console output from my previous attempt. I've redacted the switch name and MAC address, and removed a couple of irrelevant ROMMON commands (e.g. a dir listing), but aside from that it should be complete.

[Leo Laohoo]

---

@JohnCKirk wrote:

 Non-Redundant system or peer not running IOS
 System Uplinks & Linecards have been reset!!

---

This is why the line card will not boot into 3.11.4. 

What firmware version is the Active line card running?

[JohnCKirk]

Just to clarify, there are 2 chassis, and each chassis has 1 supervisor. At present, both supervisors are running firmware version 3.4.0,

switch#sh redundancy
Redundant System Information :

------------------------------
Available system uptime = 8 years, 19 weeks, 2 hours, 9 minutes
Switchovers system experienced = 2
Standby failures = 1
Last switchover reason = user_forced

Hardware Mode = Duplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
------------------------------
Active Location = slot 1/5
Current Software state = ACTIVE
Uptime in current state = 4 days, 6 hours, 35 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.00.SG RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Wed 05-Dec-12 04:38 by prod
BOOT = bootflash:cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin,1;
Configuration register = 0x2102

Peer Processor Information :
------------------------------
Standby Location = slot 2/5
Current Software state = STANDBY HOT
Uptime in current state = 4 days, 5 hours, 31 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.00.SG RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Wed 05-Dec-12 04:38 by pro
BOOT = bootflash:cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin,1;
Configuration register = 0x2102

[Reza Sharifi · Accepted solution]

Hi John,

The output looks good. Since you are doing this Manually, in order for it to work correctly you would need to reboot both chassis at the same time with the command I provided if not once one switch reboots, there will be an IOS mismatch between the chassis and with VSS that is not a good thing. The other option is using ISSU, in which you will reboot one switch at a time but I have not seen it work flawlessly (no downtime). If you have a maintenance window and can effort a 15 min downtime, the manual way is the best way to go.

https://community.cisco.com/t5/switching/4500x-vss-upgrade-ios/td-p/2635131

HTH 

[JohnCKirk]

Thanks Reza - that's not what I wanted to hear, but unfortunately I think you're correct.

ISSU won't work in this case, because the current version is prior to 3.6.0. See:
Configuring the Cisco IOS XE In Service Software Upgrade Process
"An ISSU upgrade from any release prior to IOS XE 3.6.0E to 3.6.0E or later is unsupported for IOS XE
supervisor engines (SUP7-E, SUP7L-E and SUP8-E)."

Looking at the link you posted, someone suggested a workaround (breaking the VSS link, then recreating it afterwards) but that looks quite risky. So, I have some planning to do...

[Reza Sharifi]

Looking at the link you posted, someone suggested a workaround (breaking the VSS link, then recreating it afterwards) but that looks quite risky. 

That is exactly why I am suggesting the manual procedure if you can get a maintenance window because we have seen it work. On the other hand, with ISSU, you are going to be rebooting multiple times, and not sure at the end it will all be seamless. Depending on the number of modules you have installed, the full reload time for the 4500 VSS is about 15 to 20 minutes.

HTH

[Leo Laohoo]

Everyone in this forums (will) know I am not a big believer of ISSU or FSU/eFSU/xFSU.

@Reza Sharifi and I have been recommending people to use the manual method of upgrading chassis in VSS (regardless if they are 4500, 6500 and even 9400/9500/9600).  

A good guide on how to upgrade the switch (in VSS), including ROMMON, can be found HERE.

Hope this works.  

[JohnCKirk]

Sorry for the delayed reply - I was waiting for a maintenance window. Just to confirm, rebooting both chassis together fixed the problem, and the upgrade was successful. Thanks to Leo and Reza for all their help!