Re: Cisco 6880-X VSS Upgrade procedure

Nagra_net · ‎07-06-2018

Hello,

I have two Catalyst 6880-X in a VSS pair that I need to upgrade to one of the suggested release. I don't have any FEX attached. I'm currently running version 15.1(2)SY6 and would need to go to either 15.2.1-SY6 or 15.5.1-SY1.

I have a maintenance window to do this upgrade this weekend, but I still need to minimize downtime/impact as much as possible.

The documentation I found is a bit confusing, it mentions ISSU and eFSU (enhanced Fast Software Upgrade). I'm not sure what the difference is between the two. I found a compatibility matrix that shows the different minor versions in the same major version and if they are compatible for ISSU/eFSU, but no indication anywhere if you can do a hit-less or minimal downtime upgrade between major versions.

Can someone confirm what the best way to upgrade this pair of switches is to minimize downtime?

Also, is there something that would make it better to go to one version or the other (15.2 or 15.5)?

Thank you,

Luke

Leo Laohoo · ‎07-06-2018

There are several ways of upgrading a VSS pair. Cisco's way is to use FSU/eFSU or ISSU. Results can vary between it works and ... not.

Some of us prefer to use a different method:

Copy the IOS to the primary bootflash:
Copy the IOS to the secondary bootflash:
Remove the boot variable string pointing to the old IOS
Add the new boot variable string pointing to the new IOS
Insert the boot variable string pointing to the old IOS
Make sure the config-registry is 0x2102
Save the config
Reboot

Hope this helps.

Nagra_net · ‎07-09-2018

Hello Leo,

Thank you for your answer. We tried that method, it didn't work on the first go, but worked the second time.

We changed the boot statements as you indicated and rebooted using "redundancy reload shelf". The primary switch reloaded, but the secondary didn't. And the primary failed to boot with the new IOS so ended up booting with the old IOS and coming up as standby. We then had to console in to the second device (now primary) and reload it using "reload". This time both of them reloaded and booted up with the new image.

I'm not sure why it failed the first time and why only one of the devices rebooted.

It's great that Cisco has a way of doing upgrades with minimal or no downtime, it's just too bad it doesn't work well, and doesn't work at all between major versions (from what I can find online). The documentation for the 6880-X ISSU/eFSU is also quite poor when compared with the Nexus ISSU documentation.

I would think a lot of people need to upgrade their switches from one major version to another, every now and again, and want to do that with as little downtime as possible - or even better, no downtime at all.

If anyone else has information on the recommended way to do these upgrades for minimal downtime, it would be very helpful. We are now running 15.5(1)SY1.

Thanks,

Luke

Reza Sharifi · ‎07-09-2018

Hi,

"redundancy reload shelf" should have worked. Did you have Cisco TAC online during the upgrade?

Also, did you capture the output of "sh redundancy" before the upgrade?

HTH

Nagra_net · ‎07-10-2018

Hi,

I didn't have TAC online no, we don't have Smartnet, but have a contract through a Cisco partner. We have to go through them to get a TAC case opened.

Here's the output of "show redundancy" before the upgrade:

SWC01#show redundancy 
Redundant System Information :
------------------------------
Available system uptime = 1 year, 32 weeks, 1 day, 21 hours, 13 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
-------------------------------
Active Location = slot 1/5
Current Software state = ACTIVE
Uptime in current state = 1 year, 32 weeks, 1 day, 21 hours, 13 minutes
Image Version = Cisco IOS Software, c6880x Software (c6880x-IPSERVICESK9-M), Version 15.1(2)SY6, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 10-Sep-15 01:14 by prod_rel_team
BOOT = bootdisk:c6880x-ipservicesk9-mz.SPA.151-2.SY6.bin,12;
Configuration register = 0x2102

Peer Processor Information :
----------------------------
Standby Location = slot 2/5
Current Software state = STANDBY HOT 
Uptime in current state = 1 year, 32 weeks, 1 day, 21 hours, 9 minutes
Image Version = Cisco IOS Software, c6880x Software (c6880x-IPSERVICESK9-M), Version 15.1(2)SY6, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 10-Sep-15 01:14 by prod_rel_team
BOOT = bootdisk:c6880x-ipservicesk9-mz.SPA.151-2.SY6.bin,12;
CONFIG_FILE = 
BOOTLDR = 
Configuration register = 0x2102

Luke

Reza Sharifi · ‎07-10-2018

Hi,

The output looks good. The only thing I can think of is that maybe there was a bug in the previous version causing only one to chassis to reboot and not both.

HTH

shamg1974 · ‎05-21-2021

Hello Reza,

Thanks for your help in the past. I had a question regarding this post.

I will be going through the process of upgrading the IOS on the Cisco 6840-X VSS. Specifically Step #8. After steps 1-7 (listed by Leo above) are completed, what do you recommend for reboot? I'm reading several different commands, some of which reboot one chassis at a time and then Redundancy reload shelf which will reload then entire VSS at once?

Can you clarify which command to run to reboot? Basically, should I just reboot the whole VSS at one time?

Thanks,

Shameer

Leo Laohoo · ‎07-09-2018

Thanks for the update and we are happy to see the upgrade went well (albeit a few hiccups).

wilkins_95 · ‎05-06-2019

Hello Luke

I see that your post is about a year old, did you go through an upgrade again since? and if so, was FSU still the method used and preferred? Next week I will be upgrading my 6880x VSS pair for the 1st time from 15.1(02)SYS01 to 15.5-1SY3. It will be a scheduled outage so I'm not concern about up-time, thus eliminating the ISSU method. Any feedback via your experience would be appreciated. Thanks

perkin · ‎08-27-2019

Hello Wilkins

I read this thread and I planned to do the same too, may I know if any special procedure or summary you can share with us? :)

I want to get rid fo the "watermark" bug on our 15.5.1...

Leo Laohoo · ‎08-28-2019

Read the Release Notes carefully. There are releases that require a ROMMON upgrade.
Follow the procedures above otherwise raise a TAC Case and get TAC to WebEx and do the upgrade.

perkin · ‎08-28-2019

thank you Leo, looks the way so standard, I guess all our nervous came from VSS rather than the single system, thanks!

Leo Laohoo · ‎08-28-2019

@Reza Sharifi and I have been guiding people how to do the upgrade, standalone or VSS, using the "non standard" ISSU, FSU/eFSU method.
So far, no one has complained.

Martin Morken · ‎09-02-2019

I upgraded my 6880 in VVS this morning from 15.2(1)SY6 to 15.5(1)SY3.

It went fine.

Here is the guide I got and followed:

	Command	Purpose
Step 1	Router# copy tftp disk_name	Uses TFTP to copy the new software image to flash memory on the VSS active and VSS standby chassis (disk0: and slavedisk0:). Answer the prompts to identify the name and location of the new software image.
Step 2	Router# config terminal	Enters global configuration mode.
Step 3	Router(config)# no boot system	Removes any previously assigned boot variables.
Step 4	Router(config)# config-register 0x2102	Sets the configuration register.
Step 5	Router(config)# boot system flash device:file_name	Configures the chassis to boot the new image.
Step 6	Router(config)# end	Returns to privileged EXEC mode.
Step 7	Router# copy running-config startup-config	Saves the configuration.
Step 8	Router# redundancy reload peer	Reloads the VSS standby chassis and brings it back online running the new version of the Cisco IOS software. Due to the software version mismatch between the two chassis, the VSS standby chassis will be in RPR redundancy mode. Note Before reloading the VSS standby chassis, make sure you wait long enough to ensure that all configuration synchronization changes have completed.
Step 9	Router# redundancy force-switchover	Forces the VSS standby chassis to assume the role of the VSS active chassis running the new Cisco IOS image. The modules are reloaded and the module software is downloaded from the new VSS active chassis. The old VSS active chassis reboots with the new image and becomes the VSS standby chassis.

"redundancy reload peer" forced the standby chassis to boot and Upgrade. No downtime.

"redundancy force-switchover" forced the standby chassis to take over. About 5 minutes downtime.

perkin · ‎09-03-2019

many thanks!
that is really good information!
re "redundancy reload peer" forced the standby chassis to boot and Upgrade. No downtime.
meaning that the Layer 2 on the standby unit without any downtime? is that only *one* RP on your secondary unit?