Cat4500x VSS ISSU issue

[Erik Molenaar]

Hi,

I'm having a bit of a problem doing an ISSU. Perhaps someone can give me some clues.

I'm trying to do an ISSU upgrade of a Cat 4500x VSS cluster, from version cat4500e-universalk9.SPA.03.05.03.E.152-1.E3 to version cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.

I loaded the image into bootflash: and slavebootflash:

Futhermore i used these commands to check sso, config-register and other states, as mentioned in the docs:

Switch# show issu state [detail]
Switch# show redundancy
Switch# show switch virtual
Switch# show switch virtual slot-map

Both VSL port-channels Po10 (switch 1) and Po20 (switch 2) are in the up state and cdp is disabled on the actual links.

The following VSL configuration commands are inserted automatically on all VSL member ports:

switchport mode trunk
switchport nonegotiate
no lldp transmit
no lldp receive
no cdp enable
service-policy output VSL-Queuing-Policy

ROMmon version on both units is 15.0(1r)SG11 (minimum required is 15.0(1r)SG10)

So, all looks well. After this i issue the command:

Switch# changeversion bootflash:cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.bin

As expected, the slave loads the new software and starts rebooting. I can see this happening on the console of the slave. After it has loaded the software it starts loading the configuration. What i see at this point is an error message: % Invalid input detected at '^' marker.

After a few minutes the slave console says it's not able to make contact with the active unit. On the active unit Port-channel 20 - the port-channel on switch 2 - stays down. After a while the slave reboots and starts loading the original image, and rolls back as expected.

I'm expecting to find some configuration issue, so what i did was to install a stand alone 4500x with the new software, using the VSS configuration.

This works, but comparing the pre-upgrade config with the post-upgrade config i notice that the command 'no cdp ena' on the VSL links is gone.

Perhaps that's the reason the VSL's are not activated during an ISSU? Ofcourse i tried to turn off cdp on the VSL links on the upgraded switch manually, but this doesn't work.

Am i hitting a bug or incompatibility issue here?

Erik

[Philip D'Ath]

Which supervisor engine do you have?

You could try turning on CDP on the VSL link.  It is only your boxes and a very critical link, so no real security implications.

Is there plenty of spare space on the flash card for the ISSU process to unpack the image once it starts?

[Erik Molenaar]

I am running a 4500-x, not sure which supervisor it contains. 

There's also lots of space on the bootflash: (1692037120 bytes total (1339691008 bytes free))

Tried turning off cdp on the VSL links but there's only a limited command set that you can use on VSL links.

Switch(config-if)#?
virtual link interface commands (restricted):
 channel-group Etherchannel/port bundling configuration
 default Set a command to its defaults
 description Interface specific description
 exit Exit from virtual link interface configuration mode
 load-interval Specify interval for load calculation for an interface
 logging Configure logging for interface
 no Negate a command or set its defaults
 service-policy Configure CPL Service Policy
 shutdown Shutdown the selected interface

The command "no cdp enable" is not part of this.

[Philip D'Ath]

"show inventory" will tell you which supervisor you have,

[Erik Molenaar]

Hmmm thanks Philip, but not much on that either:

Switch#show inventory

NAME: "Switch1 System", DESCR: "Cisco Systems, Inc. WS-C4500X-16 2 slot switch "
PID: , VID: , SN: JAE1909086SF

NAME: "Switch1 Supervisor 1 (virtual slot 1)", DESCR: "4500X-16 10GE (SFP+)"
PID: WS-C4500X-16 , VID: V03 , SN: JAE197364CB

Anyway, the new software version does work on this platform. It does install in a stand alone situation. 

Next thing i did was to upgrade both units separately to the new version and then reconnect the VSL's. Actually, this works for VSS!

But it lacks the advantage of doing an ISSU which is not having to deal with downtime.

[paul driver]

Hello

How are you reloading the VSS?

1) download the bin file to your bootflash: and slavebootflash:
2) delete and reset you bootvar
no bootvar
wr
boot system bootflash:xxxxx.bin
wr

3) verfiy bin file - verfiy bootflash:xxxxx.bin
4) reload vss using redundancy reload shelf


res
Paul

That should be it

[Erik Molenaar]

Thanks Paul,

Using the redundancy reload shelf command reloads the whole VSS, without having the advantage of ISSU. Perhaps you mean redundancy reload peer to first reload the peer unit using the new bootvar?

ISSU documentation states that one should use the changeversion command (or use the four-step procedure). That makes the peer do a reload to the new software. Actually this works. On the console of the peer unit i can observe that the switch loads the new software and fully boots up, but it doens't give me a login prompt and doensn't form a VSS. It only keeps saying:

The communication with the active supervisor cannot be established.

Possible reasons:

-Active sup. not completely initialized.

-Faulty active supervisor.

-Error on the communication channel.

And then it automatically revert back to the old software version, restoring VSS functionality

Well, the Active unit is still working, so that's not the issue.  I'm suspecting the VSL links, but how to troubleshoot this is the question....

[Rolf Fischer · Accepted solution]

Erik,

I'm trying to do an ISSU upgrade of a Cat 4500x VSS cluster, from version cat4500e-universalk9.SPA.03.05.03.E.152-1.E3 to version cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.

Have a look at the Prerequisites to Performing ISSU

"A permanent “ISSU barrier" exists between pre-IOS XE 3.6.0E and IOS XE 3.6.0 releases:

ISSU is supported between versions on the same side of the barrier but it is not supported between versions on opposite sides.

This restriction applies to Catalyst 4500X in a VSS, (...).

If you are running a release prior to IOS XE 3.6.0E (3.5.1E, for example), you cannot perform an ISSU upgrade to IOS XE 3.6.0E."

So I'd try the manual procedure with redundancy reload peer and redundancy force-switchover.

HTH
Rolf

[Erik Molenaar]

Thanks Rolf, now i see it too! That's a pity!

Makes you wonder what good is ISSU when it's broken between versions hahaha :-)

Anyway, this solved my problem! Thanks everyone!!

Erik

[Rolf Fischer]

You're welcome! The good news is that there are no such barriers when you want to update 3.6 to 3.7 or 3.8. Thanks for marking the question as answered.

Rolf