Upgrading Quad supervisor 9606

richyvrlimited · ‎06-24-2024

Can anyone advise the best way to upgrade a quad supervisor 9600 chassis?

I've used ISSU sucessfully once following cisco documentation, but that was as soon as the chassis was put into production as a test and it worked perfectly, the 2nd time 18 months later the process failed and caused downtime FWIW the upgrade was from 17.3.4 to 17.12.3 which is listed in the upgrade matix as no problem.

Is there another metohod to update the supervisors and not incur any downtime? i.e. can I manually updated and reload the ICS and then the standby, then manually failover and finally update the old active, or is there a foolproof method to updating and not incurring downtime.

The more I read up on ISSU the more flakey and unreliable it seems to be,. I've been advised multiple times to ignore ISSU and just do a standard install / take the 10 minute outage, this really defeats the point of quad supervisors in a dual chassis in a virtual stack to me.

Leo Laohoo · ‎06-24-2024

I never use ISSU nor FSU/eFSU/xFSU. And I do not use DNAC or PI.

This is the solution I've developed:

https://community.cisco.com/t5/switching/ios-xe-upgrade-discussion-quot-install-quot-mode-vs-quot-request/m-p/4789971/highlight/true#M540019

richyvrlimited · ‎06-24-2024

Hi Leo,

I came across that process before I posted this discussion, however unless I'm misunderstanding you'll still experience a period of downtime, all you're doing is manually postponing the automatic reboot of an install mode process.

So once the reboot is initiated I'll still have a 10-15 minute period of downtime whilst the chassis reboot and reconverge into VSS.

Leo Laohoo · ‎06-24-2024

@richyvrlimited wrote:
however unless I'm misunderstanding you'll still experience a period of downtime, all you're doing is manually postponing the automatic reboot of an install mode process.

Actually, the one-hit-wonder method achieves two things:

1. The operator can guarantee the both chassis will boot the correct firmware. That's because the process calls for the "inspection" of the "packages.conf" file.

2. Delayed reboot at a specific time and date of my choosing.

Look at it this way: If ISSU goes awry, the outage time will exceed 15 minutes.

What kind of network is this? Is this a medical facility or a 24-hour manufacturing plant?

richyvrlimited · ‎06-24-2024

@Leo Laohoo wrote:
@richyvrlimited wrote:
however unless I'm misunderstanding you'll still experience a period of downtime, all you're doing is manually postponing the automatic reboot of an install mode process.
Actually, the one-hit-wonder method achieves two things:
1. The operator can guarantee the both chassis will boot the correct firmware. That's because the process calls for the "inspection" of the "packages.conf" file.
2. Delayed reboot at a specific time and date of my choosing.
Look at it this way:  If ISSU goes awry, the outage time will exceed 15 minutes.
What kind of network is this? Is this a medical facility or a 24-hour manufacturing plant?

Medical

shambhu.kumar · ‎06-24-2024

When you enter the “install add file file-name activate commit” command, the new software version is activated onto all the supervisors at the same time and it requires reload.

Another option is in-service software upgrade(ISSU) or non-disruptive upgrade.
For ISSU to work our current release and target release must be compatible and it will perform sanity check after running the issu commit command.

I have tried with ISSU but faced multiple issues then later I tried manual method and it works

Switch 1 is active and switch 2 is standby with ICS in slot 3.

Switch Number 1

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port 10GE / 25GE C9600-LC-48YL FDO244301DV

2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO244312ZJ

3 0 Supervisor 1 Module C9600-SUP-1 FDO244218U0

4 0 Supervisor 1 Module C9600-SUP-1 CAT2311L4E0

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 0476.B0E8.B780 to 0476.B0E8.B7FF 1.1 17.8.1r[FC1] 17.06.05 ok

2 40F0.7803.6900 to 40F0.7803.697F 1.1 17.8.1r[FC1] 17.06.05 ok

3 689E.0B0D.D680 to 689E.0B0D.D6FF 1.2 17.8.1r[FC1] 17.06.05 ok

4 -- -- N/A -- Provisioned

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Active sso sso

4 InChassis-Standby rpr rpr

Switch Number 2

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port 10GE / 25GE C9600-LC-48YL FDO24390PM7

2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO244312ZU

3 0 Supervisor 1 Module C9600-SUP-1 CAT2328L2G5

4 0 Supervisor 1 Module C9600-SUP-1 CAT2330L4JW

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 689E.0B0D.5D00 to 689E.0B0D.5D7F 1.1 17.8.1r[FC1] 17.06.05 ok

2 40F0.7803.7980 to 40F0.7803.79FF 1.1 17.8.1r[FC1] 17.06.05 ok

3 -- -- N/A -- Provisioned

4 084F.A9A2.3180 to 084F.A9A2.31FF 1.0 17.8.1r[FC1] 17.06.05 ok

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 InChassis-Standby rpr rpr

4 Standby sso sso

As switches were running on 17.6.5, decided to move to 17.3.4.

This time upgradation took longer than expected, it took around 30-40 minutes.

When switch came up on 17.3.4, what I observed that, Switchover happened and now switch 2 is active peer and ICS in switch 2 was in slot 3, now slot 4 SUP is ICS, before moving to 17.3.4 switch 1 was active now switch 2 is active. Please refer below output:

CAT9600_ACTIVE#show module

Chassis Type: C9606R

Switch Number 1

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port 10GE / 25GE C9600-LC-48YL FDO244301DV

2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO244312ZJ

3 0 Supervisor 1 Module C9600-SUP-1 FDO244218U0

4 0 Supervisor 1 Module C9600-SUP-1 CAT2311L4E0

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 0476.B0E8.B780 to 0476.B0E8.B7FF 1.1 17.8.1r[FC1] 17.03.04 ok

2 40F0.7803.6900 to 40F0.7803.697F 1.1 17.8.1r[FC1] 17.03.04 ok

3 689E.0B0D.D680 to 689E.0B0D.D6FF 1.2 17.8.1r[FC1] 17.03.04 ok

4 -- -- -- -- Provisioned

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Standby sso sso

4 InChassis-Standby rpr rpr

Switch 2---Active now after moving from 17.6.5 to 17.3.4

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port 10GE / 25GE C9600-LC-48YL FDO24390PM7

2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO244312ZU

3 0 Supervisor 1 Module C9600-SUP-1 CAT2328L2G5---before upgrade it was ICS, now Active

4 0 Supervisor 1 Module C9600-SUP-1 CAT2330L4JW

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 689E.0B0D.5D00 to 689E.0B0D.5D7F 1.1 17.8.1r[FC1] 17.03.04 ok

2 40F0.7803.7980 to 40F0.7803.79FF 1.1 17.8.1r[FC1] 17.03.04 ok

3 084F.A9A1.8580 to 084F.A9A1.85FF 1.0 17.8.1r[FC1] 17.03.04 ok

4 -- -- -- -- Provisioned

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Active sso sso

4 InChassis-Standby rpr rpr

So, after upgrade, supervisor roles got changed and in both the switches slot 4 becomes ICS.

This is obviously not the expected behavior that after upgrade the peers got switched over.

To reverify, I decided to upgrade the switch with same setup as switch 2 as active with slot 4 SUP with ICS role and switch 1 as standby and slot 4 SUP in switch 1 as ICS.

But this time, upgrade finished quite quickly considering last one and this time slot placement and SUP roles remains intact, no peers switchover neither any abrupt delays:

Please refer below output after moving from 17.3.4 to 17.6.5 when switch 2 was active and on both the switches slot 4 was Inchassis-standby(ICS).

CAT9600_ACTIVE#show module

Chassis Type: C9606R

Switch Number 1

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port 10GE / 25GE C9600-LC-48YL FDO244301DV

2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO244312ZJ

3 0 Supervisor 1 Module C9600-SUP-1 FDO244218U0

4 0 Supervisor 1 Module C9600-SUP-1 CAT2311L4E0

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 0476.B0E8.B780 to 0476.B0E8.B7FF 1.1 17.8.1r[FC1] 17.06.05 ok

2 40F0.7803.6900 to 40F0.7803.697F 1.1 17.8.1r[FC1] 17.06.05 ok

3 689E.0B0D.D680 to 689E.0B0D.D6FF 1.2 17.8.1r[FC1] 17.06.05 ok

4 -- -- N/A -- Provisioned

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Standby sso sso

4 InChassis-Standby rpr rpr

Switch Number 2----Active as earlier before upgrade. No switchover of active peers.

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port 10GE / 25GE C9600-LC-48YL FDO24390PM7

2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO244312ZU

3 0 Supervisor 1 Module C9600-SUP-1 CAT2328L2G5

4 0 Supervisor 1 Module C9600-SUP-1 CAT2330L4JW

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 689E.0B0D.5D00 to 689E.0B0D.5D7F 1.1 17.8.1r[FC1] 17.06.05 ok

2 40F0.7803.7980 to 40F0.7803.79FF 1.1 17.8.1r[FC1] 17.06.05 ok

3 084F.A9A1.8580 to 084F.A9A1.85FF 1.0 17.8.1r[FC1] 17.06.05 ok

4 -- -- N/A -- Provisioned

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Active sso sso

4 InChassis-Standby rpr rpr

From this result so far, I would say when active, standby and ICS placement is in sync, we don’t see any discrepancies like switchover or unexpected delays during upgrade. But upgrade failure I didn’t observed so far with either of the slot placement.

So before further upgrade attempts, requesting you to delete “packages.conf” pointing to old and other release and clean the unused files.

richyvrlimited · ‎06-24-2024

This is what I expected to happen, however what actually occured was that both ISC upgraded and rebooted, then the standby sup upgraded and rebooted. Then when the Primary attempted to upgrade and reboot it crashed and the entire VSS rebooted.

shambhu.kumar · ‎06-24-2024

When I run "install add file flash:cat9k_iosxe.17.06.05.SPA.bin activate commit, it appeared message

FAILED: install_add_activate_commit /bootflash/cat9k-wlc.17.06.05.SPA.pkg /bootflash/cat9k-webui.17.06.05.SPA.pkg /bootflash/cat9k-srdriver.17.06.05.SPA.pkg /bootflash/cat9k-sipspa.17.06.05.SPA.pkg /bootflash/cat9k-sipbase.17.06.05.SPA.pkg /bootflash/cat9k-rpboot.17.06.05.SPA.pkg /bootflash/cat9k-rpbase.17.06.05.SPA.pkg /bootflash/cat9k-lni.17.06.05.SPA.pkg /bootflash/cat9k-guestshell.17.06.05.SPA.pkg /bootflash/cat9k-espbase.17.06.05.SPA.pkg /bootflash/cat9k-cc_srdriver.17.06.05.SPA.pkg exit(1)

Then I execute CLI "clear install state" It started upgrading and upgradation was successful it took 19 minutes, Attaching logs,

C9600#install add file flash:cat9k_iosxe.17.06.05.SPA.bin activate commit
install_add_activate_commit: START
install_add_activate_commit: Adding PACKAGE
install_add_activate_commit: Checking whether new add is allowed ....

--- Starting initial file syncing ---
[1]: Copying bootflash:cat9k_iosxe.17.06.05.SPA.bin from chassis 1/R0 to chassis 1/R1 2/R0 2/R1
!![1 2 2]: Finished copying to chassis 1/R1 chassis 2/R2 chassis 2/R2
Info: Finished copying bootflash:cat9k_iosxe.17.06.05.SPA.bin to the selected chassis
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
[1] Add package(s) on chassis 1/R0
[1] Finished Add on chassis 1/R0
[1] Add package(s) on chassis 1/R1
[1] Finished Add on chassis 1/R1
[2] Add package(s) on chassis 2/R0
[2] Finished Add on chassis 2/R0
[2] Add package(s) on chassis 2/R1
[2] Finished Add on chassis 2/R1
Checking status of Add on [1/R0 1/R1 2/R0 2/R1]
Add: Passed on [1/R0 1/R1 2/R0 2/R1]
Finished Add

Image added. Version: 17.06.05.0.5797
install_add_activate_commit: Activating PACKAGE
Following packages shall be activated:
/bootflash/cat9k-wlc.17.06.05.SPA.pkg
/bootflash/cat9k-webui.17.06.05.SPA.pkg
/bootflash/cat9k-srdriver.17.06.05.SPA.pkg
/bootflash/cat9k-sipspa.17.06.05.SPA.pkg
/bootflash/cat9k-sipbase.17.06.05.SPA.pkg
/bootflash/cat9k-rpboot.17.06.05.SPA.pkg
/bootflash/cat9k-rpbase.17.06.05.SPA.pkg
/bootflash/cat9k-lni.17.06.05.SPA.pkg
/bootflash/cat9k-guestshell.17.06.05.SPA.pkg
/bootflash/cat9k-espbase.17.06.05.SPA.pkg
/bootflash/cat9k-cc_srdriver.17.06.05.SPA.pkg

This operation may require a reload of the system. Do you want to proceed? [y/n]y
--- Starting Activate ---
Performing Activate on all members
[1] Activate package(s) on chassis 1/R0
--- Starting list of software package changes ---
Old files list:
Removed cat9k-cc_srdriver.17.03.04.SPA.pkg
Removed cat9k-espbase.17.03.04.SPA.pkg
Removed cat9k-guestshell.17.03.04.SPA.pkg
Removed cat9k-lni.17.03.04.SPA.pkg
Removed cat9k-rpbase.17.03.04.SPA.pkg
Removed cat9k-rpboot.17.03.04.SPA.pkg
Removed cat9k-sipbase.17.03.04.SPA.pkg
Removed cat9k-sipspa.17.03.04.SPA.pkg
Removed cat9k-srdriver.17.03.04.SPA.pkg
Removed cat9k-webui.17.03.04.SPA.pkg
Removed cat9k-wlc.17.03.04.SPA.pkg
New files list:
Added cat9k-cc_srdriver.17.06.05.SPA.pkg
Added cat9k-espbase.17.06.05.SPA.pkg
Added cat9k-guestshell.17.06.05.SPA.pkg
Added cat9k-lni.17.06.05.SPA.pkg
Added cat9k-rpbase.17.06.05.SPA.pkg
Added cat9k-rpboot.17.06.05.SPA.pkg
Added cat9k-sipbase.17.06.05.SPA.pkg
Added cat9k-sipspa.17.06.05.SPA.pkg
Added cat9k-srdriver.17.06.05.SPA.pkg
Added cat9k-webui.17.06.05.SPA.pkg
Added cat9k-wlc.17.06.05.SPA.pkg
Finished list of software package changes
[1] Finished Activate on chassis 1/R0
[1] Activate package(s) on chassis 1/R1
FAILED: install_activate : Lock held for excessive time for install_add_activate_commit.
Please run 'install commit' or 'install abort' to complete current operation..
[1] Finished Activate on chassis 1/R1
[2] Activate package(s) on chassis 2/R0
FAILED: install_activate : Lock held for excessive time for install_add_activate_commit.
Please run 'install commit' or 'install abort' to complete current operation..
[2] Finished Activate on chassis 2/R0
[2] Activate package(s) on chassis 2/R1
--- Starting list of software package changes ---
Old files list:
Removed cat9k-cc_srdriver.17.03.04.SPA.pkg
Removed cat9k-espbase.17.03.04.SPA.pkg
Removed cat9k-guestshell.17.03.04.SPA.pkg
Removed cat9k-lni.17.03.04.SPA.pkg
Removed cat9k-rpbase.17.03.04.SPA.pkg
Removed cat9k-rpboot.17.03.04.SPA.pkg
Removed cat9k-sipbase.17.03.04.SPA.pkg
Removed cat9k-sipspa.17.03.04.SPA.pkg
Removed cat9k-srdriver.17.03.04.SPA.pkg
Removed cat9k-webui.17.03.04.SPA.pkg
Removed cat9k-wlc.17.03.04.SPA.pkg
New files list:
Added cat9k-cc_srdriver.17.06.05.SPA.pkg
Added cat9k-espbase.17.06.05.SPA.pkg
Added cat9k-guestshell.17.06.05.SPA.pkg
Added cat9k-lni.17.06.05.SPA.pkg
Added cat9k-rpbase.17.06.05.SPA.pkg
Added cat9k-rpboot.17.06.05.SPA.pkg
Added cat9k-sipbase.17.06.05.SPA.pkg
Added cat9k-sipspa.17.06.05.SPA.pkg
Added cat9k-srdriver.17.06.05.SPA.pkg
Added cat9k-webui.17.06.05.SPA.pkg
Added cat9k-wlc.17.06.05.SPA.pkg
Finished list of software package changes
[2] Finished Activate on chassis 2/R1
Checking status of Activate on [1/R0 1/R1 2/R0 2/R1]
Activate: Passed on [1/R0 2/R1]. Failed on [1/R1 2/R0]
Finished Activate

FAILED: install_add_activate_commit /bootflash/cat9k-wlc.17.06.05.SPA.pkg /bootflash/cat9k-webui.17.06.05.SPA.pkg /bootflash/cat9k-srdriver.17.06.05.SPA.pkg /bootflash/cat9k-sipspa.17.06.05.SPA.pkg /bootflash/cat9k-sipbase.17.06.05.SPA.pkg /bootflash/cat9k-rpboot.17.06.05.SPA.pkg /bootflash/cat9k-rpbase.17.06.05.SPA.pkg /bootflash/cat9k-lni.17.06.05.SPA.pkg /bootflash/cat9k-guestshell.17.06.05.SPA.pkg /bootflash/cat9k-espbase.17.06.05.SPA.pkg /bootflash/cat9k-cc_srdriver.17.06.05.SPA.pkg exit(1)

C9600#clear install state
clear_install_state: START

This command will remove all the provisioned SMUs, and rollback points. Use this command with caution.
A reload is required for this process. Press y to continue [y/n]y
--- Starting clear_install_state ---
Performing clear_install_state on all members
[1] clear_install_state package(s) on chassis 1/R0
[1] Finished clear_install_state on chassis 1/R0
[1] clear_install_state package(s) on chassis 1/R1
[1] Finished clear_install_state on chassis 1/R1
[2] clear_install_state package(s) on chassis 2/R0
[2] Finished clear_install_state on chassis 2/R0
[2] clear_install_state package(s) on chassis 2/R1
[2] Finished clear_install_state on chassis 2/R1
Checking status of clear_install_state on [1/R0 1/R1 2/R0 2/R1]
clear_install_state: Passed on [1/R0 1/R1 2/R0 2/R1]
Finished clear_install_state

Send model notification for before reload
Install will reload the system now!
Requesting RP pvp reload

C9600#sh module
Chassis Type: C9606R

Switch Number 1

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO2428194Y
2 48 48-Port 10GE / 25GE C9600-LC-48YL FDO242709MU
3 0 Supervisor 1 Module C9600-SUP-1 FDO243008PW
4 0 Supervisor 1 Module C9600-SUP-1 FDO243008S0
5 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO24281956

Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 3C13.CC49.6B80 to 3C13.CC49.6BFF 1.1 17.6.1r 17.06.05 ok
2 9077.EE4D.B880 to 9077.EE4D.B8FF 1.1 17.6.1r 17.06.05 ok
3 643A.EA2E.4700 to 643A.EA2E.477F 1.1 17.6.1r 17.06.05 ok
4 -- -- N/A -- Provisioned
5 3C13.CC49.6C00 to 3C13.CC49.6C7F 1.1 17.6.1r 17.06.05 ok

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Active sso sso
4 InChassis-Standby rpr rpr

Switch Number 2

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO24281954
2 48 48-Port 10GE / 25GE C9600-LC-48YL FDO242709PQ
3 0 Supervisor 1 Module C9600-SUP-1 FDO243008QV
4 0 Supervisor 1 Module C9600-SUP-1 FDO243008Q8
5 24 24-Port 40GE/12-Port 100GE C9600-LC-24C FDO24281959

Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 3C13.CC49.7100 to 3C13.CC49.717F 1.1 17.6.1r 17.06.05 ok
2 9077.EE4D.B900 to 9077.EE4D.B97F 1.1 17.6.1r 17.06.05 ok
3 643A.EA50.DE00 to 643A.EA50.DE7F 1.1 17.6.1r 17.06.05 ok
4 -- -- N/A -- Provisioned
5 3C13.CC49.7080 to 3C13.CC49.70FF 1.1 17.6.1r 17.06.05 ok

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Standby sso sso
4 InChassis-Standby rpr rpr

Chassis 1 MAC address range: 64 addresses from 3c57.31bb.e780 to 3c57.31bb.e7bf

Chassis 2 MAC address range: 64 addresses from 3c57.31bb.e7c0 to 3c57.31bb.e7ff

C9600# sh version
Cisco IOS XE Software, Version 17.06.05