C9500 SVL - ISSU via DNAC Failure

[Marco111]

Hi. I recently attempted to update 2x C9500 in SVL via DNAC from 17.6.4 to 17.9.4. It initally looked to be progressing all ok with being distributed then activated on each unit. DNAC then stated process had failed. I manually checked switches via CLI and the version was reporting all ok at 17.9.4 but was on a countdown to rollback (Operating mode: sso, terminal state reached). As it looked all ok i stopped the ISSU process via 'clear issu state' as i did not want it to rollback. I've just realised however that the new software is activated but uncommitted, can i run command 'install commit' to resolve this? I believe DNAC should have done this after the activation on each unit but failed to do so, as i cleared the ISSU state will the manual 'install commit' still work as intended?

Sorry if this is a basic question, pretty new to DNA and managing Cisco kit. May stay clear of ISSU via DNA Centre in future ha.. many thanks.

[balaji.bandi]

as per the matrix its supported :

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst_standalones/b-in-service-software-upgrade-issu.html

what DNAC Version ?

can i run command 'install commit' to resolve this? I

if you like to do traditional way you can follow below guide :

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-9/release_notes/ol-17-9-9500/upgrading_the_switch_software.html

[Marco111]

Thanks @balaji.bandi , DNAC version is 2.3.3.5.. yes think in future I will use standard update procedure when able (less things to go wrong :)) , currently the version i want '17.9.4' is on and activated, am i safe to now commit this manually, as mentioned its currently showing as uncommitted? Thanks.

[balaji.bandi]

can post

show boot

dir flash:

show install package
show install summary
show install active
show install inactive
show install committed
show install uncommitted
show version

[Marco111]

Hi @balaji.bandi  sure outputs as attached txt. Thanks.

[Marco111]

Hi again @balaji.bandi ... had a change request to try and resolve this tonight, first i reran 'show install summary' oddly it now shows Chassis 2 as Committed and Chassis 1 as Uncommitted

show install summary
[ Chassis 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 17.09.04.0.5180

[ Chassis 1/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG U 17.09.04.0.5180

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

I then attempted to manually commit which looked to complete all ok but logs shown below indicated remote (Chassis 1) had nothing to commit

[remote|install_commit]: START Tue May 14 22:10:52 UTC 2024
[remote|install_commit(CONSOLE, )]: Nothing to commit; please use 'install activate' before 'install commit'
[remote|install_commit]: END SUCCESS Tue May 14 22:10:52 UTC 2024

I then attempted to reinstall hoping that would resolve which completed ok indicating both Chassis are running the correct version.

install add file flash:cat9k_iosxe.17.09.04.SPA.bin activate issu commit
install_add_activate_commit: START Tue May 14 22:51:38 UTC 2024
install_add_activate_commit: Adding ISSU
install_add_activate_commit: Checking whether new add is allowed ....

--- Starting initial file syncing ---
[2]: Copying bootflash:cat9k_iosxe.17.09.04.SPA.bin from chassis 2/R0 to chassis 1/R0
[1]: Finished copying to chassis 1/R0
Info: Finished copying bootflash:cat9k_iosxe.17.09.04.SPA.bin to the selected chassis
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
[1] Add package(s) on chassis 1/R0
[1] Add succeed on chassis 1/R0 with reason:"Same Image File-No Change"
[1] Finished Add on chassis 1/R0
[2] Add package(s) on chassis 2/R0
[2] Add succeed on chassis 2/R0 with reason:"Same Image File-No Change"
[2] Finished Add on chassis 2/R0
Checking status of Add on [1/R0 2/R0]
Add: Passed on [1/R0 2/R0]
Finished Add

SUCCESS: install_add_activate_commit Tue May 14 22:53:53 UTC 2024

Afterwards rechecked and still only committed on Chassis 2. Unless as the package was the same it simply aborted the process etc. Think i may raise a TAC case on this.

[Leo Laohoo]

Please provide the complete output to the following commands: 

more bootflash:packages.conf | begin rp_boot
more bootflash-2:packages.conf | begin rp_boot
sh boot

[Marco111]

Sure @Leo Laohoo  as attached. thanks

[Leo Laohoo · Accepted solution]

Flash of both switches are rigged to boot 17.9.4.  

All I need to see now is the output to the command "sh boot" to confirm if the boot variable is pointed to the "packages.conf" file and nothing else. 

If it is, then reboot the entire VSS pair.  There is no way to do this using ISSU because the ISSU process has failed.  Any time wasted in trying to get ISSU back on it's feet is wasted time.  

[Marco111]

@Leo Laohoo yeah that's looking all ok, ok I'll arrange a full reboot hopefully tomorrow, tempted to do it now but this 9500 is acting as a fusion router so quite a few 24/7 offices connected so will need to arrange. Thanks for the advice. Will let you know how it goes.

show boot
BOOT variable = bootflash:packages.conf;
MANUAL_BOOT variable = no
BAUD variable = 9600
ENABLE_BREAK variable does not exist
BOOTMODE variable does not exist
IPXE_TIMEOUT variable does not exist
CONFIG_FILE variable =

Standby BOOT variable = bootflash:packages.conf;
Standby MANUAL_BOOT variable = no
Standby BAUD variable = 9600
Standby ENABLE_BREAK variable does not exist
Standby BOOTMODE variable does not exist
Standby IPXE_TIMEOUT variable does not exist
Standby CONFIG_FILE variable =

[Leo Laohoo]

Also, be aware that the choice of 17.9.4 is questionable especially when there is a known security vulnerability and is actively being exploited in the wild:  Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature

Cisco IOS XE Software Release Train	First Fixed Release	Available
17.9	17.9.4a	Yes

[Marco111]

Thanks yeah I'm aware, the SMU will be installed once main package is showing all ok. Think we had just started rolling out 17.9.4+SMU before 17.9.4a was released. We'll probably move to the next recommended release in near future. Cheers

[Leo Laohoo]

Hey @Marco111, 

Have a look at the attached document.  And, most importantly, never perform an ISSU without TAC Proactive Case.   If-and-when someone feels the need to do another ISSU, raise a Proactive TAC Case and get the TAC Agent on a WebEx call to "witness" the entire ISSU process.  One of the main task of the TAC Agent is to intervene in the likelihood event when ISSU fails and the VSS goes into a "ping pong" state and the network becomes unstable and unmanageable.

[Leo Laohoo]

CSCwj91408 if DHCP Snooping is enabled.