Re: C9300 issu upgrade failure

schulcz · ‎01-14-2021

Hi Guys,

I have some 9300 stack (with 2 switches), and I tested some features, for example in-service-software-upgrade.

I am particularly interested in the proper operation of ISSU because I have sucked it twice in recent years with 4500X VSS, even though I followed the Cisco guide, the issu update failed in either case.

I did everything according to the Cisco guide letter to letter, I received the correct output during the preparation of the devices, however I got error after image is copied to the second device, and I don't understand why.

Switch#
Switch#show version | in IOS XE 
Cisco IOS XE Software, Version 17.03.01
Switch#show ver | in INSTALL
*    1 41    C9300-24S          17.03.01          CAT9K_IOSXE           INSTALL
     2 41    C9300-24S          17.03.01          CAT9K_IOSXE           INSTALL
Switch#show redundancy 
Redundant System Information :
------------------------------
       Available system uptime = 2 days, 1 hour, 48 minutes
Switchovers system experienced = 4
              Standby failures = 0
        Last switchover reason = active unit removed

                 Hardware Mode = Duplex
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
-------------------------------
               Active Location = slot 1
        Current Software state = ACTIVE
       Uptime in current state = 2 hours, 44 minutes
                 Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Fri 07-Aug-20 21:32 by mcpre
                          BOOT = flash:packages.conf
                   CONFIG_FILE = 
        Configuration register = 0x102

Peer Processor Information :
----------------------------
              Standby Location = slot 2
        Current Software state = STANDBY HOT 
       Uptime in current state = 2 hours, 39 minutes
                 Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Fri 07-Aug-20 21:32 by mcpre
                          BOOT = flash:packages.conf
                   CONFIG_FILE = 
        Configuration register = 0x102


Switch#show boot system 
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
Switch#show issu state detail
Current ISSU Status: Disabled       
Previous ISSU Operation: N/A        
=======================================================
System Check                        Status
-------------------------------------------------------
Platform ISSU Support               Yes
Standby Online                      No
Autoboot Enabled                    Yes
SSO Mode                            Yes
Install Boot                        Yes
Valid Boot Media                    Yes
=======================================================
No ISSU operation is in progress

Switch#show install summary
[ Switch 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version
--------------------------------------------------------------------------------
IMG   C    17.03.01.0.351

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------


Switch#copy tftp: bootflash: vrf Mgmt-vrf
Address or name of remote host [10.0.80.90]? 
Source filename [cat9k_iosxe.17.03.02a.SPA.bin]? 
Destination filename [cat9k_iosxe.17.03.02a.SPA.bin]? 
Accessing tftp://10.0.80.90/cat9k_iosxe.17.03.02a.SPA.bin...
Loading cat9k_iosxe.17.03.02a.SPA.bin from 10.0.80.90 (via GigabitEthernet0/0): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 898018097 bytes]

898018097 bytes copied in 87.314 secs (10284927 bytes/sec)
Switch#$ file flash:cat9k_iosxe.17.03.02a.SPA.bin activate issu commit       
install_add_activate_commit: START Thu Jan 14 17:44:42 UTC 2021
install_add_activate_commit: Adding ISSU
install_add_activate_commit: Checking whether new add is allowed ....

--- Starting initial file syncing ---
[1]: Copying flash:cat9k_iosxe.17.03.02a.SPA.bin from switch 1 to switch 2
[2]: Finished copying to switch 2  
Info: Finished copying flash:cat9k_iosxe.17.03.02a.SPA.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

install_add_activate_commit: Activating ISSU

NOTE: Going to start Oneshot ISSU install process

STAGE 0: System Level Sanity Check
===================================================
Standby is not online. User can manually abort the ISSU
operation once Standby is online
ERROR: install_add_activate_commit  exit(2 ) Thu Jan 14 17:48:08 UTC 2021
Switch#

I tried several times from the beginning after I made "install remove inactive". I also tried it on other stack, I have no luck, got the same error. (And I tried to did it by the 3-Step Work Flow also, I got the same error during STAGE 0: System Level Sanity Check)

If I tried a normal (not issu) update, there is no problem, everything is works correctly.

Note: the "show issu state detail" command shows that standby is not online, maybe that could be the problem?

Does anybody have an idea what is the problem, or what can I do to make it work?

Leo Laohoo · ‎01-14-2021

Never, ever, believe in the hype about ISSU, FSU/eFSU.

ISSU, FSU/eFSU do not (always) work in production -- It will work in a "lab" environment but, in the wild, it normally fails most of the time.

Try the following command:

req plat soft pack inst switch all file flash:cat9k_iosxe.17.03.02a.SPA.bin on-reboot new auto-copy verb

NOTE:

The option "on-reboot" means "do not reboot the switch/stack after the package files have been unpacked".
The option "on-reboot" is a bug -- It is meant to be "no-reboot" but the BU is reluctant to fix it (CSCve94966).

WARNING:

Did someone thoroughly read the Release Notes? Because if someone did, they will come down to the ROMMON Upgrade section and realize there is, yet, another step to the upgrade process.

schulcz · ‎01-18-2021

So sorry for triplicated reply, maybe community site had some issues yesterday.

Below is my answer for your advice.

schulcz · ‎01-18-2021

So sorry for triplicated reply, maybe community site had some issues yesterday.

Below is my answer for your advice.

schulcz · ‎01-18-2021

I would like to try and see if it is works, because these switches will be installed in a production network as distribution stacks (per building central devices) in a couple of weeks. Now I can upgrade them by any method, but after they will be installed, an update can be critical, because it is a factory with 24/7/365 in production, so network outage due to upgrade not accepted.

The command you wrote made a normal or ISSU upgrade?

I issued the command without running an error, my interpretation is that the devices should be restarted now. Should I restart both at the same time or one at a time? In the latter case, should a service outage be expected?

Switch#$sh:cat9k_iosxe.17.03.02a.SPA.bin on-reboot new auto-copy verbose 
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1


*Jan 18 12:58:32.645: %INSTALL-5-OPERATION_START_INFO: Switch 1 R0/0: provision: Started install package flash:cat9k_iosxe.17.03.02a.SPA.binExpanding image file: flash:cat9k_iosxe.17.03.02a.SPA.bin

*Jan 18 12:58:53.045: %INSTALL-5-OPERATION_START_INFO: Switch 1 R0/0: packtool: Started expand package flash:cat9k_iosxe.17.03.02a.SPA.bin[1]: Copying flash:cat9k_iosxe.17.03.02a.SPA.bin from switch 1 to switch 2
[2]: Finished copying to switch 2  
[1 2]: Expanding file
[1]: Verifying parameters
[1]: Expanding superpackage flash:cat9k_iosxe.17.03.02a.SPA.bin
[1]: Validating package type
[1]: Copying package files
[1]: WARNING: cat9k-cc_srdriver.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-espbase.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-guestshell.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-lni.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-rpbase.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-rpboot.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-sipbase.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-sipspa.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-srdriver.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-webui.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: WARNING: cat9k-wlc.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[1]: SUCCESS: Finished expanding all-in-one software package.
[2]: Verifying parameters
[2]: Expanding superpackage flash:cat9k_iosxe.17.03.02a.SPA.bin
[2]: Validating package type
[2]: Copying package files
[2]: WARNING: cat9k-cc_srdriver.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-espbase.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-guestshell.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-lni.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-rpbase.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-rpboot.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-sipbase.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-sipspa.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-srdriver.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-webui.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: WARNING: cat9k-wlc.17.03.02a.SPA.pkg will replace the identical file that already exists in flash:
[2]: SUCCESS: Finished expanding all-in-one software package.
[1 2]: Finished expanding all-in-one software package in switch 1 2
SUCCESS: Finished expanding all-in-one software package.
... expand successful
[1 2]: Performing install

*Jan 18 13:00:53.303: %INSTALL-5-OPERATION_COMPLETED_INFO: Switch 1 R0/0: packtool: Completed expand package flash:cat9k_iosxe.17.03.02a.SPA.bin  SUCCESS: install finished
... Operation install completed. Checking for status
[1]: install package(s) on switch 1
--- Starting list of software package changes ---
Old files list:
  Removed cat9k-cc_srdriver.17.03.01.SPA.pkg
  Removed cat9k-espbase.17.03.01.SPA.pkg
  Removed cat9k-guestshell.17.03.01.SPA.pkg
  Removed cat9k-lni.17.03.01.SPA.pkg
  Removed cat9k-rpbase.17.03.01.SPA.pkg
  Removed cat9k-rpboot.17.03.01.SPA.pkg
  Removed cat9k-sipbase.17.03.01.SPA.pkg
  Removed cat9k-sipspa.17.03.01.SPA.pkg
  Removed cat9k-srdriver.17.03.01.SPA.pkg
  Removed cat9k-webui.17.03.01.SPA.pkg
  Removed cat9k-wlc.17.03.01.SPA.pkg
New files list:
  Added cat9k-cc_srdriver.17.03.02a.SPA.pkg
  Added cat9k-espbase.17.03.02a.SPA.pkg
  Added cat9k-guestshell.17.03.02a.SPA.pkg
  Added cat9k-lni.17.03.02a.SPA.pkg
  Added cat9k-rpbase.17.03.02a.SPA.pkg
  Added cat9k-rpboot.17.03.02a.SPA.pkg
  Added cat9k-sipbase.17.03.02a.SPA.pkg
  Added cat9k-sipspa.17.03.02a.SPA.pkg
  Added cat9k-srdriver.17.03.02a.SPA.pkg
  Added cat9k-webui.17.03.02a.SPA.pkg
  Added cat9k-wlc.17.03.02a.SPA.pkg
Finished list of software package changes
SUCCESS: Software provisioned.  New software will load on reboot.
[1]: Finished install successful on switch 1
[2]: install package(s) on switch 2
--- Starting list of software package changes ---
Old files list:
  Removed cat9k-cc_srdriver.17.03.01.SPA.pkg
  Removed cat9k-espbase.17.03.01.SPA.pkg
  Removed cat9k-guestshell.17.03.01.SPA.pkg
  Removed cat9k-lni.17.03.01.SPA.pkg
  Removed cat9k-rpbase.17.03.01.SPA.pkg
  Removed cat9k-rpboot.17.03.01.SPA.pkg
  Removed cat9k-sipbase.17.03.01.SPA.pkg
  Removed cat9k-sipspa.17.03.01.SPA.pkg
  Removed cat9k-srdriver.17.03.01.SPA.pkg
  Removed cat9k-webui.17.03.01.SPA.pkg
  Removed cat9k-wlc.17.03.01.SPA.pkg
New files list:
  Added cat9k-cc_srdriver.17.03.02a.SPA.pkg
  Added cat9k-espbase.17.03.02a.SPA.pkg
  Added cat9k-guestshell.17.03.02a.SPA.pkg
  Added cat9k-lni.17.03.02a.SPA.pkg
  Added cat9k-rpbase.17.03.02a.SPA.pkg
  Added cat9k-rpboot.17.03.02a.SPA.pkg
  Added cat9k-sipbase.17.03.02a.SPA.pkg
  Added cat9k-sipspa.17.03.02a.SPA.pkg
  Added cat9k-srdriver.17.03.02a.SPA.pkg
  Added cat9k-webui.17.03.02a.SPA.pkg
  Added cat9k-wlc.17.03.02a.SPA.pkg
Finished list of software package changes
SUCCESS: Software provisioned.  New software will load on reboot.
[2]: Finished install successful on switch 2
Checking status of install on [1 2]
[1 2]: Finished install in switch 1 2
SUCCESS: Finished install: Success on [1 2]
Ensuring that cached content is written to media

*Jan 18 13:02:10.975: %INSTALL-5-OPERATION_COMPLETED_INFO: Switch 1 R0/0: provision: Completed install package flash:{cat9k-cc_srdriver.17.03.02a.SPA.pkg,cat9k-espbase.17.03.02a.SPA.pkg,cat9k-guestshell.17.03.02a.SPA.pkg,cat9k-lni.17.03.02a.SPA.pkg,cat9k-rpbase.17.03.02a.SPA.pkg,cat9k-sipbase.17.03.02a.SPA.pkg,cat9k-sipspa.17.03.02a.SPA.pkg,cat9k-srdriver.17.03.02a.SPA.pkg,cat9k-webui.17.03.02a.SPA.pkg,cat9k-wlc.17.03.02a.SPA.pkg}... cached content flushed to media

Switch#

About ROMMON Upgrade:
Yes, I read and tried it before I wrote my first post, I had interesting experiences about that. After a successul normal upgrade (17.03.01 -> 17.03.02) I gave the rommon upgrade command based on Cisco guide, and I got error message about 3 sec later that says upgrade failure. After that I didn't got back my console access (tried to hit any key, nothing happens). After 15 mins of nothing the switches starts to reboot and starts the update:

Switch#upgrade rom-monitor capsule golden switch active R0 
This operation will reload the switch and take a few minutes to complete. Do you want to proceed (y/n)? [confirm]


Switch#
       *Jan 13 15:56:14.715: %IOSXEBOOT-4-CAPSULE_UPGRADE: (local/local): ### Wed Jan 13 15:56:14 UTC 2021 Restarting system to start capsule upgrade ####

Chassis 1 reloading, reason - System error occurred
Jan 13 15:56:17.774: %PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting

*Jan 13 15:56:17.803: %IOSXEBOOT-4-CAPSULE_UPGRADE: (local/local): ### Wed Jan 13 15:56:14 UTC 2021 Capsule upgrade failed ###





Initializing Hardware......


          Insyde H2OFFT (Flash Firmware Tool) Version (SEG) 200.00.00.02-Beta
         Copyright (C) 2018 Insyde Software Corp. All Rights Reserved.


        Platform    Name: Grangeville
        Current  Version: GRNV.05.05.15.0026
        Update   Version: GRNV.05.05.15.0026


Warning : New region (type 2) access rights will be modified

        Updating Block at FFFFF000h  100%
Restarting switch to complete capsule upgrade

Initializing Hardware...

Initializing Hardware......

System Bootstrap, Version 17.3.2r, RELEASE SOFTWARE (P)
Compiled Tue 08/25/2020 23:46:12.85 by rel

Current ROMMON image : Primary
Last reset cause     : SoftwareReload
C9300-24S platform with 8388608 Kbytes of main memory

boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf
#
###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Waiting for 120 seconds for other switches to boot

Switch number is 1
All switches in the stack have been discovered. Accelerating discovery

Switch console is now available

Press RETURN to get started.

Switch>
*Jan 13 16:22:43.444: [console-relay] Selected VTY line number 1 for console relay.
*Jan 13 16:22:43.444: [console-relay] Checking VTY access: addr(0xC0A80105) tableid(265) liin_tableid(265)
*Jan 13 16:22:43.444: [console-relay] Selected VTY line number 1 for console relay.
*Jan 13 16:22:43.444: [console-relay] Checking VTY access: addr(0xC0A80105) tableid(265) liin_tableid(265)
*Jan 13 16:22:43.445: [console-relay] LIIN tableid check passed.
*Jan 13 16:22:43.445: [console-relay] VTY-1. Relayed console using settings: terminal length: 24, width: 80, execution timeout: 600000 milliseconds, rx/tx speed: 9600/9600. Using existing?(false). Defaults exist?(false).
*Jan 13 16:22:43.445: [console-relay] Detected console relayed from member. Adding a logger.
*Jan 13 16:22:43.445: [console-relay] LIIN tableid check passed.
*Jan 13 16:22:43.445: [console-relay] VTY-1. Relayed console using settings: terminal length: 24, width: 80, execution timeout: 600000 milliseconds, rx/tx speed: 9600/9600. Using existing?(false). Defaults exist?(false).
*Jan 13 16:22:43.582: %IOSXE_REDUNDANCY-6-PEER: Active detected switch 1 as standby.
*Jan 13 16:22:43.579: %STACKMGR-6-STANDBY_ELECTED: Switch 2 R0/0: stack_mgr: Switch 1 has been elected STANDBY.
*Jan 13 16:22:48.655: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*Jan 13 16:22:48.655: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
*Jan 13 16:23:16.448: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Jan 13 16:23:17.450: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
*Jan 13 16:23:17.453: GRACEFUL-RELOAD: Wrong state: 1 to recieve chasfs event: 

Switch>

schulcz · ‎01-26-2021

I found this statement in Upgrade Guide for Cisco Catalyst 9000 Switches:

9200 and 9300 do not support ISSU. 9300 does support reloadfast

Maybe is this the reason why issu fails?

Based on this documentation, 9300 series switches DOES NOT SUPPORT ISSU method, even though the related commands are available in the software, is it right?

(As I see reloadfast option is a feature, what is not supported in Essentials license level, it requires minimum Advantage license level.)

Switch#install add file flash-1:cat9k_iosxe.17.03.02a.SPA.bin activate reloadfast commit 
install_add_activate_commit: START Tue Jan 26 09:24:29 UTC 2021
STACK_GR: Inside fast rld check if switch stack
Fast software upgrade on a stack is supported only on an Advantage license level. Please upgrade your license to use this feature.
Checking STP eligibility: Eligible

FAILED: Fast reload requirement pre-check
FAILED: install_add_activate_commit : Abort reloadfast install since requirement does not met
Switch#

Leo Laohoo · ‎01-26-2021

@schulcz wrote:

The command you wrote made a normal or ISSU upgrade?

NOTE: Before I respond, I only saw your reply today so I was not able to make any comments to your previous response.

I will never write any commands regarding ISSU &/or FSU/eFSU.

@schulcz wrote:

and I got error message about 3 sec later that says upgrade failure. After that I didn't got back my console access (tried to hit any key, nothing happens). After 15 mins of nothing the switches starts to reboot and starts the update:

LOL

geder · ‎10-18-2022

Got the sollution without any impact from TAC:

Configure service internal, then you can clear the ISSU state and do the Upgrade:

Core-SW(config)#service internal
Core-SW(config)#exit
Core-SW#clear
Core-SW#clear ins
Core-SW#clear install state
clear_install_state: START Tue Oct 18 08:20:20 CEST 2022
--- Starting clear_install_state ---
Performing clear_install_state on all members
[1] clear_install_state package(s) on chassis 1/R0
[1] Finished clear_install_state on chassis 1/R0
[2] clear_install_state package(s) on chassis 2/R0
[2] Finished clear_install_state on chassis 2/R0
Checking status of clear_install_state on [1/R0 2/R0]
clear_install_state: Passed on [1/R0 2/R0]
Finished clear_install_state

Core-SW#show issu state detail
Current ISSU Status: Enabled
Previous ISSU Operation: Abort Failed
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
Operational Mode HA-REMOTE
=======================================================
No ISSU operation is in progress

mitard · ‎12-29-2022

Hi,

Does it mean you eventually manage to perform ISSU thanks to the "service internal" command ? What about the statement from the upgrade documentation :

9200 and 9300 do not support ISSU. 9300 does support reloadfast

Thanks for your feedback, Vincent