Cannot upgrade IOS on switch C9200L

[james!!!]

Hi Everyone,

I have been trying to upgrade the IOS for one C9200L switch from 16.9.4 to 17.3.3. The switch was on install mode and unfortunately it was failed.

From the below installation log message that the install activate was failed. 

#install add file flash:cat9k_lite_iosxe.17.03.04b.SPA.bin activate commit
install_add_activate_commit: START Sat Feb 19 16:10:01 met 2022
install_add_activate_commit: Adding PACKAGE

--- Starting initial file syncing ---
Info: Finished copying flash:cat9k_lite_iosxe.17.03.04b.SPA.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
[1] Add package(s) on switch 1
[1] Finished Add on switch 1
Checking status of Add on [1]
Add: Passed on [1]
Finished Add

install_add_activate_commit: Activating PACKAGE

gzip: initramfs.cpio.gz: decompression OK, trailing garbage ignored
Following packages shall be activated:
/flash/cat9k_lite-webui.17.03.04b.SPA.pkg
/flash/cat9k_lite-srdriver.17.03.04b.SPA.pkg
/flash/cat9k_lite-rpboot.17.03.04b.SPA.pkg
/flash/cat9k_lite-rpbase.17.03.04b.SPA.pkg

This operation requires a reload of the system. Do you want to proceed? [y/n]y
--- Starting Activate ---
Performing Activate on all members
[1] Activate package(s) on switch 1
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1
--- Starting file path checking ---
Finished file path checking
--- Starting image file verification ---
Checking image file names
Locating image files and validating name syntax
Found cat9k_lite-webui.17.03.04b.SPA.pkg
Found cat9k_lite-srdriver.17.03.04b.SPA.pkg
Found cat9k_lite-rpboot.17.03.04b.SPA.pkg
Found cat9k_lite-rpbase.17.03.04b.SPA.pkg
Verifying image file locations
Inspecting image file types
WARNING: In-service installation of RP Boot package
WARNING: requires software reboot of target RP
WARNING: Automatically setting the on-reboot flag
WARNING: In-service installation of RP Base package
WARNING: requires software reboot of target RP
WARNING: In-service installation of srdriver package
WARNING: requires software reboot of target
Processing image file constraints
Creating candidate provisioning file
Finished image file verification
--- Starting candidate package set construction ---
Verifying existing software set
Processing candidate provisioning file
Production Image
Constructing working set for candidate package set
Constructing working set for running package set
Checking command output
Constructing merge of running and candidate packages
Checking if resulting candidate package set would be complete
Finished candidate package set construction
--- Starting ISSU compatibility verification ---
WARNING: Ignoring ISSU version compatibility checks
WARNING: Setting onreboot flag
Verifying image type compatibility
WARNING: Ignoring ISSU subpackage compatibility checks
IPC Compatibility checks are not required for Install Mode
Skipping IPC compatibility with running software
Checking candidate package set infrastructure compatibility
Checking infrastructure compatibility with running software
Checking infrastructure compatibility with running software ... skipped
Checking package specific compatibility
Finished ISSU compatibility verification
--- Starting list of software package changes ---
Old files list:
Removed cat9k_lite-rpbase.16.09.04.SPA.pkg
Removed cat9k_lite-rpboot.16.09.04.SPA.pkg
Removed cat9k_lite-srdriver.16.09.04.SPA.pkg
Removed cat9k_lite-webui.16.09.04.SPA.pkg
New files list:
Added cat9k_lite-rpbase.17.03.04b.SPA.pkg
Added cat9k_lite-rpboot.17.03.04b.SPA.pkg
Added cat9k_lite-srdriver.17.03.04b.SPA.pkg
Added cat9k_lite-webui.17.03.04b.SPA.pkg
Finished list of software package changes
--- Starting commit of software changes ---
Updating provisioning rollback files
Creating pending provisioning file
Committing provisioning file
Finished commit of software changes
SUCCESS: Software provisioned. New software will load on reboot.
FAILED: install_activate exit(1) Sat Feb 19 16:17:38 met 2022
[1] Finished Activate on switch 1
Checking status of Activate on [1]
Activate: Passed on []. Failed on [1]
Finished Activate

FAILED: install_add_activate_commit /flash/cat9k_lite-webui.17.03.04b.SPA.pkg /flash/cat9k_lite-srdriver.17.03.04b.SPA.pkg /flash/cat9k_lite-rpboot.17.03.04b.SPA.pkg /flash/cat9k_lite-rpbase.17.03.04b.SPA.pkg exit(1)

Then I tried to install again and below log messages were pop-up.

#install add file flash:cat9k_lite_iosxe.17.03.04b.SPA.bin activate commit
install_add_activate_commit: START Sat Feb 19 16:20:25 met 2022
install_add_activate_commit: Adding PACKAGE
FAILED: install_add_activate_commit : Super package already added. Add operation not allowed. 'install remove inactive' can be used to discard added packages

Next I performed 'install remove inactive' command

#install remove inactive
install_remove: START Sat Feb 19 16:22:04 met 2022
FAILED: Clean inactive packages: previous provisioning action is pending reboot.
SUCCESS: install_remove Sat Feb 19 16:22:13 met 2022

Therefore, how can I deal with this case, can I reboot this switch or any other commands need to be performed?

Any help would be much appreciated. Thanks in advance!

[balaji.bandi]

Next I performed 'install remove inactive' command

This needs to be performed before any upgrade take place or after a successful upgrade. not when it failed.

check below information :

show boot

dir flash:

check the content of packages.conf file.

Note: if you reload now. you may see the device go into ROMMON, if this is a live network, suggest taking config backup out of the switch, keep ready USB with IOS image in case required or accidentally rebooting the device.

For reference of upgrade steps :

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9200/software/release/17-3/release_notes/ol-17-3-9200.html

[Leo Laohoo]

Post the complete output to the following commands: 

• sh version
• dir
• dir flash:*.conf
• sh boot

[james!!!]

#sh version
Cisco IOS XE Software, Version 16.09.04
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:30 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.1r [FC2], RELEASE SOFTWARE (P)

delinswh10401 uptime is 22 weeks, 1 day, 17 hours, 8 minutes
Uptime for this control processor is 22 weeks, 1 day, 17 hours, 10 minutes
System returned to ROM by Power Failure or Unknown
System restarted at 10:54:31 mest Sat Sep 25 2021
System image file is "flash:packages.conf"
Last reload reason: Power Failure or Unknown

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-essentials Smart License network-essentials
dna-essentials Subscription Smart License dna-essentials

Smart Licensing Status: UNREGISTERED/EVAL EXPIRED

cisco C9200L-48P-4X (ARM64) processor with 519008K/3071K bytes of memory.
Processor board ID JAE23480R7U
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
1998948K bytes of physical memory.
819200K bytes of Crash Files at crashinfo:.
1941504K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : c4:b2:39:3d:9e:80
Motherboard Assembly Number : 73-19245-06
Motherboard Serial Number : JAE23480R7U
Model Revision Number : G0
Motherboard Revision Number : A0
Model Number : C9200L-48P-4X
System Serial Number : JAE23480R7U

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 52 C9200L-48P-4X 16.9.4 CAT9K_LITE_IOSXE INSTALL

Configuration register is 0x102

#dir
Directory of flash:/

56676 -rw- 427189272 Feb 19 2022 16:11:28 +01:00 cat9k_lite-rpbase.17.03.04b.SPA.pkg
48579 drwx 4096 Feb 27 2022 04:18:53 +01:00 .installer
48581 -rw- 481 Sep 25 2021 10:54:42 +02:00 bootloader_evt_handle.log
48582 drwx 4096 Nov 28 2019 18:41:14 +01:00 core
72865 drwx 4096 Nov 28 2019 18:58:15 +01:00 .prst_sync
89057 drwx 4096 Nov 28 2019 18:54:16 +01:00 .rollback_timer
97153 drwx 4096 Nov 28 2019 18:38:46 +01:00 tech_support
105249 drwx 4096 Sep 25 2021 10:54:58 +02:00 dc_profile_dir
48580 -rw- 140 Sep 25 2021 10:54:59 +02:00 boothelper.log
48584 -rw- 123888 Sep 25 2021 10:55:17 +02:00 memleak.tcl
48586 drwx 4096 Nov 28 2019 18:41:35 +01:00 .dbpersist
48587 -rw- 0 Sep 25 2021 10:56:52 +02:00 rdope_out.txt
48588 drwx 4096 Nov 28 2019 18:40:53 +01:00 onep
48589 -rw- 89 Sep 25 2021 10:57:06 +02:00 rdope.log
48591 -rw- 2097152 Feb 27 2022 04:19:24 +01:00 nvram_config
48592 -rw- 2097152 Feb 27 2022 04:19:24 +01:00 nvram_config_bkup
48593 -rw- 35 Apr 14 2021 15:23:57 +02:00 pnp-tech-time
48594 -rw- 94314 Apr 14 2021 15:24:01 +02:00 pnp-tech-discovery-summary
48583 -rw- 140 Apr 14 2021 15:15:29 +02:00 boothelper.log.old
48622 -rw- 4930 Feb 19 2022 16:17:28 +01:00 packages.conf
48596 -rw- 976 Sep 25 2021 11:23:13 +02:00 vlan.dat
48616 -rw- 299574284 Nov 28 2019 18:51:20 +01:00 cat9k_lite-rpbase.16.09.04.SPA.pkg
48619 -rw- 35549545 Nov 28 2019 18:51:52 +01:00 cat9k_lite-rpboot.16.09.04.SPA.pkg
48617 -rw- 3101708 Nov 28 2019 18:51:20 +01:00 cat9k_lite-srdriver.16.09.04.SPA.pkg
48618 -rw- 7586824 Nov 28 2019 18:51:20 +01:00 cat9k_lite-webui.16.09.04.SPA.pkg
48600 -rw- 482856473 Jan 14 2022 04:09:10 +01:00 cat9k_lite_iosxe.17.03.04b.SPA.bin
48615 -rw- 0 Feb 19 2022 16:17:35 +01:00 .onreboot
56679 -rw- 40716729 Feb 19 2022 16:12:07 +01:00 cat9k_lite-rpboot.17.03.04b.SPA.pkg
56677 -rw- 4256792 Feb 19 2022 16:11:29 +01:00 cat9k_lite-srdriver.17.03.04b.SPA.pkg
56678 -rw- 11035672 Feb 19 2022 16:11:29 +01:00 cat9k_lite-webui.17.03.04b.SPA.pkg
56675 -rw- 4930 Feb 19 2022 16:12:07 +01:00 cat9k_lite_iosxe.17.03.04b.SPA.conf

#dir flash:*.conf
Directory of flash:/*.conf

Directory of flash:/

48622 -rw- 4930 Feb 19 2022 16:17:28 +01:00 packages.conf
56675 -rw- 4930 Feb 19 2022 16:12:07 +01:00 cat9k_lite_iosxe.17.03.04b.SPA.conf

#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

[Leo Laohoo]

---

@james!!! wrote:

48622 -rw- 4930 Feb 19 2022 16:17:28 +01:00 packages.conf

---

Beautiful.  Thanks for the outputs.  Extremely useful. 

Please post the complete output to the following command:  more flash:packages.conf | begin for CAT

[james!!!]

#more flash:packages.conf | begin CAT
# pkginfo: Platform: CAT9K_LITE
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 17.03.04b
# pkginfo: .SupportedBoards: quake
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: .UnifiedPlatformList: quake1,quake2
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /scratch/mcpre/release/BLD-V17_03_04B_FC1/binos/linkfarm/stage-quake/hard/rp_super_universalk9.arm32
# pkginfo: .Version: 17.03.04b.0.13.1634640213..Amsterdam
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

[Leo Laohoo · Accepted solution]

Looks good to me. 
The switch will boot 17.3.4b in the next reboot/reload.

[balaji.bandi]

reboot the device and test it.

Once boot ok, then clean up old packages.

[fredlnewton]

One thing of note if you find your C9200 reacts with not enough space

[1] Switch 1 FAILED: /mnt/sd3/user requires 459651 KB of free space, but only 334684 KB is available
FAILED: add_activate_commit /mnt/sd3/user/cat9k_lite_iosxe.17.09.04a.SPA.bin Thu Jan 18 10:12:18 est 2024

Follow the Cisco upgrade procedure to the point of 'install remove inactive'

Then

copy usbflash0:/cat9k_lite_iosxe.17.09.04a.SPA.bin flash:/

I have found this allows enough space to install the new code.

[balaji.bandi]

what is the existing IOS code - what mode it was bundle or installed mode

post below output :

dir flash:   (this give you more information what files are occupied the space).

Or you can also clean up  #install remove inactive  (if this was installed mode).