9300 IOS update - packages.conf corrupted after update

[Blair C]

Hi all,

Recently I updated a 9300-24U from IOS 17.9.4a to 17.12.04.

Normally I have found the update process to take 10-15 minutes for 9300s and then a further 10 mins to reboot after. 

What was different for this switch however, is that it finished the update in about 5 minutes and then rebooted, which I thought was suspicious. It followed all the update process just fine. However it then failed to boot back up. Had to connect to it locally and found it was stuck in rommon mode. "show boot" showed correctly as flash:packages.conf. However when trying to read packages.conf it just spat out complete garbled nonsense.

I booted off the 17.12.04 bin file and that was fine. Was then able to copy a conf file from another 9300 with the same IOS version. Rebooted and was then all sorted.

So the original command I used was:
install add file flash:cat9k_iosxe.17.12.04.SPA.bin activate commit

From what I've read, some say the packages.conf file corrupting after update is a random bug that occurs. Is there anyway to manually prevent that from happening?

From what I understand if you do the step by step process, is that the packages.conf file doesn't get updated till you do "install activate", however this also reboots the switch. So if the packages.conf file is corrupted during this process, your switch will fail to rommon mode post reboot.


The only way I've found so far is the method from Leo: Solved: Re: Install add command for later reboot - Cisco Community

However, the problem with this method is that, although you are able to manually confirm the conf file. After you reboot, if you want to install an SMU after it. It then doesn't allow you, saying you need to activate and commit, because it technically never done this during the update process. Luckily just doing "install activate commit" works after that error message trying to install the SMU. But then have to wait 20 mins for it to "update" and reboot despite already being updated. When really only the SMU requires updating. Since I'm having to do the update process once more, my worry is this could then have the potential to break the packages.conf file.

[Leo Laohoo]

1.  Post the complete output to the following commands:

dir flash:
sh romvar

2.  For 3850 & 9300, I never use the "install add file".  Instead, I use the command "request platform software package install switch all file flash: on-reboot new auto-copy verbose":  Cisco 3850 IOS-XE/Firmware Upgrade (Install Mode)

[Blair C]

switch#dir flash:
Directory of flash:/

630787 -rwx 2097152 Feb 19 2025 16:03:31 +00:00 nvram_config_bkup
630786 -rwx 2097152 Feb 19 2025 16:03:31 +00:00 nvram_config
679937 drwx 4096 Feb 19 2025 15:56:52 +00:00 .installer
630808 -rw- 16957 Feb 19 2025 14:51:35 +00:00 rdope_out.txt
630790 -rw- 0 Feb 19 2025 14:51:35 +00:00 dope_hist
630809 -rw- 89 Feb 19 2025 14:51:34 +00:00 rdope.log
630791 -rw- 856 Feb 19 2025 14:51:28 +00:00 vlan.dat
696321 drwx 4096 Feb 19 2025 14:51:19 +00:00 .prst_sync
630798 -rw- 15 Feb 19 2025 14:50:55 +00:00 rmon_var_template
630807 -rwx 2049 Feb 19 2025 14:50:43 +00:00 svl_ipc.tcl
630801 -rw- 138115 Feb 19 2025 14:50:43 +00:00 memleak.tcl
630788 -rw- 437 Feb 19 2025 14:50:26 +00:00 boothelper.log
24581 drwx 4096 Feb 19 2025 14:50:24 +00:00 dc_profile_dir
630800 -rw- 210 Feb 19 2025 14:50:16 +00:00 mcelog.txt
630799 -rw- 777 Feb 19 2025 14:50:16 +00:00 bootloader_evt_handle.log
630826 -rw- 425 Feb 19 2025 14:48:36 +00:00 mcu_debug.txt
630824 -rw- 1198 Feb 19 2025 14:48:36 +00:00 .packages.verify.log
630838 -rw- 2734 Feb 19 2025 14:48:34 +00:00 fpgahelper.log
630823 -rw- 116 Feb 19 2025 14:48:34 +00:00 .mb_fpgapackages.log
630821 -rw- 412 Feb 19 2025 14:48:34 +00:00 .bootpackages.verify.log
630822 -rw- 1247 Feb 19 2025 14:48:34 +00:00 preboothelper.log
630818 -rw- 152 Feb 19 2025 14:48:33 +00:00 .iox_alt_hdd_cleanup.log
630829 drwx 4096 Feb 19 2025 14:48:28 +00:00 .PATCH
8193 drwx 4096 Feb 19 2025 14:48:28 +00:00 .rollback_timer
630831 drwx 4096 Feb 19 2025 14:48:25 +00:00 .PATCH-backup
630828 -rw- 7542 Feb 19 2025 14:43:18 +00:00 packages.conf
630815 drwx 4096 Feb 19 2025 14:11:15 +00:00 pcap
630817 -rw- 7542 Feb 19 2025 11:49:37 +00:00 cat9k_iosxe.17.12.04.SPA.conf
98307 -rw- 61891845 Feb 19 2025 11:49:28 +00:00 cat9k-rpboot.17.12.04.SPA.pkg
630816 drwx 4096 Feb 19 2025 11:49:28 +00:00 .images
630797 -rw- 5596 Jan 24 2025 15:54:53 +00:00 cat9k_iosxe.17.12.04.CSCwk25954.SPA.smu.bin
98314 -rw- 768689152 Jul 23 2024 18:03:45 +01:00 cat9k-rpbase.17.12.04.SPA.pkg
98311 -rw- 169464836 Jul 23 2024 18:01:58 +01:00 cat9k-espbase.17.12.04.SPA.pkg
98316 -rw- 56431616 Jul 23 2024 18:01:40 +01:00 cat9k-sipspa.17.12.04.SPA.pkg
98315 -rw- 48747524 Jul 23 2024 18:01:39 +01:00 cat9k-sipbase.17.12.04.SPA.pkg
98313 -rw- 111498240 Jul 23 2024 18:01:38 +01:00 cat9k-lni.17.12.04.SPA.pkg
98317 -rw- 41288708 Jul 23 2024 18:01:30 +01:00 cat9k-srdriver.17.12.04.SPA.pkg
98318 -rw- 17884160 Jul 23 2024 18:01:30 +01:00 cat9k-webui.17.12.04.SPA.pkg
98310 -rw- 29471752 Jul 23 2024 18:01:12 +01:00 cat9k-cc_srdriver.17.12.04.SPA.pkg
98312 -rw- 2155524 Jul 23 2024 18:01:05 +01:00 cat9k-guestshell.17.12.04.SPA.pkg
98319 -rw- 9216 Jul 23 2024 18:01:04 +01:00 cat9k-wlc.17.12.04.SPA.pkg
630813 drwx 4096 Jul 17 2024 02:02:09 +01:00 .rommon_sync
630812 -rw- 1261 Jul 17 2024 02:02:06 +01:00 Baltimore.ca
630811 -rw- 1606 Jul 17 2024 02:02:06 +01:00 StarfieldServicesRootCA.ca
630810 -rw- 1924 Jul 17 2024 02:02:06 +01:00 ISRGRootX1CrossSignDST.ca
630806 -rw- 1338 Jul 17 2024 02:02:06 +01:00 DigiCertGlobalRootCA.ca
98305 drwx 4096 Jul 17 2024 02:02:00 +01:00 fp_cc_crash
24577 drwx 4096 Jul 17 2024 02:01:59 +01:00 tech_support
630796 drwx 4096 Jul 17 2024 02:01:59 +01:00 EDScisco
630795 -rwx 0 Jul 17 2024 02:01:05 +01:00 mode_event_log
630794 drwx 4096 Jul 17 2024 02:01:05 +01:00 SHARED-IOX
32769 drwx 4096 Apr 9 2022 19:55:36 +01:00 .dbpersist
73729 drwx 4096 Apr 9 2022 19:50:15 +01:00 Tbot
696324 drwx 4096 Apr 9 2022 19:50:06 +01:00 sys_report
630789 drwx 4096 Sep 16 2020 15:11:26 +01:00 mkdir
630805 -rw- 50972 Sep 8 2020 13:13:50 +01:00 pnp-tech-discovery-summary
630804 -rw- 34 Sep 8 2020 13:13:42 +01:00 pnp-tech-time
630802 -rw- 312 Sep 8 2020 13:12:10 +01:00 tam_client_app.log
688129 drwx 4096 Feb 22 2019 05:00:54 +00:00 core
40961 drwx 4096 Feb 22 2019 05:00:34 +00:00 onep

11353194496 bytes total (9426931712 bytes free)

------------------------------------------------------------

Switch#sh romvar
----------------------------------
ROMMON variables for Active Switch
----------------------------------
ABNORMAL_RESET_COUNT=0
AUTO_SWITCH_CONSOLE_DISABLE=0
BAUD=9600
BOARDID=24662
BOOT=flash:packages.conf;
BSI=0
CONSOLE_MIRRORING_DISABLE=0
DC_COPY=yes
IMAGE_UPGRADE=no
LICENSE_BOOT_LEVEL=network-advantage+dna-advantage,all:C9300_24P;
MAC_ADDR=00:87:64:02:3F:80
MANUAL_BOOT=no
MODEL_NUM=C9300-24U
MODEL_REVISION_NUM=A0
MOTHERBOARD_ASSEMBLY_NUM=73-18272-03
MOTHERBOARD_REVISION_NUM=A0
MOTHERBOARD_SERIAL_NUM=FOC23040HGF
RANDOM_NUM=963403521
RET_2_RCALTS=
RET_2_RTS=14:43:53 GMT Wed Feb 19 2025
ROMMON_AUTOBOOT_ATTEMPT=3
ReloadReason=SMU Install RET_2_RCALTS=1739976511
SWITCH_NUMBER=1
SWITCH_PRIORITY=15
SYSTEM_SERIAL_NUM=FCW2306CH1G
TEMPLATE=access
USB_DISABLE=0
VERSION_ID=V02
Switch#

[Leo Laohoo]

If you are worried the "packages.conf" file is corrupt (I am not because the file size is the same), then we can "re-create" the file: 

del /f /r flash:packages.conf
copy flash:cat9k_iosxe.17.12.04.SPA.conf flash:packages.conf

[m_pfeiler]

Hello Leo,
thx for your posts.

attn2(9300): Can you please tell us why you NEVER use the "install add file" (although Cisco recommended at least with 17.x.x) and still use "re, quest platform software package install switch all file flash: on-reboot new auto-copy verbose" , instead?

Thx in advance,
Martin

[Leo Laohoo]

---

@m_pfeiler wrote:

Can you please tell us why you NEVER use the "install add file" (although Cisco recommended at least with 17.x.x) and still use "request platform software package install switch all file flash: on-reboot new auto-copy verbose" , instead?

---

The command "install add file" has a "bug".  The process forks with a "y/n" question.  If the operator presses "y" the switch will finalize the install process and reboot.  What's wrong with this?  Because there is a known bug where the process to rename or swap the "packages.conf" gets stuck and the swap never occurs (see below).  This means when the stack reboots, there is a strong chance the stack will still boot the old firmware.  

When the command "install add file" is used and when the swap of the "packages.conf" fails, the process continues and reboots and it becomes a flip of a coin to discover if the bug has/was hit or not.  

If I use the "request platform software package install" and coupled with the "on-reboot" option, the entire install process will STOP and dump a message to let the operator know if the swap of the "packages.conf" file was successful or not.  

Finally, one of the biggest "benefit" with the use of "request platform software package install" when combined with the "on-reboot" option is the process STOPS.  The switch/stack does not reboot.  Why is this beneficial?  Because I have the luxury of rebooting the switch/stack at a later date.  

Does this help?

EDIT (24 March 2025):  Have a look at CSCwk54116.  This bug is very, very similar to what I'm saying about "failure to swap the packages.conf file".

[Leo Laohoo]

---

@Blair C wrote:
After you reboot, if you want to install an SMU after it. It then doesn't allow you, saying you need to activate and commit

---

I want to see this complete error message when it happens. 

[Blair C]

Okay, this error only occurred after updating using your method from: Solved: Re: Install add command for later reboot - Cisco Community

I'll do the same for another switch I'll be updating shortly

[Blair C]

switch#install add file flash:cat9k_iosxe.17.12.04.CSCwk25954.SPA.smu.bin activate commit
install_add_activate_commit: START Fri Feb 21 12:12:35 GMT 2025
install_add: START Fri Feb 21 12:12:35 GMT 2025
install_add: Adding IMG
--- Starting initial file syncing ---
Copying flash:cat9k_iosxe.17.12.04.CSCwk25954.SPA.smu.bin from Switch 1 to Switch 1 2 3
Info: Finished copying to the selected Switch
Finished initial file syncing

--- Starting SMU Add operation ---
Performing SMU_ADD on all members
Checking status of SMU_ADD on [1 2 3]
SMU_ADD: Passed on [1 2 3]
Finished SMU Add operation

install_activate: START Fri Feb 21 12:12:43 GMT 2025
install_activate: Activating SMU

This operation may require a reload of the system. Do you want to proceed? [y/n]y

--- Starting SMU Activate operation ---
Performing SMU_ACTIVATE on all members
[1] SMU_ACTIVATE package(s) on Switch 1
[2] SMU_ACTIVATE package(s) on Switch 2
[3] SMU_ACTIVATE package(s) on Switch 3
[3] Finished SMU_ACTIVATE on Switch 3
[2] Finished SMU_ACTIVATE on Switch 2
[1] Finished SMU_ACTIVATE on Switch 1
Checking status of SMU_ACTIVATE on [1 2 3]
SMU_ACTIVATE: Passed on [1 2 3]
Finished SMU Activate operation

--- Starting Commit ---
Performing Commit on all members
--- Starting SMU Commit ---
[1] Commit package(s) on Switch 1
[3] Commit package(s) on Switch 3
[2] Commit package(s) on Switch 2
FAILED: Cannot commit; please use install activate to activate added packages OR install remove inactive to remove added subpackages.
FAILED: Cannot commit; please use install activate to activate added packages OR install remove inactive to remove added subpackages.
FAILED: Cannot commit; please use install activate to activate added packages OR install remove inactive to remove added subpackages.
Checking status of Commit on [1 2 3]
Commit: Passed on []. Failed on [1 2 3]
Finished Commit operation

FAILED: install_add_activate_commit Fri Feb 21 12:13:04 GMT 2025
switch#

[Leo Laohoo · Accepted solution]

Oh I know how to fix this: 

conf t
 service internal
end
clear install state

The last command will reboot the switch. 

After the switch re-boots, repeat the SMU install and see if it working or not.

[Blair C]

Thank you, this worked like a charm.

Even tried it on another switch and entered your above command before doing the first reload after unpacking the IOS image to save on a reload and it works then as well. So happy days. I can now confirm the .conf file and save on a reload installing the SMU:

Switch#install add file flash:cat9k_iosxe.17.12.04.SPA.bin activate commit

*Feb 21 13:18:40.231: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_mgr: Started install add_activate_commit flash:cat9k_iosxe.17.12.04.SPA.bininstall_add_activate_commit: START Fri Feb 21 13:18:40 UTC 2025
install_add: Adding IMG
--- Starting initial file syncing ---
Copying flash:cat9k_iosxe.17.12.04.SPA.bin from Switch 1 to Switch 1
Info: Finished copying to the selected Switch
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
[1] Finished Add package(s) on Switch 1
Checking status of Add on [1]
Add: Passed on [1]
Finished Add

Image added. Version: 17.12.04.0.4826

Warning: ISSU compatibility check failed for 17.12.04.0.4826
install_activate: Activating IMG
Following packages shall be activated:
/flash/cat9k-cc_srdriver.17.12.04.SPA.pkg
/flash/cat9k-espbase.17.12.04.SPA.pkg
/flash/cat9k-guestshell.17.12.04.SPA.pkg
/flash/cat9k-lni.17.12.04.SPA.pkg
/flash/cat9k-rpbase.17.12.04.SPA.pkg
/flash/cat9k-sipbase.17.12.04.SPA.pkg
/flash/cat9k-sipspa.17.12.04.SPA.pkg
/flash/cat9k-srdriver.17.12.04.SPA.pkg
/flash/cat9k-webui.17.12.04.SPA.pkg
/flash/cat9k-wlc.17.12.04.SPA.pkg
/flash/cat9k-rpboot.17.12.04.SPA.pkg
Following packages shall be activated:
/flash/cat9k-cc_srdriver.17.12.04.SPA.pkg
/flash/cat9k-espbase.17.12.04.SPA.pkg
/flash/cat9k-guestshell.17.12.04.SPA.pkg
/flash/cat9k-lni.17.12.04.SPA.pkg
/flash/cat9k-rpbase.17.12.04.SPA.pkg
/flash/cat9k-sipbase.17.12.04.SPA.pkg
/flash/cat9k-sipspa.17.12.04.SPA.pkg
/flash/cat9k-srdriver.17.12.04.SPA.pkg
/flash/cat9k-webui.17.12.04.SPA.pkg
/flash/cat9k-wlc.17.12.04.SPA.pkg
/flash/cat9k-rpboot.17.12.04.SPA.pkg

This operation may require a reload of the system. Do you want to proceed? [y/n]n

FAILED: install_add_activate_commit Fri Feb 21 13:19:44 UTC 2025

Switch#rename packages.conf packages.conf.0--
Switch#copy cat9k_iosxe.17.12.04.SPA.conf packages.conf
Destination filename [packages.conf]?
Copy in progress...C
7542 bytes copied in 0.021 secs (359143 bytes/sec)

-------------------------------
Checked conf file and was good
-------------------------------
Switch#config t
Switch(config)#service internal
Switch(config)#end

Switch#clear install state
clear_install_state: START Fri Feb 21 13:21:43 UTC 2025

This command will remove all the provisioned SMUs, and rollback points. Use this command with caution.
A reload is required for this process. Press y to continue [y/n]y
--- Starting clear_install_state ---
Performing clear_install_state on all members
[1] clear_install_state package(s) on switch 1
[1] Finished clear_install_state on switch 1
Checking status of clear_install_state on [1]
clear_install_state: Passed on [1]
Finished clear_install_state

-------------------------------------------
switch reboots

This also updated the rommon to version 17.12.1r (if reload without clear install state, stays on old rommon version. Can also fix by doing install activate, then post reboot, install commit as that redoes the full installation)
-------------------------------------------

Switch#show install summary
[ Switch 1 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 17.12.04.0.4826
SMU C flash:cat9k_iosxe.17.12.04.CSCwk25954.SPA.smu.bin

--------------------------------------------------------------------------------


So now able to check and confirm .conf file before reload. reboot to new ISO. Then install SMU. All in just 2 reloads. Happy days!

I wonder if this method would also work on a 9400...

[Leo Laohoo]

The method you're currently using is what I call the "One-Hit-Wonder + NSFW" or the One-Hit-Wonder (for short).  This will work on 9200, 9400, 9500 (high performance) and 9600.