Solved: C9500 Install Mode issue - Page 2

philasam79 · ‎11-25-2021

Hi folks. I'm trying to upgrade my C9500 that is in VSS to IOS image Cat9k_iosxe.17.03.04.SPA.bin in install mode. when i issue the command "install add file flash:Cat9k_iosxe.17.03.04.SPA.bin activate ISSU commit. It fails to perform the upgrade. I discovered that the "packages.conf" file isn't in the flash.

How can i get it back in the flash to be able to perform the ISSU upgrade successfully?

philasam79 · ‎11-26-2021

Hi Leo,

I'm not sure what you are trying to advise here. I have tried the steps mentioned in the PDF. Right after your last comment came in. I issue "install remove inactive" command on the switch. This command was completed. However, there was no install files to be removed as the same files in my "Flash" are still the file there after the install remove inactive was completed. The question i wish you could answer is "If there is no packages.conf" file or any ".PKG" file in the flash, where will the install process find packages.conf to boot from?

See below result of my boot variable. this was after i had issued the install remove inactive command and set the boot variable. However, after this the appropriate image was tranfered to the switch and when i issued the "install add file flash:cat9k_iosxe.17.03.04.SPA.bin activate commit" I get this error"c9500_core#$ file flash:cat9k_iosxe.17.03.04.SPA.bin activate commit
install_add_activate_commit: START Sat Nov 27 01:38:18 UTC 2021
FAILED: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed
c9500_core#
*Nov 27 01:38:18.546: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install one-shot flash:cat9k_iosxe.17.03.04.SPA.bin
*Nov 27 01:38:18.845: %INSTALL-3-OPERATION_ERROR_MESSAGE: Switch 2 R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_iosxe.17.03.04.SPA.bin, Error: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed"

c9500_core#sh boot
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

81921 drwx 4096 Nov 27 2021 01:26:31 +00:00 .installer
442381 -rw- 616 Nov 27 2021 01:23:12 +00:00 stby-vlan.dat
49155 -rw- 2097152 Nov 27 2021 01:21:12 +00:00 nvram_config_bkup
49154 -rw- 2097152 Nov 27 2021 01:21:12 +00:00 nvram_config
49159 -rw- 0 Nov 27 2021 01:21:10 +00:00 dope_hist
442383 -rw- 16984 Nov 27 2021 01:21:10 +00:00 rdope_out.txt
442384 -rw- 89 Nov 27 2021 01:21:09 +00:00 rdope.log
442387 -rw- 5242880 Nov 27 2021 01:19:44 +00:00 ssd
442376 -rwx 2049 Nov 27 2021 01:19:24 +00:00 svl_ipc.tcl
442377 -rw- 134458 Nov 27 2021 01:19:24 +00:00 memleak.tcl
49156 -rw- 312 Nov 27 2021 01:19:14 +00:00 boothelper.log
131073 drwx 4096 Nov 27 2021 01:19:13 +00:00 dc_profile_dir
442375 -rw- 1739 Nov 27 2021 01:19:04 +00:00 bootloader_evt_handle.log
98305 drwx 4096 Nov 26 2021 19:00:16 +00:00 .prst_sync
49162 -rw- 439 Nov 26 2021 18:16:03 +00:00 capsule-upgrade.log
229377 drwx 4096 Nov 25 2021 13:47:57 +00:00 .rommon_sync
49160 -rw- 616 Nov 25 2021 13:32:36 +00:00 vlan.dat
147457 drwx 4096 Nov 24 2021 23:02:23 +00:00 .dbpersist
49161 -rw- 12242 Nov 24 2021 23:01:55 +00:00 running_config
49158 -rw- 907450095 Nov 24 2021 18:33:51 +00:00 cat9k_iosxe.17.03.04.SPA.bin
221185 drwx 4096 Nov 18 2021 03:06:34 +00:00 Tbot
81967 drwx 4096 Nov 18 2021 03:06:28 +00:00 sys_report
81922 drwx 4096 Nov 18 2021 03:06:28 +00:00 tech_support
122881 drwx 4096 Nov 18 2021 03:06:28 +00:00 gs_script
81960 drwx 4096 Nov 18 2021 03:06:26 +00:00 ss_disc
442380 -rw- 116459 Nov 17 2021 22:39:07 +00:00 pnp-tech-discovery-summary
442379 -rw- 35 Nov 17 2021 22:39:04 +00:00 pnp-tech-time
155649 drwx 4096 Apr 30 2021 00:20:10 +00:00 onep
90113 drwx 4096 Jan 28 2019 10:32:38 +00:00 core
114689 drwx 4096 Jan 28 2019 10:31:12 +00:00 .rollback_timer

!

FYI: This is not the first time i'm doing this install. I have more than 50 Cat9k in my infrastructure. I have done this 10s to 50 times. However, i have never come across a situation where, "packages.conf" or Pkg files aren't in the flash. If you can tell me how to get "packages.conf" in the flash, i could so much appreciate that.

philasam79 · ‎11-26-2021

c9500_core#install add file flash:cat9k_iosxe.17.03.03.SPA.bin activate commit
install_add_activate_commit: START Sat Nov 27 01:54:01 UTC 2021

System configuration has been modified.
Press Yes(y) to save the configuration and proceed.
Press No(n) for proceeding without saving the configuration.
Press Quit(q) to exit, you may save configuration and re-enter the command. [y/n/q]yBuilding configuration...

[OK]Modified configuration has been saved
FAILED: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed

c9500_core#
*Nov 27 01:54:06.663: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install one-shot flash:cat9k_iosxe.17.03.03.SPA.bin
*Nov 27 01:54:06.959: %INSTALL-3-OPERATION_ERROR_MESSAGE: Switch 1 R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_iosxe.17.03.03.SPA.bin, Error: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed

Leo Laohoo · ‎11-26-2021

@philasam79 wrote:

*Nov 27 01:54:06.959: %INSTALL-3-OPERATION_ERROR_MESSAGE: Switch 1 R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_iosxe.17.03.03.SPA.bin, Error: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed

Can you try the following command:

install abort issu

Can I see the complete output to the command "sh version". What I want to see are the following:

What is the current IOS-XE version; and
What is uptime

@philasam79 wrote:

If you can tell me how to get "packages.conf" in the flash, i could so much appreciate that.

The Install Mode will extract the necessary package files from the BIN file. If the process works, it will also create a packages.conf file. Remember, "if the process works". In some documents, I have a "gotchas" section and this only applies when the system is unable to create the "packages.conf" file and manual intervention is required.

philasam79 · ‎11-26-2021

I have ran this command "install abort issue" more than 5 times today and tried to attempt the install over and over again. Everytime, i abort the install, i get the same error right when i tried to install the image.

!

c9500_core#install abort issu
install_abort: START Sat Nov 27 02:26:45 UTC 2021

NOTE: Going to start Abort ISSU install process

STAGE 0: System Level Sanity Check
===================================================
/usr/binos/conf/install_util.sh: line 688: /bootflash/.installer/install_add_pkg_list.txt: No such file or directory
Standby is not online. User can manually abort the ISSU
operation once Standby is online
ERROR: install_abort exit(2 ) Sat Nov 27 02:26:45 UTC 2021
!
!
c9500_core#sh version
Cisco IOS XE Software, Version 17.03.04
Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.4, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Sat 03-Jul-21 01:55 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2021 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)

c9500_core uptime is 1 hour, 6 minutes
Uptime for this control processor is 1 hour, 8 minutes
System returned to ROM by SSO Switchover
System image file is "flash:cat9k_iosxe.17.03.04.SPA.bin"
Last reload reason: PowerOn

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage

Smart Licensing Status: Registration Not Applicable/Not Applicable

cisco C9500-40X (X86) processor with 1331366K/6147K bytes of memory.
Processor board ID xxxxxxxx
2 Virtual Ethernet interfaces
96 Ten Gigabit Ethernet interfaces
4 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
16777216K bytes of physical memory.
1638400K bytes of Crash Files at crashinfo:.
11264000K bytes of Flash at flash:.
15662080K bytes of USB Flash at usbflash0:.

Base Ethernet MAC Address : 00:2f:5c:0x:0c:00
Motherboard Assembly Number : 73-18140-03
Motherboard Serial Number : T
Model Revision Number : D0
Motherboard Revision Number : B0
Model Number : C9500-40X
System Serial Number :
CLEI Code Number :

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 50 C9500-40X 17.03.04 CAT9K_IOSXE BUNDLE

Leo Laohoo · ‎11-26-2021

@philasam79 wrote:

I have ran this command "install abort issue" more than 5 times today and tried to attempt the install over and over again. Everytime, i abort the install, i get the same error right when i tried to install the image.

Right, this leaves only one option left: Reboot the appliance one more time and then re-run the Install Mode command.

install add file flash:cat9k_iosxe.17.03.04.SPA.bin activate commit

philasam79 · ‎11-26-2021

I set the boot variable to "boot flash:packages.conf".
Remember that i had said that if packages.confi is not present in the flash:, the install will not know where to locate the file? Hence, during a reboot, there will be an ERROR.

This is the boot ERROR.
Initializing Hardware...

System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)
Compiled Wed 10/23/2019 16:35:17.50 by rel

Current ROMMON image : Primary
Last reset cause : PowerOn
C9500-40X platform with 16777216 Kbytes of main memory

boot: attempting to boot from [flash:packages.conf]
ERROR: failed to boot from flash:packages.conf (Not Found)
!
Now that the switch is in ROMMON, i had to boot it with the image directly.
!
switch:
switch: boot flash:cat9k_iosxe.17.03.03.SPA.bin
boot: attempting to boot from [flash:cat9k_iosxe.17.03.03.SPA.bin]
boot: reading file cat9k_iosxe.17.03.03.SPA.bin
!
THIS IS THE OUTPUT OF THE SHOW VERSION AFTER I DECIDED TO BOOT WITH THE .BIN FROM ROMMON.
!
Cisco IOS XE Software, Version 17.03.03
Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.3, RELEASE SOFTWARE (fc7)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Thu 04-Mar-21 12:32 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2021 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)

c9500_core uptime is 0 minutes
Uptime for this control processor is 4 minutes
System returned to ROM by Reload Command
System image file is "flash:cat9k_iosxe.17.03.03.SPA.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage

Smart Licensing Status: Registration Not Applicable/Not Applicable

cisco C9500-40X (X86) processor with 1331521K/6147K bytes of memory.
Processor board ID XXXXXXXXXXXXX
2 Virtual Ethernet interfaces
96 Ten Gigabit Ethernet interfaces
4 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
16777216K bytes of physical memory.
1638400K bytes of Crash Files at crashinfo:.
11264000K bytes of Flash at flash:.
15662080K bytes of USB Flash at usbflash0:.

Base Ethernet MAC Address : 00:2f:5c:03:2c:80
Motherboard Assembly Number : 73-18140-03
Motherboard Serial Number : XXXXXXXXXXXXXXX
Model Revision Number : D0
Motherboard Revision Number : B0
Model Number : C9500-40X
System Serial Number : XXXXXXXXXXXXXXXXX
CLEI Code Number :

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 50 C9500-40X 17.03.03 CAT9K_IOSXE BUNDLE

Configuration register is 0x102

Leo Laohoo · ‎11-26-2021

After the reboot, execute this command:

install add file flash:cat9k_iosxe.17.03.04.SPA.bin activate commit

If this does not work, I have one more trick to do, however, export the switch config externally.

philasam79 · ‎11-26-2021

Ran the command after the reboot. This is the ERROR. The same error is what i will also get after i have ran "install abort issu.

!

c9500_core#install add file flash:cat9k_iosxe.17.03.04.SPA.bin activate commit
install_add_activate_commit: START Sat Nov 27 03:00:32 UTC 2021
FAILED: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed
c9500_core#
*Nov 27 03:00:33.199: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install one-shot flash:cat9k_iosxe.17.03.04.SPA.bin
*Nov 27 03:00:33.491: %INSTALL-3-OPERATION_ERROR_MESSAGE: Switch 1 R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_iosxe.17.03.04.SPA.bin, Error: install_add_activate_commit : Install ISSU in progress. Abort the ISSU upgrade to proceed

Leo Laohoo · ‎11-26-2021

Ok, one last trick -- And this is going to be "NSFW".

Format the flash of the switch (Command: format flash:)
Reboot the switch, and because packages.conf is not present, it will go into ROMMON.
After the flash gets formatted, copy the IOS-XE from the USB thumb drive to the switch.
Boot the IOS-XE BIN file from ROMMON.
Re-run the Install Mode command.

philasam79 · ‎11-26-2021

Thank you Leo for your consistent followup. It didn't appear to me that my images in the flash may be corrupt. After i formatted the flash. I i rebooted the chassis from ROMMON with the .BIN extension. After a complete boot, i copied the image to flash and repeated the install and this time, it was successful. Much appreciation.

Leo Laohoo · ‎11-26-2021

@philasam79 wrote:

It didn't appear to me that my images in the flash may be corrupt

The BIN file was-not and never-was corrupt.

Here are several "take-aways" to consider:

Please "minimize" using ISSU, FSU/eFSU.
Cold reboot is more valuable than "FastReload".

DonaldWB · ‎08-03-2023

I spoke with TAC on this and if you verify that your "more packages.conf" and "more flash-2:packages.conf" are pointing to the same file (can verify via looking to see if they have the same shalsum marker if nothing else.)

then you should be able to perform the following commands in order to clear the state that the switch is in and allow you to proceed:

conf t

service internal

exit

clear install state

conf t

no service internal

Afterwards when I did a "show issu stat detail" it gave me the current ISSU Status: Enabled

conf t
no boot system
boot system bootflash:packages.conf
exit
install add file bootflash:FirmwareFile.bin activate commit
y (to save the configuration and proceed)

And viola Updated firmware just fine after that

yang yang · ‎08-31-2023

this works. you save my ass. thank bro

Pacerfan9 · ‎12-26-2023

Saved my ass as well. Beer is on me at Cisco Live!