Cisco 9200 Out of Space and Cannot Upgrade

jschaff · ‎01-03-2022

Hello,

I have a stack of 3 C9200 s 2 are 48 port and 1 is a 24 port. I am trying to do an IOS-XE upgrade on them to cat9k_lite_iosxe.17.03.04b.SPA.bin. All of my other stacks have done the upgrade no problem, but this stack is just not having it. I have run the "install remove inactive" command to clean up space, but when I run "install add file flash:cat9k_lite_iosxe.17.03.04b.SPA.bin active commit" I am met with that switch 2 and 3 do not have sufficient space to unpack the file. Switch 1 has plenty of space and I would have thought that with switch 2 being the exact same model and port count that it would have enough space as well, but it does not. I am not seeing anything on switch2 or 3 that would take up the space or would allow me to free up the space needed to unpack the file. Below is the error I am getting and what is in flash on switch 2 and 3. Any help on this would be greatly appreciated.

--- Starting Add ---

Performing Add on all members

[1] Add package(s) on switch 1

[1] Finished Add on switch 1

[2] Add package(s) on switch 2

Expanding image file: flash:cat9k_lite_iosxe.17.03.04b.SPA.bin

[2]: Expanding file

... expanding file completed

[2]: Verifying parameters

[2]: Expanding superpackage flash:cat9k_lite_iosxe.17.03.04b.SPA.bin

[2]: ... parameters verified

[2]: Validating package type

[2]: FAILED: flash: requires 471540 KB of free space, but only 451624 KB is available

FAILED: [2]: FAILED: Expanding all-in-one software package failed in switch 2

FAILED: install_add exit(1) Mon Jan 3 06:30:55 MST 2022

[2] Finished Add on switch 2

[3] Add package(s) on switch 3

Expanding image file: flash:cat9k_lite_iosxe.17.03.04b.SPA.bin

[3]: Expanding file

... expanding file completed

[3]: Verifying parameters

[3]: Expanding superpackage flash:cat9k_lite_iosxe.17.03.04b.SPA.bin

[3]: ... parameters verified

[3]: Validating package type

[3]: FAILED: flash: requires 471540 KB of free space, but only 450248 KB is available

FAILED: [3]: FAILED: Expanding all-in-one software package failed in switch 3

FAILED: install_add exit(1) Mon Jan 3 06:30:55 MST 2022

[3] Finished Add on switch 3

Checking status of Add on [1 2 3]

Add: Passed on [1]. Failed on [2 3]

Finished Add

FAILED: install_add_activate_commit exit(1) Mon Jan 3 06:34:11 MST 2022

Jan 3 06:34:12.722 %INSTALL-3-OPERATION_ERROR_MESSAGE: R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_lite_iosxe.17.03.04b.SPA.bin, Error: FAILED: [2]: FAILED: Expanding all-in-one software package failed in switch 2

Jan 3 06:34:12.783 %INSTALL-3-OPERATION_ERROR_MESSAGE: R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_lite_iosxe.17.03.04b.SPA.bin, Error: FAILED: install_add exit(1) Mon Jan 3 06:30:55 MST 2022

Jan 3 06:34:12.854 %INSTALL-3-OPERATION_ERROR_MESSAGE: R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_lite_iosxe.17.03.04b.SPA.bin, Error: FAILED: [3]: FAILED: Expanding all-in-one software package failed in switch 3

Jan 3 06:34:12.916 %INSTALL-3-OPERATION_ERROR_MESSAGE: R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_lite_iosxe.17.03.04b.SPA.bin, Error: FAILED: install_add exit(1) Mon Jan 3 06:30:55 MST 2022

Jan 3 06:34:12.975 %INSTALL-3-OPERATION_ERROR_MESSAGE: R0/0: install_engine: Failed to install_add_activate_commit package flash:cat9k_lite_iosxe.17.03.04b.SPA.bin, Error: [2 3]: FAILED: Add failed in switch 2 3

Switch 2 flash:

Directory of flash-2:/

32390 -rw- 2097152 Jan 3 2022 07:11:41 -07:00 nvram_config

64769 drwx 4096 Jan 3 2022 06:47:12 -07:00 .installer

89057 drwx 4096 Jan 3 2022 06:15:19 -07:00 .prst_sync

56679 -rw- 15088 Jan 3 2022 06:13:21 -07:00 rdope_out.txt

56682 -rw- 0 Jan 3 2022 06:13:21 -07:00 dope_hist

56680 -rw- 89 Jan 3 2022 06:13:18 -07:00 rdope.log

56676 -rw- 134458 Jan 3 2022 06:11:46 -07:00 memleak.tcl

32387 -rw- 2130 Jan 3 2022 06:11:30 -07:00 boothelper.log

34 drwx 4096 Jan 3 2022 06:11:28 -07:00 dc_profile_dir

56673 -rw- 555 Jan 3 2022 06:11:09 -07:00 bootloader_evt_handle.log

32386 -rw- 2130 Dec 3 2021 06:06:09 -07:00 boothelper.log.old

32389 drwx 4096 Dec 3 2021 06:04:35 -07:00 .rommon_sync

32391 -rw- 820 Jul 26 2021 11:19:31 -07:00 vlan.dat

56683 -rw- 2097152 Jul 26 2021 08:37:08 -07:00 nvram_config_bkup

72866 drwx 4096 Jul 26 2021 08:37:03 -07:00 license_evlog

64780 drwx 4096 Jul 26 2021 08:32:47 -07:00 pnp-tech

72865 drwx 4096 Jun 3 2021 20:38:46 -07:00 .rollback_timer

64807 -rw- 40648801 Jun 3 2021 20:33:29 -07:00 cat9k_lite-rpboot.17.03.03.SPA.pkg

64803 -rw- 4919 Jun 3 2021 20:33:29 -07:00 packages.conf

64806 -rw- 11031572 Jun 3 2021 20:32:08 -07:00 cat9k_lite-webui.17.03.03.SPA.pkg

64805 -rw- 4133912 Jun 3 2021 20:32:08 -07:00 cat9k_lite-srdriver.17.03.03.SPA.pkg

64804 -rw- 426931224 Jun 3 2021 20:32:08 -07:00 cat9k_lite-rpbase.17.03.03.SPA.pkg

64774 drwx 4096 Jun 3 2021 20:28:17 -07:00 .dbpersist

64772 drwx 4096 Jun 3 2021 20:25:25 -07:00 core

72867 drwx 4096 Jun 3 2021 20:25:23 -07:00 guest-share

64777 drwx 4096 Jun 3 2021 20:25:04 -07:00 onep

64776 drwx 4096 Jun 3 2021 20:25:03 -07:00 pnp-info

8097 drwx 4096 Jun 3 2021 20:24:13 -07:00 .USWAP

40481 drwx 4096 Jun 3 2021 20:19:37 -07:00 Tbot

24290 drwx 4096 Jun 3 2021 20:19:36 -07:00 .CRFT

36 drwx 4096 Jun 3 2021 20:19:31 -07:00 sys_report

11 drwx 4096 Jun 3 2021 20:19:29 -07:00 tech_support

105249 drwx 4096 Jun 3 2021 20:19:29 -07:00 ss_disc

56674 -rw- 5242880 Jun 3 2021 20:19:29 -07:00 ssd

89060 drwx 4096 Jun 3 2021 20:19:29 -07:00 gs_script

1957167104 bytes total (846725120 bytes free)

Switch 3 flash:

Directory of flash-3:/

113355 -rw- 2097152 Jan 3 2022 07:11:45 -07:00 nvram_config_bkup

89066 -rw- 2097152 Jan 3 2022 07:11:45 -07:00 nvram_config

11 drwx 4096 Jan 3 2022 06:47:12 -07:00 .installer

89065 -rw- 820 Jan 3 2022 06:16:05 -07:00 vlan.dat

8099 drwx 4096 Jan 3 2022 06:15:59 -07:00 license_evlog

24289 drwx 4096 Jan 3 2022 06:15:37 -07:00 .prst_sync

113351 -rw- 15088 Jan 3 2022 06:13:21 -07:00 rdope_out.txt

89061 -rw- 0 Jan 3 2022 06:13:21 -07:00 dope_hist

113352 -rw- 89 Jan 3 2022 06:13:18 -07:00 rdope.log

113348 -rw- 134458 Jan 3 2022 06:11:46 -07:00 memleak.tcl

89059 -rw- 2130 Jan 3 2022 06:11:30 -07:00 boothelper.log

56696 drwx 4096 Jan 3 2022 06:11:28 -07:00 dc_profile_dir

113345 -rw- 481 Jan 3 2022 06:11:09 -07:00 bootloader_evt_handle.log

89058 -rw- 2130 Dec 3 2021 06:06:08 -07:00 boothelper.log.old

89060 drwx 4096 Dec 3 2021 06:04:35 -07:00 .rommon_sync

23 drwx 4096 Jul 21 2021 14:23:08 -07:00 pnp-tech

16193 drwx 4096 May 29 2021 07:26:36 -07:00 .rollback_timer

50 -rw- 40648801 May 29 2021 07:21:15 -07:00 cat9k_lite-rpboot.17.03.03.SPA.pkg

46 -rw- 4919 May 29 2021 07:21:15 -07:00 packages.conf

49 -rw- 11031572 May 29 2021 07:19:42 -07:00 cat9k_lite-webui.17.03.03.SPA.pkg

48 -rw- 4133912 May 29 2021 07:19:42 -07:00 cat9k_lite-srdriver.17.03.03.SPA.pkg

47 -rw- 426931224 May 29 2021 07:19:42 -07:00 cat9k_lite-rpbase.17.03.03.SPA.pkg

16 drwx 4096 May 29 2021 07:15:48 -07:00 .dbpersist

22 drwx 4096 May 29 2021 07:13:01 -07:00 guest-share

14 drwx 4096 May 29 2021 07:13:00 -07:00 core

20 drwx 4096 May 29 2021 07:12:39 -07:00 pnp-info

18 drwx 4096 May 29 2021 07:12:38 -07:00 onep

64769 drwx 4096 May 29 2021 07:11:48 -07:00 .USWAP

97153 drwx 4096 May 29 2021 07:07:11 -07:00 Tbot

80961 drwx 4096 May 29 2021 07:07:10 -07:00 .CRFT

56698 drwx 4096 May 29 2021 07:07:05 -07:00 sys_report

56673 drwx 4096 May 29 2021 07:07:03 -07:00 tech_support

40481 drwx 4096 May 29 2021 07:07:03 -07:00 ss_disc

113346 -rw- 5242880 May 29 2021 07:07:03 -07:00 ssd

24292 drwx 4096 May 29 2021 07:07:03 -07:00 gs_script

1957167104 bytes total (845152256 bytes free)

Just for reference here is switch 1 flash:

Directory of flash:/

16199 drwx 4096 Jan 3 2022 07:13:43 -07:00 .installer

16211 -rw- 2097152 Jan 3 2022 07:11:41 -07:00 nvram_config_bkup

16210 -rw- 2097152 Jan 3 2022 07:11:41 -07:00 nvram_config

16214 -rw- 482856473 Jan 3 2022 06:58:42 -07:00 cat9k_lite_iosxe.17.03.04b.SPA.bin

16213 -rw- 820 Jan 3 2022 06:15:35 -07:00 stby-vlan.dat

16212 -rw- 0 Jan 3 2022 06:13:21 -07:00 dope_hist

16206 -rw- 15088 Jan 3 2022 06:13:21 -07:00 rdope_out.txt

16207 -rw- 89 Jan 3 2022 06:13:18 -07:00 rdope.log

48616 drwx 4096 Jan 3 2022 06:13:14 -07:00 license_evlog

64769 drwx 4096 Jan 3 2022 06:13:00 -07:00 .prst_sync

16204 -rw- 134458 Jan 3 2022 06:11:47 -07:00 memleak.tcl

16194 -rw- 2130 Jan 3 2022 06:11:31 -07:00 boothelper.log

48603 drwx 4096 Jan 3 2022 06:11:29 -07:00 dc_profile_dir

16195 -rw- 407 Jan 3 2022 06:11:11 -07:00 bootloader_evt_handle.log

16196 -rw- 2130 Dec 3 2021 06:06:08 -07:00 boothelper.log.old

16208 -rw- 820 Jul 26 2021 11:19:31 -07:00 vlan.dat

48619 drwx 4096 Jul 21 2021 14:29:51 -07:00 pnp-tech

56673 drwx 4096 May 21 2021 08:16:26 -07:00 .rollback_timer

48638 -rw- 40648801 May 21 2021 08:11:07 -07:00 cat9k_lite-rpboot.17.03.03.SPA.pkg

48634 -rw- 4919 May 21 2021 08:11:07 -07:00 packages.conf

48637 -rw- 11031572 May 21 2021 08:09:45 -07:00 cat9k_lite-webui.17.03.03.SPA.pkg

48636 -rw- 4133912 May 21 2021 08:09:45 -07:00 cat9k_lite-srdriver.17.03.03.SPA.pkg

48635 -rw- 426931224 May 21 2021 08:09:45 -07:00 cat9k_lite-rpbase.17.03.03.SPA.pkg

16197 drwx 4096 May 21 2021 08:03:03 -07:00 core

48617 drwx 4096 May 21 2021 08:03:01 -07:00 guest-share

48614 drwx 4096 May 21 2021 08:02:41 -07:00 onep

48613 drwx 4096 May 21 2021 08:02:41 -07:00 pnp-info

48606 drwx 4096 May 21 2021 08:01:42 -07:00 Tbot

48605 drwx 4096 May 21 2021 08:01:37 -07:00 sys_report

48580 drwx 4096 May 21 2021 08:01:35 -07:00 tech_support

48578 drwx 4096 May 21 2021 08:01:35 -07:00 ss_disc

16202 -rw- 5242880 May 21 2021 08:01:35 -07:00 ssd

16201 drwx 4096 May 21 2021 08:01:35 -07:00 gs_script

16219 drwx 4096 May 21 2021 06:12:22 -07:00 .dbpersist

1956839424 bytes total (875577344 bytes free)

Georg Pauwen · ‎01-03-2022

Hello,

try and run:

request platform software package clean switch all

as the first command before anything else.

jschaff · ‎01-03-2022

This is what I get when trying to run request platform software package clean switch all:

SW_AMS_IDFQuad_STK#request platform software package clean switch all

^

% Invalid input detected at '^' marker.

SW_AMS_IDFQuad_STK#request platform software ?

console Request console actions

crft Request CRFT actions

iox-manager IOx Manager

license CSL Software license

process Request process actions

python Request a python interactive shell

shell Request shell actions

system Request system actions

trace Request trace actions

vty Request vty actions

yang-management YANG management

Package does not seem to be an option under software.

The current version on the switch is 17.3.3, so I am not sure if that makes a difference with this command, but it is not running for me.

Reza Sharifi · ‎01-03-2022

How about:

"install remove inactive" command

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9200/software/release/17-3/release_notes/ol-17-3-9200.html

HTH

jschaff · ‎01-03-2022

Before attempting to install the software I did run the install remove inactive, as noted in my original post. There still is not enough space on switch 2 or 3.

Leo Laohoo · ‎01-03-2022

This is a known "behaviour" with IOS-XE.

Reboot the entire stack and try again. Let us know if it still does not work, I have a few trick left.

jschaff · ‎01-03-2022

Thanks for the reply! I did try reloading them this morning, with no change, but I will try again tomorrow morning and report back.

Leo Laohoo · ‎01-03-2022

@jschaff wrote:

I did try reloading them this morning, with no change

For the stack to boot into Bundle mode.

Once the stack boots into Bundle mode, convert it to Install mode and then reboot again.

If this does not work, let us know -- I have two more tricks.

jschaff · ‎01-04-2022

@Leo Laohoo wrote:
For the stack to boot into Bundle mode.
Once the stack boots into Bundle mode, convert it to Install mode and then reboot again.
If this does not work, let us know -- I have two more tricks.

I got the switches into bundle mode and was able to upgrade all 3 through bundle mode, then run install remove inactive to get rid of the old IOS-XE .pkgs. From there I had enough space to get the .bin expand on all of the switches, so it is now set back to install mode.

The weird thing I am seeing is even after removing all of the files, I still have 516ish MB of unaccounted for flash being used. When looking at all the files in flash, it is adding up to a total of 495ish MB. The switches are both showing the the use of about 1gb of their flash, which when doing the math of what they are have total and what all the files are totaled up to, I have 516ish MB unaccounted for. Even after the IOS-XE upgrade and several reboots. Do you know where that flash might be being used or if this is just a bug that Cisco needs to fix, etc?

Reza Sharifi · ‎01-04-2022

I think this is just a bug, as your flash has plenty of space and upgrading should not be a problem. I recommend opening a ticket with TAC as this may be a known bug to them.

HTH

Leo Laohoo · ‎01-04-2022

@jschaff wrote:
Do you know where that flash might be being used or if this is just a bug that Cisco needs to fix, etc?

This is a known bug with IOS-XE. I have seen this before on a number of times, hence, I have a few "tricks".

IMPORTANT WARNING: It is of great benefit if, every 6 months (or so), appliances running IOS-XE gets rebooted (cold reboot preferred).

jschaff · ‎01-05-2022

Any more tricks you have would be great, as I was able to get them to update but that are still showing about 500MB of used flash that cannot be accounted for.

Leo Laohoo · ‎01-05-2022

@jschaff wrote:

that are still showing about 500MB of used flash that cannot be accounted for.

Look at each stack member's flash: for the following:

dir flash-X:core
dir crashinfo-X:

The files in there can be deleted.

udifa · ‎12-01-2022

Hi jschaff,

we had similar probplem with lack if free flash space on one of our stack members.

we manage to solve this issiues by running "guestshell enable" (some time it throws error about interface) after this you can run sh "app-hosting detail" and you will get status DEPLOYED then running "guestshell destroy" command and check flash free space by "show file systems".

this fix the free space issue on the switch for us

Have a good day!

Fahad Afzal · ‎12-20-2022

Thanks very much udifa. Your solution resolved the space issue.