Cisco 9200 ZTP fails to complete firmware upgrade - flash space issue

[ROBERT BROOKS]

Hello,

I have a number of Cisco 9200 switches to build and would normally build these manually ( old school ).  Decided to be all modern on this one and try to automate the build process for the Cisco 9200 switches.  Big mistake !

There are a number of guides for ZTP with Python scripts and details for DHCP / Apache / TFTP setup and I have managed to get this all working with no issue.

The issue I appear to be having - and the reason for this discussion - is regarding the firmware upgrade failing due to lack of flash memory space.  Just wondering if ZTP, using Guestshell and a Python script on a 9200, it is even possible to upgrade the firmware (9200 has 4GB flash, 9200 has 16GB flash).  Guestshell grabs a large amount of flash memory and "install" mode commands require 1GB free of flash to run correctly.  It appears that with the 9200 4GB Flash memory limit, there is just not enough as to start with I only have 1.3GB remaining on the drive and that drops to 700MB when Guestshell is loaded - so not enough to then run an "install" command.

I have cleaned the flash, there are no old images etc.

Just wondering if anyone has managed to automate the build process of the 9200 using ZTP that both upgrades the firmware and loads a defined configuration, or am I trying to do the impossible and should have purchased to 9300 instead ?  

[Leo Laohoo]

Post the complete output to the following commands: 

1.  sh version
2.  dir flash:
3.  dir flash:core

[Sebastian Hoefer]

Hi,

I'm experiencing exactly the same issue. I'm wondering if you solved the issue somehow? I saw that the issue is not with older versions of IOS-XE. Running version 17.3.4 and upgrading to 17.9.4 is working perfectly. But for example upgrading from 17.9.3 to 17.9.4 is not working because of the too less space.

Kind regards,

Sebastian

[ROBERT BROOKS]

I was on some Cisco Partner training last week and asked the question there. The verbal answer was the 9200 does not have enough flash memory to complete ZTP firmware upgrade in install mode, with guestshell enabled. They have put me in touch with someone else in their team but yet to hear back from them as to any way to get it to work. If I get an answer I will post it back.

[Leo Laohoo]

---

@ROBERT BROOKS wrote:
The verbal answer was the 9200 does not have enough flash memory to complete ZTP firmware upgrade in install mode, with guestshell enabled.

---

This must be another one of those "undocumented feature".

Thanks for this information.

[Rich R]

Did you mean "9300 has 16GB flash" @ROBERT BROOKS ?

My 9200L have 2GB of flash <smile> and 9300 have about 12GB so maybe it varies by specific models and revisions within the range.

I'd also like to see the outputs Leo asked for below because something doesn't add up.  I'd also like to see "sh file sys"?
Same for yours @Sebastian Hoefer 

[Sebastian Hoefer]

Hi All,

Thank you very much for your feedback.
Find the mentioned output below.
Don't be confused about, that the flash is almost empty. That was part of my script to erase everything in order to get the needed space for the upgrade.
But as you can see, there is basically nothing on the flash, but more then 1 GB is used.

Kind regards,
Sebastian

Switch#show version
Cisco IOS XE Software, Version 17.09.03
Cisco IOS Software [Cupertino], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 17.9.3, RELEASE SOFTWARE (fc6)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Tue 14-Mar-23 18:12 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2023 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 17.9.1r [FC13], RELEASE SOFTWARE (P)

Switch uptime is 16 hours, 27 minutes
Uptime for this control processor is 16 hours, 29 minutes
System returned to ROM by Reload Command
System image file is "flash:packages.conf"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-essentials Smart License network-essentials
dna-essentials Subscription Smart License dna-essentials

Smart Licensing Status: Smart Licensing Using Policy

cisco C9200-48P (ARM64) processor with 646364K/3071K bytes of memory.
Processor board ID JAE26161B1A
1 Virtual Ethernet interface
104 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4014172K bytes of physical memory.
819200K bytes of Crash Files at crashinfo:.
819200K bytes of Crash Files at crashinfo-2:.
1941504K bytes of Flash at flash:.
1941504K bytes of Flash at flash-2:.

Base Ethernet MAC Address : 48:2e:72:18:7e:00
Motherboard Assembly Number : 73-20259-03
Motherboard Serial Number : JAE26161B1A
Model Revision Number : D0
Motherboard Revision Number : A0
Model Number : C9200-48P
System Serial Number : JAE26161B1A
CLEI Code Number : INM9200ERB

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 C9200-48P 17.09.03 CAT9K_LITE_IOSXE INSTALL
2 56 C9200-48P 17.09.03 CAT9K_LITE_IOSXE INSTALL

Switch 02
---------
Switch uptime : 16 hours, 29 minutes

Base Ethernet MAC Address : c0:f8:7f:90:63:00
Motherboard Assembly Number : 73-20259-03
Motherboard Serial Number : JAE26161BJT
Model Revision Number : D0
Motherboard Revision Number : A0
Model Number : C9200-48P
System Serial Number : JAE26161BJT
reload history file is NULL
Last reload reason : Error while obtaining reason
CLEI Code Number : INM9200ERB

Configuration register is 0x102

Switch#dir flash:
Directory of flash:/

48585 drwx 4096 Nov 22 2023 16:17:24 +00:00 pnp-tech
48583 drwx 4096 Nov 22 2023 16:17:08 +00:00 .rommon_sync
8098 drwx 4096 Nov 22 2023 16:11:29 +00:00 pnp-info
8099 drwx 4096 Nov 22 2023 16:11:14 +00:00 .installer

1956839424 bytes total (850620416 bytes free)

Switch#dir flash:core
%Error opening flash:/core (No such file or directory)

Switch#show file sys
File Systems:

Size(b) Free(b) Type Flags Prefixes
- - opaque rw system:
- - opaque rw tmpsys:
825638912 742932480 disk rw crashinfo: crashinfo-1:
825753600 752353280 disk rw crashinfo-2: stby-crashinfo:
* 1956839424 850620416 disk rw flash: bootflash: flash-1:
1957167104 1364197376 disk rw flash-2: stby-flash:
1321074688 1186070528 disk ro webui:
- - opaque rw null:
- - opaque ro tar:
- - network rw tftp:
2097152 2088908 nvram rw nvram:
- - opaque wo syslog:
- - network rw rcp:
- - network rw http:
- - network rw ftp:
- - network rw scp:
- - network rw sftp:
- - network rw https:
- - opaque ro cns:
2097152 2085336 nvram rw stby-nvram:
- - nvram rw stby-rcsf:
- - opaque rw revrcsf:

[Leo Laohoo]

Please post the complete output to the following command: 

sh platform software mount switch <SWITCH MEMBERS> r0 | include ^tmpfs.*tmp | exclude /tmp/

[Rich R]

I don't think that double pipe regex will work @Leo Laohoo ?
I think the result of that will always be blank won't it?

[Leo Laohoo]

It will work.  

[Sebastian Hoefer]

For both switches in my stack empty output:

Switch#$rm software mount switch 1 r0 | include ^tmpfs.*tmp | exclude /tmp/
Switch#$rm software mount switch 2 r0 | include ^tmpfs.*tmp | exclude /tmp/

Same without the exclude /tmp/

[Rich R]

Cool - you also have 2GB flash so no surprise it's running out.

This bug might interest you guys too ...
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa71156

Mine shows:
* 1956839424 1256603648 disk rw flash: flash-1: bootflash:
  1956904960 1256456192 disk rw flash-2: stby-flash:
You do seem to be missing about 400M on that primary flash.  Might be worth getting TAC to check it out for you in case you've been hit by that bug.