Any idea on how to recover a Catalyst9300 from ROMMON

[gnijs]

It boots into ROMMON (after platform install software xxx)

i guess flash is corrupt

switch: dir flash:

Size Attributes Name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
4096 drw- .EXPAND_MOUNT_QgCJ
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch:

dir flash:.EXPAND_MOUNT_QgCJ
switch:

switch: dir sda9:

Size Attributes Name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
21680202 -rw- cat9k-recovery.SSA.bin
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch:

i tried

DEFAULT_ROUTER=10.96.56.17
IP_ADDRESS=10.96.56.19
IP_MASK=255.255.255.248

but ping still fails.

switch: ?

emergency-install Initiate Disaster Recovery

switch: emergency-install sda9:
WARNING: The system partition (bootflash:) will be erased during the system recovery install process.
Are you sure you want to proceed? [y] y/n [n]: y
Starting system recovery (sda9:) ...

boot: attempting to boot from [sda9:cat9k-recovery.SSA.bin]
boot: reading file cat9k-recovery.SSA.bin
###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

PLATFORM_TYPE C9300 speed 9600

Booting Recovery Image 16.6.3

Initiating Emergency Installation of bundle sda9:

Emergency Install failed - bundle sda9: does not exist
Will reboot now

[Leo Laohoo · Accepted solution]

Copy the IOS into a USB flash drive and perform the emergency-install from it.

[gnijs]

Thanks that worked.

(however, i have a new problem, my ZTP upgrade script always corrupts the flash in this matter. It is reproducible)

[Leo Laohoo]

Could be a bug. ZeroTouch is not meant to work with the 9K.

[gnijs]

Not on Fuji release, see here: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/16-5/configuration_guide/prog/b_165_prog_9300_cg.pdf

By the way, flash corruption was caused by the fact the script rebooted too fast. After "request platform package install....." you need to wait at least some seconds before rebooting (or use some other sort of ready checking). If you boot immediately -> flash corruption (or image not ready)

[dal]

This didn't work for me.

The process just stopped after showing:

boot: attempting to boot from [sda9:cat9k-recovery.SSA.bin]
boot: reading file cat9k-recovery.SSA.bin

What I did, was booting the switch directly from the USB stick:

boot usbflash0:cat9k_iosxe.17.06.03.SPA.bin

And then flashing the switch the normal way after the boot was complete.

[nicholaswojciechowski]

I am trying to pull my 9300-24T out of ROMMON and I tried using the TFTP and after 99% it stops and according to the logs of the TFTP server it says "user aborted transfer "and then the switch reboots .I have also tried to boot from usbflash0: and after a few minutes of transfer it stops and also reboots. I do a dirflash: and it does not exist nor does the sda9. I am completely lost and I cannot open a TAC case as the switch isnt covered under any support.

[Leo Laohoo]

WARNING:  Before proceeding any further, make sure to export the configuration file(s).

Ok, let's start with something simple.  

In ROMMON, can the switch "see" the USB flash drive (dir usbflash0:)?

If the answer is "yes", then do the following: 

1. Remove the USB thumb drive from the switch. 
2. Power down the switch.  Wait for the power supplies to stop "screaming" and then put the power back on. 
3. Once the switch boots into ROMMON, put the USB thumb drive in. 
4. Use this command: 

emergency-install usbflash0:IOS_filename.bin

Let us know how you go.

[nicholaswojciechowski]

I tried what you said and it didn't work. I have never had this much trouble getting out of ROMMON.

switch: dir
usage:
dir <path-to-directory>
supported filesystem(s):
flash: ro flash
bootflash: ro system partition
usbflash0: rw front-panel USB
usbflash1: rw back-panel USB
switch: dir usbflash0:

Size Attributes Name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
810207530 -rw- cat9k_iosxe.16.12.01.SPA.bin
711506601 -rw- cat9k_iosxe.16.09.05.SPA.bin
823865016 -rw- cat9k_iosxe.16.12.06.SPA.bin
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch: emergency-install usbflash0:cat9k_iosxe.16.12.01.SPA.bin
WARNING: The system partition (bootflash:) will be erased during the system recovery install process.
Are you sure you want to proceed? [y] y/n [n]: y
Starting system recovery (usbflash0:cat9k_iosxe.16.12.01.SPA.bin) ...

boot: attempting to boot from [sda9:cat9k-recovery.SSA.bin]
media drive sda9: not present.
ERROR: failed to boot from sda9:cat9k-recovery.SSA.bin (Not Found)

ERROR: Failed to perform system recovery from sda9:cat9k-recovery.SSA.bin
Contact Cisco Systems Technical Support.
switch:

[nicholaswojciechowski]

I have been through this over a dozen time, does anyone see something I missed in here?

switch: set
ABNORMAL_RESET_COUNT=0
BAUD=9600
BOARDID=24664
BSI=0
DC_COPY=yes
DEFAULT_GATEWAY=192.168.1.1
IP_ADDRESS=192.168.1.2
IP_SUBNET_MASK=255.255.255.0
LICENSE_BOOT_LEVEL=network-essentials+dna-essentials,all:C9300_24P;
MAC_ADDR=0C:D0:F8:53:C3:00
MANUAL_BOOT=yes
MODEL_NUM=C9300-24T
MODEL_REVISION_NUM=A0
MOTHERBOARD_ASSEMBLY_NUM=73-18270-03
MOTHERBOARD_REVISION_NUM=A0
MOTHERBOARD_SERIAL_NUM=******
RANDOM_NUM=577002102
RET_2_RCALTS=
RET_2_RTS=
ROMMON_AUTOBOOT_ATTEMPT=3
SWITCH_IGNORE_STARTUP_CFG=1
SWITCH_NUMBER=1
SYSTEM_SERIAL_NUM=*********
TEMPLATE=access
TFTP_BLKSIZE=1024
TFTP_SERVER=192.168.1.5
TFTP_TIMEOUT=100000
VERSION_ID=V01
switch:

[Leo Laohoo]

Try again: 

emergency-install usbflash0:cat9k_iosxe.16.12.06.SPA.bin

[nicholaswojciechowski]

switch: emergency-install usbflash0:cat9k_iosxe.16.12.06.SPA.bin
WARNING: The system partition (bootflash:) will be erased during the system recovery install process.
Are you sure you want to proceed? [y] y/n [n]: y
Starting system recovery (usbflash0:cat9k_iosxe.16.12.06.SPA.bin) ...
boot: attempting to boot from [sda9:cat9k-recovery.SSA.bin]
media drive sda9: not present.
ERROR: failed to boot from sda9:cat9k-recovery.SSA.bin (Not Found)

ERROR: Failed to perform system recovery from sda9:cat9k-recovery.SSA.bin
Contact Cisco Systems Technical Support.
switch:

[Leo Laohoo]

Ok, for my next trick, do this: 

1.  Copy the IOS-XE file, from the USB thumb drive, to the flash:

2.  While in ROMMON, force the switch to boot the file in Bundle Mode:  boot flash:cat9k_iosxe.16.12.06.SPA.bin

3.  Once the switch boots up, convert to Install Mode. 

4.  Reboot to make sure the switch boots into Install Mode.

[nicholaswojciechowski]

switch: dir usbflash0:

Size Attributes Name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
711506601 -rw- cat9k_iosxe.16.09.05.SPA.bin
4096 drw- System Volume Information
799150673 -rw- cat9k_iosxe_npe.16.12.02.SPA.bin
907450095 -rw- cat9k_iosxe.17.03.04.SPA.bin
823865016 -rw- cat9k_iosxe.16.12.06.SPA.bin
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin
usage:
copy <path-to-source-file> <path-to-destination-file>
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin flash:
Write access to "flash:" unsupported.
Unsupported destination device/filesystem.
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin flash
Unsupported destination device/filesystem.
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin bootflash:
Write access to "bootflash:" unsupported.
Unsupported destination device/filesystem.
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin bootflash
Unsupported destination device/filesystem.
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin boot flash:
usage:
copy <path-to-source-file> <path-to-destination-file>
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin boot flash
usage:
copy <path-to-source-file> <path-to-destination-file>
switch: copy usbflash0:cat9k_iosxe.16.12.06.SPA.bin boot flash

[Leo Laohoo]

Ok, try this:  

boot usbflash0:cat9k_iosxe.16.12.06.SPA.bin