C9500 Stuck in ROMMON Mode

[tltl4866]

Hello all,

One of our C9500 in the stack went down and after hooking up the console, we noticed that the C9500 is in ROMMON mode (we are currently running on 17.03.04)

I have tried booting directly from bootflash and usb, but neither of it seems to work. I have also tried to use the command emergency-install, and while it shows promises at beginning, upon rebooting, the switch went back into ROMMON mode. see below for the messages:

Initiating Emergency Installation of bundle usbflash0:cat9k_iosxe.17.03.04.SPA.bin


Reading bundle usbflash0:cat9k_iosxe.17.03.04.SPA.bin...

Validating bundle usbflash0:cat9k_iosxe.17.03.04.SPA.bin...
Installing bundle usbflash0:cat9k_iosxe.17.03.04.SPA.bin....
Verifying bundle usbflash0:cat9k_iosxe.17.03.04.SPA.bin...
Package cat9k-cc_srdriver.17.03.04.SPA.pkg /temp//stage/cat9k-cc_srdriver.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-espbase.17.03.04.SPA.pkg /temp//stage/cat9k-espbase.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-guestshell.17.03.04.SPA.pkg /temp//stage/cat9k-guestshell.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-lni.17.03.04.SPA.pkg /temp//stage/cat9k-lni.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-rpbase.17.03.04.SPA.pkg /temp//stage/cat9k-rpbase.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-sipbase.17.03.04.SPA.pkg /temp//stage/cat9k-sipbase.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-sipspa.17.03.04.SPA.pkg /temp//stage/cat9k-sipspa.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-srdriver.17.03.04.SPA.pkg /temp//stage/cat9k-srdriver.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-webui.17.03.04.SPA.pkg /temp//stage/cat9k-webui.17.03.04.SPA.pkg is Digitally Signed
Package cat9k-wlc.17.03.04.SPA.pkg /temp//stage/cat9k-wlc.17.03.04.SPA.pkg is Digitally Signed
Package /cat9k-rpboot.17.03.04.SPA.pkg /temp//rpboot/cat9k-rpboot.17.03.04.SPA.pkg is Digitally Signed
Preparing flash....
Flash filesystem unmounted successfully /dev/sdb3
Syncing device....
Emergency Install successful... Rebooting
Will reboot now

Initializing Hardware...

SNP: failed to initialize MAC address (not found/zero)
Please set a value for MAC_ADDR and restart the device before proceeding

Both copies of Nvram are corrupted.

Warning: monitor nvram area is corrupt ... using default values

System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)
Compiled Wed 10/23/2019 16:35:17.50 by rel

Current ROMMON image : Primary
Last reset cause     : CpuReset
C9500-40X platform with 16777216 Kbytes of main memory

Setting MODEL_NUM [C9500-40X]
Setting SYSTEM_SERIAL_NUM []
Setting MOTHERBOARD_SERIAL_NUM []
Setting VERSION_ID [V01]
Setting MOTHERBOARD_REVISION_NUM []
Setting MOTHERBOARD_ASSEMBLY_NUM []
Setting MODEL_REVISION_NUM []
WARNING: Bootable URL's in BOOT variable not found or exhausted.
         Please check the ROMMON configuration or boot command usage.
switch:

I also noticed that it says both nvram are corrupted. Does that mean the switch itself is bad? Or is there something else I can try?

[Leo Laohoo]

---

@tltl4866 wrote:

Both copies of Nvram are corrupted.

---

Wow.  I have never seen this before.  

If the switch is in ROMMON, post the complete output to the command "dir" and "set".

[tltl4866]

Yeah. Right now I'm leaning towards the switch being bricked because this happened out of nowhere.

output for dir:

switch: dir
usage:
        dir <path-to-directory>
supported filesystem(s):
         flash: ro flash
     bootflash: ro system partition
     usbflash0: rw front-panel USB
     usbflash1: rw back-panel USB

output for set (removed serial #):

switch: set
BOARDID=20610
MAC_ADDR=68:7D:B4:A8:CD:00
MODEL_NUM=C9500-40X
MODEL_REVISION_NUM=H0
MOTHERBOARD_ASSEMBLY_NUM=73-18140-03
MOTHERBOARD_REVISION_NUM=D0
MOTHERBOARD_SERIAL_NUM=[REMOVED]
PS1=switch:
ROMMON_AUTOBOOT_ATTEMPT=0
SWITCH_NUMBER=1
SYSTEM_SERIAL_NUM=[REMOVED]
VERSION_ID=V01

[Leo Laohoo]

Can I see the complete output to the command "dir flash:"?

[tltl4866]

My bad!

Here's both flash and bootflash

switch: dir flash:

Size       Attributes Name
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2097152      -rw-     nvram_config
2097152      -rw-     nvram_config_bkup
18289676     -rw-     cat9k-cc_srdriver.17.03.04.SPA.pkg
104526856    -rw-     cat9k-espbase.17.03.04.SPA.pkg
2262024      -rw-     cat9k-guestshell.17.03.04.SPA.pkg
5124         -rw-     cat9k-lni.17.03.04.SPA.pkg
599221252    -rw-     cat9k-rpbase.17.03.04.SPA.pkg
34874376     -rw-     cat9k-sipbase.17.03.04.SPA.pkg
57082884     -rw-     cat9k-sipspa.17.03.04.SPA.pkg
29717512     -rw-     cat9k-srdriver.17.03.04.SPA.pkg
14431236     -rw-     cat9k-webui.17.03.04.SPA.pkg
9220         -rw-     cat9k-wlc.17.03.04.SPA.pkg
6802         -rw-     packages.conf
47407779     -rw-     cat9k-rpboot.17.03.04.SPA.pkg
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch: dir bootflash:

Size       Attributes Name
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2097152      -rw-     nvram_config
2097152      -rw-     nvram_config_bkup
18289676     -rw-     cat9k-cc_srdriver.17.03.04.SPA.pkg
104526856    -rw-     cat9k-espbase.17.03.04.SPA.pkg
2262024      -rw-     cat9k-guestshell.17.03.04.SPA.pkg
5124         -rw-     cat9k-lni.17.03.04.SPA.pkg
599221252    -rw-     cat9k-rpbase.17.03.04.SPA.pkg
34874376     -rw-     cat9k-sipbase.17.03.04.SPA.pkg
57082884     -rw-     cat9k-sipspa.17.03.04.SPA.pkg
29717512     -rw-     cat9k-srdriver.17.03.04.SPA.pkg
14431236     -rw-     cat9k-webui.17.03.04.SPA.pkg
9220         -rw-     cat9k-wlc.17.03.04.SPA.pkg
6802         -rw-     packages.conf
47407779     -rw-     cat9k-rpboot.17.03.04.SPA.pkg
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[Leo Laohoo]

Do you have a copy of versions 16.X.X?

[tltl4866]

Yes I do. Should I grab that and do another emergency-install?

[Leo Laohoo]

No, don't. 

Boot the 16.X.X OS via the USB and in Bundle Mode.  

IF the switch boots into Bundle Mode, convert to Install Mode.  
Reboot again. 
Let us see what the switch responds with.

I think this is CSCvq47878.

[marce1000]

          - Check the value of the config-register , and or set to standard again if it was changed to something else.

 M.

[tltl4866]

Hi,

Thank you for your reply. I'm not exactly sure how to set the config-register on this one. ROMMON says "confreg" is not found.

The closest command according to "help" is set. However, I'm not sure which variable I need to input to set the config-register.

[marce1000]

 - Actually , you may be suffering from this bug (or similar) : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr86782  which apparently is not viewable for the general public. Contact TAC , discuss and work with tac for implementing the solution.

 M.

[65471203]

Hi I also met this issue. I even cannot boot from usbflash0:

switch: boot usbflash0:cat9k_iosxe.16.06.01.SPA.bin
boot: attempting to boot from [usbflash0:cat9k_iosxe.16.06.01.SPA.bin]
boot: reading file cat9k_iosxe.16.06.01.SPA.bin

May 9 07:25:55.368 RP0/0: %PMAN-3-PROCHOLDDOWN: The process sif_mgr has been helddown (rc 139)
May 9 07:25:55.396 RP0/0: %PMAN-0-PROCFAILCRIT: A critical process sif_mgr has failed (rc 139)
May 9 07:25:55.418 RP0/0: %PMAN-3-RELOAD_RP: Reloading: Switch will be reloaded

Chassis 0 reloading, reason - Non participant detected
May 9 07:26:02.580 FP0/0: %PMAN-3-PROCHOLDDOWN: The process fed has been helddown (rc 139)
May 9 07:26:04.593 FP0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
May 9 07:26May 9 07:26:11.855 RP0/0: %PMAN-3-PROCESS_NOTIFICATION: Deleting old files in /crashinfo ... System report /crashinfo/system-report_local_20230509-072610-Universal.tar.gz (size: 11776 KB) generated

Initializing Hardware...

SNP: failed to initialize MAC address (not found/zero)
Please set a value for MAC_ADDR and restart the device before proceeding

Both copies of Nvram are corrupted.

Warning: monitor nvram area is corrupt ... using default values

System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)
Compiled Wed 10/23/2019 16:35:17.50 by rel

Current ROMMON image : Primary
Last reset cause : CpuReset
C9500-40X platform with 16777216 Kbytes of main memory

Guys, do we have any idea?

[Leo Laohoo]

---

@65471203 wrote:
Both copies of Nvram are corrupted.

---

Contact Cisco TAC and organize for an RMA.