Re: Cisco 9500 Boot Fail

saxenanitesh8522 · ‎04-08-2018

Hi,

we have a cisco 9500 it says BOOT FAIL ROMMON INVALID or TIMEOUT

Is it possible to recover this switch or its dead?

Leo Laohoo · ‎04-09-2018

Use the "emergency-install" command.

antwist · ‎08-14-2018

Not Sure of the situation for the original post, however, on the switch that I am working with (same error msg) the switch does not make it to ROMMON.

Leo Laohoo · ‎08-14-2018

Please attach a console cable and reboot the appliance. We want to see what the appliance is doing.

Jan Wachsmuth · ‎02-20-2023

I get the same error message, although it is not the same with every power on:

Console output:

Initializing Hardware...

Initializing Hardware......

->> after a few minutes the fans go out for a fraction of a second, then the same again:

Initializing Hardware...

Initializing Hardware......

***BOOT FAIL***: Rommon invalid or timeout.

-->> until power is turned off. There is no way to press any key to get into rommon.

After one or two further reboots/restarts I get:

Initializing Hardware...

Initializing Hardware......

->> after a few minutes the fans go out for a fraction of a second, then the same again:

Initializing Hardware...

Initializing Hardware......

System Bootstrap, Version 17.6.1r, RELEASE SOFTWARE (P)
Compiled 18-03-2021 12:00:00.00 by rel

Current ROMMON image : Golden Rommon Image

Last reset cause:SecurebootReset
C9500-48Y4C platform with 16777216 Kbytes of main memory

Preparing to autoboot. [Press Ctrl-C to interrupt] 0
boot: attempting to boot from [bootflash:packages.conf]
boot: reading file packages.conf
#
############### ... #########

*Feb 20 17:16:00.336: %IOSXEBOOT-4-BOOTLOADER_UPGRADE: (rp/0): Mon Feb 20 17:16:00 Universal 2023 BOOTING FROM GOLDEN, PRIMARY CORRUPTED

Waiting for remote chassis to join
##############################

In the last output the switch was connected to a virtual stack, but it makes no difference for the error to appear. We have disconnected ALL cables and got the same results. Finally the switch was able to join the virtual stack, but the "PRIMARY CORRUPTED" message is something that should not appear. So far the new switches are not yet productive, but a reboot should work in any case and even after a power failure the switch should come back properly.

I found a bug CSCvo21693 that looks to match exactly, but that one was filed in 2019 and I would expect that Cisco has fixed any hardware issues until now. Tomorrow I'll try to open a TAC case. An RMA might be the only option.

Leo Laohoo · ‎02-20-2023

RMA the switch. The Primary ROMMON image is corrupt and the secondary/backup ROMMON, aka Golden Capsule, is also corrupt.

Jan Wachsmuth · ‎02-21-2023

I had a Webex session with a TAC engineer today. The switch had shown the same symptoms as explained above during the Webex session. He has confirmed that it's a hardware fault and while we were working to fill the forms for the RMA, I've power cycled the switch again and this time the switch was able to boot with the golden rommon image, so the engineer was able to take some show outputs. Especially the following command was new to me and quite interesting:

show logging onboard switch 2 rp active uptime detail

This command shows all successful or failed booting processes of that switch (PowerOn, Reload vs. SecureBootFail).

Leo Laohoo · ‎02-21-2023

What is the age of the switch? If the switch is <12 months old, ask TAC to send replacement under "DoA" (Dead on Arrival).

DoA replacements are taken directly off the production line -- They are brand new.

RMA replacements are taken off the refurbishment line.

Jan Wachsmuth · ‎02-26-2023

The 9500 switches are quite new. Due to delivery problems for the other switches (9300 series) in the project they were produced in Aug 2022. We did a normal RMA, because quick delivery was important. I remember another (different) case where a faulty item was classified as DoA and it took about a week to get the replacement, because it has to be processed via delivery channel and therefore via distributor.

The replacement C9500-48Y4C switch that was delivered directly from Cisco via RMA came from Amsterdam (I'm located in Germany). It looked brand new and has a production date of Jan 2023, so it is even newer than the old one. Of course, it may be different in another case.

So at the end I'm happy that my problem is solved and the new switch is working properly. Thank you to everybody for your input!