Re: Number of ISR4K devices not recognizing upgraded flash under ROMMO

pietro manicioto · ‎12-16-2022

Hello All,

I am facing what seems like a unique problem in which I have a number of ISR4K devices (4331's and 4351's) that when we upgrade the flash, they will not boot properly from flash.

Going through the process and booting to USB, when in IOSXE you can see the Flash and write the image to it and confirm it wrote with no issues. You set the boot command to boot from the new flash and it fails and goes into ROMMON. When in ROMMON you try to boot to flash or DIR flash and it just tells you the "media drive is unreadable"

I have tried multiple versions of IOSXE with no luck. I have tried format/FSCK the flash and rewrite with same issues. These devices cannot be downgraded to older ROMMON, so they are on 17.6.1. We have swapped flash multiple times as well to confirm this is NOT flash related. We have moved known working flash from another 4300 upgrade to a problematic chassis and replicated the problem and vice versa fine.

Has anyone run into this before that could lend some insight? TAC wants me to RMA all the routers, which I am hesitant to do at this very moment as I feel this is software related in some fashion.

Leo Laohoo · ‎12-16-2022

Console into the router and reboot.
Post the entire boot-up process so we can see what is going on.

pietro manicioto · ‎12-16-2022

Initializing Hardware ...

Checking for PCIe device presence...done
System integrity status: 0x610
Rom image verified correctly

Current image running: Boot ROM0

Last reset cause: PowerOn
ISR4331/K9 platform with 8388608 Kbytes of main memory

........
no valid BOOT image found
Final autoboot attempt from default boot device...
media drive bootflash: not readable

........
autoboot: boot failed, restarting...

Resetting .......

Initializing Hardware ...

Checking for PCIe device presence...done
System integrity status: 0x610
Rom image verified correctly

Current image running: Boot ROM0

Last reset cause: LocalSoft
ISR4331/K9 platform with 8388608 Kbytes of main memory

rommon 1 > dir flash:

media drive flash: not readable
rommon 2 >

Leo Laohoo · ‎12-16-2022

@pietro manicioto wrote:
no valid BOOT image found
Final autoboot attempt from default boot device...
media drive bootflash: not readable

Post the complete output to the following ROMMON commands:

set
dir bootflash:

pietro manicioto · ‎12-16-2022

Here is the output when in IOSXE on USB0:

Router#dirRouter#show run | i boot
boot-start-marker
boot system flash isr4300-universalk9.16.03.10.SPA.bin
boot-end-marker
diagnostic bootup level minimal

____________________________________________________________________________
Router# dir flash:
Directory of bootflash:/

245761 drwx 24576 Dec 16 2022 14:13:11 +00:00 tracelogs
581633 drwx 4096 Dec 16 2022 14:12:59 +00:00 license_evlog
16 -rw- 30 Dec 16 2022 14:12:52 +00:00 throughput_monitor_params
13 -rw- 134899 Dec 16 2022 14:12:42 +00:00 memleak.tcl
606209 drwx 4096 Dec 16 2022 14:12:41 +00:00 .prst_sync
12 -rw- 364 Dec 16 2022 14:12:10 +00:00 mode_event_log
368641 drwx 4096 Dec 16 2022 14:11:39 +00:00 .installer
892929 drwx 4096 Dec 16 2022 14:11:03 +00:00 sysboot
638977 drwx 4096 Dec 15 2022 22:29:35 +00:00 core
565249 drwx 4096 Dec 15 2022 22:29:31 +00:00 onep
344065 drwx 4096 Dec 15 2022 22:29:09 +00:00 virtual-instance
647169 drwx 4096 Dec 15 2022 22:29:09 +00:00 .dbpersist
15 -rw- 1923 Dec 15 2022 22:29:04 +00:00 trustidrootx3_ca_092024.ca
14 -rw- 20109 Dec 15 2022 22:29:04 +00:00 ios_core.p7b
311297 drwx 4096 Dec 15 2022 22:28:13 +00:00 .rollback_timer
11 -rw- 459981294 Dec 15 2022 22:16:29 +00:00 isr4300-universalk9.16.03.10.SPA.bin

________________________________________________________________________________

Router#show boot
BOOT variable = isr4300-universalk9.16.03.10.SPA.bin,1;
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x2102

Standby not ready to show bootvar

________________________________________________________________________________

Here is the output in ROMMON:

rommon 1 > dir bootflash:

media drive bootflash: not readable

________________________________________________________________________________

rommon 2 > set
PS1=rommon ! >
THRPUT=
RET_2_RTS=
TFTP_BLKSIZE=512
MCP_STARTUP_TRACEFLAGS=00000000:00000000
TFTP_RETRY_COUNT=20
LICENSE_BOOT_LEVEL=
DEVICE_MANAGED_MODE=autonomous
BOOT=isr4300-universalk9.16.03.10.SPA.bin,1;
BSI=0
RANDOM_NUM=575514630
RET_2_RCALTS=1671200280

Leo Laohoo · ‎12-16-2022

@pietro manicioto wrote:

boot system flash isr4300-universalk9.16.03.10.SPA.bin

That is the wrong boot-variable string.

Try this:

conf t
 NO boot system flash isr4300-universalk9.16.03.10.SPA.bin
 boot system flash bootflash:isr4300-universalk9.16.03.10.SPA.bin
end
wr

And then reboot the router.

NOTE:

Please refer to THIS.
This "feature" is brought to you by CSCvg37458.
Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode)

pietro manicioto · ‎12-16-2022

I believe I tried this already.

Bootflash and flash are synonmous as I understand it. Bootflash is an alias for flash in ROMMON.

That being said, that article you linked is for older IOS or ROMMON version that is not applicable here as I read it.

I tried this again to show this does not work

_______________________________________________________________________________

Router(config)#no boot system flash isr4300-universalk9.16.03.10.SPA.bin
Router(config)#$m flash bootflash:isr4300-universalk9.16.03.10.SPA.bin
Router(config)#
Router(config)#
Router(config)#
Router(config)#do show run | i boot
boot-start-marker
boot system flash bootflash:isr4300-universalk9.16.03.10.SPA.bin
boot-end-marker
diagnostic bootup level minimal

_________________________________________________________________________________

Router#reload
Proceed with reload? [confirm]

*Dec 16 14:37:32.734: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.Dec 16 14:37:45.761: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload chassis code

Initializing Hardware ...

Checking for PCIe device presence...done
System integrity status: 0x610
Rom image verified correctly

Current image running: Boot ROM0

Last reset cause: LocalSoft
ISR4331/K9 platform with 8388608 Kbytes of main memory

....
rommon 1 >
rommon 1 > boot bootflash:
media drive bootflash: not readable
rommon 2 >

Leo Laohoo · ‎12-16-2022

@pietro manicioto wrote:
System Bootstrap, Version 17.6.1, RELEASE SOFTWARE

Did this router came with 17.6.X (or later) or did someone upgrade the router to 17.6.X (and later)?

Please provide the first seven characters of the serial number of this router?

pietro manicioto · ‎12-16-2022

I believe it came with 17.6.1. All newer IOS versions have any bootstrap updates cooked in. This device cannot be downgraded though due to build date. I tried this already in my diagnosis

FLM2539

Leo Laohoo · ‎12-16-2022

Try 17.6.4.

pietro manicioto · ‎12-16-2022

17.6.4 iosxe upgrade was the first image we tried to put on the devices as that is our Gold Standard currently. The bootstrap does not update, I guess because the system never boots to the OS from flash.

Leo Laohoo · ‎12-16-2022

Ok, so the router is going to be RMA-ed, right? I guess the worst that we can do is "fix it".

The "latest" ROMMON for the 4331 that can be downloaded is 16.12(2r).

So, boot the router into IOS and use the ROMMON Upgrade procedure to attempt to downgrade the ROMMON to 16.12(2r). And then reboot.

Log the console to record what happens and then attach it.

I am going to sleep (2am already).

pietro manicioto · ‎12-16-2022

As mentioned prior, this device will not allow me to downgrade ROMMON. This is not allowed on devices of a certain build number. I have tried this already.

Note

For devices with manufacturing date equal or later than 2535, the minimum supported ROMMON version is 17.6.1. These devices cannot downgrade to older ROMMON versions.

pietro manicioto · ‎12-16-2022

I just got word that Cisco sent us an RMA for one of the affected devices and same issue. It appears there is something with Bootloader 17.6.1 that is causing this issue, because the ones we did successfully upgrade had 16.12 on them.

Shot in the dark, but I asked TAC if there is someway Cisco can unlock the Bootloader downgrade limitation as this is looking more and more like a bug, atleast starting from 17.6.1 as the base.

pietro manicioto · ‎12-16-2022

I confirmed my theories and it looks like atleast bootloader ver 17.6.1 is problematic with reading from flash.

We were able to find an older device prior to the 2532 I think was the hardware cutoff and rolled it back to 16.12 and ROMMON can read flash properly and allow upgrade.

So now I need Cisco to either unlock older hardware to be rolled back to this version for Flash upgrades, or unpack a newer BootLoader from newer IOS images than 17.6.1 to see if those work.

Number of ISR4K devices not recognizing upgraded flash under ROMMON