Think I've bricked an ISR 4331

[Gary Keaney]

Ok so the order of events. Had a task to re-configure a 4331, it had an existing config and an older ios. As is my usual practice I backed up the config to flash just in case, with a copy command, giving no file extension.

I then fttp'd the latest recommended image over (isr4300-universalk9.16.09.03.SPA.bin) and did a verify on it. Deleted the old image and did a write erase.

Once I reloaded the router it entered a loop. It appears that I should have done a ROMMON upgrade first. I cannot get into rommon due to password recovery being enabled. Ordinarily, I would just answer y to the question "Do you want to reset the router to the factory default configuration and proceed [y/n] ?"

But for some reason, during the boot process I think it is picking up the config saved on flash and re-enabling password recovery. I thought it only looked for .cfg files.

Here's a condensed excerpt of the boot process;

Router#wr er
Erase of nvram: complete
Router#reload

System configuration has been modified. Save? [yes/no]: no
Proceed with reload? [confirm]

Initializing Hardware ...

System integrity status: 00000610
Rom image verified correctly

System Bootstrap, Version 15.4(3r)S5, RELEASE SOFTWARE
Copyright (c) 1994-2015  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: LocalSoft

........

no valid BOOT image found
Final autoboot attempt from default boot device...
File size is 0x00000180

Located old_config_saved_to_flash_with_password_rec_dis_command_in_it

Located isr4300-universalk9.16.09.03.SPA.bin 
Image size 583810543 inode num 16, bks cnt 142532 blk size 8*512
#################################################################

Package header rev 3 structure detected
Calculating SHA-1 hash...done
validate_package: SHA-1 hash:
        calculated fc8707af:7758180b:25f16323:3830ea98:3c16accc
        expected   fc8707af:7758180b:25f16323:3830ea98:3c16accc

RSA Signed RELEASE Image Signature Verification Successful.

Unknown image structure
boot: cannot determine first file name on device "bootflash:/"
autoboot: boot failed, restarting...

Initializing Hardware ...

The above will loop round indefinitely. If I enter the break sequence;

PASSWORD RECOVERY IS DISABLED.
Do you want to reset the router to the factory default
configuration and proceed [y/n] ?y

Router clearing configuration.  Please wait for ROMMON prompt...

Initializing Hardware ...

It does not drop to ROMMON but instead restarts and continues to loop.

I've opened her up to see if there is a removable flash, not in this model. Tried answering n and continued to hit the break sequence after the image uncompresses. I'm out of ideas.

Help!

[balaji.bandi]

Power Off the device and connect the Console cable use CTRL+BREAK to get in ROMMON

Other side Copy correct Image to USB Flash

From ROMMON > boot usb0:image.bin ( replace USB and image name)

once booted copy the image to flash, change the boot system 

let us know how it goes ?

[Leo Laohoo]

---

@Gary Keaney wrote:

System Bootstrap, Version 15.4(3r)S5, RELEASE SOFTWARE

---

I know the reason why:  The ROMMON has not yet been upgraded.  

Did anyone read the Release Notes?  It is stated in the 16.9.3 Release Notes what is the minimum recommended ROMMON version (LINK).

It will go into a boot loop unless you force the router to boot the old version.  

One final thing, this one is a "gotcha!" if this router is going to do a NAT:  Has anyone read the Release Notes or not?   Look at the Open Caveats section.  Pay close attention to CSCvo36188.

[Gary Keaney]

Yep, I realise this now (stated as much in the 3rd paragraph) and this problem is further compounded by the fact I cannot get into ROMMON. Crtl-Break was issued in the excerpt I showed above, it either immediately reboots or still attempts to load the IOS - after picking up a config backup from flash that has password recovery disabled in it.

Unless there are some pins I can jumper on the board to disable flash or cause it to drop to ROMMON I think I'm pretty screwed.

[Leo Laohoo]

Power down the router and power it back up.
Bash the Ctrl+C continuously. You'll get there.
Otherwise, organize an RMA.

[Gary Keaney]

Ctrl+C does nothing. Crtl+pause\break brings me to this;

PASSWORD RECOVERY IS DISABLED.
Do you want to reset the router to the factory default
configuration and proceed [y/n] ?y

Router clearing configuration.  Please wait for ROMMON prompt...

Pressing y does one of two things. Either immediately resets:

Initializing Hardware ...

Or tries to load the ios:

PASSWORD RECOVERY IS DISABLED.
Do you want to reset the router to the factory default
configuration and proceed [y/n] ?y

Router clearing configuration.  Please wait for ROMMON prompt...

no valid BOOT image found
Final autoboot attempt from default boot device...
File size is 0x00000180
... [omitted for brevity]
Located isr4300-universalk9.16.09.03.SPA.bin 
Image size 583810543 inode num 16, bks cnt 142532 blk size 8*512
################################################################

Both of which continue the boot loop. Calling TAC now, will let you know if there is even a fix for this. Spoke to our resident CCIE and he's admitted to never needing to upgrade ROMMON before. Seems strange that a fairly innocuous series of events would exclude you from getting into ROMMON to fix the issue.

[Leo Laohoo]

---

@Gary Keaney wrote:

PASSWORD RECOVERY IS DISABLED.

---

And this is the ONLY reason why you are not able to get into ROMMON. 

I know that disabling password recovery is one of their "best practices" but it is also the dumbest recommendation.  Right now, your case is the best example of why I would not recommend the practice of disabling password recovery.  

[Gary Keaney]

Totally agree. Not something I ever do on my own projects. I work in a managed service environment and lead engineers often stray from our company policy on a client by client basis. Very frustrating.

Could be a £5k paperweight now. 🤣

[Leo Laohoo]

(Maybe) I was a bit "harsh" with my previous response.  Let me put a little bit of clarification down: 

In the past, before the 900, 1000 and 4K series routers came out, there was a way to perform password recovery even though password recovery was disabled in the config.  And if one follows the instructions properly, it was relatively very easy to achieve.  

Word of this "feature" must've hit Cisco because they've actually recognize how useless the "no service password-recovery" command that a bug was raised:  CSCvo02389

This bug recognizes that for "no password recovery" to work, the Ctrl+C must be disabled the moment the appliance boots up.   Genius ain't it?  (Not!)

So now, in your case, the only way to "gain control" back of the router is to RMA the appliance.  Cisco logistics should see a significant uptick in appliance being RMAed because some "genius" to configure "no password recovery" and locked themselves out.  

Like I said in my previous post, "no password recovery" is in one of Cisco's "recommended" best practice guide about how to harden the network (LINK).  I will never recommend anyone use "no service password-recovery" in their config.  It'll bite you and will bite hard.  

Hope this helps.

NOTE:  

In my personal opinion, the answer to this dilemma is a patch only published by TAC.  Why?  There are legitimate users, like defense- &/or intelligence related services/customers, where they do have the need for the appliance to be locked down.  With a patch, it stops anyone from enabling "no service password-recovery" and only to find themselves locked out under the rain, hail or snow.  

[balaji.bandi]

Personally i would still raise an TAC case ask for RMA if you have contract or provide alternative option to recover.

since this feature was there so as user point of view enabled. Yes  other side effect we do know  that it was not recoverable. it does not mean that device get wasted due to brick (with the command was allowed to do).

Cisco developped some feature which not intented to the general deployment, that should be removed from general release and provide only specific to the customer base or client base requirement.

This is just my personal views when we look at the protecting money as Cisco Valued customer.

[Leo Laohoo · Accepted solution]

Ok, I might have a solution for you and this all hinges on whether or not the eMMC (flash) can be removed or not. 

Go HERE and look at item #2.  See if this is removable or not. 

If it is, then here's the process: 

1.  Power down the appliance.  Make sure the power cable is pulled and wear a anti-static strap.

2.  Remove the flash. 

3.  Power up the appliance and see if you can go into ROMMON. 

4.  In ROMMON, issue the command "confreg 0x2100". 

5.  Power down the appliance and remove the power cord. 

6.  Insert the flash. 

7.  Power up the appliance.  

8.  The router should now boot into ROMMON. 

9.  Issue the command "dir bootflash:" (or "dir flash:") and see if you can find the offending IOS. 

10.  Delete the IOS (delete bootflash:filename.bin).  

11.  Issue the command "confreg 0x2102" and reboot the router.  

ALTERNATIVELY, after #8 you can insert a supported USB thumb drive with the old IOS and instruct the router to boot the IOS found in the USB thumb drive.  Before you do that, issue the command "confreg 0x2102" and then followed by "boot usb0:IOS_filename.bin".

Let us know how this works.

[Leo Laohoo]

Did this work? Please let us know the result.

[Gary Keaney · Accepted solution]

Thank you Leo for your two detailed responses. Thanks to your pointers in the post I've marked as the solution I have successfully 'unbricked' the device.

The 4331 has an integrated flash but it does have the port for the flash module shown in the field upgrade guide. So I located a 4431 that we had in stock and this one has the module as it's primary flash. Oddly on the 4431 there is also a CF flash slot that you would need to remove the fans to insert anything; but irrelevant here.

So I copied the older 4331 ios to the flash on the 4431, powered down, removed the module and inserted it into the bricked 4331. The hope here is that the boot process will locate the ios and be able to boot it with the version of ROMMON it has.

From further investigation, I have confirmed that when adding this flash module to the 4331 it becomes the only flash in use (i.e. bootflash:) you do not get an additional file system such as flash1:

During boot it does indeed locate the ios, validates it but then I ended up in a new boot loop;

Initializing Hardware ...

System integrity status: 00000610
Rom image verified correctly


System Bootstrap, Version 15.4(3r)S5, RELEASE SOFTWARE
Copyright (c) 1994-2015  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: LocalSoft
Cisco ISR4331/K9 platform with 4194304 Kbytes of main memory


........

no valid BOOT image found
Final autoboot attempt from default boot device...
File size is 0x190a6f44
Located isr4300-universalk9.03.13.04.S.154-3.S4-ext.SPA.bin 
Image size 420114244 inode num 12, bks cnt 102567 blk size 8*512
################################################################
Boot image size = 420114244 (0x190a6f44) bytes

Package header rev 1 structure detected
Calculating SHA-1 hash...done
validate_package: SHA-1 hash:
        calculated 83acd4f8:dc03c892:f243621c:06872286:6c9f0cf5
        expected   83acd4f8:dc03c892:f243621c:06872286:6c9f0cf5

RSA Signed RELEASE Image Signature Verification Successful.
Package Load Test Latency : 7647 msec 
Image validated
%IOSXEBOOT-4-BOOT_SRC: (rp/0): mounting /boot/super.iso to /tmp/sw/isos
%IOSXEBOOT-4-PLATFORM_RECOVERY: (rp/0): Started platform recovery...
%IOSXEBOOT-4-PLATFORM_RECOVERY: (rp/0): Finished platform recovery...


Initializing Hardware ...

I left it to reload a fair few times in the hope it would self rectify. Similar to the 12 reboots you get before autoboot will attempt to load another ios other than those specified in specific statements.

I also tried the break sequence answering yes each time but it continued to just reload and loop. In the end and I cannot understand why but it was answering no that got it to boot.

PASSWORD RECOVERY IS DISABLED.
Do you want to reset the router to the factory default
configuration and proceed [y/n] ?y

Router clearing configuration.  Please wait for ROMMON prompt...

no valid BOOT image found
Final autoboot attempt from default boot device...
File size is 0x190a6f44
Located isr4300-universalk9.03.13.04.S.154-3.S4-ext.SPA.bin 
Image size 420114244 inode num 12, bks cnt 102567 blk size 8*512
#################################################################
Boot image size = 420114244 (0x190a6f44) bytes

Package header rev 1 structure detected
Calculating SHA-1 hash...done
validate_package: SHA-1 hash:
        calculated 83acd4f8:dc03c892:f243621c:06872286:6c9f0cf5
        expected   83acd4f8:dc03c892:f243621c:06872286:6c9f0cf5

RSA Signed RELEASE Image Signature Verification Successful.
Package Load Test Latency : 7638 msec 
Image validated
%IOSXEBOOT-4-BOOT_SRC: (rp/0): mounting /boot/super.iso to /tmp/sw/isos
%IOSXEBOOT-4-PLATFORM_RECOVERY: (rp/0): Started platform recovery...
%IOSXEBOOT-4-PLATFORM_RECOVERY: (rp/0): Finished platform recovery...


Initializing Hardware ...

System integrity status: 00000610
Rom image verified correctly


System Bootstrap, Version 15.4(3r)S5, RELEASE SOFTWARE
Copyright (c) 1994-2015  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: LocalSoft
Cisco ISR4331/K9 platform with 4194304 Kbytes of main memory


.


PASSWORD RECOVERY IS DISABLED.
Do you want to reset the router to the factory default
configuration and proceed [y/n] ?n

Router continuing with existing configuration...

no valid BOOT image found
Final autoboot attempt from default boot device...
File size is 0x190a6f44
Located isr4300-universalk9.03.13.04.S.154-3.S4-ext.SPA.bin 
Image size 420114244 inode num 12, bks cnt 102567 blk size 8*512
################################################################
Boot image size = 420114244 (0x190a6f44) bytes

Package header rev 1 structure detected
Calculating SHA-1 hash...done
validate_package: SHA-1 hash:
        calculated 83acd4f8:dc03c892:f243621c:06872286:6c9f0cf5
        expected   83acd4f8:dc03c892:f243621c:06872286:6c9f0cf5

RSA Signed RELEASE Image Signature Verification Successful.
Package Load Test Latency : 7641 msec 
Image validated
%IOSXEBOOT-4-BOOT_SRC: (rp/0): mounting /boot/super.iso to /tmp/sw/isos

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S4, RELEASE SOFTWARE (fc3)
! YAY :)

I also had a small issue when I inserted the flash back into the 4431. Rommon confreg was reported as corrupt but it booted into rommon and I was able to reset it.

I also took the time to rommon upgrade half a dozen 4K routers to 16.9 to avoid another engineer falling into this trap and I also patched against Thrangycat 😼

Actually on that, I came across another issue. On a a fresh out the box 4K router I copied the patch bin file to flash and reloaded, did the patch but from then on it constantly located the file at boot, checked to upgrade, realised it did not need to and rebooted again. Had this device also had password recovery disabled then I would also be in a bad situation. Even though the guide advises to do this via usb or flash, I would always advise via usb now.

no valid BOOT image found
Final autoboot attempt from default boot device...
Located tracelogs.776
#

Zero Image size
Located throughput_monitor_params
#

Unsupported package header version (0)
Located isr4300_cpld_update_v1.1_SPA.bin
########################################

Package header rev 1 structure detected
IsoSize = 0
Calculating SHA-1 hash...Validate package: SHA-1 hash:
        calculated EB5A86C4:04A30F06:7601152E:A78A1691:10F2DB8E
        expected   EB5A86C4:04A30F06:7601152E:A78A1691:10F2DB8E
RSA Signed RELEASE Image Signature Verification Successful.
Image validated


    Cisco ISR4300 CPLD Programming Utility

  ******************************************
  **                                      **
  **     DO NOT TURN OFF THE POWER OR     **
  **   RESET THE BOX DURING THE UPGRADE   **
  **                                      **
  ******************************************

Detected platform: ISR4331
CPLD version: 19040541
The CPLD is locked.

An update isn't required.


Resetting the platform ... 


Initializing Hardware ...

So thank you Leo, very much :)

[Gary Keaney]

Also worth a mention is where has the delete command gone in rommon?

rommon 8 > dir bootflash:

0   0         drw-     lost+found
0   0         drw-     .prst_sync
0   0         drw-     .installer
0   0         drw-     core
0   0         drw-     .rollback_timer
0   0         -rw-     tracelogs.776
0   0         drw-     tracelogs
0   30        -rw-     throughput_monitor_params
0   4015456   -rw-     isr4300_cpld_update_v1.1_SPA.bin
0   5010380   -rw-     isr4200_4300_rommon_169_1r_SPA.pkg
0   583810543 -rw-     isr4300-universalk9.16.09.03.SPA.bin
rommon 9 > delete bootflash:isr4300_cpld_update_v1.1_SPA.bin
monitor: command "delete" not found
rommon 10 > ? 
alias               set and display aliases command
boot                boot up an external process
confreg             configuration register utility
dev                 list the device table
dir                 list files in file system
help                monitor builtin command help
history             monitor command history
meminfo             main memory information
repeat              repeat a monitor command
reset               system reset
set                 display the monitor variables
showmon             display currently selected ROM monitor
sync                write monitor environment to NVRAM
token               display board's unique token identifier
unalias             unset an alias
unset               unset a monitor variable

[Leo Laohoo]

---

@Gary Keaney wrote:

In the end and I cannot understand why but it was answering no that got it to boot.

---

That's weird.  

Also, I noticed in ROMMON, the "format" command is also missing.  

Thanks, Gary.  Happy to see that it's finally working.