ASA5516 crashing, reimage not possible bec password rec is disabled.

[edsge teenstra]

Hi all 

We have a quite expensive ASA5516-FPWR-K9 crashing right after the boot process.

After some troubleshooting I get the impression the issue can be fixed by reinstalling the image.

Only since the password recovery is disabled we are not able to go to the ROMMON.

With many routers the password recovery disabling can be solved by a key combination at a "secret"  moment while the router is booting.

But I tried an enormous amount of different key combinations on all possible moments that the switch is booting but nothing helps. 

Any suggestions?

Since if I would not be able to fix this rebooting ASA, this would be totally waste of a good product and not helping the e-waste problem  

See attached the log of the item with the issue. 

Thanks in advance!! 

[balaji.bandi]

what option you have tried to get on to ROMMON :

try below :

https://community.cisco.com/t5/security-documents/asa-password-recovery/ta-p/3126046

[edsge teenstra]

Now when it is booting there is no text saying you can press ESC to go to ROMMON. < I checked and other ASA do have this when just starting to boot. 

And we have tried dozens of key combinations (ESC, Break, Break+ESC, Ctrl+Break, Ctrl+C, Ctrl+Z etc etc) to be able to go to ROMMON but it just does not want to go to ROMMON, so we cannot install a new image, and now this ASA5516-FPWR-K9 is eating dust.

Any idea's ? 

Spamming ESC all the time has no impact.

I have tried with a working tested regular rj45 console cable, and I tried the mini USB console cable and same result. 

After the individual packages are booted the text from spamming the ESC button it sent to the terminal screen so the ASCII protocol data is sent to the device. (see below in red the ESC and Break button being pressed)

See log below : 

Rom image verified correctly

Cisco Systems ROMMON, Version 1.1.14, RELEASE SOFTWARE
Copyright (c) 1994-2018 by Cisco Systems, Inc.
Compiled Tue 06/05/2018 22:45:19.61 by builder

Current image running: Boot ROM0
Last reset cause: PowerOn
DIMM Slot 0 : Present
DIMM Slot 1 : Present

Platform ASA5516 with 8192 Mbytes of main memory
MAC Address: 28:6f:7f:03:b1:a2

Located '.boot_string' @ cluster 200582.

#
Attempt autoboot: "boot disk0:/asdm-7101.bin"
Located 'asdm-7101.bin' @ cluster 958584.

################################################################################ ################################################################################ ################################################################################ ################################################################################ #############

boot: error executing "boot disk0:/asdm-7101.bin"
Attempt autoboot: "boot disk0:"
Located 'asasfr-5500x-boot-6.2.2-3.img' @ cluster 1200252.

################################################################################ ################################################################################ ################################################################################ ################################################################################ ################################################################################ #########
Located '.boot_string' @ cluster 200582.

Located 'asdm-7101.bin' @ cluster 958584.

################################################################################ ################################################################################ ################################################################################ ################################################################################ #############
Located 'crashinfo_20220511_152027_UTC' @ cluster 200585.

#######
Located 'asa5500-firmware-1114.SPA' @ cluster 966920.

################################################################################ ##########
LFBFF signature verified.
Objtype: lfbff_object_rommon (0x800000 bytes @ 0x74c9b238)
Objtype: lfbff_object_fpga (0xd0100 bytes @ 0x7549b258)
Located 'asa9-14-1-10-lfbff-k8.SPA' @ cluster 1026945.

########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
LFBFF signature verified.
INIT: version 2.88 booting
Starting udev
^[Configuring network interfaces... done.
Populating dev cache
^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[fsck.fat 3.0.28 (2015-05-16)
^[Starting check/repair pass.
^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[Starting verification pass.
^[^[^[^[^[/dev/sdb1: 74 files, 843002/1798211 clusters
dosfsck(/dev/sdb1) returned 0
Mounting /dev/sdb1
^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[Starting random number generator daemon.
^[^[^[^[Running postinst /etc/rpm-postinsts/100-rng-tool^[^[IO Memory Nodes: 1
IO Memory Per Node: 610271232 bytes num_pages = 148992 page_size = 4096

Global Reserve Memory Per Node: 314572800 bytes Nodes=1

^[^[^[^[^[^[^[^[^[^[LCMB: got 610271232 bytes on numa-id=0, phys=0x1eb800000, virt=0x7f81a0200000
^[^[^[^[LCMB: HEAP-CACHE POOL got 312475648 bytes on numa-id=0, virt=0x7f818d600000

total_reserved_mem = 610271232

total_heapcache_mem = 312475648
total mem 4029635417 system 8238256128 kernel 36143339 image 99075856
new 4188461845 old 4498944906 reserve 610271232 priv new 3614333952 priv old 3790923776
Processor memory: 4029635417
M_MMAP_THRESHOLD 65536, M_MMAP_MAX 61487
^[^[^[^[POST started...
POST finished, result is 0 (hint: 1 means it failed)

Compiled on Tue 26-May-20 09:39 PDT by builders
^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[
Total NICs found: 14
i354 rev03 Gigabit Ethernet @ irq255 dev 20 index 08 MAC: 286f.7f03.b1a2
ivshmem rev03 Backplane Data Interface @ index 09 MAC: 0000.0001.0002
en_vtun rev00 Backplane Control Interface @ index 10 MAC: 0000.0001.0001
en_vtun rev00 Backplane Int-Mgmt Interface @ index 11 MAC: 0000.0001.0003
en_vtun rev00 Backplane Ext-Mgmt Interface @ index 12 MAC: 0000.0000.0000
en_vtun rev00 Backplane Tap Interface @ index 13 MAC: 0000.0100.0001
WARNING: Attribute already exists in the dictionary.
^[Verify the activation-key, it might take a while...
Running Permanent Activation Key: 0x8a2df867 0xf0f977b2 0x00c2e544 0x979c3088 0xc72d0b9c

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 150 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 2 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 4 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 300 perpetual
Total VPN Peers : 300 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 1000 perpetual
Botnet Traffic Filter : Disabled perpetual
Cluster : Enabled perpetual
Cluster Members : 2 perpetual
VPN Load Balancing : Enabled perpetual

^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[^[cnnic_asa_exit_cb: Accelerator boot err Accelerator boot failed status 4.

--- Begin of accelerator boot log ---
Using user supplied board name: CUST_CLARK, number: 20003
Using user supplied DDR 0 spd address(es)/file(s): /asa/cavium/accelerator_spd
Read 128 values from spd file: /asa/cavium/accelerator_spd
PCIE port 0
All cores in reset, skipping soft reset.
Using bootloader image: /asa/cavium/u-boot.bin
Notice: Using board default DDR clock of: 0 hertz.
Warning: Using generic default DDR clock of 533000000 hertz.
Initialized 1024 MBytes of DRAM
Setting dram_size in env
Starting cores 0x1
Powering up additional cores.
Timeout waiting for boot completion!

--- End of accelerator boot log ---
Invalid log size 0
(set_exptime) Timer not a leaf 0x00007f8188df9210. Traceback: 0x0000562437e7263e 0x0000562437e69edd 0x0000562437e7a0ea 0x0000562437e7453c 0x00005624398a6aab 0x00005624398a82a3 0x00007f81cac7ec60 0x0000562437e6ce16 0x0000562438ac1053 0x00005624398ac1e1 0x0000562437e7d6f6 0x00007f81c62b5340 0x00005624398acd0b 0x0000562437e47e16 0x00007f81c62918f0 0x41d589495541f689
mgd_timer_set_exptime: Not a leaf called from 0x0000562437e7a0ea
core0 same core snap_count=1 signo=11 RIP=562437e7a12b

-----------------------------------------------
Traceback output aborted.
Flushing first exception frame:
r8 0x000056243fe0dd50

[Marius Gunnerud]

it should just be a matter of spamming the ESC key upon boot, and then enter YES when prompted to delete the ASA's configuration.  Do you not get this option?

[edsge teenstra]

Thanks you for taking the moment to reply !

Yes I have tried that while connected by console using putty. I pressed ESC so much that even after a while symbols appear on the command line  

Let me try that again. 

[edsge teenstra]

Nope, nothing happens, except after all different images are extracted the ESC or Break is sent to the screen, but the unit just proceeds and crashes and reloads again. 

Rom image verified correctly

Cisco Systems ROMMON, Version 1.1.14, RELEASE SOFTWARE
Copyright (c) 1994-2018 by Cisco Systems, Inc.
Compiled Tue 06/05/2018 22:45:19.61 by builder

Current image running: Boot ROM0
Last reset cause: PowerOn
DIMM Slot 0 : Present
DIMM Slot 1 : Present

Platform ASA5516 with 8192 Mbytes of main memory
MAC Address: 28:6f:7f:03:b1:a2

Located '.boot_string' @ cluster 200582.

#
Attempt autoboot: "boot disk0:/asdm-7101.bin"
Located 'asdm-7101.bin' @ cluster 958584.

################################################################################ ################################################################################ ################################################################################ ################################################################################ #############

<truncated log text> 

15: 0x0000562437e47e16
16: 0x00007f81c62918f0
17: 0x41d589495541f689
-----------------------------------------------
core0 same core snap_count=2 signo=11 RIP=562438ac3e74
Process shutdown finished
Rebooting... (status 0x8b)
^[^[^[^[^[^[^[^[^[^[^[^[^[..
INStopping Advanced Configuration and Power Interface daemon: no /usr/sbin/acpid found; none killed
acpid.
Deconfiguring network interfaces... done.
Sending all processes the TERM signal...
^[^Z^Z^Z^Z^Z^Z^Z^Z^Z^[^[^[^[^Z^Z^Z^Z^Z^Z^Z^Z/^Z^Z^Z^Z^Z^ZSending all processes the KILL signal...
Deactivating swap...
Unmounting local filesystems...
^Z^ZRebooting... ^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z
Rom image verified correctly

Cisco Systems ROMMON, Version 1.1.14, RELEASE SOFTWARE
Copyright (c) 1994-2018 by Cisco Systems, Inc.
Compiled Tue 06/05/2018 22:45:19.61 by builder

Current image running: Boot ROM0
Last reset cause: PowerCycleRequest
DIMM Slot 0 : Present
DIMM Slot 1 : Present

Platform ASA5516 with 8192 Mbytes of main memory
MAC Address: 28:6f:7f:03:b1:a2

[Jitendra Kumar]

hope this will help you

connect console and access on the putty -reboot ASA
-Press ESC (ctrl+Break if using hyperter )- once you enter in romman#0>confreg

you will get current confguration register:0x00000001 (copy 0x00000001 and save it )

will ask you do you want to chang configuration? Y/n:y
enable boot to romman prompt y/n:n
enable ftp netboot y/n:n
enable flash boot y/n :n
select specific flash image index:n
disable system configuration?y/n:y

rest all just enter or no n

once you done you will get new current congiguration register:0x00000040

romman#1> boot

once boot done go to the enable mode there will be blank password

drage config from flash memory to ram

ciscoasa#copy startup-config running-config
ciscoasa# config t
ciscoasa(config)#enable password ayx
ciscoasa(config)#pass xyz
ciscoasa(config)#write mem
ciscoasa(config)#config-register 0x00000001
ciscoasa(config)#write mem
ciscoasa(config)#reload

[Marius Gunnerud · Accepted solution]

@Jitendra Kumar - the issue is that when password recovery is disabled you cannot enter rommon without erasing the full configuration of the ASA (for security reasons).  So you will be prompted if you want to erase the configuration when you break the boot process at which point you will enter rommon with a blank ASA configuration and can boot from a fresh ASA image.

[Jitendra Kumar]

Yes, if password recovery is disabled as per my knowledge there no way to recover need to re-image or factor reset then restore the config file if available 

[edsge teenstra]

So I try to go to ROMMON to do that like you say , but if the unit does not let me go to ROMMON then I cannot reinstall the image... little frustrating that it is possible to brick a Cisco device these days while in the old days you were always able to bypass password recovery disabled devices.

[edsge teenstra]

Somehow I do not manage to get into Rommon> at all.

I have tried for more then a hour with all sort of key combinations. 

I have compared units that do let me do this and the difference is that the ones I manage it on also show (while booting) that i have to press ESC. My unit shows nothing while booting so that's why I assume the password recovery is disabled.