Re: Cisco ASA 5525X reloads after SFR recover boot command

ramon1987 · ‎06-13-2020

I was tryng to reimage SFR module on Cisco ASA 5525-X, I followed this procedure:

https://community.cisco.com/t5/security-blogs/reimage-firepower-module-in-cisco-5500-x-firewall-models/ba-p/3760395

I used that in a lot of appliances, but this time, after issue the command:

sw-module module sfr recover boot

it shows a few logs like this:

Mod-sfr 22> *** EVENT: Creating the Disk Image...
Mod-sfr 23> *** TIME: 08:17:25 col Jun 13 2020
Mod-sfr 24> ***
Mod-sfr 25> ***Cisco ASA 5500X
Mod-sfr 26> *** EVENT: The module is being recovered.
Mod-sfr 27> *** TIME: 08:17:26 col Jun 13 2020
Mod-sfr 28> ***

And that's all, it reloads the whole ASA.

Tryed a lot of boot giles, 5.4, 6.0, 6.1, 6.2, 6.3, 6.4 it's always the same issue.

After every reload it is creating .REC files:

180 0 Jun 12 2020 19:13:06 FSCK0008.REC
181 0 Jun 12 2020 19:24:30 FSCK0009.REC
182 0 Jun 12 2020 19:37:34 FSCK0010.REC
183 0 Jun 13 2020 08:26:22 FSCK0013.REC
184 0 Jun 13 2020 08:44:36 FSCK0014.REC
185 0 Jun 13 2020 08:53:56 FSCK0015.REC
186 0 Jun 13 2020 09:31:42 FSCK0016.REC
187 0 Jun 13 2020 09:43:00 FSCK0017.REC

Hope you can help! thanks

Marvin Rhoads · ‎06-13-2020

Can you provide more details on the system you're working with? For instance:

show inventory
show version
show module details
dir disk0:

...as well as the exact commands you entered on your system.

ramon1987 · ‎06-14-2020

Hi Marvin, thank you, These are the outputs for the commands:

FW-IN# sh inventory
Name: "Chassis", DESCR: "ASA 5525-X with SW, 8 GE Data, 1 GE Mgmt, AC"
PID: ASA5525 , VID: V03 , SN: FGL162741L8

Name: "Storage Device 1", DESCR: "Model Number: Micron_M600_MTFDDAK128MBF"
PID: N/A , VID: N/A , SN: MSA204905QH

FW-IN# sh version

Cisco Adaptive Security Appliance Software Version 9.8(3)18
Firepower Extensible Operating System Version 2.2(2.111)
Device Manager Version 7.12(1)

Compiled on Wed 12-Dec-18 17:03 PST by builders
System image file is "disk0:/asa983-18-smp-k8.bin"
Config file at boot was "startup-config"

FW-IN up 1 day 9 hours
failover cluster up 3 years 1 day

Hardware: ASA5525, 8192 MB RAM, CPU Lynnfield 2393 MHz, 1 CPU (4 cores)
ASA: 4179 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
Boot microcode : CNPx-MC-BOOT-2.00
SSL/IKE microcode : CNPx-MC-SSL-SB-PLUS-0005
IPSec microcode : CNPx-MC-IPSEC-MAIN-0026
Number of accelerators: 1
Baseboard Management Controller (revision 0x1) Firmware Version: 2.4

0: Int: Internal-Data0/0 : address is a493.4caa.d7ec, irq 11
1: Ext: GigabitEthernet0/0 : address is a493.4caa.d811, irq 5
2: Ext: GigabitEthernet0/1 : address is a493.4caa.d7ed, irq 5
3: Ext: GigabitEthernet0/2 : address is a493.4caa.d812, irq 10
4: Ext: GigabitEthernet0/3 : address is a493.4caa.d7ee, irq 10
5: Ext: GigabitEthernet0/4 : address is a493.4caa.d813, irq 5
6: Ext: GigabitEthernet0/5 : address is a493.4caa.d7ef, irq 5
7: Ext: GigabitEthernet0/6 : address is a493.4caa.d814, irq 10
8: Ext: GigabitEthernet0/7 : address is a493.4caa.d7eg, irq 10
9: Int: Internal-Data0/1 : address is 0000.0001.0002, irq 0
10: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 0
11: Int: Internal-Data0/2 : address is 0000.0001.0003, irq 0
12: Ext: Management0/0 : address is a493.4caa.d7ec, irq 0
13: Int: Internal-Data0/3 : address is 0000.0100.0001, irq 0

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 200 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 2 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 750 perpetual
Total VPN Peers : 750 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 2 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual
Cluster Members : 2 perpetual

This platform has an ASA5525 VPN Premium license.

Failover cluster licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 200 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 4 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 4 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 750 perpetual
Total VPN Peers : 750 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Disabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 4 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual

This platform has an ASA5525 VPN Premium license.

Serial Number: FCH16247GYN
Running Permanent Activation Key: 0x4f10e450 0xf44b600f 0xd872265d 0x9cc889c1 0x80113gb3
Configuration register is 0x1

Image type : Release
Key version : A

Configuration last modified by enable_1 at 11:37:03.489 col Sat Jun 13 2020

FW-IN# sh module sfr details
Getting details from the Service Module, please wait...
Unable to read details from module sfr

Card Type: Unknown
Model: N/A
Hardware version: N/A
Serial Number: FCH16247GYN
Firmware version: N/A
Software version:
MAC Address Range: a493.4caa.d7e0 to a493.4caa.d7e0
Data Plane Status: Not Applicable
Console session: Ready
Status: Unresponsive

FW-IN# dir disk0:

Directory of disk0:/

12 drwx 4096 08:52:28 Mar 27 2017 log
25 drwx 4096 08:13:08 Dec 01 2015 crypto_archive
118 -rwx 0 08:13:10 Dec 01 2015 nat_ident_migrate
28 drwx 4096 08:13:10 Dec 01 2015 coredumpinfo
119 -rwx 72771616 01:14:38 Jun 06 2020 anyconnect-win-4.8.03052-webdeploy-k9.pkg
120 -rwx 0 11:35:48 Jun 13 2020 FSCK0000.REC
121 -rwx 108914688 23:32:02 Apr 06 2017 asa971-4-smp-k8.bin
122 -rwx 41852928 06:41:16 May 17 2017 asasfr-5500x-boot-6.1.0-330.img
123 drwx 4096 09:16:14 May 17 2017 tmp
124 -rwx 111034368 05:55:04 Feb 23 2019 asa983-18-smp-k8.bin
125 -rwx 34143680 05:55:16 Feb 23 2019 asdm-7101.bin
26 drwx 4096 19:11:48 May 12 2020 snmp
126 -rwx 34183584 22:33:00 Jun 05 2020 asdm-7121.bin
127 -rwx 42956800 11:16:34 Jun 13 2020 asasfr-5500x-boot-6.4.0-1.img

9 file(s) total size: 445857664 bytes
7989768192 bytes total (7540854784 bytes free/94% free)

The commands I used were:

sw-module module sfr uninstall

sw-module module sfr recover configure image disk0:/asasfr-5500x-boot-6.4.0-1.img

debug module-boot

sw-module module sfr recover boot

3 seconds after the last command the whole ASA is reloaded so I can't even try to install the image.

Thank you again.

Sheraz.Salim · ‎06-14-2020

just to add what Marvin mentioned could you also share the show crash file of this unit. also share the output of show flash.FSCK0008.REC files would indicate that the file system had errors that needed to be fixed. I would recommend running 'fsck' to check and fix any file system issues that might exist.

please do not forget to rate.

ramon1987 · ‎06-14-2020

Hi Sheraz, thank you, the 'show crash file' didn't throw anything.

The others commands:

FW-IN# sh flash
--#-- --length-- -----date/time------ path
12 4096 Mar 27 2017 08:52:28 log
14 7457 Jun 13 2020 11:36:37 log/asa-appagent.log
25 4096 Dec 01 2015 08:13:08 crypto_archive
118 0 Dec 01 2015 08:13:10 nat_ident_migrate
28 4096 Dec 01 2015 08:13:10 coredumpinfo
29 59 Dec 01 2015 08:13:10 coredumpinfo/coredump.cfg
119 72771616 Jun 06 2020 01:14:38 anyconnect-win-4.8.03052-webdeploy-k9.pkg
120 0 Jun 13 2020 11:35:48 FSCK0000.REC
121 108914688 Apr 06 2017 23:32:02 asa971-4-smp-k8.bin
122 41852928 May 17 2017 06:41:16 asasfr-5500x-boot-6.1.0-330.img
123 4096 May 17 2017 09:16:14 tmp
124 111034368 Feb 23 2019 05:55:04 asa983-18-smp-k8.bin
125 34143680 Feb 23 2019 05:55:16 asdm-7101.bin
26 4096 May 12 2020 19:11:48 snmp
27 4 Jun 13 2020 11:37:10 snmp/single_vf
126 34183584 Jun 05 2020 22:33:00 asdm-7121.bin
127 42956800 Jun 13 2020 11:16:34 asasfr-5500x-boot-6.4.0-1.img

7989768192 bytes total (7540854784 bytes free)

FW-IN# fsck flash:
umount: /mnt/disk0: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
There are differences between boot sector and its backup.
Differences: (offset:original/backup)
65:01/00
Not automatically fixing this.
/dev/sdb1: 71 files, 109598/1950627 clusters
mount: /dev/sdb1 is already mounted or /mnt/disk0 busy
/dev/sdb1 is already mounted on /mnt/disk0

fsck of flash: complete

I deleted all the .REC files the last time trying to clean the flash but it is creating a new one when I try to boot the sfr module and the ASA gets reloaded.

Thanks again.

Marvin Rhoads · ‎06-14-2020

I had asked about "show module details" - not "show module sfr details" - so that we can verify that neither an ips or cxsc module is installed.

Other than that, you commands and img file appear to be correct. I might suggest trying it without first turning on the debug.

The fsck (file system check) .rec files are normal when an ASA crashes or reloads unexpectedly. They can be safely deleted.

If there's no other module installed and doing the recover without debugging on doesn't allow you to recover then I would suggest opening a TAC case.

ramon1987 · ‎06-16-2020

Hi Marvin, I did try the command 'show module details' but it was not accepted, that is why I sent the 'show module sfr details':

FW-IN# show module details
^
ERROR: % Invalid input detected at '^' marker.

FW-IN# show module ?

Available module ID(s):
0 Module ID
all show all module information for all slots
cxsc Module ID
ips Module ID
sfr Module ID
| Output modifiers
<cr>

Maybe these outputs can help:

FW-IN# show module ips details
Getting details from the Service Module, please wait...
Unable to read details from module ips

Card Type: Unknown
Model: N/A
Hardware version: N/A
Serial Number: FCH16247FXM
Firmware version: N/A
Software version:
MAC Address Range: a493.4caa.d6f9 to a493.4caa.d6f9
Data Plane Status: Not Applicable
Console session: Not ready
Status: Unresponsive No Image Present
License: IPS Module Disabled perpetual

FW-IN# show module cxsc details
Getting details from the Service Module, please wait...
Unable to read details from module cxsc

Card Type: Unknown
Model: N/A
Hardware version: N/A
Serial Number: FCH16247FXM
Firmware version: N/A
Software version:
MAC Address Range: a493.4caa.d6f9 to a493.4caa.d6f9
Data Plane Status: Not Applicable
Console session: Not ready
Status: Unresponsive No Image Present

I'll try the recover without the debug enabled and let you know.

Thank you.

Sheraz.Salim · ‎06-15-2020

Since when you having issue with this unit? this seem like very strange behavior. if you have a support contract escalate this to cisco tac. curious could be the SSD had some issue.

please do not forget to rate.

ramon1987 · ‎06-16-2020

Hi Sheraz, we realized about this since last saturday, it was the first time in a long time that we tryed to upgrade the appliances, first they didin't allow an upgrade from 6.1.X to 6.2.X so we tryed the reimage...

Do you know if there is a way to check the SSD by CLI? like the command 'fsck flash'?

Thank you.

Sheraz.Salim · ‎06-20-2020

The SSD is not user-addressable/accessible storage. It is used for storage by the module. Is your issue fixed or still going on?

please do not forget to rate.

ramon1987 · ‎06-23-2020

Hi Sheraz,Marvin, thanks for all your help, the TAC has raised the RMA for the SSD, as soon as we change it I'll let you know.

ramon1987 · ‎06-26-2020

Hi guys, well, the SSD replacement didn't work, we had to change the whole ASA.

Thank you for everything.