Hello,
Some time ago both FPR 4110 appliances located on the same site went down...L3-wise, i guess. At the same time.
Physical interfaces of core devices that were connected to appliances remained up/up, but contexts on Virtual ASA stopped responding. GUI became unavailable.
After the lookup, i could see that DME process went down on the FPR:
Firepower4110# scope firmware
Software Error: Exception during execution: [Error: Timed out communicating with DME]
Firepower4110(local-mgmt)# show pmon state
{output omitted}
svc_sam_dme failed 4(4) 0 6 no
{output omitted}
Additionally it appears that there are some issues with the file system:
Firepower4110(local-mgmt)# ls
/bin/ls: reading directory '/workspace/': Input/output error
Firepower4110(fxos)# show startup-config
No startup configuration
Firepower4110(fxos)# sh logg nvram
2023 Jan 11 10:08:31 %$ VDC-1 %$ (none) kernel: [38370840.345670] EXT4-fs error (device sda7): ext4_find_entry:1442: inode #2: comm svc_sam_dcosAG: reading directory lblock 0
2023 Jan 11 10:08:31 %$ VDC-1 %$ (none) kernel: [38370840.345786] EXT4-fs error (device sda7): ext4_find_entry:1442: inode #2: comm svc_sam_dcosAG: reading directory lblock 0
2023 Jan 11 10:08:32 %$ VDC-1 %$ (none) kernel: [38370841.487203] EXT4-fs error (device sda7): ext4_find_entry:1442: inode #2: comm httpd.sh: reading directory lblock 0
2023 Jan 11 10:08:34 %$ VDC-1 %$ (none) kernel: [38370843.048717] EXT4-fs error (device sda3): ext4_find_entry:1442: inode #2: comm lacp: reading directory lblock 0
2023 Jan 11 10:08:35 %$ VDC-1 %$ (none) kernel: [38370844.050561] EXT4-fs error (device sda3): ext4_find_entry:1442: inode #2: comm lacp: reading directory lblock 0
2023 Jan 11 10:08:35 %$ VDC-1 %$ (none) kernel: [38370844.050855] EXT4-fs error (device sda3): ext4_find_entry:1442: inode #2: comm lacp: reading directory lblock 0
I attempted to reboot the secondary node. It went straight to ROMMON mode. Commands like "dir" or "ls" returned an error saying "No mapping".
I decided to go according to Cisco FPR4100 Troubleshooting article, section "Recovering from corrupted file system":
Link
First two steps went well. Between 2nd and 3rd steps, i could see the following:
Checking all filesystems.r.r.r.r.rrr done.
##############################################################
Boot has been interrupted or internal disk is not initialized!
##############################################################
##############################################################
Do you want to initialize it? (y/n) y
fdisk: cannot open /dev/sda: No
And when i tried to do the 3rd step (initialyze the flash):
switch(boot)# init system
This command is going to erase your startup-config, licenses as well as the contents of your bootflash:.
Do you want to continue? (y/n) [n] y
fdisk: cannot open /dev/sda: No such device or address
/isanboot/sbin/init-system: line 144: [: -gt: unary operator expected
There is a problem with the flash device. It could not be initialized.
So my question is:
Is there any way to go further with recovery attempts? Is there a hope?
Thank you and hope for some hint going forward.
