FirePower 1010 ASA 'write error: No space left on device'

[andrew.butterworth]

I am not having much luck with the FirePower 1010 platform.  I have had to RMA two devices so far and I've had an unrecoverable software crash on a third.

I'm now back to two devices - one from the original delivery and a second new one as we have had two DoAs.  As a workaround I have put in a temporary pair of ASA5515-X's that had been decommissioned from another customer but I need to get them swapped out for these FP1010's.  I currently have them offline, although there is a cable connecting them for HA and I have console access to them both.

I logged onto the existing one and the console was full of these messages 

[1355831.468897] spi_read offset:0x812000  offset:0x400
[1355831.531000] Read: address 0x812000, length 0x400
[1355831.591794] read ret length = 1024
[1355833.117410] spi_read offset:0x812400  offset:0x400
[1355833.179538] Read: address 0x812400, length 0x400
[1355833.240349] read ret length = 1024
[1355834.562105] spi_read offset:0x812800  offset:0x110
[1355834.624197] Read: address 0x812800, length 0x110
[1355834.683706] read ret length = 272
[1355837.242258] spi_fs_ioctl spi_block_size
[1355837.554652] spi_read offset:0x810000  offset:0x400
[1355837.616753] Read: address 0x810000, length 0x400
[1355837.677513] read ret length = 1024
[1355839.850695] spi_read offset:0x810400  offset:0x400
[1355839.912783] Read: address 0x810400, length 0x400
[1355839.973549] read ret length = 1024
[1355841.049534] spi_read offset:0x810800  offset:0x400
[1355841.111607] Read: address 0x810800, length 0x400
[1355841.172367] read ret length = 1024
[1355841.718938] spi_read offset:0x810c00  offset:0x400
[1355841.781017] Read: address 0x810c00, length 0x400
[1355841.841774] read ret length = 1024

I managed to login and reload, however it didn't reload and just hung with the messages repeating continually.  I power reset the device and on boot up got various error messages - sample of some here:

/bin/ps: write error: No space left on device
/bin/ps: write error: No space left on device

/etc/init.d/start_monitor_confreg.sh: line 29: echo: write error: No space left on device
SSP-Security-Module is shutting down ...
Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Beginning System Shutdown request for CSP Apps
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Continue System Shutdown request for CSP Apps
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/bin/ls: write error: No space left on device

/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device
grep: write error
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 49: echo: write error: No space left on device
grep: write error
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device
grep: write error

/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device

/bin/rm: cannot remove '/tmp/openssl.conf': No such file or directory
 httpdRegister INFO: [httpd.3744 -s -4 0.0.0.0 -n localhost]
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 httpdRegister INFO: SKIP httpd syntax check
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 httpdRegister INFO: Starting httpd setup/registration...
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 httpdRegister INFO: Completed httpd setup/registration!
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 INFO: httpdRegister [httpd.3744 script exit]
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
INFO: manager_startup: Completed manager httpd setup!
Starting crond: OK
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device
1:/opt/cisco/csp/cores
/opt/cisco/csp/cores 31457280
/bin/cat: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_vnic_validate.sh: line 55: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 76: echo: write error: No space left on device
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1498
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 734
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1504
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 919
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1528
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1530
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1327
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1352
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1367
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1409
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1435
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1444
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1453
/opt/csp/csp_utils.sh: line 1333: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1336: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1337: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1340: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1347: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/opt/csp/csp_utils.sh: line 1382: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1386: echo: write error: No space left on device
/opt/csp/csp_startup.sh: line 75: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 138: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 139: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 166: echo: write error: No space left on device
/bin/grep: write error: No space left on device
/opt/csp/csp_mgr.sh: line 156: echo: write error: No space left on device
▒▒▒▒▒55)▒csp_mgr.sh: line Q▒▒K▒ɥѕ▒▒▒ɽ▒▒9▒▒▒▒
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 46: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 11: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 12: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 14: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 15: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 17: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 18: echo: write error: No space left on device
firepower-1010 login: admin (automatic login)

I did a 'write erase' and reloaded it and I get the same 'out of space' errors.  I booted the new unit and I don't see any of these errors on boot.

As both devices are running ASA software I have no control over the underlying FXOS.  I can reimage it, however as its remote it might be a bit difficult.

Does this look like another faulty device?  Are there inherent issues with the FirePower 1010 platform?

Andy

[balaji.bandi]

Not sure what Code of FXOS or FTD or ASA running, But there are a couple of bugs reported as below : ( please if this affecting).

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34966/?rfs=iqvred

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi61729/?rfs=iqvred

[Sheraz.Salim]

Dont you have a TAC support? If TAC support avaiable worth open a case with them. seem like you hitting a bug on this problemetic unit https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34966/?rfs=iqvred

[harmesh88]

Please try to upgrade Device with Latest version , Bug is related to FXOS so when you have latest FXOS it will remove bug .

if you will get same issue again then only tac will be last option.

I hope it will help you !!

[ang7127]

My TAC case resulted in CSCvx34966 being filed - running latest ASA (9.14(2)8) on FP1010. TAC is telling me to factory-reset the device to hopefully clear the full FXOS partitions in question and restore from backup, or RMA the device so they can further diagnose. This issue prevents login to the underlying FXOS subsystem, so I cannot remove the logs in question. All attempts to 'connect fxos admin' fail and there appears to be no work-around other than factory-reset. The ASA running on FXOS appears stable, for now...

[andrew.butterworth]

I managed to 'factory reset' them both and from ROMMON via a USB stick loaded with the 9.15(1)7 software, I re-imaged them.  So far they have not experienced the errors I was seeing with 9.15(1).  I have configured them up as a HA pair and connected the management interfaces and then added them to our management platform.  I'm going to leave them for a couple of weeks with just management connected and see what happens.

[balaji.bandi]

that is good work, glad all good for now.

[Sheraz.Salim]

Happy day. all the best..

[kerstin-534]

 "no space left on device" problems can be resolved with TAC and only with TAC. They do such command

firepower-1010(local-mgmt)# secure-login

Challenge String (Please copy everything between the asterisk lines exclusively) :

********************************************************************************
Iuf//wYAAAAp3+0C5t20r1fsCfIdgminySBekByLUrD2zpJzPVJdcbv51bZGWE9T

and delete /var/log/messages.*

There problem is here that customer has to access to Linux. They should have access to Linux to resolve this problem themselves.

[enrf]

This can be easily fixed without TAC. Just pull out the SSD, plug it into a Linux computer, mount the filesystem and delete the log files.

[balaji.bandi]

good hardware trick - some time in critical environment process wont allow this - but good catch

[zuerchermi]

Hi all,

I just ran into a similar or even the same issue.
It's an active/standby cluster of FPR2130 running the ASA image.
The primary node suddenly stopped to work last week and the secondary node took over. Unfortunately only the primary firewall is connected to the internet which is why I lost my remote connection. On site we then saw that the primary ASA was still running and the active LED was green. After rebooting the ASA, I got following console output:

*******************************************************************************
Cisco System ROMMON, Version 1.0.12, RELEASE SOFTWARE
Copyright (c) 1994-2019  by Cisco Systems, Inc.
Compiled Mon 06/17/2019 16:23:23.36 by builder
*******************************************************************************

Current image running: Boot ROM0
Last reset cause: PowerCycleRequest (0x00002000)
DIMM_1/1 : Present
DIMM_2/1 : Present

Platform FPR-2130 with 32768 MBytes of main memory

WARNING: This board is using a temporary MAC address.
WARNING: The temporary MAC address override value = 00:11:22:33:44:30
WARNING: Please clear this value to use the programmed MAC address.
WARNING: Use the following two CLI commands:
WARNING:   unset MACADDR
WARNING:   sync

BIOS has been successfully locked !!
MAC Address: 44:88:16:e0:40:00

Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.


Located '.boot_string' @ cluster 465251.


Attempt autoboot: "boot disk0:installables/switch/fxos-k8-fp2k-lfbff.2.10.1.241.SPA"
Located 'installables/switch/fxos-k8-fp2k-lfbff.2.10.1.241.SPA' @ cluster 586282.

###########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

+-------------------------------------------------------------------+
+------------------------- SUCCESS ---------------------------------+
+-------------------------------------------------------------------+
|                                                                   |
|             LFBFF signature authentication passed !!!             |
|                                                                   |
+-------------------------------------------------------------------+
LFBFF signature verified.
+-------------------------------------------------------------------+
+------------------------- SUCCESS ---------------------------------+
+-------------------------------------------------------------------+
|                                                                   |
|              LFBFF controller type check passed !!!               |
|                                                                   |
+-------------------------------------------------------------------+

Linux version: 4.18.45-yocto-standard (oe-user@oe-host) #1 SMP Fri Feb 3 23:29:01 UTC 2023
kernel_image = 0x8daf7da8, kernel_size=0x6452a0
Image validated
[    9.349010] Disabling IRQ #16
INIT: version 2.88 booting
Starting udev
Hardware tweak APPLIED: Disable SATA Throttle.1
Hardware tweak APPLIED: Disable SATA Throttle.2
Configuring network interfaces... done.
Starting random number generator daemon.
Starting Power Off Shutdown Handler (poshd)
poshd: using FPGA version  and PSEQ version
Starting TAm services ...
Device configuration status = TAM_SUCCESS
TAm Services started successfully
Primary SSD discovered
fsck from util-linux 2.32.1
[/sbin/fsck.ext3 (1) -- /dev/sda1] fsck.ext3 -a /dev/sda1
/dev/sda1: recovering journal
/dev/sda1: clean, 8796/61056 files, 244224/244224 blocks
fsck(/dev/sda1) returned 0
fsck from util-linux 2.32.1
[/sbin/fsck.ext3 (1) -- /dev/sda2] fsck.ext3 -a /dev/sda2
/dev/sda2: recovering journal
/dev/sda2: clean, 61056/61056 files, 243968/243968 blocks
fsck(/dev/sda2) returned 0
fsck from util-linux 2.32.1
[/sbin/fsck.ext3 (1) -- /dev/sda3] fsck.ext3 -a /dev/sda3
/dev/sda3: recovering journal
/dev/sda3: clean, 14/1831424 files, 258124/7324416 blocks
fsck(/dev/sda3) returned 0
mount_disk_xfs. device: /dev/sda4, dir: /opt/cisco/csp, mount returned: 0.
fsck from util-linux 2.32.1
[/sbin/fsck.vfat (1) -- /dev/sdb1] fsck.vfat -a /dev/sdb1
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Performing changes.
/dev/sdb1: 63 files, 460050/1919062 clusters
fsck(/dev/sdb1) returned 1
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 286: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 276: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 286: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 286: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 286: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 286: echo: write error: No space left on device
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
/bin/cp: error writing '/var/log/faillog.sav': No space left on device
/bin/sed: couldn't flush /opt/cisco/config/platform/hardening/sedhqbe18: No space left on device
/bin/sed: couldn't flush /opt/cisco/config/platform/hardening/sedK0Rua9: No space left on device
/bin/sed: couldn't flush /opt/cisco/config/platform/hardening/sedji2Tp9: No space left on device
/bin/sed: couldn't flush /opt/cisco/config/platform/hardening/sed76Qam8: No space left on device
/bin/sed: couldn't flush /opt/cisco/config/platform/hardening/sedpFN7j8: No space left on device
FIPS POST Test Script
NOTICE: The FIPS POST is not run because the FIPS feature is not enabled
INIT: Entering runlevel: 3rst bo
Starting system message bus: dbus.
Starting OpenBSD Secure Shell server: sshd
done.
Starting rpcbind daemon...done.
starting statd: done
Starting Advanced Configuration and Power Interface daemon: acpid.
acpid: starting up with netlink and the input layer
acpid: 1 rule loaded
acpid: waiting for events: event logging is off
Starting DHCP server: .
starting 8 nfsd kernel threads: done
starting mountd: done
Starting ntpd: done
Starting internet superserver: xinetd.
Starting Octeon NPU ...
Starting Octeon NPU ... success
Starting fan control daemon: fancontrol... done.
INFO: beginning of manager_install
INFO: manager_install: fxmgr=/mnt/boot/installables/switch/fxos-k9-fp2k-manager.2.10.1.241.SPA chmgr=/mnt/boot/installables/switch/fxos-k9-mgmtext.2.10.1.60.SPA update=false
INFO: manager_install: fxmgr is dummy, skip_fxmgr_install=true
INFO: in validating image ...
INFO: manager_validate_image: fxmgr_absfilename /mnt/boot/installables/switch/fxos-k9-fp2k-manager.2.10.1.241.SPA
INFO: Validating image /mnt/boot/installables/switch/fxos-k9-fp2k-manager.2.10.1.241.SPA signature ...
: File /mnt/boot/installables/switch/fxos-k9-fp2k-manager.2.10.1.241.SPA size 1296
Done!
Computed Hash   SHA2: 4f6310e3e179421fae5ccfc31194fcdd
                      cdbb2e4bcdaaa2c1e47397fb03872845
                      dabfb514ceb92cb30a7244b5b0349459
                      31200dfafcdadad89eb3cadd0a2e4c52

Embedded Hash   SHA2: 4f6310e3e179421fae5ccfc31194fcdd
                      cdbb2e4bcdaaa2c1e47397fb03872845
                      dabfb514ceb92cb30a7244b5b0349459
                      31200dfafcdadad89eb3cadd0a2e4c52

The digital signature of the file: fxos-k9-fp2k-manager.2.10.1.241.SPA verified successfully
INFO: manager_validate_image: chmgr_absfilename /mnt/boot/installables/switch/fxos-k9-mgmtext.2.10.1.60.SPA
INFO: Validating image /mnt/boot/installables/switch/fxos-k9-mgmtext.2.10.1.60.SPA signature ...
: File /mnt/boot/installables/switch/fxos-k9-mgmtext.2.10.1.60.SPA size 37135504
Done!
Computed Hash   SHA2: 152e65b198a7b98491ebc4d3212d9b5f
                      47a3a10dfe42296d9f095cbb45994c15
                      7a97413e849739cc25a47e5bffbf7ea4
                      b6e81eaf2c0890f2dea162cc62ef8e4b

Embedded Hash   SHA2: 152e65b198a7b98491ebc4d3212d9b5f
                      47a3a10dfe42296d9f095cbb45994c15
                      7a97413e849739cc25a47e5bffbf7ea4
                      b6e81eaf2c0890f2dea162cc62ef8e4b

The digital signature of the file: fxos-k9-mgmtext.2.10.1.60.SPA verified successfully
INFO: manager_install: skip_fxmgr_install=true - delete unnecessary files and skip
INFO: deleting unnecessary xml file..!!
INFO: deleted unnecessary xml file..!!
INFO: manager_post_install ...
INFO: manager_post_install: fxmgr=/mnt/boot/installables/switch/fxos-k9-fp2k-manager.2.10.1.241.SPA chmgr=/mnt/boot/installables/switch/fxos-k9-mgmtext.2.10.1.60.SPA update=false
INFO: manager_post_install: fxmgr is dummy
INFO: manager_post_install: Linking libraries ...
INFO: manager_post_install: Linking binaries ...
INFO: Creating directory /tmp/chmgr
INFO: creating /isan/apache/chassis-mgr/
INFO: Change permission /isan/apache/chassis-mgr/.deploy_onbox.sh
INFO: Change permission /isan/apache/chassis-mgr/.httpd.conf
INFO: Change permission /isan/apache/chassis-mgr/kpmgmt/onbox-version.txt
INFO: manager_post_install: succesful install chassis mgr
INFO: Trying to add iptables and ip6tables rules ...
INFO: Set up Application Diagnostic Interface ...
INFO: Configure management0 interface ...

2023-12-20T14:48:40 [WARN/lldpctl] unknown command from argument 4: `status`

INFO: Configure system files ...
INFO: System Name is: firepower-2130
Starting sensors logging daemon: sensord... done.
INFO: /mnt/boot/installables/switch/fxos-k8-fp2k-npu.2.10.1.241.SPA
INFO: Need to validate the image
: File /mnt/boot/installables/switch/fxos-k8-fp2k-npu.2.10.1.241.SPA size 73750272
Done!
Computed Hash   SHA2: 6e4fcb544bb75bf933bc663ce0e8179d
                      64a98883c3117f3a776cedbecbdb1d3e
                      bd2b0b250841b3acddb5dfb5d09cf667
                      bcb58d78c1e2629cb1657b8538ba8af4

Embedded Hash   SHA2: 6e4fcb544bb75bf933bc663ce0e8179d
                      64a98883c3117f3a776cedbecbdb1d3e
                      bd2b0b250841b3acddb5dfb5d09cf667
                      bcb58d78c1e2629cb1657b8538ba8af4

The digital signature of the file: fxos-k8-fp2k-npu.2.10.1.241.SPA verified successfully
INFO: Creating directory /tmp/npu
INFO: all files are there ...
INFO: fp2100 asa copy appliance mode
INFO: cur reboot count is 1
INFO: max reboot count is 7
INFO: console : ttyS0, speed : 9600
INFO: manager_startup: setting up fxmgr apache ...
INFO: manager_startup: Start manager httpd setup...
INFO: manager_startup: using HTTPD_INFO persistent cache
/bin/rm: cannot remove '/tmp/openssl.conf': No such file or directory
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdRegister INFO: [httpd.2578 -s -4 0.0.0.0 -n localhost]
/isan/bin/httpdRegister.sh: line 75: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdRegister INFO: SKIP httpd syntax check
/isan/bin/httpdRegister.sh: line 75: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdRegister INFO: Starting httpd setup/registration...
/isan/bin/httpdRegister.sh: line 75: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdRegister INFO: Completed httpd setup/registration!
/isan/bin/httpdRegister.sh: line 75: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 INFO: httpdRegister [httpd.2578 script exit]
/isan/bin/httpdRegister.sh: line 75: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
INFO: manager_startup: Completed manager httpd setup!
INFO: manager_startup: configuring chassis manager
INFO: unconfig older conf files
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: [httpd.2644 -d /isan/apache/.httpd.conf]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf [fpr21xx] PARAMS: [GLOBAL_DEL:/isan/apache/.httpd.conf]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: /isan/apache/.httpd.conf changes already removed
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: httpd.conf GLOBAL_DEL update for /isan/apache/.httpd.conf already applied
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 INFO: httpdAppconf [httpd.2644 script exit]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: [httpd.2682 -V -d /isan/apache/.httpd.conf]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf [fpr21xx] PARAMS: [VHOST_DEL:/isan/apache/.httpd.conf]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: SUCCESSFUL httpd.conf VHOST_DEL update for /isan/apache/.httpd.conf
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/isan/apache/logs/apachectl.synchk.log': No space left on device
/isan/bin/httpdAppconf.sh: line 234: /isan/apache/logs/apachectl.synchk.log: No space left on device
/isan/bin/httpdAppconf.sh: line 235: /isan/apache/logs/apachectl.synchk.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf ERR: Failure in httpd config syntax check
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 ERR: httpdAppconf [httpd.2682 script error:1]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
INFO: Configuring httpd
/bin/touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: [httpd.2724 -V -a /isan/apache/.httpd.conf]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf [fpr21xx] PARAMS: [VHOST_ADD:/isan/apache/.httpd.conf]
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
touch: cannot touch '/opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log': No space left on device
 httpdAppconf INFO: SUCCESSFUL httpd.conf VHOST_ADD update for /isan/apache/.httpd.conf
/isan/bin/httpdAppconf.sh: line 76: /opt/cisco/platform/logs/sysmgr/apache/kp_httpd.log: No space left on device
/bin/touch: cannot touch '/isan/apache/logs/apachectl.synchk.log': No space left on device
/isan/bin/httpdAppconf.sh: line 234: /isan/apache/logs/apachectl.synchk.log: No space

/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 62: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 14: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 15: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 22: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 23: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 26: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 27: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 64: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 14: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 15: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 22: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 23: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 26: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 27: echo: write error: No space left on device
/bin/echo: write error: No space left on device
firepower-2130 login: admin (automatic login)

Last login: Wed Dec 13 17:21:10 UTC 2023 on ttyS0
Successful login attempts for user 'admin' : 1
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 45: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
INFO: System Disks /dev/sda is present. Status: Operable. /dev/sdb is present. Status: Inoperable.
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_mgmt_startup.sh: line 37: echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_create_cgroup.sh: line 23: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_create_cgroup.sh: line 23: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
top: write error
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
Building ldconfig cache ...

Waiting for Application infrastructure to be ready...
Verifying the signature of the Application image...
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/grep: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_create_swap.sh: line 23: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
top: write error
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_stat.sh: line 35: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_heimdall_init: line 653: echo: write error: No space left on device

Cisco ASA: CMD=-bootup, CSP-ID=cisco-asa.9.16.4.14__asa_001_FCZ2637R7QDJIONFE1, FLAG='fromHconfFile'
Cisco ASA booting up ...
Cisco ASA started successfully.
Please wait for Cisco ASA to come online...1...
/bin/grep: write error: No space left on device
/bin/echo: write error: No space left on device
/bin/grep: write error: No space left on device
/bin/echo: write error: No space left on device
cat: /proc//stack: No such file
Cisco ASA: CMD=-stop, CSP-ID=cisco-asa.9.16.4.14__asa_001_FCZ2637R7QDJIONFE1, FLAG=''
Cisco ASA stopping ...

Via rommon i then initiated a reset to factory-defaults which formatted the SSD. After that, the ASA booted correctly. I configured the failover commands (including failover lan unit primary). At the moment, the secondary active FW is shutdown. My plan is to:
- Bring up the secondary active FW
- Then boot the primary with all IF (foremost the HA-IF) connected
- Assuming the primary firewall will see the secondary as active and load the config

I think and hope like this, the empty config of the former primary FW won't be written to the secondary active.

Does someone know if this bug has already been fixed in a newer release? Or should I just give it a try with a newer SW version?

Thanks,
Michael