FirePower 1010 ASA 'write error: No space left on device'

andrew.butterworth · ‎02-19-2021

I am not having much luck with the FirePower 1010 platform. I have had to RMA two devices so far and I've had an unrecoverable software crash on a third.

I'm now back to two devices - one from the original delivery and a second new one as we have had two DoAs. As a workaround I have put in a temporary pair of ASA5515-X's that had been decommissioned from another customer but I need to get them swapped out for these FP1010's. I currently have them offline, although there is a cable connecting them for HA and I have console access to them both.

I logged onto the existing one and the console was full of these messages

[1355831.468897] spi_read offset:0x812000  offset:0x400
[1355831.531000] Read: address 0x812000, length 0x400
[1355831.591794] read ret length = 1024
[1355833.117410] spi_read offset:0x812400  offset:0x400
[1355833.179538] Read: address 0x812400, length 0x400
[1355833.240349] read ret length = 1024
[1355834.562105] spi_read offset:0x812800  offset:0x110
[1355834.624197] Read: address 0x812800, length 0x110
[1355834.683706] read ret length = 272
[1355837.242258] spi_fs_ioctl spi_block_size
[1355837.554652] spi_read offset:0x810000  offset:0x400
[1355837.616753] Read: address 0x810000, length 0x400
[1355837.677513] read ret length = 1024
[1355839.850695] spi_read offset:0x810400  offset:0x400
[1355839.912783] Read: address 0x810400, length 0x400
[1355839.973549] read ret length = 1024
[1355841.049534] spi_read offset:0x810800  offset:0x400
[1355841.111607] Read: address 0x810800, length 0x400
[1355841.172367] read ret length = 1024
[1355841.718938] spi_read offset:0x810c00  offset:0x400
[1355841.781017] Read: address 0x810c00, length 0x400
[1355841.841774] read ret length = 1024

I managed to login and reload, however it didn't reload and just hung with the messages repeating continually. I power reset the device and on boot up got various error messages - sample of some here:

/bin/ps: write error: No space left on device
/bin/ps: write error: No space left on device

/etc/init.d/start_monitor_confreg.sh: line 29: echo: write error: No space left on device
SSP-Security-Module is shutting down ...
Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Beginning System Shutdown request for CSP Apps
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Continue System Shutdown request for CSP Apps
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/bin/ls: write error: No space left on device

/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device
grep: write error
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 49: echo: write error: No space left on device
grep: write error
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device
grep: write error

/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device

/bin/rm: cannot remove '/tmp/openssl.conf': No such file or directory
 httpdRegister INFO: [httpd.3744 -s -4 0.0.0.0 -n localhost]
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 httpdRegister INFO: SKIP httpd syntax check
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 httpdRegister INFO: Starting httpd setup/registration...
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 httpdRegister INFO: Completed httpd setup/registration!
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
 INFO: httpdRegister [httpd.3744 script exit]
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
INFO: manager_startup: Completed manager httpd setup!
Starting crond: OK
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device
1:/opt/cisco/csp/cores
/opt/cisco/csp/cores 31457280
/bin/cat: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_vnic_validate.sh: line 55: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 76: echo: write error: No space left on device
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1498
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 734
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1504
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 919
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1528
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1530
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1327
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1352
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1367
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1409
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1435
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1444
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1453
/opt/csp/csp_utils.sh: line 1333: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1336: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1337: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1340: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1347: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/opt/csp/csp_utils.sh: line 1382: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1386: echo: write error: No space left on device
/opt/csp/csp_startup.sh: line 75: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 138: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 139: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 166: echo: write error: No space left on device
/bin/grep: write error: No space left on device
/opt/csp/csp_mgr.sh: line 156: echo: write error: No space left on device
▒▒▒▒▒55)▒csp_mgr.sh: line Q▒▒K▒ɥѕ▒▒▒ɽ▒▒9▒▒▒▒
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 46: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 11: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 12: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 14: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 15: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 17: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 18: echo: write error: No space left on device
firepower-1010 login: admin (automatic login)

I did a 'write erase' and reloaded it and I get the same 'out of space' errors. I booted the new unit and I don't see any of these errors on boot.

As both devices are running ASA software I have no control over the underlying FXOS. I can reimage it, however as its remote it might be a bit difficult.

Does this look like another faulty device? Are there inherent issues with the FirePower 1010 platform?

Andy

balaji.bandi · ‎02-19-2021

Not sure what Code of FXOS or FTD or ASA running, But there are a couple of bugs reported as below : ( please if this affecting).

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34966/?rfs=iqvred

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi61729/?rfs=iqvred

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Sheraz.Salim · ‎02-19-2021

Dont you have a TAC support? If TAC support avaiable worth open a case with them. seem like you hitting a bug on this problemetic unit https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34966/?rfs=iqvred

please do not forget to rate.

harmesh88 · ‎02-20-2021

Please try to upgrade Device with Latest version , Bug is related to FXOS so when you have latest FXOS it will remove bug .

if you will get same issue again then only tac will be last option.

I hope it will help you !!

ang7127 · ‎03-02-2021

My TAC case resulted in CSCvx34966 being filed - running latest ASA (9.14(2)8) on FP1010. TAC is telling me to factory-reset the device to hopefully clear the full FXOS partitions in question and restore from backup, or RMA the device so they can further diagnose. This issue prevents login to the underlying FXOS subsystem, so I cannot remove the logs in question. All attempts to 'connect fxos admin' fail and there appears to be no work-around other than factory-reset. The ASA running on FXOS appears stable, for now...

andrew.butterworth · ‎03-02-2021

I managed to 'factory reset' them both and from ROMMON via a USB stick loaded with the 9.15(1)7 software, I re-imaged them. So far they have not experienced the errors I was seeing with 9.15(1). I have configured them up as a HA pair and connected the management interfaces and then added them to our management platform. I'm going to leave them for a couple of weeks with just management connected and see what happens.

balaji.bandi · ‎03-02-2021

that is good work, glad all good for now.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Sheraz.Salim · ‎03-02-2021

Happy day. all the best..

please do not forget to rate.

kerstin-534 · ‎01-18-2023

"no space left on device" problems can be resolved with TAC and only with TAC. They do such command

firepower-1010(local-mgmt)# secure-login

Challenge String (Please copy everything between the asterisk lines exclusively) :

********************************************************************************
Iuf//wYAAAAp3+0C5t20r1fsCfIdgminySBekByLUrD2zpJzPVJdcbv51bZGWE9T

and delete /var/log/messages.*

There problem is here that customer has to access to Linux. They should have access to Linux to resolve this problem themselves.