02-19-2021 04:30 AM
I am not having much luck with the FirePower 1010 platform. I have had to RMA two devices so far and I've had an unrecoverable software crash on a third.
I'm now back to two devices - one from the original delivery and a second new one as we have had two DoAs. As a workaround I have put in a temporary pair of ASA5515-X's that had been decommissioned from another customer but I need to get them swapped out for these FP1010's. I currently have them offline, although there is a cable connecting them for HA and I have console access to them both.
I logged onto the existing one and the console was full of these messages
[1355831.468897] spi_read offset:0x812000 offset:0x400 [1355831.531000] Read: address 0x812000, length 0x400 [1355831.591794] read ret length = 1024 [1355833.117410] spi_read offset:0x812400 offset:0x400 [1355833.179538] Read: address 0x812400, length 0x400 [1355833.240349] read ret length = 1024 [1355834.562105] spi_read offset:0x812800 offset:0x110 [1355834.624197] Read: address 0x812800, length 0x110 [1355834.683706] read ret length = 272 [1355837.242258] spi_fs_ioctl spi_block_size [1355837.554652] spi_read offset:0x810000 offset:0x400 [1355837.616753] Read: address 0x810000, length 0x400 [1355837.677513] read ret length = 1024 [1355839.850695] spi_read offset:0x810400 offset:0x400 [1355839.912783] Read: address 0x810400, length 0x400 [1355839.973549] read ret length = 1024 [1355841.049534] spi_read offset:0x810800 offset:0x400 [1355841.111607] Read: address 0x810800, length 0x400 [1355841.172367] read ret length = 1024 [1355841.718938] spi_read offset:0x810c00 offset:0x400 [1355841.781017] Read: address 0x810c00, length 0x400 [1355841.841774] read ret length = 1024
I managed to login and reload, however it didn't reload and just hung with the messages repeating continually. I power reset the device and on boot up got various error messages - sample of some here:
/bin/ps: write error: No space left on device /bin/ps: write error: No space left on device /etc/init.d/start_monitor_confreg.sh: line 29: echo: write error: No space left on device SSP-Security-Module is shutting down ... Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Beginning System Shutdown request for CSP Apps /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Continue System Shutdown request for CSP Apps /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device /bin/ls: write error: No space left on device /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device /bin/ls: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device grep: write error /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 49: echo: write error: No space left on device grep: write error /usr/bin/tr: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device grep: write error /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device /bin/rm: cannot remove '/tmp/openssl.conf': No such file or directory httpdRegister INFO: [httpd.3744 -s -4 0.0.0.0 -n localhost] /isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device httpdRegister INFO: SKIP httpd syntax check /isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device httpdRegister INFO: Starting httpd setup/registration... /isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device httpdRegister INFO: Completed httpd setup/registration! /isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device INFO: httpdRegister [httpd.3744 script exit] /isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device INFO: manager_startup: Completed manager httpd setup! Starting crond: OK /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device 1:/opt/cisco/csp/cores /opt/cisco/csp/cores 31457280 /bin/cat: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_vnic_validate.sh: line 55: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 76: echo: write error: No space left on device Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1498 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 734 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1504 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 919 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1528 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1530 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1327 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1352 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1367 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1409 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1435 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1444 Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file ssp_rmdb_mgmt.py, line 1453 /opt/csp/csp_utils.sh: line 1333: echo: write error: No space left on device /opt/csp/csp_utils.sh: line 1336: echo: write error: No space left on device /opt/csp/csp_utils.sh: line 1337: echo: write error: No space left on device /opt/csp/csp_utils.sh: line 1340: echo: write error: No space left on device /opt/csp/csp_utils.sh: line 1347: echo: write error: No space left on device /bin/ls: write error: No space left on device /opt/csp/csp_utils.sh: line 1382: echo: write error: No space left on device /opt/csp/csp_utils.sh: line 1386: echo: write error: No space left on device /opt/csp/csp_startup.sh: line 75: echo: write error: No space left on device /opt/csp/csp_mgr.sh: line 138: echo: write error: No space left on device /opt/csp/csp_mgr.sh: line 139: echo: write error: No space left on device /opt/csp/csp_mgr.sh: line 166: echo: write error: No space left on device /bin/grep: write error: No space left on device /opt/csp/csp_mgr.sh: line 156: echo: write error: No space left on device ▒▒▒▒▒55)▒csp_mgr.sh: line Q▒▒K▒ɥѕ▒▒▒ɽ▒▒9▒▒▒▒ /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 46: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 11: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 12: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 14: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 15: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 17: echo: write error: No space left on device /opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 18: echo: write error: No space left on device firepower-1010 login: admin (automatic login)
I did a 'write erase' and reloaded it and I get the same 'out of space' errors. I booted the new unit and I don't see any of these errors on boot.
As both devices are running ASA software I have no control over the underlying FXOS. I can reimage it, however as its remote it might be a bit difficult.
Does this look like another faulty device? Are there inherent issues with the FirePower 1010 platform?
Andy
02-19-2021 05:03 AM
Not sure what Code of FXOS or FTD or ASA running, But there are a couple of bugs reported as below : ( please if this affecting).
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34966/?rfs=iqvred
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi61729/?rfs=iqvred
02-19-2021 06:00 AM - edited 02-19-2021 06:01 AM
Dont you have a TAC support? If TAC support avaiable worth open a case with them. seem like you hitting a bug on this problemetic unit https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34966/?rfs=iqvred
02-20-2021 11:00 PM - edited 02-20-2021 11:00 PM
Please try to upgrade Device with Latest version , Bug is related to FXOS so when you have latest FXOS it will remove bug .
if you will get same issue again then only tac will be last option.
I hope it will help you !!
03-02-2021 02:47 PM - edited 03-02-2021 03:00 PM
My TAC case resulted in CSCvx34966 being filed - running latest ASA (9.14(2)8) on FP1010. TAC is telling me to factory-reset the device to hopefully clear the full FXOS partitions in question and restore from backup, or RMA the device so they can further diagnose. This issue prevents login to the underlying FXOS subsystem, so I cannot remove the logs in question. All attempts to 'connect fxos admin' fail and there appears to be no work-around other than factory-reset. The ASA running on FXOS appears stable, for now...
03-02-2021 03:41 PM
I managed to 'factory reset' them both and from ROMMON via a USB stick loaded with the 9.15(1)7 software, I re-imaged them. So far they have not experienced the errors I was seeing with 9.15(1). I have configured them up as a HA pair and connected the management interfaces and then added them to our management platform. I'm going to leave them for a couple of weeks with just management connected and see what happens.
03-02-2021 04:36 PM
that is good work, glad all good for now.
03-02-2021 11:48 PM
Happy day. all the best..
01-18-2023 01:06 AM
"no space left on device" problems can be resolved with TAC and only with TAC. They do such command
firepower-1010(local-mgmt)# secure-login
Challenge String (Please copy everything between the asterisk lines exclusively) :
********************************************************************************
Iuf//wYAAAAp3+0C5t20r1fsCfIdgminySBekByLUrD2zpJzPVJdcbv51bZGWE9T
and delete /var/log/messages.*
There problem is here that customer has to access to Linux. They should have access to Linux to resolve this problem themselves.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: