I am not having much luck with the FirePower 1010 platform. I have had to RMA two devices so far and I've had an unrecoverable software crash on a third.
I'm now back to two devices - one from the original delivery and a second new one as we have had two DoAs. As a workaround I have put in a temporary pair of ASA5515-X's that had been decommissioned from another customer but I need to get them swapped out for these FP1010's. I currently have them offline, although there is a cable connecting them for HA and I have console access to them both.
I logged onto the existing one and the console was full of these messages
[1355831.468897] spi_read offset:0x812000 offset:0x400
[1355831.531000] Read: address 0x812000, length 0x400
[1355831.591794] read ret length = 1024
[1355833.117410] spi_read offset:0x812400 offset:0x400
[1355833.179538] Read: address 0x812400, length 0x400
[1355833.240349] read ret length = 1024
[1355834.562105] spi_read offset:0x812800 offset:0x110
[1355834.624197] Read: address 0x812800, length 0x110
[1355834.683706] read ret length = 272
[1355837.242258] spi_fs_ioctl spi_block_size
[1355837.554652] spi_read offset:0x810000 offset:0x400
[1355837.616753] Read: address 0x810000, length 0x400
[1355837.677513] read ret length = 1024
[1355839.850695] spi_read offset:0x810400 offset:0x400
[1355839.912783] Read: address 0x810400, length 0x400
[1355839.973549] read ret length = 1024
[1355841.049534] spi_read offset:0x810800 offset:0x400
[1355841.111607] Read: address 0x810800, length 0x400
[1355841.172367] read ret length = 1024
[1355841.718938] spi_read offset:0x810c00 offset:0x400
[1355841.781017] Read: address 0x810c00, length 0x400
[1355841.841774] read ret length = 1024
I managed to login and reload, however it didn't reload and just hung with the messages repeating continually. I power reset the device and on boot up got various error messages - sample of some here:
/bin/ps: write error: No space left on device
/bin/ps: write error: No space left on device
/etc/init.d/start_monitor_confreg.sh: line 29: echo: write error: No space left on device
SSP-Security-Module is shutting down ...
Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Beginning System Shutdown request for CSP Apps
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
Fri Feb 19 11:54:22 UTC 2021 SHUTDOWN WARNING: Continue System Shutdown request for CSP Apps
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 74: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 77: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_utils.sh: line 58: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device
grep: write error
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 49: echo: write error: No space left on device
grep: write error
/usr/bin/tr: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 13: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_pm_killall.sh: line 26: echo: write error: No space left on device
grep: write error
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/bin/rm: cannot remove '/tmp/openssl.conf': No such file or directory
httpdRegister INFO: [httpd.3744 -s -4 0.0.0.0 -n localhost]
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
httpdRegister INFO: SKIP httpd syntax check
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
httpdRegister INFO: Starting httpd setup/registration...
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
httpdRegister INFO: Completed httpd setup/registration!
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
INFO: httpdRegister [httpd.3744 script exit]
/isan/bin/httpdRegister.sh: line 75: echo: write error: No space left on device
INFO: manager_startup: Completed manager httpd setup!
Starting crond: OK
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 260: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_utils.sh: line 250: echo: write error: No space left on device
1:/opt/cisco/csp/cores
/opt/cisco/csp/cores 31457280
/bin/cat: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_vnic_validate.sh: line 55: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 66: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_rm_db.sh: line 76: echo: write error: No space left on device
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1498
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 734
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1504
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 919
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1528
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1530
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1327
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1352
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1367
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1409
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1435
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1444
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file ssp_rmdb_mgmt.py, line 1453
/opt/csp/csp_utils.sh: line 1333: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1336: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1337: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1340: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1347: echo: write error: No space left on device
/bin/ls: write error: No space left on device
/opt/csp/csp_utils.sh: line 1382: echo: write error: No space left on device
/opt/csp/csp_utils.sh: line 1386: echo: write error: No space left on device
/opt/csp/csp_startup.sh: line 75: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 138: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 139: echo: write error: No space left on device
/opt/csp/csp_mgr.sh: line 166: echo: write error: No space left on device
/bin/grep: write error: No space left on device
/opt/csp/csp_mgr.sh: line 156: echo: write error: No space left on device
▒▒▒▒▒55)▒csp_mgr.sh: line Q▒▒K▒ɥѕ▒▒▒ɽ▒▒9▒▒▒▒
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 46: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 11: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 12: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 14: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 15: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 17: echo: write error: No space left on device
/opt/ssp-mgmt-scripts/ssp_system_storage.sh: line 18: echo: write error: No space left on device
firepower-1010 login: admin (automatic login)
I did a 'write erase' and reloaded it and I get the same 'out of space' errors. I booted the new unit and I don't see any of these errors on boot.
As both devices are running ASA software I have no control over the underlying FXOS. I can reimage it, however as its remote it might be a bit difficult.
Does this look like another faulty device? Are there inherent issues with the FirePower 1010 platform?
Andy
