FirePower 1010 hardware failures?

[andrew.butterworth]

I recently installed a HA pair of FirePower 1010 devices running ASA software.

After working out how to enable the license tiers and Security-Plus feature they were working OK and I tested failover several times as well as upgraded the software to the latest ASA 9.15.1 & ASDM 7.15.1.  These were not in production and I was scheduled to replace a pair of ASA 5508-X's this morning.

The official rackmount kit was ordered but delivered after the original installation and this was installed on Friday.  This was completed by one of the DC engineers.  On powering up the devices one of them failed and the Power LED was cycling between Green & Orange.  I got a console connected but there was nothing.  I had the DC engineer take the case off the failed FP1010 and check if any components (SODIMM RAM I thought?) had come loose, however everything looked fine so we have RMA'd this device.

This morning however I have attempted to logon to the 2nd FP1010 via its OoB managment interface and cannot. The OoB L2 switch where the management interface is connected to is up but not showing any MAC addresses and the link has remained up since we powered the device back on following the rack-mount kit installation.

Our monitoring platforms show two CPU warning messages for 02:31 and 04:22 from yesterday morning followed by a loss of communication at 06:58.

I am awaiting on one of our DC engineers to do a visual inspection and to get a console connected, however I suspect this unit has also failed as our DC is unmanned at the time the events occurred.

My concern here is that this 2nd unit has failed within 2-days of the 1st so is this an inherent hardware issue with this platform, possibly a faulty batch or are we just unlucky?

This is my 1st FirePower 1010 installation, however I am familiar with ASA's and the larger FirePower platforms so I don't think I am missing anything with regards to their operation.

Andy

[Mohammed al Baqari]

Hi,

I have seen this on some 1010 devices (not on others within the same
series). It might be DB corrupted for many reasons or hardware failures.
This should be indicated during console boot if you can get console logs.

You can try to reimage the unit if you want with ASA or FTD images. If its
not working, then go for RMA.

Here is the link to do this.

https://www.cisco.com/c/en/us/td/docs/security/firepower/quick_start/reimage/asa-ftd-reimage.html

***** please remember to rate useful posts

[andrew.butterworth]

The unit that failed doesn't output anything to the console and the Power LED cycling Green & Orange I believe indicates a major hardware failure.

Hopefully I'll get a console connected to the other unit and I'll see if this responds on the console.  I must admit though I am concerned about the reliability of this platform?

[Mohammed al Baqari]

I got reports about 1010 not performing well. However, my experience on
1140 and 1150 is excellent. With 15 units deployed, not single failure with
6.6.1 image.

**** please remember to rate useful

[andrew.butterworth]

These are running ASA software, however there should be some output to the console on boot up.

I have a couple of ASA5506-X's in the lab that I have reimaged a few times and there is always output on the console during boot up.

I think I have a pair of lemons here tbh....

Andy

[andrew.butterworth]

Managed to get a console connected but no response.  However rebooted it and it has come back.  There is a Crashinfo file in the flash that corresponds with when our monitoring platform lost contact with it.  This is in the file:

------------------ show crashinfo ------------------

Thread Name: Cluster event-queue processing thread
Page fault: Address not mapped

r8 0x0000000000000000
r9 0x0000000000000000
r10 0x0000000000000008
r11 0x0000000000003246
r12 0x00007f96d3201640
r13 0x000000000000f79f
r14 0x0000000000000000
r15 0x00007f96d1cc4d00
rdi 0x000055661d123e7c
rsi 0x00007f9761f92ed0
rbp 0x00007f9761f92f70
rbx 0x00007f96e8e66640
rdx 0x0000000000000000
rax 0x0000000600000000
rcx 0x00007f9761d9e180
rsp 0x00007f9761f92eb8
rip 0x0000000600000000
eflags 0x0000000000013206
csgsfs 0x002b000000000033
error code 0x0000000000000014
vector 0x000000000000000e
old mask 0xffffffde3e3ada05
cr2 0x0000000600000000

I am guessing this is either a bug in the OS or a hardware issue?

I have 9.15.1 installed on a couple of ASA5508-X's for a few months and haven't seen this issue.  Just checked and one has been up 73-days running 9.15.1.

Cheers

Andy

[Mohammed al Baqari]

Hi,

This is a memory error. Try to reimage and see bit seems that memory is
corrupted

[ewaizel]

We had the same problem and this was caused by using the wrong screws that were included in the Cisco mounting kit. The silver screws are a few millimeters longer and damaged the motherboard. The black ones should be used instead.

This is a mechanical design defect. Cisco should not include exactly 6 silver screws of the same diameter of the black ones. I suggest Cisco to redesign this and use ticker silver screws for the cable manager. These will never fit into the FW and will prevent any damage.

[muryelvelasco]

I have the same issue here this year..04 firewalls 1000 series .... so a realized the motherboard are damaged thanks to this wrong sized screws.  Thanks for the advice about this issue. 

[muryelvelasco]

Did have any support from Cisco to solve this issue?