Firepower ASA 2110 completely broken

[StephanBretonCNS]

Hello,

I have 2 x FPR2110-ASA-K9 and I'm trying to set up an active/standby state between them but unfortunately I face a lot of issues.

Versions :

• Cisco Adaptive Security Appliance Software Version 9.8(2)
• Firepower Extensible Operating System Version 2.2(2.52)
• Device Manager Version 7.8(2)

1. FXOS CLI is broken (Timed out communicating with DME)

I cannot run (almost) any command through FXOS CLI. I constantly get the error message "Software Error: Exception during execution: [Error: Timed out communicating with DME]". I found some related bugs (especially CSCvs61701 and CSCul61847) but either there is no workaround or the command doesn't exist. I wanted to do a fresh install but apparently it needs some commands to be applied on FXOS side... which I can't do.

Ex :

firepower# show eth-uplink
Software Error: Exception during execution: [Error: Timed out communicating with DME]

DME service is set to failed state (which is different compared to crashed state) :

firepower(local-mgmt)# show pmon state

SERVICE NAME STATE RETRY(MAX) EXITCODE SIGNAL CORE
------------ ----- ---------- -------- ------ ----
svc_sam_dme failed 4(4) 0 11 yes
svc_sam_dcosAG running 0(4) 0 0 no
svc_sam_portAG running 0(4) 0 0 no
svc_sam_statsAG running 0(4) 0 0 no
svc_sam_licenseAG running 0(4) 0 0 no
httpd.sh running 0(4) 0 0 no
svc_sam_sessionmgrAG running 0(4) 0 0 no
sam_core_mon running 0(4) 0 0 no
svc_sam_svcmonAG running 0(4) 0 0 no
svc_sam_serviceOrchAG running 0(4) 0 0 no
svc_sam_appAG running 0(4) 0 0 no
svc_sam_envAG running 0(4) 0 0 no

Obviously I tried to reboot many many times and nothing changed. Since I cannot upgrade or do a fresh install, what should I do ? Do I miss something ?

2. FCM GUI infinite loading

Firepower Chassis Management GUI is also broken because it shows an infinite loading on the login page, like the "Login" button is grey and it is impossible to click on it. Probably related to the first point. I tried with different computers and browsers.

3. No route on my routing table (ASA)

I know the conditions that a route needs to match in order to be installed in the routing table. But the thing is that I am currently remotely connected to the firewall but no route seems installed on my routing table :

firepower# sh route

[...]
Gateway of last resort is not set


firepower# sh run | i route
route INSIDE 10.0.0.0 255.0.0.0 10.1.1.1 1
route MGMT 10.0.0.0 255.0.0.0 10.2.2.2 2
timeout igp stale-route 0:01:10

It is expected that I don't have a gateway of last resort (because I cant enable the Internet Interface on FXOS side...).

I suspect this to be my issue when trying to communicate with our License server (On-Prem Manager) because I can't see any request that goes outside the FW when forcing the token or renewing the authentication.

If you need anything just tell me, thank you!

[johnlloyd_13]

hi,

does your FW have smartnet? it would be prudent to open a TAC case to resolve your problems.

[StephanBretonCNS]

We do not have order them so I believe we don't have. How can I make sure ? Can I try to look for it somewhere with serial numbers ?

[Sheraz.Salim]

This sound very strange issue. I have worked with FTD-2110 and encounter issue. but this seem to be the strangest one. If you have contract in place with cisco (smartNet) or any PSS with your partner (cisco partner) escalate it to them. So you can get involve the TAC or get the RMA for these appliances. 

The bug  CSCvs61701 FIPS/trustpoint but in your case you cant access the config due to error you getting.

[StephanBretonCNS]

"Workaround: None" .....

I will try to open a case but we need to buy a maintenance contract first.

[Sheraz.Salim]

If they are brand new firewall. Just let cisco know they are not fit for purpose due to these issues you encounter. if these arrived within 2 to 4 week. you can raise the issue with cisco or the third party who were involved to bought these appliances. As it stand its not your issue the appliances shipped faulty.

[StephanBretonCNS]

We ordered them back in November 2021 and we were only able to configure them this month... My supplier don't want/can't open a case.

[Sheraz.Salim]

Yes I understand due to project pipe line and other thing inlne. sometime the hardware is just in IT store room. I completely understand this.
On the other side I guess you have not much options to get these appliances under support with cisco and RMA them. Shame the appliance does not seem to be fit in purpose. And this will postpone all the planning work you was going to under take.

[Marvin Rhoads]

Are these new (out of the box) and behaving this way or have they been reimaged from an earlier working state?

I've worked with a number of Firepower devices running ASA image and have never seen errors like the ones you cite.

[StephanBretonCNS]

These are brand new Firepower that have never been reimaged.

[tvotna]

Obviously, if DME process constantly fails on your appliance, you cannot use CLI or FCM and entire appliance becomes unusable.

You can try to reimage it using ROMMON:

https://www.cisco.com/c/en/us/td/docs/security/firepower/2100/troubleshoot_fxos/b_2100_CLI_Troubleshoot/b_2100_CLI_Troubleshoot_chapter_011.html#task_ryc_5wm_1jb

This procedure will replace both FXOS and ASA versions. Configuration will be lost. You can try latest ASA 9.16.3 interim which is currently marked goldstar release. This will include FXOS 2.10.1.something. Or you can try older ASA 9.12 and this will include FXOS 2.6.1.something. I wouldn't recommend other versions. BTW, I'm not sure how Cisco updates FXOS when new ASA or FTD version for FP2100/1000 is released and if we have a table somewhere which documents all version pairs. Having this table would help to workaround few known FXOS bugs. Other board members are welcome to comment on this.

[Marvin Rhoads]

@tvotna there is a table with the ASA to bundled FXOS version found here:

https://www.cisco.com/c/en/us/td/docs/security/asa/compatibility/asamatrx.html#id_65802

[tvotna]

Hi Marvin,

Right, but it doesn't list ASA interims. Only minor versions are listed there. I'm not really sure whether Cisco updates FXOS when ASA interim is released, or not.

[Marvin Rhoads]

@tvotna I'm not sure about interim builds updating FXOS (or not).

They do only include bug fixes; but ostensibly there could be an FXOS bug that needs to be fixed and Cisco uses an interim ASA build to accomplish that. However, looking at several ASA interim build release notes I didn't notice any that also mention FXOS bug fixes. So I'd be inclined to believe ASA interim builds do not update FXOS.

For example:

https://www.cisco.com/web/software/280775065/160837/ASA-9163-Interim-Release-Notes.html

[tvotna]

Upon some thinking I realized they do update FXOS. For example, let's take a look at famous https://www.cisco.com/c/en/us/support/docs/field-notices/722/fn72282.html and the defect https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvu84127. The defect is fixed in FXOS 002.006(001.245) and ASA 009.012(004.039). Compatibility table https://www.cisco.com/c/en/us/td/docs/security/asa/compatibility/asamatrx.html shows the following:

9.12(4)

2.6(1.198)

This means that FXOS was updated when interim was released. But it looks like the correspondence between ASA interims and FXOS versions isn't documented in a single place and we need to reverse engineer it from bug toolkit. Typical for Cisco.