Firepower SFR Module - No connection events in FMC event viewer?

[mattw]

Hi,

I'm looking into a issue where no connection events are shown in the FMC event viewer despite the configuration of the ASA and FMC looking good:

ASA5515 running 9.12(3)12

SFR module running 6.4.0.9

FMCv running 6.4.0.9

---

Service policy is configured in fail-open monitor-only mode and I see packets incrementing:

policy-map global_policy
  class sfr_map
    sfr fail-open monitor-only

FW# show service-policy sfr

Global policy:
Service-policy: global_policy
Class-map: sfr_map
SFR: card status Up, mode fail-open monitor-only
packet input 0, packet output 102498529867, drop 0, reset-drop 0

---

There is an any any MONITOR rule at the top of the ACP but I see absolutely no events in the event viewer on the FMC. I'm really struggling to figure out why nothing is being logged.

Can anyone suggest what to look at??

Many thanks in advance,

Matt.

[Marvin Rhoads]

Does your rule in the ACP have logging turned on? It's off by default.

If that's OK then check the clock on both the ASA and FMC to ensure they match. The Firepower service module will take its clock time from the ASA.

[mattw]

Hi Marvin,

Thank you for the response. ACP rule does have logging enabled. Looks like the timezones are different:

FW# show clock
13:01:34.359 GMT/BDT Fri Oct 15 2021

---

> show time
UTC - Fri Oct 15 12:01:36 UTC 2021
Localtime - Fri Oct 15 08:01:37 EDT 2021

Do you think this could be part of the problem?

[Marvin Rhoads]

The time zone shouldn't affect the logging as events are sent with UTC timestamps and adjusted per the timezone of the user's settings in FMC.

Has this ever worked? Is it the only managed device in your FMC?

[mattw]

OK cool. I don't believe this has ever worked, no.

There are 2 ASAs configured as an active/standby pair.

Both FPR modules are known by the FMC but only these two, nothing else.

Any chance some old module config on the ASA for IPS and CXSC could be causing an issue?:

---

FW# sh run
service-module 0 keepalive-timeout 4
service-module 0 keepalive-counter 6
service-module ips keepalive-timeout 4
service-module ips keepalive-counter 6
service-module cxsc keepalive-timeout 4
service-module cxsc keepalive-counter 6
service-module sfr keepalive-timeout 4
service-module sfr keepalive-counter 6

bdavpn# show module

Mod Card Type Model Serial No.
---- -------------------------------------------- ------------------ -----------
0 ASA 5515-X with SW, 6 GE Data, 1 GE Mgmt, AC ASA5515 FCH19257K3C
ips Unknown N/A FCH19419K3D
cxsc Unknown N/A FCH19419K3D
sfr FirePOWER Services Software Module ASA5515 FCH19419K3D

Mod MAC Address Range Hw Version Fw Version Sw Version
---- --------------------------------- ------------ ------------ ---------------
0 188b.9d72.9b38 to 188b.9d72.9b3f 1.0 2.1(9)8 9.12(3)12
ips 188b.9d72.9b36 to 188b.9d72.9b36 N/A N/A
cxsc 188b.9d72.9b36 to 188b.9d72.9b36 N/A N/A
sfr 188b.9d72.9b36 to 188b.9d72.9b36 N/A N/A 6.4.0.9-62

Mod SSM Application Name Status SSM Application Version
---- ------------------------------ ---------------- --------------------------
ips Unknown No Image Present Not Applicable
cxsc Unknown No Image Present Not Applicable
sfr ASA FirePOWER Up 6.4.0.9-62

Mod Status Data Plane Status Compatibility
---- ------------------ --------------------- -------------
0 Up Sys Not Applicable
ips Unresponsive Not Applicable
cxsc Unresponsive Not Applicable
sfr Up Up

Mod License Name License Status Time Remaining
---- -------------- --------------- ---------------
ips IPS Module Disabled perpetual

[Marvin Rhoads]

The old modules types are not installed - they're only listed in the output for consistency across older and newer ASAs.

Do you have your class-map defined?

"show run class-map sfr_map" will give us that info.

By the way, it should pretty much follow this guide:

https://www.cisco.com/c/en/us/support/docs/security/asa-firepower-services/118644-configure-firepower-00.html#anc12

[mattw]

Hi @Marvin Rhoads,

Thank you again for your suggestions. Today we carried out some testing:

In the applied ACP, I added a specific block rule from x.x.x.x to y.y.y.y and observed that this DID block the traffic flow so I know that the traffic is punted up from the ASA to the SFR module and the SFR module is doing what it's supposed to do.

However, despite logging being enabled on that rule to both the FMC event viewer and also a syslog server, we did not see any events logged.

I can also see from the output of "show access-control-config" that there are hits against the rule and confirmation that logging is enabled:

####################################################################

===============[ Rule Set: (User) ]================

--------------[ Rule: matt-allow-any ]--------------
Action : Allow
Intrusion Policy : Matt-IPS-Policy
  ISE Metadata :

Logging Configuration
DC : Enabled
  Beginning : Enabled
  End : Enabled
  Files : Disabled
Safe Search : No
Rule Hits : 48287
Variable Set : Matt-Variable-Set

####################################################################

One thing that is odd is that sometimes when I run "show summary", it says "Access control policy not yet applied." but then after I execute "show access-control-config", the output of "show summary" then changes to be what you'd expect (policy info, plus list of interfaces). 

Host discover is correctly configured but there are no hosts listed in the network map.

There is "No data" listed in any of the traffic related dashboard widgets in the FMC.

We tried rebooting the FMC - no change.

We tried rebooting the SFR module - no change.

I'm out of ideas here, it's driving me crazy!

Any help that you, or anybody else can provide is very much appreciated. I'm running out of hair to pull out here!

Many thanks,
Matt.

[Marius Gunnerud]

I know you said that logging is enabled, but could you verify that logging to Event Viewer is enabled under logging on the ACP rule.

[mattw]

Thank you for your suggestion @Marius Gunnerud. I can confirm that logging to event viewer is enabled under logging on the ACP rule.

[mattw]

Hi all,

Anybody got any ideas on this?

Unfortunately the kit is not under TAC support at this time otherwise I'd be raising a case. I've asked the customer to look into getting TAC support but in the meantime, I'm thinking this must be either a bug or some sort of corruption of a database somewhere as I'm fairly sure the config is good?

I've read things about corruption occurring if you make certain changes through expert mode CLI (like clock or NTP changes for example).

I'm wondering if there is any way to prove this as the cause or to rule it out?

Would I be right in thinking that all the event logging from the FPR module to the FMC goes via the SF_tunnel so I wouldn't be able to verify if events are being sent from the module or received at the FMC as it's all encrypted?

Finally, what about next step assuming we can't get TAC support? I'm thinking:

1. Upgrade the firepower modules to the latest supported & compatible image (not sure this will fix any corruption of FPR DBs??)
2. Upgrade the ASAs to the latest supported & compatible image
3. Perform a full reimage on the Firepower modules using the latest version
4. Build a new FMCv to replace the existing FMCv in case there is some corruption on the FMC?

Any thoughts?

Thanks,

Matt.

[Marvin Rhoads]

If this is a new setup then I would go with your suggestion #4 and #3 in that order.

[mattw]

Hi all,

Solution (from TAC) to this was as follows just in case it help anyone else in the future...

=============================

-Checked the FMC messages:

Nov 17 12:07:38 firepower SF-IMS[11905]: [11905] SFDataCorrelator:RNAEventDatabase [ERROR] failed to initialize rna_flow_stats table

Nov 17 12:07:38 firepower SF-IMS[11905]: [11905] SFDataCorrelator:SFDataCorrelator [ERROR] Failed to initialize the dispatcher: Unhandled database error

Nov 17 12:08:39 firepower SF-IMS[12106]: [12106] SFDataCorrelator:RNAEventDatabase [ERROR] failed to initialize rna_flow_stats table

Nov 17 12:08:39 firepower SF-IMS[12106]: [12106] SFDataCorrelator:SFDataCorrelator [ERROR] Failed to initialize

- Checked similar cases and found that the command to repair:

root@firepower:/var/log#repair_table.pl -arms --optimize rna_flow_stats

- Checked the SFDataCorrelator:

 root@firepower:/var/log# pmtool status | egrep -i sfdata

SFDataCorrelator (normal) - Running 15612

- The SFDataCorrelator is running and the FMC GUI is showing data

=============================

Cheers,

Matt.