topic Re: Firepower Congestion in Network Security

Firepower Congestion

sysad43 — Tue, 13 Aug 2024 13:40:27 GMT

ASA 5555x with firepower - 9.14 latest interim
SFR modules and FPMCv at 6.4 latest interim

Suddenly on a Monday morning at 8am, latency through our ASA 5555x firewall spiked from its usual <10ms to over 3000ms and started dropping packets. Voice calls failed. We started getting alerts from systems we monitored, and internet connectivity was slow to nonexistent. This is when most of our users log in, connection rates go up to 100-200/s total connections go from about 8000 to 20000. We had not made any recent changes other than upgrading some anyconnect clients to 5.1 version.

Now, the problem happens every monday at 8am, and calms down by about 9am. And also happens a little less at 8am every work day. It happend on a tuesday at 8am when we were closed on monday, so its certainly correlated to increased internal traffic going out.

To try and resolve we made sure we were on latest fixes (firepower pending), looked for DDOS, shunned some internal IPs tripping threat detection rates, but nothing has worked except bypassing the firepower module entirely. So, we think the issue is thus with the firepower module, or the config, but since nothing changed, we cant find the reason.

Where should I look to try and get more information on whats going on?

Re: Firepower Congestion

sysad43 — Tue, 13 Aug 2024 13:41:14 GMT

Also, we are working with our 3rd party cisco support engineer, but have not been able to resolve this yet.

Re: Firepower Congestion

MHM Cisco World — Tue, 13 Aug 2024 15:04:38 GMT

You mentioned that the traffic internal make asa high utilize' then you need to change the GW of host into other L3 device and make asa only inspect traffic when host want to access internet.

MHM

Re: Firepower Congestion

james.king14 — Tue, 13 Aug 2024 15:58:30 GMT

Did you try the troubleshooting of the ASA modules? Of course! Try these
steps on the module to id what is causing this.

ASA FirePOWER Module (SFR) Troubleshoot File Generation Procedures using
ASDM (On-box Management) (cisco.com)
<>

Firepower Data Path Troubleshooting: Overview - Cisco
<>

Troubleshooting ASA FirePOWER modules | CCIE Security Blog (cciesecblog.com)
<>

Re: Firepower Congestion

sysad43 — Tue, 13 Aug 2024 16:49:29 GMT

Thanks, I will take a look.

Re: Firepower Congestion

sysad43 — Thu, 15 Aug 2024 20:22:54 GMT

No real new information from testing, so next step is we are going to update FPMC and SFR modules and then change the service policy to bypass inspection for some traffic. So far, disabling inspection entirely is only thing that resolves this.

Re: Firepower Congestion

MHM Cisco World — Sat, 17 Aug 2024 14:10:02 GMT

Show conn long

See the top ten traffic rate' if the traffic is internal then as I suggest before make internal traffic bypass ftd (change GW of internal host)

MHM

Re: Firepower Congestion

sysad43 — Mon, 19 Aug 2024 13:41:09 GMT

Updating FPMC and SFR and/or bypassing SFR for some traffic so far seems to have resolved it.