topic Re: Firepower FTD high CPU in Network Security

Firepower FTD high CPU

David Rollins — Fri, 01 Mar 2024 21:30:29 GMT

Firepower FTD CPU 07 spiked to 100% earlier today. And it corresponds to the same time there was a spike on snort03. Snort-busy Frame drops - Snort busy started averaging 100 drops/sec.

Is there a way for me to identify what traffic may have started this?

I have looked into https://community.cisco.com/t5/network-security/high-cpu-usage-in-ftd/td-p/4793469 and https://bst.cisco.com/bugsearch/bug/CSCwh58213

as possible issues. But disabling TLS server identity discovery didn't resolve it. I even tried failing over to secondary HA. Issue came right back on secondary.

So I'm certain there is current network traffic causing it. But I'm unsure how to find out. Is there a way to do a packet capture on the ASP drops? Or a command in Lina to find out what traffic is tying up snort?

We are currently running FP 4112 version 7.0.5

Re: Firepower FTD high CPU

Ruben Cocheno — Fri, 01 Mar 2024 22:53:07 GMT

@David Rollins

Another thought I have could be related to the rules if they are using the app IDs instead of the service ports?! sometimes when using the app IDs the firewall needs to see more traffic before it can understand what app ID is inside the payload, that could potentially cause some temp drops.

Re: Firepower FTD high CPU

tvotna — Sat, 02 Mar 2024 16:43:57 GMT

Good question. Lina ASP drop capture should be able to capture "snort-busy" drops. Try "capture cap-asp type asp-drop snort-busy".

Also, "show asp inspect-dp" commands have instance-id argument and can display statistics per Snort instance. The "show asp inspect-dp snort" can show which instance has high CPU and "show asp inspect-dp snort queues" displays instance queue utilization. The "show asp inspect-dp snort counters rate" displays per-queue traffic rate. If some queue is exhausted, Lina should take a snapshot of the queue automatically: "show asp inspect-dp snort queue-exhaustion". The snapshot in .pcap format can be offloaded for analysis: "show capture", "show capture <name>", "show asp inspect-dp snort queue-exhaustion export ftp://...". This can help identify problematic flows.

FMC Analysis > Search should be able to search connection events by Snort instance-id. In case of Snort3 Elephant Flow Detection feature can help identify elephant flows and display them:

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/snort/740/snort3-configuration-guide-v74/m_configure-elephant-flow-detection-and-remediation.html#view-events-for-elephant-flows

Re: Firepower FTD high CPU

David Rollins — Tue, 05 Mar 2024 19:58:40 GMT

What is the no form of the "capture cap-asp" command. Because I stopped it and wanted to redo it. I tried configuring another capture, but it says "error: another capture associated with this drop"

Re: Firepower FTD high CPU

tvotna — Wed, 06 Mar 2024 14:56:10 GMT

no capture cap-asp

Re: Firepower FTD high CPU

Max Jobs — Wed, 06 Mar 2024 15:06:28 GMT

Hi David,

This is what I know:

Use CLI Commands to Monitor CPU Usage:

Log in to the FTD device's CLI and use commands like show cpu usage or show processes cpu-usage sorted to monitor CPU utilization. Look for any processes or threads consuming excessive CPU resources.
Check Snort Statistics:

Since you noticed a spike in Snort CPU usage, check Snort statistics using the show snort statistics command. Look for any anomalies or excessive activity that might be causing the CPU spike.
Packet Capture for ASP Drops:

You can use packet capture functionality to capture traffic associated with ASP (Accelerated Security Path) drops. Use the following command to configure a capture:

capture capture_name interface ingress capture_filter

Replace capture_name with a name for the capture, interface with the interface where drops are occurring, and capture_filter with a filter to capture specific traffic (if needed).
After capturing packets, you can use Wireshark or similar tools to analyze the captured traffic and identify any anomalies or excessive traffic causing drops.
Inspect Logs for Anomalies:

Check system logs (show logging) for any unusual events or error messages that might indicate the cause of the CPU spike.
Review Configuration Changes:

Check for any recent configuration changes or updates that might have triggered the CPU spike. Roll back any recent changes if necessary to isolate the issue.
Update to Latest Version:

Consider upgrading your FTD device to the latest software version (if feasible) as newer versions often include bug fixes and performance improvements.

Re: Firepower FTD high CPU

David Rollins — Wed, 06 Mar 2024 15:23:21 GMT

I thought I had tried "no capture cap-asp" that already, but it took this time. What I did differently this time was exit out of Lina and then connected back to FTD. Thank you.

Re: Firepower FTD high CPU

MHM Cisco World — Wed, 06 Mar 2024 16:06:00 GMT

Sorry what traffic effect the Snort ?

Can you share more details

Thanks

MHM

Re: Firepower FTD high CPU

David Rollins — Wed, 06 Mar 2024 17:45:07 GMT

I wasn't able to capture the traffic in time. By the time I started capturing, whoever it was had stopped transferring the data. I really just wanted to know, so the next time it happens I can capture the traffic and identify.
The other way I thought of identifying the traffic, was through looking at the interfaces with high input and output rates. Then through FMC, doing a packet capture. Seeing what IP address was showing up the most in the packet capture. Then from Lina, doing a "show conn address x.x.x.x"" to confirm how much data has been sent.

Re: Firepower FTD high CPU

MHM Cisco World — Wed, 06 Mar 2024 17:53:09 GMT

I was just to confirm,

The capture IN and Out show traffic ingress and egress the Lina

Capture-traffic this use for traffic punt to snort,

But this way we know traffic punt to snort but we don't know which traffic high punt to snort, this from my opinion.

Can you share show conn

Only few lines

MHM

Re: Firepower FTD high CPU

tvotna — Thu, 07 Mar 2024 06:30:33 GMT

Yeah, regular capture can be helpful too. You can also use "show traffic" to see the rate per interface as well as aggregated rate on Internal-Data0/0 which connects ASA to the internal switch. But do "clear traffic" first, otherwise the rate won't be correct.

You can also try EXEC CLI "conn data-rate". This enables per conn traffic rate tracking. Then use:

show conn {long | detail} [data-rate-filter {lt | eq | gt} <value in bytes per second>]