Firepower Management Center - very high CPU usage

[fabian.seeber]

hello team,

i want to use the FMC with Firepower. Now i tested the system with only 1 firepower module (hardware) and i always have a CPU usage with an average of 80%. It's so high, that my system is to slow to handle it. The Software of the FMC and firepower is all actual and updated.

The FMC has its own server. ESXI, 8core CPU, 32GB RAM and 500GB HD. So i doubled the hardware that you need to handle it. no effect!

I added 2 pictures from the usage.

Please help me. Dont know what can help now. Thats only one module and i want to use maybe 5 in the future with the same FMC.

best regards, Fabian

[Jetsy Mathew · Accepted solution]

Hello Fabian,

Based on the software version that you are using check the compatibility of the software versions that you are using. Make sure the system requirements met according to the release notes. Also use the top command in the Firepower cli to confirm the process which are consuming high cpu.

Also check the policies that you have configured. You can try creating a test rule and apply the Balanced Security & Connectivity rules  to confirm if the policies are causing the CPU spike.

Let me know if you have any questions.

Rate if this answer helps.

Regards

Jetsy 

[fabian.seeber]

Hello Jetsy,

thanks for your answer. I checked the the asa processes and took a screenshot. There is an average of maybe 3% CPU usage. As you can see is that the command "snort" uses the most CPU. I tried dirrent software version and always got this high usage. sometimes its more, and sometimes its not that much.

What can you say about the campatibility? There is no information at the cisco hp about the hardware. Only: 8GB RAM, 4 core CPU and 250GB HD. And i think i have enough power. the FMC is running at a single virtual machine on his own server! there is nothing else on this machine.

I think that the database is using this high cpu level. because every time i logged in at the web interface, the usage raises up.

best regards,

Fabian

[Jetsy Mathew]

Hello Fabian ,

During the traffic inspection if the snort consumes the CPU , then its very normal . Snort handles the traffic inspection and thus if the inspection is happening and if the CPU is bit high that time , then its very normal. Is the usage is always high every-time or is it goes down gradually during off business hours ?

If the usage goes down during off hours then there is nothing to worry. Snort will consume CPU when the  detection is happening and its fine.

Regards with the database usage , can you see the usage kind of hitting 90% or above ?

Are you seeing some latency in the Web UI , that we can verify using the database troubleshooting . Run the following command from the FMC cli.

FMC@123# DBCheck.pl

See if you are observing any fatal errors in the output.If you are seeing any Fatal errors then it can be a problem with the database,then please open a TAC case to troubleshoot it .

Rate if this answer helps.

Regards

Jetsy 

[Marvin Rhoads]

Nice tip Jetsy - I hadn't seen that one.

n.b. note the # prompt - it must be run with root privileges so just "sudo su" first to change to superuser.

Here's the output from a healthy FMC:

Cisco Fire Linux OS v6.2.1 (build 6)
Cisco Firepower Management Center for VMWare v6.2.1 (build 342)

admin@sfvdc:~$ sudo su
Password: 
root@sfvdc:/Volume/home/admin# DBCheck.pl
running database integrity check with the following options:
- use exception directory /usr/local/sf/etc/db_exceptions
- check refererences
- check enterprise objects
- check schema
- check required data
- log to stderr
getting filenames from [/usr/local/sf/etc/db_updates/index]
getting filenames from [/usr/local/sf/etc/db_updates/base-6.2.1]
getting exceptions from [/usr/local/sf/etc/db_exceptions/db_exceptions.yaml]
After Checking DB, Warnings: 0, Fatal Errors: 0
root@sfvdc:/Volume/home/admin# 

[Jetsy Mathew]

Hello Fabian

If you are facing issues while accessing the GUI , then this can be due to slow queries.

We need the troubleshoot file to verify the database slow queries.

Regards

Jetsy 

[fabian.seeber]

where can i get the troubleshoot file?

[Jetsy Mathew]

Hello Fabian ,

Open a cisco TAC case and submit the troubleshoot file as per the following link.

http://www.cisco.com/c/en/us/support/docs/security/sourcefire-defense-center/117663-technote-SourceFire-00.html

Let me know if you have any questions

Regards

Jetsy 

[fabian.seeber]

dear jetsy and marvin,

thanks for the answers. I will check the DB in the next hour. But i already did that weeks ago with no effect and without any errors.

Marvin, whats your CPU usage average and which hardware do you use with your FMC?

best regards

[fabian.seeber]

i did it - no effect. no fatal errors. I attached a screenshot.

@Jetsy

Yes, the health monitor at the webinterface always alerts the CPU usage at over 90%.

I attached a screenshot for that, too.

[Marvin Rhoads]

[@fabian.seeber@web.de]  

My office FMC runs as a VM with the minimal specs. (8 GB of RAM, 4 vCPUs and 250 GB hard drive). Load is averaging about 8% per core.

I'm only monitoring one 5512-X (corporate office with about 50 users and 20 Mbps of Internet) a couple of lab ASAs that don't have much if any traffic day to day.

I also checked a customer FMC that's another VM. They have 2 production data centers with ASA 5525-X HA pairs in each and a corporate office with a third. They are pushing a good bit more traffic and their FMC load is 2% per core across 8 cores.

Haver you customized your IPS policies much? Most of my IPS policies are "Balanced Security and Connectivity" and we almost never do much customization at the IPS policy level. I could see the Snort process being affected by customization there.

[fabian.seeber]

Thats crazy. I doubled the minimal hardware specs with the VM.

There is only a 5506-X ASA connected. Nothing special. Only about 50 users in the network an 5 Mbps internet. So we dont have much traffic. Everxthing is normal (HD, RAM, Network...) only the CPU using is that high on every CPU.

I tried now to sync the firepower and FMC time from the DC. Maybe that helps - atm it restarts the FMC.

We dont have that much IPS policies. At the beginning we disabled IPS and had the same problem.

[Marvin Rhoads]

I wonder if it could be something with respect to VMware and the hardware you are using.

Is the server type and associated hardware one you have used in other ESXi installations?

[fabian.seeber]

the hardware is okay... i used it for other ESXi installations in the past.

Yesterday, I sync the FMC and FP time with the domaincontroler. In the night, the avarage was at about 20%. Now after i logged in to the webinterface - 90%. Everytime i want to see something from the web or the database. its slow and the average raises up to 95% per CPU.

I tried to reinstall - no effect.

tried another server - no effect.

tried with or without updates - no effect.

[NeWGuy1109]

Sorry to ask this here.. but i think my issue is some what similar.
In my FMC managing 2 FTD devices i am getting the error "High Memory Utilization Physical + Swap". How can i resolve this ? is it possible to free memory from CLI without disrupting production environment.