Firepower 4145 (FPR4K-SM-44S) ASA software high cpu usage?

[balazs.nacsa]

Hello,

I have trouble with Firepower 4145 ASA software high CPU usage.

Hardware: FPR4K-SM-44S, 348160 MB RAM, CPU Xeon 4100/6100/8100 series 2100 MHz, 2 CPUs (88 cores)

Cisco Adaptive Security Appliance Software Version 9.12(4)10
SSP Operating System Version 2.9(1.131)
Device Manager Version 7.15(1)150

Compiled on Thu 19-Nov-20 19:05 PST by builders
System image file is "disk0:/fxos-lfbff-k8.2.9.1.131.SPA"
Config file at boot was "startup-config"

High CPU usage Core 43-85, why? What is the problem?:

FPR-ASA/pri/act# sh cpu core
Core 5 sec 1 min 5 min
Core 0 11.6% 14.0% 16.8%
Core 1 8.4% 9.2% 9.3%
Core 2 9.4% 9.9% 10.1%
Core 3 8.6% 9.6% 9.7%
Core 4 8.8% 9.5% 9.6%
Core 5 9.0% 9.8% 10.0%
Core 6 8.8% 9.6% 9.7%
Core 7 9.0% 9.7% 9.9%
Core 8 9.2% 9.6% 9.7%
Core 9 9.0% 9.8% 10.0%
Core 10 9.0% 9.6% 9.7%
Core 11 8.8% 9.7% 9.8%
Core 12 9.0% 9.6% 9.8%
Core 13 9.0% 9.7% 9.8%
Core 14 9.0% 9.7% 9.9%
Core 15 9.0% 9.7% 9.9%
Core 16 8.8% 9.5% 9.7%
Core 17 8.8% 9.4% 9.5%
Core 18 8.6% 9.3% 9.4%
Core 19 8.4% 9.2% 9.3%
Core 20 8.6% 9.2% 9.4%
Core 21 11.0% 12.9% 15.4%
Core 22 5.4% 5.4% 5.5%
Core 23 5.6% 5.6% 5.7%
Core 24 5.6% 5.6% 5.7%
Core 25 5.6% 5.7% 5.8%
Core 26 5.6% 5.7% 5.8%
Core 27 5.8% 5.8% 5.9%
Core 28 5.6% 5.7% 5.9%
Core 29 5.8% 5.8% 5.9%
Core 30 5.8% 5.7% 5.8%
Core 31 5.8% 5.8% 5.9%
Core 32 5.8% 5.8% 5.9%
Core 33 5.8% 5.8% 5.9%
Core 34 6.0% 5.8% 5.9%
Core 35 5.8% 5.8% 5.9%
Core 36 5.6% 5.7% 5.8%
Core 37 5.6% 5.7% 5.8%
Core 38 5.6% 5.7% 5.8%
Core 39 5.8% 5.8% 5.9%
Core 40 5.8% 5.8% 5.9%
Core 41 5.8% 5.7% 5.9%
Core 42 5.6% 5.7% 5.8%
Core 43 88.6% 88.6% 88.6%
Core 44 88.6% 88.6% 88.6%
Core 45 88.6% 88.6% 88.6%
Core 46 88.4% 88.6% 88.6%
Core 47 88.4% 88.4% 88.4%
Core 48 88.6% 88.6% 88.6%
Core 49 88.8% 88.8% 88.8%
Core 50 89.0% 89.0% 89.0%
Core 51 88.8% 88.8% 88.8%
Core 52 89.0% 89.0% 89.0%
Core 53 89.0% 89.0% 89.0%
Core 54 89.0% 89.0% 89.0%
Core 55 88.8% 89.0% 89.0%
Core 56 89.0% 89.0% 89.0%
Core 57 88.8% 88.8% 88.8%
Core 58 88.8% 88.8% 88.8%
Core 59 88.8% 88.8% 88.8%
Core 60 88.8% 88.8% 88.8%
Core 61 88.8% 88.8% 88.8%
Core 62 88.8% 89.0% 89.0%
Core 63 88.8% 88.8% 88.8%
Core 64 88.4% 88.4% 88.4%
Core 65 88.8% 88.8% 88.8%
Core 66 88.8% 88.8% 88.8%
Core 67 88.8% 88.8% 88.8%
Core 68 88.8% 88.8% 88.8%
Core 69 88.6% 88.6% 88.6%
Core 70 88.8% 88.8% 88.8%
Core 71 88.6% 88.6% 88.6%
Core 72 88.8% 88.8% 88.8%
Core 73 88.6% 88.7% 88.7%
Core 74 88.6% 88.6% 88.6%
Core 75 88.6% 88.7% 88.7%
Core 76 88.6% 88.8% 88.8%
Core 77 88.6% 88.7% 88.7%
Core 78 88.8% 88.8% 88.8%
Core 79 88.8% 88.8% 88.8%
Core 80 88.8% 88.8% 88.8%
Core 81 88.8% 88.8% 88.8%
Core 82 88.8% 88.8% 88.8%
Core 83 88.8% 88.8% 88.8%
Core 84 88.8% 88.8% 88.8%
Core 85 88.8% 88.8% 88.8%
FPR-ASA/pri/act#

HA pair is the same:

FPR-ASA/sec/stby# sh cpu core
Core 5 sec 1 min 5 min
Core 0 6.4% 11.1% 5.6%
Core 1 2.6% 2.2% 2.1%
Core 2 2.4% 2.2% 2.1%
Core 3 2.4% 2.3% 2.2%
Core 4 2.0% 2.3% 2.2%
Core 5 2.4% 2.3% 2.2%
Core 6 2.2% 2.2% 2.2%
Core 7 2.2% 2.3% 2.2%
Core 8 2.2% 2.3% 2.2%
Core 9 2.4% 2.4% 2.3%
Core 10 2.4% 2.3% 2.2%
Core 11 2.0% 2.2% 2.1%
Core 12 2.4% 2.2% 2.1%
Core 13 2.4% 2.3% 2.2%
Core 14 2.0% 2.3% 2.2%
Core 15 2.6% 2.4% 2.3%
Core 16 2.2% 2.2% 2.1%
Core 17 2.2% 2.2% 2.1%
Core 18 2.0% 2.1% 2.0%
Core 19 2.4% 2.2% 2.2%
Core 20 2.4% 2.4% 2.2%
Core 21 5.8% 9.6% 4.8%
Core 22 1.2% 1.3% 1.2%
Core 23 1.2% 1.3% 1.2%
Core 24 1.2% 1.3% 1.2%
Core 25 1.2% 1.3% 1.2%
Core 26 1.2% 1.2% 1.2%
Core 27 1.2% 1.3% 1.2%
Core 28 1.2% 1.3% 1.2%
Core 29 1.2% 1.3% 1.2%
Core 30 1.2% 1.3% 1.2%
Core 31 1.2% 1.3% 1.2%
Core 32 1.2% 1.3% 1.2%
Core 33 1.2% 1.3% 1.2%
Core 34 1.2% 1.3% 1.2%
Core 35 1.2% 1.3% 1.2%
Core 36 1.2% 1.3% 1.2%
Core 37 1.2% 1.3% 1.2%
Core 38 1.2% 1.3% 1.2%
Core 39 1.2% 1.3% 1.2%
Core 40 1.2% 1.3% 1.2%
Core 41 1.2% 1.3% 1.2%
Core 42 1.2% 1.2% 1.2%
Core 43 89.4% 89.5% 89.6%
Core 44 89.4% 89.4% 89.4%
Core 45 89.4% 89.4% 89.4%
Core 46 89.4% 89.4% 89.4%
Core 47 89.4% 89.4% 89.4%
Core 48 89.4% 89.4% 89.4%
Core 49 89.8% 89.7% 89.8%
Core 50 89.8% 89.8% 89.8%
Core 51 89.8% 89.8% 89.8%
Core 52 89.8% 89.8% 89.8%
Core 53 89.8% 89.8% 89.8%
Core 54 89.8% 89.8% 89.8%
Core 55 89.8% 89.8% 89.8%
Core 56 89.8% 89.7% 89.8%
Core 57 89.6% 89.6% 89.6%
Core 58 89.8% 89.8% 89.8%
Core 59 89.6% 89.6% 89.6%
Core 60 89.6% 89.6% 89.6%
Core 61 89.6% 89.6% 89.6%
Core 62 89.8% 89.8% 89.8%
Core 63 89.6% 89.6% 89.6%
Core 64 89.6% 89.5% 89.6%
Core 65 89.8% 89.8% 89.8%
Core 66 89.8% 89.8% 89.8%
Core 67 89.8% 89.8% 89.8%
Core 68 89.8% 89.8% 89.8%
Core 69 89.8% 89.8% 89.8%
Core 70 89.8% 89.8% 89.8%
Core 71 89.8% 89.8% 89.8%
Core 72 89.8% 89.8% 89.8%
Core 73 89.8% 89.8% 89.8%
Core 74 89.8% 89.8% 89.8%
Core 75 89.8% 89.7% 89.6%
Core 76 89.6% 89.7% 89.7%
Core 77 89.8% 89.8% 89.8%
Core 78 89.8% 89.8% 89.8%
Core 79 89.8% 89.8% 89.8%
Core 80 89.8% 89.8% 89.8%
Core 81 89.8% 89.8% 89.8%
Core 82 89.8% 89.8% 89.8%
Core 83 89.8% 89.8% 89.8%
Core 84 89.8% 89.8% 89.8%
Core 85 89.8% 89.8% 89.8%
FPR-ASA/sec/stby#

[Sheraz.Salim]

run this command and see which process is taking more cpu.

show processes cpu-usage sorted non-zero

since when the cpu is gone up any recent change at network?

[balazs.nacsa]

Hello,

FPR-ASA/pri/act# show processes cpu-usage sorted non-zero
Hardware: FPR4K-SM-44S
Cisco Adaptive Security Appliance Software Version 9.12(4)10
ASLR enabled, text region 55b2b3b33000-55b2b8ec43c5
PC Thread 5Sec 1Min 5Min Process
- - 0.1% 0.1% 0.1% DATAPATH-2-15875
- - 0.1% 0.1% 0.1% DATAPATH-3-15876
- - 0.1% 0.1% 0.1% DATAPATH-4-15877
- - 0.1% 0.1% 0.1% DATAPATH-5-15878
- - 0.1% 0.1% 0.1% DATAPATH-6-15879
- - 0.1% 0.1% 0.1% DATAPATH-7-15880
- - 0.1% 0.1% 0.1% DATAPATH-8-15881
- - 0.1% 0.1% 0.1% DATAPATH-9-15882
- - 0.1% 0.1% 0.1% DATAPATH-10-15883
- - 0.1% 0.1% 0.1% DATAPATH-11-15884
- - 0.1% 0.1% 0.1% DATAPATH-12-15885
- - 0.1% 0.1% 0.1% DATAPATH-13-15886
- - 0.1% 0.1% 0.1% DATAPATH-14-15887
- - 0.1% 0.1% 0.1% DATAPATH-15-15888
- - 0.1% 0.1% 0.1% DATAPATH-16-15889
- - 0.1% 0.1% 0.1% DATAPATH-17-15890
- - 0.1% 0.1% 0.1% DATAPATH-18-15891
- - 0.1% 0.1% 0.1% DATAPATH-19-15892
- - 0.1% 0.1% 0.1% DATAPATH-20-15893
- - 0.1% 0.1% 0.1% DATAPATH-1-15874
- - 0.1% 0.1% 0.1% DATAPATH-22-15895
- - 0.1% 0.1% 0.1% DATAPATH-23-15896
- - 0.1% 0.1% 0.1% DATAPATH-24-15897
- - 0.1% 0.1% 0.1% DATAPATH-25-15898
- - 0.1% 0.1% 0.1% DATAPATH-26-15899
- - 0.1% 0.1% 0.1% DATAPATH-27-15900
- - 0.1% 0.1% 0.1% DATAPATH-28-15901
- - 0.1% 0.1% 0.1% DATAPATH-29-15902
- - 0.1% 0.1% 0.1% DATAPATH-30-15903
- - 0.1% 0.1% 0.1% DATAPATH-31-15904
- - 0.1% 0.1% 0.1% DATAPATH-32-15905
- - 0.1% 0.1% 0.1% DATAPATH-33-15906
- - 0.1% 0.1% 0.1% DATAPATH-34-15907
- - 0.1% 0.1% 0.1% DATAPATH-35-15908
- - 0.1% 0.1% 0.1% DATAPATH-36-15909
- - 0.1% 0.1% 0.1% DATAPATH-37-15910
- - 0.1% 0.1% 0.1% DATAPATH-38-15911
- - 0.1% 0.1% 0.1% DATAPATH-39-15912
- - 0.1% 0.1% 0.1% DATAPATH-40-15913
- - 0.1% 0.1% 0.1% DATAPATH-41-15914
- - 0.1% 0.1% 0.1% DATAPATH-42-15915
- - 0.1% 0.1% 0.1% DATAPATH-0-15873
- - 0.1% 0.1% 0.1% DATAPATH-21-15894
0x000055b2b6dfe6f4 0x00007fd0cb0a6e80 0.1% 0.1% 0.1% Logger
FPR-ASA/pri/act#

FPR-ASA/sec/stby# show processes cpu-usage sorted non-zero
Hardware: FPR4K-SM-44S
Cisco Adaptive Security Appliance Software Version 9.12(4)10
ASLR enabled, text region 5584c2ec5000-5584c82563c5
PC Thread 5Sec 1Min 5Min Process
0x00005584c48964bc 0x00007f21cb09dd80 0.2% 0.4% 0.1% tmatch compile thread
0x00005584c6da723d 0x00007f21cb0aa140 0.1% 0.0% 0.0% Checkheaps
FPR-ASA/sec/stby#

This is a new install.

[Marvin Rhoads]

It's normal to see high CPU utilization for the LINA (ASA) engine when running on a Firepower appliance. It's an artifiact of how the traffic is processed in the system architecture.

Reference: https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/200950-Clarifying-the-Firepower-Threat-Defense.html

[balazs.nacsa]

Hello,

thank you very much for your answer, I know that the LINA engine is high CPU usage when Firepower runs FTD, but now Firepower 4145 runs ASA software directly, not FTD.

This is very frustrating:

We use Firepower 2110 similar to ASA software too, and there is no high CPU usage there.

[Sheraz.Salim]

run this command again please

show processes cpu-usage non-zero sorted

have you configured any SNMP or syslog (syslogs to external server).

show what does it show your when you give a command show logging

[balazs.nacsa]

FPR-ASA/pri/act# show processes cpu-usage non-zero sorted
Hardware: FPR4K-SM-44S
Cisco Adaptive Security Appliance Software Version 9.12(4)10
ASLR enabled, text region 55b2b3b33000-55b2b8ec43c5
PC Thread 5Sec 1Min 5Min Process
0x000055b2b5503bc3 0x00007fd0cb09dd80 0.9% 0.1% 0.0% tmatch compile thread
0x000055b2b6746548 0x00007fd0cb0ad060 0.1% 0.0% 0.0% idfw_proc
- - 0.1% 0.1% 0.1% DATAPATH-3-15876
- - 0.1% 0.1% 0.1% DATAPATH-4-15877
- - 0.1% 0.1% 0.1% DATAPATH-5-15878
- - 0.1% 0.1% 0.1% DATAPATH-8-15881
- - 0.1% 0.1% 0.1% DATAPATH-9-15882
- - 0.1% 0.1% 0.1% DATAPATH-10-15883
- - 0.1% 0.1% 0.1% DATAPATH-11-15884
- - 0.1% 0.1% 0.1% DATAPATH-13-15886
- - 0.1% 0.1% 0.1% DATAPATH-15-15888
- - 0.1% 0.1% 0.1% DATAPATH-18-15891
- - 0.1% 0.1% 0.1% DATAPATH-20-15893
- - 0.1% 0.1% 0.1% DATAPATH-39-15912
- - 0.1% 0.1% 0.1% DATAPATH-1-15874
- - 0.1% 0.1% 0.1% DATAPATH-2-15875
- - 0.1% 0.1% 0.1% DATAPATH-6-15879
- - 0.1% 0.1% 0.1% DATAPATH-7-15880
- - 0.1% 0.1% 0.1% DATAPATH-12-15885
- - 0.1% 0.1% 0.1% DATAPATH-14-15887
- - 0.1% 0.1% 0.1% DATAPATH-16-15889
- - 0.1% 0.1% 0.1% DATAPATH-17-15890
- - 0.1% 0.1% 0.1% DATAPATH-19-15892
- - 0.1% 0.1% 0.1% DATAPATH-24-15897
- - 0.1% 0.1% 0.1% DATAPATH-25-15898
- - 0.1% 0.1% 0.1% DATAPATH-26-15899
- - 0.1% 0.1% 0.1% DATAPATH-27-15900
- - 0.1% 0.1% 0.1% DATAPATH-28-15901
- - 0.1% 0.1% 0.1% DATAPATH-30-15903
- - 0.1% 0.1% 0.1% DATAPATH-31-15904
- - 0.1% 0.1% 0.1% DATAPATH-32-15905
- - 0.1% 0.1% 0.1% DATAPATH-34-15907
- - 0.1% 0.1% 0.1% DATAPATH-35-15908
- - 0.1% 0.1% 0.1% DATAPATH-36-15909
- - 0.1% 0.1% 0.1% DATAPATH-37-15910
- - 0.1% 0.1% 0.1% DATAPATH-40-15913
- - 0.1% 0.1% 0.1% DATAPATH-41-15914
- - 0.1% 0.1% 0.1% DATAPATH-42-15915
- - 0.1% 0.1% 0.1% DATAPATH-22-15895
- - 0.1% 0.1% 0.1% DATAPATH-23-15896
- - 0.1% 0.1% 0.1% DATAPATH-29-15902
- - 0.1% 0.1% 0.1% DATAPATH-33-15906
- - 0.1% 0.1% 0.1% DATAPATH-38-15911
- - 0.1% 0.1% 0.1% DATAPATH-0-15873
- - 0.1% 0.1% 0.1% DATAPATH-21-15894
0x000055b2b6dfe6f4 0x00007fd0cb0a6e80 0.1% 0.1% 0.1% Logger

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act#

FPR-ASA/pri/act# sh cpu core
Core 5 sec 1 min 5 min
Core 0 8.6% 13.1% 11.6%
Core 1 6.4% 6.4% 6.1%
Core 2 6.6% 6.6% 6.3%
Core 3 6.8% 6.6% 6.3%
Core 4 6.8% 6.6% 6.3%
Core 5 6.8% 6.5% 6.3%
Core 6 6.6% 6.6% 6.3%
Core 7 6.8% 6.6% 6.4%
Core 8 6.8% 6.6% 6.3%
Core 9 6.6% 6.6% 6.3%
Core 10 6.6% 6.6% 6.4%
Core 11 6.6% 6.6% 6.4%
Core 12 6.6% 6.6% 6.3%
Core 13 6.6% 6.6% 6.4%
Core 14 6.8% 6.6% 6.3%
Core 15 6.6% 6.6% 6.3%
Core 16 6.8% 6.6% 6.4%
Core 17 6.4% 6.5% 6.2%
Core 18 6.4% 6.5% 6.2%
Core 19 6.4% 6.4% 6.2%
Core 20 6.4% 6.4% 6.2%
Core 21 10.4% 14.0% 12.4%
Core 22 6.8% 6.4% 5.8%
Core 23 7.0% 6.6% 5.9%
Core 24 7.0% 6.6% 6.0%
Core 25 7.2% 6.7% 6.1%
Core 26 7.0% 6.6% 6.0%
Core 27 7.0% 6.7% 6.1%
Core 28 7.2% 6.7% 6.1%
Core 29 7.2% 6.7% 6.1%
Core 30 7.2% 6.6% 6.0%
Core 31 7.2% 6.7% 6.0%
Core 32 7.2% 6.7% 6.1%
Core 33 7.2% 6.7% 6.1%
Core 34 7.0% 6.7% 6.1%
Core 35 7.2% 6.7% 6.1%
Core 36 7.0% 6.7% 6.1%
Core 37 7.2% 6.7% 6.1%
Core 38 7.0% 6.7% 6.0%
Core 39 7.2% 6.7% 6.1%
Core 40 7.2% 6.7% 6.1%
Core 41 7.2% 6.7% 6.0%
Core 42 7.2% 6.5% 6.0%
Core 43 88.8% 88.8% 88.8%
Core 44 88.6% 88.6% 88.6%
Core 45 88.6% 88.6% 88.6%
Core 46 88.6% 88.6% 88.6%
Core 47 88.6% 88.6% 88.6%
Core 48 88.6% 88.6% 88.6%
Core 49 89.0% 89.0% 89.0%
Core 50 89.0% 89.0% 89.0%
Core 51 89.0% 89.0% 89.0%
Core 52 89.0% 89.0% 89.0%
Core 53 89.0% 89.0% 89.0%
Core 54 89.0% 89.0% 89.0%
Core 55 89.0% 89.0% 89.0%
Core 56 89.0% 89.0% 89.0%
Core 57 89.0% 89.0% 89.0%
Core 58 89.0% 89.0% 89.0%
Core 59 89.0% 89.0% 89.0%
Core 60 89.0% 89.0% 89.0%
Core 61 89.0% 89.0% 89.0%
Core 62 89.0% 89.0% 89.0%
Core 63 88.8% 88.9% 89.0%
Core 64 88.4% 88.4% 88.4%
Core 65 88.8% 88.8% 88.8%
Core 66 88.6% 88.8% 88.8%
Core 67 88.6% 88.7% 88.8%
Core 68 88.6% 88.7% 88.7%
Core 69 88.6% 88.6% 88.6%
Core 70 88.6% 88.6% 88.7%
Core 71 88.6% 88.6% 88.6%
Core 72 88.6% 88.6% 88.6%
Core 73 88.6% 88.6% 88.6%
Core 74 88.6% 88.6% 88.6%
Core 75 88.6% 88.6% 88.6%
Core 76 88.6% 88.6% 88.6%
Core 77 88.6% 88.6% 88.6%
Core 78 88.8% 88.8% 88.8%
Core 79 88.6% 88.6% 88.6%
Core 80 88.6% 88.7% 88.8%
Core 81 88.6% 88.6% 88.6%
Core 82 88.6% 88.8% 88.8%
Core 83 88.6% 88.8% 88.8%
Core 84 88.6% 88.8% 88.8%
Core 85 88.6% 88.7% 88.8%
FPR-ASA/pri/act#

[Sheraz.Salim]

seem like to be cosmetic issue. your process looks fine in good health. I wont worried much. however if it a production network which i assume it is. open a TAC case.

as Marvin mentioned it more of a system architecture desgin.

[NetworkNinjaAC]

I'm having similar issues with our Firepower 2140 running ASA on top of FXOS, no FTD. Is it normal for it to run at 70-95% cpu during busy hours and hover around 19-25% during low traffic hours?

[Marvin Rhoads]

It's normal on both FTD as well as when running natively as an ASA logical device on FXOS. It has to do with how the parent OS allocates the multicore CPUs to processes.

[NetworkNinjaAC]

Hi Marvin, thanks for responding so promptly. That makes sense and it brings about 2 additional questions for me. If it is normal behavior for the ASA to use up so much CPU , then is it also normal for it to occasionally reach 100% and trigger a fail over? Also, we have 2 Firepower 4110s that are running ASA natively as well but I don't see a high CPU issue with those 2. They're not sitting at the edge of the network like the 2140 pair so they don't have as high a count on xlates and connections, but their CPU usage is extremely low like at 2%