FPR9300-ASA-K9 CPU Core usage distribution

ahunsolo — Wed, 10 Aug 2022 09:31:45 GMT

Hello,

I have trouble with Firepower FPR9300-ASA-K9 CPU Core usage distribution
Hardware: FPR9K-SM-40, 347915 MB RAM, CPU Xeon 4100/6100/8100 series 2000 MHz, 2 CPUs (80 cores)

Cisco Adaptive Security Appliance Software Version 9.12(4)18 <system>
SSP Operating System Version 2.8(1.143)
Device Manager Version 7.12(2)14

Compiled on Sun 28-Mar-21 22:57 GMT by builders
System image file is "disk0:/fxos-lfbff-k8.2.8.1.143.SPA"
Config file at boot was "startup-config"

From 80 cores, only half of them shows CPU usage, from Core 0 to Core 38, but other half core 39-77 shows zero all time

------------------ show cpu detailed ------------------

Break down of per-core data path versus control point cpu usage:
Core 5 sec 1 min 5 min
Core 0 25.6 (12.2 + 13.4) 14.8 (12.9 + 1.9) 14.2 (13.0 + 1.0)
Core 1 24.2 (24.2 + 0.0) 24.3 (24.3 + 0.0) 24.5 (24.5 + 0.0)
Core 2 24.6 (24.6 + 0.0) 24.9 (24.9 + 0.0) 25.1 (25.1 + 0.0)
Core 3 25.0 (25.0 + 0.0) 25.1 (25.1 + 0.0) 25.3 (25.3 + 0.0)
Core 4 24.8 (24.8 + 0.0) 24.8 (24.8 + 0.0) 24.9 (24.9 + 0.0)
Core 5 24.6 (24.6 + 0.0) 24.8 (24.8 + 0.0) 25.0 (25.0 + 0.0)
Core 6 25.2 (25.2 + 0.0) 25.3 (25.3 + 0.0) 25.5 (25.5 + 0.0)
Core 7 24.8 (24.8 + 0.0) 24.9 (24.9 + 0.0) 25.1 (25.1 + 0.0)
Core 8 24.2 (24.2 + 0.0) 24.4 (24.4 + 0.0) 24.6 (24.6 + 0.0)
Core 9 24.8 (24.8 + 0.0) 25.2 (25.2 + 0.0) 25.4 (25.4 + 0.0)
<output ommited>
Core 36 24.2 (24.2 + 0.0) 24.3 (24.3 + 0.0) 24.6 (24.6 + 0.0)
Core 37 23.8 (23.8 + 0.0) 23.7 (23.7 + 0.0) 24.0 (24.0 + 0.0)
Core 38 24.6 (24.6 + 0.0) 24.4 (24.4 + 0.0) 24.7 (24.7 + 0.0)
Core 39 0.2 ( 0.0 + 0.2) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 40 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 41 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 42 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 43 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 44 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 45 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 46 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 47 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 48 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 49 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 50 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
<output ommited>
Core 76 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)
Core 77 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0) 0.0 ( 0.0 + 0.0)

From the same time, the sh cpu core output shows

# sh cpu core
Core 5 sec 1 min 5 min
Core 0 13.8% 14.6% 14.6%
Core 1 24.6% 24.7% 24.6%
Core 2 25.2% 25.2% 25.1%
Core 3 25.4% 25.5% 25.4%
Core 4 25.2% 25.1% 25.0%
Core 5 24.8% 25.1% 25.0%
Core 6 25.4% 25.6% 25.6%
Core 7 25.0% 25.3% 25.2%
Core 8 25.0% 24.8% 24.7%
Core 9 25.2% 25.5% 25.4%
<output ommited>
Core 31 24.4% 24.6% 24.6%
Core 32 24.2% 24.2% 24.2%
Core 33 24.4% 24.7% 24.7%
Core 34 24.2% 24.3% 24.3%
Core 35 24.0% 24.2% 24.1%
Core 36 24.6% 24.4% 24.5%
Core 37 23.8% 24.0% 24.0%
Core 38 24.4% 24.7% 24.7%
Core 39 95.2% 95.2% 95.2%
Core 40 94.8% 94.8% 94.8%
Core 41 94.8% 94.8% 94.8%
Core 42 94.8% 94.8% 94.8%
Core 43 94.8% 94.8% 94.8%
Core 44 94.8% 94.8% 94.8%
Core 45 95.0% 95.0% 95.0%
Core 46 95.0% 95.0% 95.0%
Core 47 95.0% 95.0% 95.0%
Core 48 95.0% 95.0% 95.0%
Core 49 95.0% 95.0% 95.0%
Core 50 95.0% 95.0% 95.0%
Core 51 95.0% 95.0% 95.0%
<output ommited>
Core 73 95.2% 95.2% 95.2%
Core 74 95.2% 95.2% 95.2%
Core 75 95.2% 95.2% 95.2%
Core 76 95.2% 95.2% 95.2%
Core 77 95.2% 95.2% 95.2%
#
The Cores 39 - 77 shows High CPU usage

What is the problem? Why cores 39-77 didn't process the ASA traffic?

# show processes cpu-usage non-zero sorted

Hardware: FPR9K-SM-40
Cisco Adaptive Security Appliance Software Version 9.12(4)18
ASLR enabled, text region 559b32d60000-559b380ef0f5
PC Thread 5Sec 1Min 5Min Process
- - 0.3% 0.3% 0.3% DATAPATH-2-14057
- - 0.3% 0.3% 0.3% DATAPATH-6-14061
- - 0.3% 0.3% 0.3% DATAPATH-9-14064
- - 0.3% 0.3% 0.3% DATAPATH-12-14067
- - 0.3% 0.3% 0.3% DATAPATH-1-14056
- - 0.3% 0.3% 0.3% DATAPATH-3-14058
- - 0.3% 0.3% 0.3% DATAPATH-4-14059
- - 0.3% 0.3% 0.3% DATAPATH-5-14060
- - 0.3% 0.3% 0.3% DATAPATH-7-14062
- - 0.3% 0.3% 0.3% DATAPATH-10-14065
- - 0.3% 0.3% 0.3% DATAPATH-11-14066
- - 0.3% 0.3% 0.3% DATAPATH-13-14068
- - 0.3% 0.3% 0.3% DATAPATH-14-14069
- - 0.3% 0.3% 0.3% DATAPATH-15-14070
- - 0.3% 0.3% 0.3% DATAPATH-17-14072
- - 0.3% 0.3% 0.3% DATAPATH-18-14073
- - 0.3% 0.3% 0.3% DATAPATH-24-14079
- - 0.3% 0.3% 0.3% DATAPATH-27-14082
- - 0.3% 0.3% 0.3% DATAPATH-28-14083
- - 0.3% 0.3% 0.3% DATAPATH-29-14084
- - 0.3% 0.3% 0.3% DATAPATH-30-14085
- - 0.3% 0.3% 0.3% DATAPATH-33-14088
- - 0.3% 0.3% 0.3% DATAPATH-36-14091
- - 0.3% 0.3% 0.3% DATAPATH-38-14093
- - 0.3% 0.3% 0.3% DATAPATH-8-14063
- - 0.3% 0.3% 0.3% DATAPATH-16-14071
- - 0.3% 0.3% 0.3% DATAPATH-20-14075
- - 0.3% 0.3% 0.3% DATAPATH-21-14076
- - 0.3% 0.3% 0.3% DATAPATH-22-14077
- - 0.3% 0.3% 0.3% DATAPATH-23-14078
- - 0.3% 0.3% 0.3% DATAPATH-25-14080
- - 0.3% 0.3% 0.3% DATAPATH-26-14081
- - 0.3% 0.3% 0.3% DATAPATH-31-14086
- - 0.3% 0.3% 0.3% DATAPATH-32-14087
- - 0.3% 0.3% 0.3% DATAPATH-34-14089
- - 0.3% 0.3% 0.3% DATAPATH-35-14090
- - 0.3% 0.3% 0.3% DATAPATH-37-14092
- - 0.2% 0.2% 0.2% DATAPATH-19-14074
- - 0.2% 0.2% 0.2% DATAPATH-0-14055

Also there is another box FPR9300-ASA-K9 with same Hardware and software, but CPU core distribuion is almost equal for all cores

Here is example:

# show cpu usage detailed
Break down of per-core data path versus control point cpu usage:
Core 5 sec 1 min 5 min
Core 0 37.6 (37.2 + 0.4) 37.9 (37.4 + 0.4) 38.0 (37.6 + 0.2)
Core 1 44.0 (44.0 + 0.0) 44.1 (44.1 + 0.0) 44.4 (44.4 + 0.0)
Core 2 44.0 (44.0 + 0.0) 44.4 (44.4 + 0.0) 44.6 (44.6 + 0.0)
Core 3 44.2 (44.2 + 0.0) 44.5 (44.5 + 0.0) 44.7 (44.7 + 0.0)
Core 4 44.2 (44.2 + 0.0) 44.5 (44.5 + 0.0) 44.7 (44.7 + 0.0)
Core 5 44.0 (44.0 + 0.0) 44.4 (44.4 + 0.0) 44.7 (44.7 + 0.0)
Core 6 44.0 (44.0 + 0.0) 44.4 (44.4 + 0.0) 44.7 (44.7 + 0.0)
Core 7 44.4 (44.4 + 0.0) 44.6 (44.6 + 0.0) 44.8 (44.8 + 0.0)
Core 8 44.4 (44.4 + 0.0) 44.5 (44.5 + 0.0) 44.8 (44.8 + 0.0)
Core 9 44.0 (44.0 + 0.0) 44.4 (44.4 + 0.0) 44.6 (44.6 + 0.0)
Core 10 44.4 (44.4 + 0.0) 44.3 (44.3 + 0.0) 44.6 (44.6 + 0.0)
<output ommited>
Core 32 45.8 (45.8 + 0.0) 46.6 (46.6 + 0.0) 46.3 (46.3 + 0.0)
Core 33 46.0 (46.0 + 0.0) 46.6 (46.6 + 0.0) 46.6 (46.6 + 0.0)
Core 34 45.8 (45.8 + 0.0) 46.5 (46.5 + 0.0) 46.3 (46.3 + 0.0)
Core 35 46.0 (46.0 + 0.0) 46.8 (46.8 + 0.0) 46.6 (46.6 + 0.0)
Core 36 45.8 (45.8 + 0.0) 46.6 (46.6 + 0.0) 46.4 (46.4 + 0.0)
Core 37 45.8 (45.8 + 0.0) 46.7 (46.7 + 0.0) 46.5 (46.5 + 0.0)
Core 38 45.6 (45.6 + 0.0) 46.4 (46.4 + 0.0) 46.3 (46.3 + 0.0)
Core 39 37.4 (37.2 + 0.2) 37.2 (36.8 + 0.4) 37.3 (36.9 + 0.2)
Core 40 43.8 (43.8 + 0.0) 43.3 (43.3 + 0.0) 43.3 (43.3 + 0.0)
Core 41 44.2 (44.2 + 0.0) 43.6 (43.6 + 0.0) 43.7 (43.7 + 0.0)
Core 42 44.0 (44.0 + 0.0) 43.4 (43.4 + 0.0) 43.5 (43.5 + 0.0)
Core 43 43.8 (43.8 + 0.0) 43.2 (43.2 + 0.0) 43.3 (43.3 + 0.0)
Core 44 44.0 (44.0 + 0.0) 43.4 (43.4 + 0.0) 43.4 (43.4 + 0.0)
Core 45 43.8 (43.8 + 0.0) 43.4 (43.4 + 0.0) 43.4 (43.4 + 0.0)
Core 46 44.2 (44.2 + 0.0) 43.8 (43.8 + 0.0) 43.8 (43.8 + 0.0)
Core 47 44.2 (44.2 + 0.0) 43.5 (43.5 + 0.0) 43.6 (43.6 + 0.0)
Core 48 44.4 (44.4 + 0.0) 43.6 (43.6 + 0.0) 43.8 (43.8 + 0.0)
Core 49 43.8 (43.8 + 0.0) 43.3 (43.3 + 0.0) 43.4 (43.4 + 0.0)
Core 50 44.0 (44.0 + 0.0) 43.4 (43.4 + 0.0) 43.4 (43.4 + 0.0)
<output ommited>
Core 75 45.6 (45.6 + 0.0) 45.7 (45.7 + 0.0) 45.3 (45.3 + 0.0)
Core 76 45.8 (45.8 + 0.0) 45.8 (45.8 + 0.0) 45.4 (45.4 + 0.0)
Core 77 45.4 (45.4 + 0.0) 45.4 (45.4 + 0.0) 45.0 (45.0 + 0.0)
#

show cpu core output shows equal distibution

# show cpu core
Core 5 sec 1 min 5 min
Core 0 43.4% 44.2% 44.3%
Core 1 43.4% 44.1% 44.4%
Core 2 43.6% 44.3% 44.6%
Core 3 43.6% 44.4% 44.7%
Core 4 43.6% 44.5% 44.7%
Core 5 44.0% 44.4% 44.6%
Core 6 43.2% 44.3% 44.6%
Core 7 44.0% 44.5% 44.7%
Core 8 43.6% 44.4% 44.7%
Core 9 43.8% 44.3% 44.6%
Core 10 43.6% 44.3% 44.5%
<output ommited>
Core 30 46.6% 46.5% 46.3%
Core 31 46.4% 46.4% 46.2%
Core 32 46.2% 46.5% 46.3%
Core 33 47.0% 46.7% 46.6%
Core 34 46.6% 46.5% 46.4%
Core 35 47.0% 46.8% 46.6%
Core 36 46.6% 46.6% 46.5%
Core 37 46.8% 46.7% 46.5%
Core 38 46.4% 46.4% 46.3%
Core 39 43.0% 43.9% 44.2%
Core 40 43.2% 43.5% 43.7%
Core 41 43.8% 43.9% 44.1%
Core 42 43.6% 43.7% 44.0%
Core 43 43.4% 43.5% 43.7%
Core 44 43.6% 43.6% 43.8%
Core 45 43.6% 43.7% 43.8%
Core 46 44.0% 44.1% 44.2%
Core 47 43.8% 43.8% 44.0%
Core 48 43.8% 43.9% 44.2%
Core 49 43.4% 43.6% 43.8%
Core 50 43.6% 43.7% 43.9%
<output ommited>
Core 73 46.0% 45.9% 45.6%
Core 74 46.6% 46.5% 46.2%
Core 75 46.4% 46.0% 45.7%
Core 76 46.6% 46.2% 45.9%
Core 77 46.2% 45.8% 45.5%

What is the difference?

Re: FPR9300-ASA-K9 CPU Core usage distribution

ahunsolo — Wed, 17 Aug 2022 09:44:36 GMT

Any help... please....

topic Re: FPR9300-ASA-K9 CPU Core usage distribution in Network Security

FPR9300-ASA-K9 CPU Core usage distribution

Re: FPR9300-ASA-K9 CPU Core usage distribution