%IOSXE_QFP-2-LOAD_EXCEED: Slot: 0, QFP:0, Load 99% exceeds. ASR1k

kwuenP · ‎08-18-2023

I have had the performance issue with Cisco ASR 1001-X. We received a lot of log messages like "%IOSXE_QFP-2-LOAD_EXCEED: Slot: 0, QFP:0, Load 99% exceeds the setting threshold." few time per day but not all days.

Router throughput is 20 Gbps with 2x10Gbps port bonding LACP, and peak traffic was about 7-8 Gbps.

It seens the issue was about QuantumFlow Processor and tried to find bottlenecks with command "show platform hardware qfp active datapath infra sw-cio" as in live BRKRST-3404 https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKRST-3404.pdf . However, ASR did not support this command.

My device is "Chassis", DESCR: "Cisco ASR1001-X Chassis" PID: ASR1001-X

Cisco IOS XE Software, Version 16.09.05
Cisco IOS Software [Fuji], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.9.5, RELEASE SOFTWARE (fc1)

License Type: Permanent
License Level: adventerprise
Next reload license Level: adventerprise
The current throughput level is 20000000 kbps

cisco ASR1001-X (1NG) processor (revision 1NG) with 6925044K/6147K bytes of memory.

Anyone has the same issue ?

Leo Laohoo · ‎08-18-2023

Post the complete output to the following commands:

sh platform resource
sh platform software status con brief
sh crypto pki trustpool | include cn=

kwuenP · ‎08-21-2023

@Leo Laohoo wrote:
Post the complete output to the following commands:
sh platform resource
sh platform software status con brief
sh crypto pki trustpool | include cn=

Thank for your reply.

sh platform resource

#sh platform resource
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.649 Hanoi Tue Aug 22 2023
**State Acronym: H - Healthy, W - Warning, C - Critical                                             
Resource                 Usage                 Max             Warning         Critical        State
----------------------------------------------------------------------------------------------------
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.652 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.661 Hanoi Tue Aug 22 2023
RP0 (ok, active)                                                                               H    
 Control Processor       2.21%                 100%            80%             90%             H    
  DRAM                   2485MB(15%)           15914MB         88%             93%             H    
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.671 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.678 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.686 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.694 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.703 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.711 Hanoi Tue Aug 22 2023
ESP0(ok, active)                                                                               H    
 QFP                                                                                           H    
  TCAM                   50cells(0%)           131072cells     65%             85%             H    
  DRAM                   534897KB(14%)         3670016KB       80%             90%             H    
  IRAM                   10709KB(8%)           131072KB        80%             90%             H    
  CPU Utilization        14.00%                100%            90%             95%             H    
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.721 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.729 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.737 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.745 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.752 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.760 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.767 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.774 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.782 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.789 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.796 Hanoi Tue Aug 22 2023
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:29:46.804 Hanoi Tue Aug 22 2023

sh platform software status con brief                 
Load for five secs: 2%/0%; one minute: 2%; five minutes: 1%
Time source is NTP, 10:32:09.628 Hanoi Tue Aug 22 2023
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.07   0.08   0.09

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Healthy 16296268  2547428 (16%) 13748840 (84%)   8620772 (53%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
  RP0    0   0.90   0.40   0.00  98.69   0.00   0.00   0.00
         1   0.00   0.00   0.00 100.00   0.00   0.00   0.00
         2   1.10   0.50   0.00  98.39   0.00   0.00   0.00
         3   0.00   0.00   0.00 100.00   0.00   0.00   0.00
         4   1.09   0.39   0.00  98.50   0.00   0.00   0.00
         5   0.00   0.00   0.00 100.00   0.00   0.00   0.00
         6   2.00   0.50   0.00  97.50   0.00   0.00   0.00
         7   0.00   0.00   0.00 100.00   0.00   0.00   0.00

sh crypto pki trustpool | include cn=
    cn=QuoVadis Root CA 2
    cn=QuoVadis Root CA 2
    cn=VeriSign Class 3 Public Primary Certification Authority - G5
    cn=VeriSign Class 3 Public Primary Certification Authority - G5
    cn=Cisco RXC-R2
    cn=Cisco RXC-R2
    cn=Cisco Root CA 2048
    cn=ACT2 SUDI CA
    cn=Cisco Licensing Root CA
    cn=Cisco Licensing Root CA
    cn=Licensing Root - DEV
    cn=Licensing Root - DEV
    cn=Cisco Root CA M2
    cn=Cisco Manufacturing CA SHA2
    cn=Cisco Root CA M2
    cn=Cisco Root CA M2
    cn=Cisco Root CA 2048
    cn=Cisco Manufacturing CA
    cn=Cisco Root CA 2048
    cn=Cisco Root CA 2048
    cn=Cisco Root CA M1
    cn=Cisco Root CA M1

Leo Laohoo · ‎08-21-2023

@kwuenP wrote:

sh crypto pki trustpool | include cn=
    cn=QuoVadis Root CA 2
    cn=QuoVadis Root CA 2

Please read FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.

Whether the appliance is using Cisco Smart License or not, the expired certificate will cause a memory leak of the firmware is not upgraded or the workaround not implemented.

And another one: FN - 72578 - Cisco IOS XE - Smart Licensing Using Policy Might Cause High CPU/Memory Usage

Same as above. Cisco Smart License can cause CPU Hog and/or Memory Leak which, if left unattended, can cause the appliance to crash.

kwuenP · ‎10-30-2023

Hi Leo,

Sorry for a long delay. I've done update cert following this topic https://www.cisco.com/c/en/us/support/docs/field-notices/723/fn72323.html and waited to reboot the devices. However, it appears not to be our case

We also migrated BGP workload to other routers, and ASRs are used to NAT traffic, peak bandwidth was about 300 Mbps but log "%IOSXE_QFP-2-LOAD_EXCEED" happened .

Leo Laohoo · ‎10-30-2023

@kwuenP wrote:
%IOSXE_QFP-2-LOAD_EXCEED

This potentially means that the router is pushing more than what it is allowed.

@kwuenP wrote:
waited to reboot the devices. However, it appears not to be our case

Please read Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature.

Either upgrade the firmware of the router or apply the workaround. There are >28,000 routers compromised worldwide.

Ask yourself this: Is your router any one of them?

kwuenP · ‎10-30-2023

Be sure that HTTP Server feature was disabled at the first time configuration.

So if it is about router performance, it is quite disappointing that ASR1001-X with truly 20Gbps throughput and 16GB RAM can not do some basic NAT.

Total number of translations are between 100K to 700K but traffic loss about 0.1 - 0.3% . I'm sure that the NAT session per IP below 65K .

ip nat settings gatekeeper-size 65536
ip nat settings nonpatdrop
ip nat translation timeout 600
ip nat translation tcp-timeout 180

However, is there any limitation with nat for ASR ?

@Leo Laohoo wrote:
@kwuenP wrote:
%IOSXE_QFP-2-LOAD_EXCEED
This potentially means that the router is pushing more than what it is allowed.

@kwuenP wrote:
waited to reboot the devices. However, it appears not to be our case
Please read Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature.
Either upgrade the firmware of the router or apply the workaround. There are >28,000 routers compromised worldwide.
Ask yourself this: Is your router any one of them?

mateus.provesi · ‎07-05-2024

Have you check out this?

I'm cheking out this.

https://www.cisco.com/c/en/us/support/docs/troubleshooting/222061-troubleshoot-high-qfp-utilization-due-to.html

mateus.provesi · ‎07-02-2024

@kwuenP did you got any solution?

kwuenP · ‎07-09-2024

yes. we've tried. We did both separate the NATed and non-NATed traffic from the same interface and Increase the size of the cache on the NAT Gatekeeper feature as Cisco recommend. It did not work. We think the main reason is our heavy NAT workload.