QFP Load on C1111

[rasmus.elmholt]

I often get the following error when I load a router with traffic.

%IOSXE_QFP-2-LOAD_EXCEED: Slot: 0, QFP:0, Load 85% exceeds the setting threshold 80%.
5 secs traffic rate on QFP: Total Input: 31561 pps (31.6 kpps), 274136312 bps (274.1 mbps), Total Output: 31534 pps (31.5 kpps), 264333472 bps (264.3 mbps).

How do I figure out what is the root cause of the issue?

[Joseph W. Doherty]

I believe that message is letting you know processing capacity is nearly maxed out.

The overall traffic load looks light for such a warning, so either there is some software process consuming much processing or the dataplane is software forwarding.

Possibly Cisco has documentation addressing analysis of this.  Have you looked?

[rasmus.elmholt]

Hi @Joseph W. Doherty 

I have looked for documentation on this but could not find anything meaningful.

I have another box with almost the same configuration and software where I see the same error but that is with 900mbit/s throughput.

I have been through almost all of the "show platform hardware qfp" commands I could find, but none of them give me any usefull information about what specific configuration that is causing the high utilization of the QFP.

[Joseph W. Doherty · Accepted solution]

Have you seen 

https://www.cisco.com/c/en/us/support/docs/troubleshooting/222061-troubleshoot-high-qfp-utilization-due-to.html

Or

https://www.cisco.com/c/en/us/support/docs/technical-details/222469-reporting-problems-effectively-to-tac-se.html

Or

https://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregation-services-routers/221078-troubleshoot-router-issues-on-enterprise.html

None of the above are specific to your platform issue, but they do all appear to have some useful information for troubleshooting QFP load issues.

[rasmus.elmholt]

Hi @Joseph W. Doherty 

I have gone through some of the links but not all, I am looking into the rest now.

The router with the issue is running NAT as well and have reached the max amount of NAT translations of 16k from time to time.

I will look further into the steps in the tshoot guide.

[rasmus.elmholt]

Hi @Joseph W. Doherty 

Very useful links.

When I look at the traffic on my router some of the features take longer than other on a random packet as shown here.

R1#show platform packet-trace packet 1500 | inc Feature:|Lapsed
  Feature: IPV4(Output)
  Feature: IPV4_OUTPUT_VFR
    Lapsed time : 80 ns
  Feature: NAT
  Feature: IPV4_NAT_OUTPUT_FIA
    Lapsed time : 840 ns
  Feature: IPV4_OUTPUT_THREAT_DEFENSE
    Lapsed time : 200 ns
  Feature: IPV4_VFR_REFRAG
    Lapsed time : 80 ns
  Feature: DEBUG_COND_APPLICATION_OUT_CLR_TXT
    Lapsed time : 120 ns
  Feature: IPV4_OUTPUT_L2_REWRITE
    Lapsed time : 200 ns
  Feature: DEBUG_COND_MAC_EGRESS
    Lapsed time : 40 ns
  Feature: CFT
  Feature: NBAR
  Feature: IPV4_OUTPUT_STILE_LEGACY
    Lapsed time : 8520 ns
  Feature: DEBUG_COND_APPLICATION_OUT
    Lapsed time : 120 ns
  Feature: IPV4_OUTPUT_FRAG
    Lapsed time : 120 ns
  Feature: IPV4_OUTPUT_DROP_POLICY
    Lapsed time : 400 ns
  Feature: DEBUG_COND_OUTPUT_PKT
    Lapsed time : 120 ns
  Feature: MARMOT_SPA_D_TRANSMIT_PKT
    Lapsed time : 2760 ns

 What does the IPV4_OUTPUT_STILE_LEGACY feature do?

I have not been able to find any documentation or description of these features

[M02@rt37]

Hello @rasmus.elmholt 

Certain features such as NetFlow, NAT, ACL, QoS, IPsec, and deep packet inspection, can increase the processing load on the QFP. For instance: NetFlow: High traffic monitoring can strain the routerm and NAT: If configured for a large number of translations, NAT processing can also consume QFP resources.

Use show platform hardware qfp active feature commands to see which features are most resource-intensive.

[rasmus.elmholt]

Hi M02@rt37 

I am not quite sure how should go from here. We have had some instances where we got better performance from disabling Netflow. But I cannot find a good way to pinpoint the exact reason for the high utilization.

Let us take a look at the current router:

R1#show platform hardware qfp active datapath utilization 
  CPP 0: Subdev 0            5 secs        1 min        5 min       60 min
Input:  Priority (pps)            0            0            0            0
                 (bps)            0            0            0            0
    Non-Priority (pps)         9989        12592        12139        12479
                 (bps)     53615472     80081840     73378472     73910584
           Total (pps)         9989        12592        12139        12479
                 (bps)     53615472     80081840     73378472     73910584
Output: Priority (pps)            0            0            0            0
                 (bps)          416          568          576          568
    Non-Priority (pps)        10034        12681        12273        12625
                 (bps)     52153224     77837856     71440264     71747248
           Total (pps)        10034        12681        12273        12625
                 (bps)     52153640     77838424     71440840     71747816
Processing: Load (pct)           33           40           39           40

Crypto/IO
    Crypto: Load (pct)            7            8            7            7
        RX: Load (pct)            1            2            2            2
        TX: Load (pct)           23           27           26           26
            Idle (pct)           67           62           64           64

It is running an IPSEC tunnel, NAT but no Netflow anymore.

It looks like the QFP is loaded at 40% percent at the moment(1 min), but how do I see if it is ipsec, nat(or Non-NATed traffic as @Joseph W. Doherty mentions)

Is the Crypto/IO paragraph showing that all of the QFP load(40% 1min) is from encryption/decryption as the summary of Crypto+RX+TX is 37%(1min) ?

I was hoping there was a QFP feature summary that showed that 10% is used by NAT, 30% by IPSEC and 5% by ACL fx.

[Leo Laohoo]

QFP is Cisco's term for an "accelerator" or of sort. 

The best way to look at a snapshot of the QFP utilization is the command "sh platform resources".  

[rasmus.elmholt]

Hi @Leo Laohoo 

The show platform resources does not tell me anything about what is loading the QFP. Only that it is being utilized.

R1#show platform resources 
**State Acronym: H - Healthy, W - Warning, C - Critical                                             
Resource                 Usage                 Max             Warning         Critical        State
----------------------------------------------------------------------------------------------------
RP0 (ok, active)                                                                               H    
 Control Processor       43.36%                100%            80%             90%             H    
  DRAM                   2720MB(69%)           3897MB          88%             93%             H    
  bootflash              904MB(32%)            2831MB          70%             90%             H    
ESP0(ok, active)                                                                               H    
 QFP                                                                                           H    
  DRAM                   92095KB(46%)          196608KB        85%             95%             H    
  IRAM                   414KB(20%)            2048KB          85%             95%             H    
  CPU Utilization        35.00%                100%            90%             95%             H    
  B4Q Pool 124           4KB(0%)               1904KB          75%             85%             H    
  B4Q Pool 128           1KB(0%)               2293KB          75%             85%             H    
  B4Q Pool 256           5KB(0%)               4369KB          75%             85%             H    
  B4Q Pool 512           5KB(0%)               5767KB          75%             85%             H    
  B4Q Pool 1024          27KB(0%)              8155KB          75%             85%             H    
  B4Q Pool 1536          24KB(0%)              9678KB          75%             85%             H    
  B4Q Pool 2048          18KB(0%)              8018KB          75%             85%             H    
  B4Q Pool 4096          72KB(0%)              7624KB          75%             85%             H    
  B4Q Pool 8192          152KB(3%)             3864KB          75%             85%             H    
  B4Q Pool 16384         0KB(0%)               3248KB          75%             85%             H    
  B4Q ZNM                7KB(0%)               145312KB        75%             85%             H    

[Leo Laohoo]

---

@rasmus.elmholt wrote:

Resource                 Usage                 Max             Warning         Critical        State
----------------------------------------------------------------------------------------------------
RP0 (ok, active)                                                                               H    
 Control Processor       43.36%                100%            80%             90%             H    
  DRAM                   2720MB(69%)           3897MB          88%             93%             H    

---

Uhhhh ... WTF is this router?  What is the firmware version and what is the uptime?

The Data-Plane memory utilization is >45%.  That is a sign of a memory leak.

[rasmus.elmholt]

C1111-8P - Version 17.06.03a - Uptime is 32 weeks, 6 days, 2 hours, 38 minutes

R1#show platform resources      
**State Acronym: H - Healthy, W - Warning, C - Critical                                             
Resource                 Usage                 Max             Warning         Critical        State
----------------------------------------------------------------------------------------------------
RP0 (ok, active)                                                                               H    
 Control Processor       47.02%                100%            80%             90%             H    
  DRAM                   2726MB(69%)           3897MB          88%             93%             H    
  bootflash              903MB(32%)            2831MB          70%             90%             H    
ESP0(ok, active)                                                                               H    
 QFP                                                                                           H    
  DRAM                   110592KB(56%)         196608KB        85%             95%             H    
  IRAM                   414KB(20%)            2048KB          85%             95%             H    
  CPU Utilization        41.00%                100%            90%             95%             H    

[Leo Laohoo]

The memory utilization of 69% is not good.  I give this router about 6 months before it crashes.