2811 ISR High CPU, HQF Shaper Backg is suspect

[mmedwid]

I recently put in a Cisco 2811 for acting as a SIP gateway and it provides Zone firewall.  All is working.  I am however seeing fairly constant high CPU. The highest proc % appears to be HQF Shaper Backg.  What is this and how can I lower the CPU impact? 

Also - does anyone understand why the individual proc percents do not add up to the total CPU utilization figure?  Do tell!

c2811-1#sho proc cpu
CPU utilization for five seconds: 53%/50%; one minute: 54%; five minutes: 52%

180      121888      51382361      2      1.35%      1.43%      1.34%      0      HQF Shaper Backg
115      321248      307382      1045      1.19%      0.72%      0.85%      0      IP Input     
41      21564      207381      103      0.15%      0.15%      0.15%      0      Per-Second Jobs
2      8120      41499      195      0.07%      0.19%      0.16%      0      Load Meter
342      156972      118980      1319      0.07%      0.12%      0.14%      0      SCCP Application
90      267688      207205      1291      0.07%      0.12%      0.13%      0      tCOUNTER
190      14908      56070      265      0.07%      0.10%      0.12%      0      FW DP Inspect pr
327      9512      404794      23      0.07%      0.08%      0.07%      0      IP NAT Ager
181      14652      2069324      7      0.07%      0.07%      0.07%      0      RBSCP Background
126      6276      254228      24      0.07%      0.03%      0.02%      0      Spanning Tree
64      5752      828956      6      0.07%      0.02%      0.00%      0      Netclock Backgro
50      2116      41496      50      0.07%      0.01%      0.00%      0      Compute load avg
153      2596      38679      67      0.07%      0.01%      0.00%      0      TCP Timer
109      2048      28821      71      0.07%      0.00%      0.00%      0      CDP Protocol
188      2492      404687      6      0.07%      0.00%      0.00%      0      Inspect process
332      1792      25084      71      0.07%      0.00%      0.00%      0      IP SLA Mon Sync
230      1236      83      14891      0.00%      0.31%      0.08%      514      SSH Process
5      387280      28658      13513      0.00%      0.16%      0.17%      0      Check heaps
51      105612      3472      30418      0.00%      0.07%      0.05%      0      Per-minute Jobs
317      398956      2439      163573      0.00%      0.05%      0.06%      0      CCSIP_SPI_CONTRO
349      9776      829031      11      0.00%      0.04%      0.03%      0      MLD     
339      16228      64851      250      0.00%      0.03%      0.05%      0      NAT MIB Helper
258      7220      1036363      6      0.00%      0.03%      0.02%      0      Atheros LED Ctro
19      43352      124032      349      0.00%      0.01%      0.00%      0      ARP Input
122      904      239      3782      0.00%      0.01%      0.00%      0      AFW_application_
147      3716      318911      11      0.00%      0.01%      0.00%      0      CEF: IPv4 proces
1      392      428      915      0.00%      0.00%      0.00%      0      Chunk Manager
3      0      5      0      0.00%      0.00%      0.00%      0      Collection proce
4      0      1      0      0.00%      0.00%      0.00%      0      EDDRI_MAIN
6      40      22      1818      0.00%      0.00%      0.00%      0      Pool Manager

Tags:

[David Aicher]

The output of show process cpu is a little cryptic. For the five second field it is split into two  total/interrupt  the difference is what we see used for processes. In this case it is 53/50 so the processes should account for about 3%(not super accurate but close)

http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d0.shtml#showproccpu

As for interrupts this is typically packet forwarding.    There are other things that might fall under interrupt like alignment errors.  You can perform show align to see if there are any incrementing.  For more information see the following link.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml

HQF shaper backg is QOS shaping.  It is probably not high enough to worry about given the 1% it is using.

Hopefully that explains it.  If you feel this is higher than it should be please check out the performance information in the following document from Miercom.

http://www.miercom.com/dl.html?fid=20061201&type=report

Regards

David Aicher

[mmedwid]

If I remove all of the 0.0 utilization PIDs - note that the only ones left are near 1%.  Yes they don't

nearly add up to 24% total.  Any idea how that math works out?  It would seem to me that here in

this snapshot that HQF shaper 1.67 really means a higher percent of CPU than 1.67 percent. 

sfo-c2811-1#sho proc cpu | exc 0.0

CPU utilization for five seconds: 24%/19%; one minute: 24%; five minutes: 23%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   5       10660         653      16324  1.03%  0.24%  0.18%   0 Check heaps   

115       65640       29008       2262  0.23%  1.39%  0.86%   0 IP Input      

180       19504     1058750         18  1.67%  1.57%  1.53%   0 HQF Shaper Backg

343        5504        3578       1538  0.23%  0.21%  0.16%   0 SCCP Application

[vragotha]

CPU utilization for five seconds: 24%/19%; one minute: 24%; five minutes: 23%

24%/19% means 24% is overall CPU utilization, of which 19% is due to interrupts, meaning that 5% is utilized by processes

[mmedwid]

Thank you.  That clarifies a lot!

[mmedwid]

So that raises the question what would cause the interrupt portion of CPU utilization to go so high?  Here it looks like

processes are only taking up 4% yet the router is at 58% utilization.  Thank you.

sfo-c2811-1#sho proc cpu

CPU utilization for five seconds: 58%/54%; one minute: 47%; five minutes: 46%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   1         332         259       1281  0.00%  0.00%  0.00%   0 Chunk Manager 

   2       12416       14027        885  0.07%  0.08%  0.14%   0 Load Meter    

   3        4652        1054       4413  0.00%  0.02%  0.00%   0 AFW_application_

   4           0           1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN    

   5      156908       10102      15532  0.00%  0.21%  0.18%   0 Check heaps   

   6          64          25       2560  0.00%  0.00%  0.00%   0 Pool Manager  

   7           0           2          0  0.00%  0.00%  0.00%   0 Timers        

   8           0           1          0  0.00%  0.00%  0.00%   0 License Client N

[David Aicher]

Interrupt level CPU is typically packet forwarding.  As long as the CPU is not sustained near 100% it shouldn't cause an issue.  If you think it is higher than you expect it to be or you are planning on adding capicity then you may want to check on performance specs.

Here is a link to some third party testing.

http://www.miercom.com/dl.html?fid=20061201&type=report

Regards

Dave Aicher