CPU high utilization due to interrupts

[johnywalker]

Hi,

I have and issue in our network that router cpu usage showing as high.

When i issue "show process cpu sorted" it giving the output

CPU utilization for five seconds: 87%/73%; one minute: 78%; five minutes: 69%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 121   338610360   229817079       1473  3.59%  2.43%  2.28%   0 IP Input         
 349   424138972   252919405       1676  3.03%  1.76%  1.83%   0 IP-EIGRP: PDM    
 319    25626784      423522      60509  1.83%  0.42%  0.40%   0 L2TP mgmt daemon 
 197    24295708   612702392         39  0.79%  1.14%  1.28%   0 HQF Shaper Backg 
 336    71451564    11152148       6406  0.63%  0.51%  0.49%   0 DNS Server Input 
 354        7700         707      10891  0.55%  0.67%  0.49% 516 Virtual Exec     
 351    73787092   141822344        520  0.55%  0.38%  0.39%   0 IP-EIGRP: HELLO  
 335    55353756    14943600       3704  0.39%  0.34%  0.33%   0 DNS Server       
 328    11407336    94673937        120  0.31%  0.33%  0.34%   0 PPP manager      
  19    15587772    12813254       1216  0.31%  0.29%  0.29%   0 ARP Input        
   2     2076148      640461       3241  0.23%  0.35%  0.21%   0 Load Meter       
 118    11051532    12211664        904  0.23%  0.20%  0.18%   0 ADJ resolve proc 
  41     9637756     3197987       3013  0.23%  0.27%  0.25%   0 Per-Second Jobs  
 145    25610012     5329013       4805  0.15%  0.08%  0.09%   0 IP RIB Update    
 329     6482628    94672351         68  0.15%  0.15%  0.15%   0 PPP Events       
 321     1519860     6158214        246  0.07%  0.05%  0.06%   0 IP NAT Ager      
 343     1026372     2613651        392  0.07%  0.04%  0.01%   0 NAT MIB Helper   
 326     1242604    24193482         51  0.07%  0.03%  0.02%   0 PPPoE Background 
 153     4248308     4696498        904  0.07%  0.12%  0.15%   0 CEF: IPv4 proces 
  96     2639052     1278400       2064  0.07%  0.07%  0.07%   0 DSL State Machin 
 270      208452     3196936         65  0.07%  0.00%  0.00%   0 Crypto Device Up 
  64     1032652    12686268         81  0.07%  0.02%  0.01%   0 Netclock Backgro 
 105      191480     3170616         60  0.07%  0.00%  0.00%   0 linktest         
 208      447200     6148645         72  0.07%  0.01%  0.00%   0 CCE DP URLF cach 
 176      270304     3170565         85  0.07%  0.00%  0.00%   0 ATMSIG Timer     
  25           0           2          0  0.00%  0.00%  0.00%   0 DDR Timers       
  24           0           1          0  0.00%  0.00%  0.00%   0 Policy Manager   
  23           0           1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT 
  26           4           2       2000  0.00%  0.00%  0.00%   0 Entity MIB API   
  27         980        2581        379  0.00%  0.00%  0.00%   0 EEM ED Syslog    
  28           0           2          0  0.00%  0.00%  0.00%   0 Serial Backgroun 
  32      243984     3196903         76  0.00%  0.00%  0.00%   0 GraphIt          
  33           0           2          0  0.00%  0.00%  0.00%   0 Dialer event     
  22           0           2          0  0.00%  0.00%  0.00%   0 AAA high-capacit 
  35           0           2          0  0.00%  0.00%  0.00%   0 XML Proxy Client 
  29           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers 
  30           0           1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa 
  21           0           2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer   
  39        1212        4571        265  0.00%  0.00%  0.00%   0 Logger           
  40      265464     3170551         83  0.00%  0.00%  0.00%   0 TTY Background   
  31           0           2          0  0.00%  0.00%  0.00%   0 SMART            
  34           0           1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect  
  43          24          11       2181  0.00%  0.00%  0.00%   0 IF-MGR control p 
  44           0          21          0  0.00%  0.00%  0.00%   0 IF-MGR event pro 
  45           0           1          0  0.00%  0.00%  0.00%   0 Inode Table Dest 
  46           0           1          0  0.00%  0.00%  0.00%   0 IKE HA Mgr       
  47           0           1          0  0.00%  0.00%  0.00%   0 IPSEC HA Mgr     
  48           4           4       1000  0.00%  0.00%  0.00%   0 rf task          
  36           0           1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd   
  ........

........ output omitted..

show processes cpu | inc five
CPU utilization for five seconds: 98%/88%; one minute: 86%; five minutes: 73%

The version used in the router is Version 12.4(24)T2.

EIGRP, DMVPN, Netflow also configured in this router.

Is it a IOS bug or any other issue.. ??

I am waiting for anyone to give an answer.

Regards,

Sijoy

[Leo Laohoo]

The version used in the router is Version 12.4(24)T2

Too many garbage with your output.  Can you post the output to the command "sh proc cpu sort | ex 0.00"?

Go to the Feature Navigator and look for IOS compatible in the "M" train.

[Richard Burts]

I appreciate Leo's point that there are a lot of entries that are not using CPU cycles (have 0.00 entries) but since the output is sorted it is not difficult to ignore those entries.

The output does show a pretty busy CPU (87%/73% and 98%/88%) but most of the effort is doing Interrupt based forwarding of traffic, which is sort of confirmed by the fact the the most busy process is IP INput. There is between 10% and 14% process based activity. You might use this link to investigate possible causes of high interrupt levels

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html#causes

HTH

Rick

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Just to add to Rick's post, your interrupt CPU load is also high, which might just be due to "normal" packet forwarding for the router.  What exact model router is this and what's the aggregate traffic volume that corresponds to this CPU usage?

[johnywalker]

Hi all thanks for your comments.

It is a 2811 router.

DMVPN, EIGRP,Netflow are running on this router. There are around 95 users connected below this router.

CEF is enabled globally and it is working as packet switching.

I did CPU profile for five minutes for checking the CPU high utilization.

please find the attached.

If anyone can give me a solution it would be appreciated.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Again, what's your aggregate bandwidth being forwarded?  From my experience, a 2811 tops out at about 40 Mbps, aggregate.  Normally, I wouldn't recommend such a router for more than 10 Mbps (duplex).  I believe I recall Cisco recommends a 2811 for only a pair of T1/E1s.

Any enabled features, beyond packet forwarding, will add to the CPU load.  How much depends on the feature and how it's being used.

Of those you mention, might your DMVPN be fragmenting packets?  This can be contributor to much additional CPU loading.

[johnywalker]

Hi,

When i check show version DRAM and flash size showing as 256\64.

But the recommended for the same version of IOS is 256\128.

IOS installed here is, flash:c2800nm-advipservicesk9-mz.124-24.T2.bin

and also flash memory free space is around 1.5 MB .

This also will affect high CPU utilization or not ?

Thanks,

Sijoy

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Recommended flash insures you have sufficient space for the IOS image and perhaps some crash files.  Being low on it doesn't impact performance.

Recommend RAM insures you have sufficient space to run your IOS version in a "normal" environment.  Some features (e.g. Netflow) will use additional RAM, and if you run short, can cause problems. Some of the later IOS images support "graphing" memory usage, like CPU usage.  Don't recall whether you IOS version does.

Probably neither is a contributor to your high CPU.