2811 99% CPU High IP Input

[nfordhk]

Hello,

Our 2811 Edge router is having extremely high CPU problems. IP Input is consuming 70% of the process during business hours. Cisco express forwarding is enabled. It only shows 50% but its currently before business hours. Id say even 50% is still high.

CPU utilization for five seconds: 55%/16%; one minute: 70%; five minutes: 76%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

104     5310320     2813117       1887 36.92% 46.73% 51.43%   0 IP Input        

The below interface is our public interface and is constantly over utilized with 100% Is there any configurations that can be done to correct this? Is there any more information that is needed? Please indicate any command if it can be done live as this is production and I cannot take it down on whim.

IOS: (C2800NM-ADVSECURITYK9-M), Version 12.4(24)T4

Show IP Interface

FastEthernet0/0 is up, line protocol is up

  Internet address is x.x.x.x

  Broadcast address is x.x.x.x

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Outgoing access list is ******************

  Inbound  access list is ******************

  Proxy ARP is disabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are never sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP fast switching on the same interface is disabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: Stateful Inspection, Ingress-NetFlow, Access List, MCI Check

  Output features: Stateful Inspection, CCE Post NAT Classification, QoS Drop, IPsec or interface ACL checked on pre-encrypted cleartext packets, Post-Ingress-NetFlow

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

R-NUS-IRVINE#sh int fa0/0
FastEthernet0/0 is up, line protocol is up
  Hardware is ************ Ethernet, address is *************

  Description: *******************

  Internet address is **********************

  MTU 1500 bytes, BW 2048 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 132/255, rxload 168/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 4/75/213/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/0 (size/max total/drops)
  30 second input rate 2183000 bits/sec, 503 packets/sec
  30 second output rate 1066000 bits/sec, 436 packets/sec
     3417553 packets input, 984199842 bytes
     Received 850 broadcasts, 0 runts, 0 giants, 75 throttles
     848 input errors, 0 CRC, 0 frame, 0 overrun, 848 ignored
     0 watchdog
     0 input packets with dribble condition detected
     4432507 packets output, 3406178496 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

interface FastEthernet0/0

description *****************

bandwidth 2048

ip address ***************** *****************

ip access-group ******************

ip access-group *****************

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

load-interval 30

duplex auto

speed auto

no mop enabled

service-policy output ***************

[Joseph W. Doherty]

Disclaimer

The    Author of this posting offers the  information contained within this    posting without consideration and  with the reader's understanding that    there's no implied or expressed  suitability or fitness for any  purpose.   Information provided is for  informational purposes only and  should not   be construed as rendering  professional advice of any kind.  Usage of  this  posting's information  is solely at reader's own risk.

Liability Disclaimer

In     no event shall Author be liable for any damages whatsoever   (including,   without limitation, damages for loss of use, data or   profit) arising  out  of the use or inability to use the posting's   information even if  Author  has been advised of the possibility of  such  damage.

Posting

Have you reviewed http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml?

[nfordhk]

Yes I did, I did not feel any of those problems pretained to my router

[rsimoni]

Hi Nicholas,

try to enabled debug ip packet detail as indicated in the doc to see which type of traffic is being process-switched. that will help understand what is going on.

Riccardo

[waqas.suneel]

Hello Dears,

I have same problem as reported above. I have Cisco2811 and having same CPU utilization. as i examined , i found two process take cpu resources 1. IP input and ARP input. please tell me the solution. thanks.
 

show processes cpu         
CPU utilization for five seconds: 82%/75%; one minute: 76%; five minutes: 76%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1     1765856    660333       2674  0.00%  0.00%  0.00%   0 Chunk Manager    
   2      912092     82732      11024  0.32%  0.19%  0.15%   0 Load Meter       
   4           0         1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN       
   5     3265804    138426      23592  0.00%  0.18%  0.21%   0 Check heaps      
   6        9588      2258       4246  0.00%  0.00%  0.00%   0 Pool Manager     
   7           0         2          0  0.00%  0.00%  0.00%   0 Timers           
   8           0         1          0  0.00%  0.00%  0.00%   0 chkpt message ha
   9           0         1          0  0.00%  0.00%  0.00%   0 OIR Handler      
  10           0         1          0  0.00%  0.00%  0.00%   0 Crash writer     
  11       55672     13756       4047  0.00%  0.00%  0.00%   0 Environmental mo
  12        1760      6887        255  0.00%  0.00%  0.00%   0 IPC Dynamic Cach
  13           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager
  14      112132    389461        287  0.00%  0.00%  0.00%   0 IPC Periodic Tim
  15      107364    389455        275  0.00%  0.00%  0.00%   0 IPC Deferred Por
  16           0         1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager
  17           0         1          0  0.00%  0.00%  0.00%   0 IPC BackPressure
  18     4534248   3144033       1442  0.32%  0.31%  0.32%   0 ARP Input        
  19      183808    405827        452  0.00%  0.00%  0.00%   0 ARP Background   
  20           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer   
  21           4         2       2000  0.00%  0.00%  0.00%   0 AAA high-capacit
  22           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  23           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager   
  24           4         2       2000  0.00%  0.00%  0.00%   0 DDR Timers       
  25           4         2       2000  0.00%  0.00%  0.00%   0 Entity MIB API   
  26       11508      6934       1659  0.00%  0.00%  0.00%   0 EEM ED Syslog    
  27       94828    121114        782  0.00%  0.00%  0.00%   0 HC Counter Timer
  28           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  29           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
  30           0         1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa
  31           0         2          0  0.00%  0.00%  0.00%   0 SMART            
  32      128040    410628        311  0.00%  0.00%  0.00%   0 GraphIt          
  33           0         2          0  0.00%  0.00%  0.00%   0 Dialer event     
  34           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect  
  35           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
  36           0         2          0  0.00%  0.00%  0.00%   0 cpf_process_msg_
  37           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest
  38           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd   
  39      388128    379421       1022  0.00%  0.01%  0.00%   0 Net Background   
  40           0         2          0  0.00%  0.00%  0.00%   0 IDB Work         
  41       17068     11731       1454  0.00%  0.00%  0.00%   0 Logger           
  42      170556    389432        437  0.00%  0.00%  0.00%   0 TTY Background   
  43      691424    415384       1664  0.08%  0.03%  0.02%   0 Per-Second Jobs  
  44        7160     27448        260  0.00%  0.00%  0.00%   0 c2800 Periodic   
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  45           0         1          0  0.00%  0.00%  0.00%   0 IKE HA Mgr       
  46           0         1          0  0.00%  0.00%  0.00%   0 IPSEC HA Mgr     
  47        2056        26      79076  0.00%  0.00%  0.00%   0 rf task          
  48      122568    155877        786  0.00%  0.00%  0.00%   0 Net Input        
  49      125816     83977       1498  0.00%  0.00%  0.00%   0 Compute load avg
  50      737388      8028      91852  0.00%  0.01%  0.00%   0 Per-minute Jobs  
  51           0         1          0  0.00%  0.00%  0.00%   0 AggMgr Process   
  52           0         1          0  0.00%  0.00%  0.00%   0 Token Daemon     
  53           0         1          0  0.00%  0.00%  0.00%   0 dev_device_inser
  54           0         1          0  0.00%  0.00%  0.00%   0 dev_device_remov
  55           0         1          0  0.00%  0.00%  0.00%   0 sal_dpc_process  
  56           0         1          0  0.00%  0.00%  0.00%   0 ARL Table Manage
  57           0         2          0  0.00%  0.00%  0.00%   0 ESWPPM           
  58           0         2          0  0.00%  0.00%  0.00%   0 Eswilp Storm Con
  59           0         2          0  0.00%  0.00%  0.00%   0 ESWILPPM         
  60           0         2          0  0.00%  0.00%  0.00%   0 Eswilp Storm Con
  61      407984   1470636        277  0.00%  0.02%  0.00%   0 Netclock Backgro
  62           0         2          0  0.00%  0.00%  0.00%   0 SM Monitor       
  63           0         2          0  0.00%  0.00%  0.00%   0 Bryce I2C CMD Qu
  64         272         4      68000  0.00%  0.00%  0.00%   0 USB Startup      
  65       23356     68145        342  0.00%  0.00%  0.00%   0 mxt5100          
  66           0         2          0  0.00%  0.00%  0.00%   0 VNM DSPRM MAIN   
  67           0         1          0  0.00%  0.00%  0.00%   0 DSPFARM DSP READ
  68           0         2          0  0.00%  0.00%  0.00%   0 FLEX DNLD MAIN   
  69           0         1          0  0.00%  0.00%  0.00%   0 HDV background   
  70           0         2          0  0.00%  0.00%  0.00%   0 VMI Background   
  71           0         1          0  0.00%  0.00%  0.00%   0 RF_INTERDEV_DELA
  72           0         1          0  0.00%  0.00%  0.00%   0 RF_INTERDEV_SCTP
  73       80696    389455        207  0.00%  0.00%  0.00%   0 Ether-Switch RBC
  74           0         1          0  0.00%  0.00%  0.00%   0 IGMP Snooping Pr
  75           0         1          0  0.00%  0.00%  0.00%   0 IGMP Snooping Re
  76        3304     13757        240  0.00%  0.00%  0.00%   0 Call Management  
  77           0         1          0  0.00%  0.00%  0.00%   0 AAL2CPS TIMER_CU
  78           4         1       4000  0.00%  0.00%  0.00%   0 CES Line Conditi
  79           0         1          0  0.00%  0.00%  0.00%   0 CF_INTERDEV_SCTP
  80           0         4          0  0.00%  0.00%  0.00%   0 HWIC FE Link Mon
  81           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Mgr Proces
  82           0         1          0  0.00%  0.00%  0.00%   0 MAB Framework    
  83           0         1          0  0.00%  0.00%  0.00%   0 EAP Framework    
  84           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol     
  85       76212    389448        195  0.00%  0.00%  0.00%   0 PI MATM Aging Pr
  86        8580     41074        208  0.00%  0.00%  0.00%   0 EtherChnl        
  87           0         2          0  0.00%  0.00%  0.00%   0 Ethernet CFM     
  88      134396    382172        351  0.00%  0.00%  0.00%   0 Ethernet Timer C
  89       98100    389471        251  0.00%  0.00%  0.00%   0 linktest         
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  90           0         2          0  0.00%  0.00%  0.00%   0 Dot11 Mgmt & Ass
  91           0         2          0  0.00%  0.00%  0.00%   0 Dot11 aaa proces
  92        3360      6887        487  0.00%  0.00%  0.00%   0 pmkid            
  93           0         2          0  0.00%  0.00%  0.00%   0 Dot11 auth Dot1x
  94           0         1          0  0.00%  0.00%  0.00%   0 Dot11 Mac Auth   
  95           0         2          0  0.00%  0.00%  0.00%   0 AAA Server       
  96           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc    
  97     1710572   8928330        191  0.16%  0.16%  0.16%   0 ACCT Periodic Pr
  98           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R
  99       89312     59111       1510  0.00%  0.00%  0.00%   0 CDP Protocol     
 100           0         2          0  0.00%  0.00%  0.00%   0 Ethernet OAM Pro
 101           0         2          0  0.00%  0.00%  0.00%   0 Ethernet LMI     
 102         148       157        942  0.00%  0.00%  0.00%   0 IP ARP Adjacency
 103     1726464   8928269        193  0.08%  0.14%  0.16%   0 IP ARP Retry Age
 104   130440600  14268741       9141  3.52%  3.00%  4.13%   0 IP Input         
 105           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
 106           0         4          0  0.00%  0.00%  0.00%   0 TurboACL         
 107           0         2          0  0.00%  0.00%  0.00%   0 TurboACL chunk   
 108           0         1          0  0.00%  0.00%  0.00%   0 IPv6 RIB Redistr
 109        1456       686       2122  0.00%  0.00%  0.00%   0 MOP Protocols    
 110           0         3          0  0.00%  0.00%  0.00%   0 PPP Hooks        
 111        1008       166       6072  0.16%  0.23%  0.08% 514 Virtual Exec     
 112           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager      
 113       15184     54899        276  0.00%  0.00%  0.00%   0 SSS Test Client  
 114           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
 115      499584   1464068        341  0.00%  0.02%  0.02%   0 SSS Feature Time
 116           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
 117           0         6          0  0.00%  0.00%  0.00%   0 SSM connection m
 118           0         1          0  0.00%  0.00%  0.00%   0 AC Switch        
 119         100         2      50000  0.00%  0.00%  0.00%   0 EAPoUDP Process  
 120           0         2          0  0.00%  0.00%  0.00%   0 IP Host Track Pr
 121           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA         
 122      102268     53881       1898  0.00%  0.00%  0.00%   0 IP Background    
 123        5640      6892        818  0.00%  0.00%  0.00%   0 IP RIB Update    
 124           0         2          0  0.00%  0.00%  0.00%   0 PPP IP Route     
 125           0         2          0  0.00%  0.00%  0.00%   0 PPP IPCP         
 126           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant
 127           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant
 128           4         2       2000  0.00%  0.00%  0.00%   0 Dot1x Supplicant
 129           0         1          0  0.00%  0.00%  0.00%   0 Asy FS Helper    
 130      186500    373039        499  0.00%  0.00%  0.00%   0 Socket Timers    
 131       20556     15069       1364  0.00%  0.00%  0.00%   0 TCP Timer        
 132       11304      2491       4537  0.00%  0.00%  0.00%   0 TCP Protocols    
 133           0         1          0  0.00%  0.00%  0.00%   0 COPS             
 134      256720    520961        492  0.00%  0.02%  0.01%   0 CEF process      
         

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

BTW, if your IOS supports it, sorted would have been a nice option to use when displaying your CPU stats.

Hmm, looking at your stats .  .  .

CPU utilization for five seconds: 82%/75%; one minute: 76%; five minutes: 76%

Most of your utilization appears to be interrupt CPU.  What's the aggregate traffic load?

  18     4534248   3144033       1442  0.32%  0.31%  0.32%   0 ARP Input   

You're really worried about ARP Input at 3/10 of a percent?

104   130440600  14268741       9141  3.52%  3.00%  4.13%   0 IP Input 

IP Input isn't too high.  Maybe a little higher than we would like.  Generally, it represents non fast-path packet forwarding.  It can have many causes, including the nature of the traffic.  Have you reviewed the whitepapers on troubleshooting high IP Input?