Solved: Troubleshooting High CPU on 1841

jwbensley · ‎08-15-2012

Hi All,

I am trying to troubleshoot high CPU usage on an 1841. It has a five minute average (as you can see below) of 37% CPU usage, not no process has a CPU usage that high, so how can this be? An invisible process is tying up the CPU? This router typically has a sustained CPU of 4-5%, now its circa 40%.

Any ideas people can give to diagnose this would be greatly appreciated

1841#show processes cpu sorted

CPU utilization for five seconds: 47%/44%; one minute: 42%; five minutes: 37%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

63 2455548 828599 2963 1.30% 1.43% 1.34% 0 IP Input

196 946308 19241151 49 0.81% 0.90% 0.94% 0 IP SLA Mon Event

203 364616 2545972 143 0.24% 0.20% 0.21% 0 PPP manager

204 254988 2546391 100 0.16% 0.14% 0.15% 0 PPP Events

54 99092 66335 1493 0.16% 0.09% 0.08% 0 DSL State Machin

110 11132 83074 134 0.08% 0.00% 0.00% 0 RUDPV1 Main Proc

2 29676 16674 1779 0.08% 0.02% 0.00% 0 Load Meter

126 94244 815990 115 0.08% 0.04% 0.06% 0 RBSCP Background

52 94880 65786 1442 0.08% 0.09% 0.08% 0 DSL State Machin

47 5104 16844 303 0.08% 0.00% 0.00% 0 ATM Periodic

198 36060 172104 209 0.08% 0.03% 0.01% 0 IP NAT Ager

215 636 85 7482 0.08% 0.14% 0.06% 195 SSH Process

73 36544 325090 112 0.08% 0.01% 0.00% 0 SSS Feature Time

13 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

12 0 5 0 0.00% 0.00% 0.00% 0 AAA high-capacit

14 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

11 4 2 2000 0.00% 0.00% 0.00% 0 ATM Idle Timer

18 6560 24952 262 0.00% 0.00% 0.00% 0 HC Counter Timer

15 2136 5055 422 0.00% 0.00% 0.00% 0 DDR Timers

16 4 2 2000 0.00% 0.00% 0.00% 0 Entity MIB API

21 0 2 0 0.00% 0.00% 0.00% 0 SMART

17 124 479 258 0.00% 0.00% 0.00% 0 EEM ED Syslog

23 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

19 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

25 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

26 564 77 7324 0.00% 0.05% 0.03% 194 SSH Process

27 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

10 96164 133931 718 0.00% 0.04% 0.05% 0 ARP Input

1841#show processes memory sorted

Processor Pool Total: 187088432 Used: 25946504 Free: 161141928

I/O Pool Total: 28311552 Used: 4829408 Free: 23482144

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 41699136 11015608 24831508 0 0 *Init*

39 0 654496 1272 635224 0 0 USB Startup

0 0 0 0 396868 0 0 *MallocLite*

26 194 668188 305088 355188 0 0 SSH Process

0 0 82213740 85010240 327468 384284 4680 *Dead*

206 0 5649108 226327924 304828 240700 0 DNS Server

17 0 256856 0 266828 113400 0 EEM ED Syslog

5 0 1124620 1508800 261576 660112 841924 Pool Manager

1 0 541068 319832 228208 0 0 Chunk Manager

208 0 228498968 6552940 193324 12924 37700 DNS Server Input

201 0 175812 196 182652 0 0 PPPoE Background

204 0 474392 402664 135540 0 0 PPP Events

145 0 116528 496 125004 0 0 Crypto WUI

29 0 134284 184 100436 0 0 Net Background

152 0 666584 611404 80544 0 0 Crypto ACL

84 0 66676 0 76648 0 0 IP RIB Update

87 0 73552 0 73552 0 0 CEF process

3 0 629332 43544 59636 0 0 Dynamic DNS Upda

215 195 1564748 1515020 44248 0 0 SSH Process

221 196 813376 720008 43636 0 0 SSH Process

195 0 40556 196 39380 0 0 DHCPD Receive

159 0 24912 0 37884 0 0 Crypto Delete Ma

193 0 14216 27956 29036 0 0 VTEMPLATE Backgr

202 0 158520 12996 25996 0 0 PPPoE Discovery

147 0 284 0 25256 0 0 RPMS_PROC_MAIN

192 0 196 0 25168 0 0 tHUB

Joseph W. Doherty · ‎08-15-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

jwbensley wrote:

Hi All,

I am trying to troubleshoot high CPU usage on an 1841. It has a five minute average (as you can see below) of 37% CPU usage, not no process has a CPU usage that high, so how can this be? An invisible process is tying up the CPU? This router typically has a sustained CPU of 4-5%, now its circa 40%.

Any ideas people can give to diagnose this would be greatly appreciated

That appears to be mostly interrupt CPU, which ought to be fast path packet forwarding.

What's you overall interface loading when this is happening?

View solution in original post

Joseph W. Doherty · ‎08-15-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

jwbensley wrote:

Hi All,

I am trying to troubleshoot high CPU usage on an 1841. It has a five minute average (as you can see below) of 37% CPU usage, not no process has a CPU usage that high, so how can this be? An invisible process is tying up the CPU? This router typically has a sustained CPU of 4-5%, now its circa 40%.

Any ideas people can give to diagnose this would be greatly appreciated

That appears to be mostly interrupt CPU, which ought to be fast path packet forwarding.

What's you overall interface loading when this is happening?

francisco_1 · ‎08-15-2012

Use CPU profiling to determine where the CPU spends its time when it increased by interrupts. You will need cisco to decode output.

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

jwbensley · ‎08-16-2012

Of course, interrupt processing, mental slip I think!

Although the router was only pushing around 5Mbps of traffic (in one interface and out the other, and it only has these two Ethernet interfaces. There are 2 WIC-1ADSL cards which weren't doing anything, a few Kbps or bps).

Due to the wonders of CEF shouldn't CPU processing only occur for control packets like ICMP, or BGP and IGP updates etc. Despite the low traffic volume, do you think (as a side question) the type of traffic passing through was causing the high CPU load as apposed to the volume of traffic?

Many thanks,

James.

Joseph W. Doherty · ‎08-16-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

CEF is just an optimal fast path routing lookup, still uses CPU, and I believe it would be in the "interrupt" category.

The 1841 is rated at 75 Kpps (fast path), so 5 Mbps duplex shouldn't need much more than about 25% CPU, but much depends on your traffic characteristics and router configuration.

jwbensley · ‎08-17-2012

Great, thanks for the info!