08-15-2012 05:23 AM - edited 03-07-2019 08:21 AM
Hi All,
I am trying to troubleshoot high CPU usage on an 1841. It has a five minute average (as you can see below) of 37% CPU usage, not no process has a CPU usage that high, so how can this be? An invisible process is tying up the CPU? This router typically has a sustained CPU of 4-5%, now its circa 40%.
Any ideas people can give to diagnose this would be greatly appreciated
1841#show processes cpu sorted
CPU utilization for five seconds: 47%/44%; one minute: 42%; five minutes: 37%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
63 2455548 828599 2963 1.30% 1.43% 1.34% 0 IP Input
196 946308 19241151 49 0.81% 0.90% 0.94% 0 IP SLA Mon Event
203 364616 2545972 143 0.24% 0.20% 0.21% 0 PPP manager
204 254988 2546391 100 0.16% 0.14% 0.15% 0 PPP Events
54 99092 66335 1493 0.16% 0.09% 0.08% 0 DSL State Machin
110 11132 83074 134 0.08% 0.00% 0.00% 0 RUDPV1 Main Proc
2 29676 16674 1779 0.08% 0.02% 0.00% 0 Load Meter
126 94244 815990 115 0.08% 0.04% 0.06% 0 RBSCP Background
52 94880 65786 1442 0.08% 0.09% 0.08% 0 DSL State Machin
47 5104 16844 303 0.08% 0.00% 0.00% 0 ATM Periodic
198 36060 172104 209 0.08% 0.03% 0.01% 0 IP NAT Ager
215 636 85 7482 0.08% 0.14% 0.06% 195 SSH Process
73 36544 325090 112 0.08% 0.01% 0.00% 0 SSS Feature Time
13 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
12 0 5 0 0.00% 0.00% 0.00% 0 AAA high-capacit
14 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
11 4 2 2000 0.00% 0.00% 0.00% 0 ATM Idle Timer
18 6560 24952 262 0.00% 0.00% 0.00% 0 HC Counter Timer
15 2136 5055 422 0.00% 0.00% 0.00% 0 DDR Timers
16 4 2 2000 0.00% 0.00% 0.00% 0 Entity MIB API
21 0 2 0 0.00% 0.00% 0.00% 0 SMART
17 124 479 258 0.00% 0.00% 0.00% 0 EEM ED Syslog
23 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
19 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
25 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
26 564 77 7324 0.00% 0.05% 0.03% 194 SSH Process
27 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
10 96164 133931 718 0.00% 0.04% 0.05% 0 ARP Input
1841#show processes memory sorted
Processor Pool Total: 187088432 Used: 25946504 Free: 161141928
I/O Pool Total: 28311552 Used: 4829408 Free: 23482144
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 41699136 11015608 24831508 0 0 *Init*
39 0 654496 1272 635224 0 0 USB Startup
0 0 0 0 396868 0 0 *MallocLite*
26 194 668188 305088 355188 0 0 SSH Process
0 0 82213740 85010240 327468 384284 4680 *Dead*
206 0 5649108 226327924 304828 240700 0 DNS Server
17 0 256856 0 266828 113400 0 EEM ED Syslog
5 0 1124620 1508800 261576 660112 841924 Pool Manager
1 0 541068 319832 228208 0 0 Chunk Manager
208 0 228498968 6552940 193324 12924 37700 DNS Server Input
201 0 175812 196 182652 0 0 PPPoE Background
204 0 474392 402664 135540 0 0 PPP Events
145 0 116528 496 125004 0 0 Crypto WUI
29 0 134284 184 100436 0 0 Net Background
152 0 666584 611404 80544 0 0 Crypto ACL
84 0 66676 0 76648 0 0 IP RIB Update
87 0 73552 0 73552 0 0 CEF process
3 0 629332 43544 59636 0 0 Dynamic DNS Upda
215 195 1564748 1515020 44248 0 0 SSH Process
221 196 813376 720008 43636 0 0 SSH Process
195 0 40556 196 39380 0 0 DHCPD Receive
159 0 24912 0 37884 0 0 Crypto Delete Ma
193 0 14216 27956 29036 0 0 VTEMPLATE Backgr
202 0 158520 12996 25996 0 0 PPPoE Discovery
147 0 284 0 25256 0 0 RPMS_PROC_MAIN
192 0 196 0 25168 0 0 tHUB
Solved! Go to Solution.
08-15-2012 07:41 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
jwbensley wrote:
Hi All,
I am trying to troubleshoot high CPU usage on an 1841. It has a five minute average (as you can see below) of 37% CPU usage, not no process has a CPU usage that high, so how can this be? An invisible process is tying up the CPU? This router typically has a sustained CPU of 4-5%, now its circa 40%.
Any ideas people can give to diagnose this would be greatly appreciated
That appears to be mostly interrupt CPU, which ought to be fast path packet forwarding.
What's you overall interface loading when this is happening?
08-15-2012 07:41 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
jwbensley wrote:
Hi All,
I am trying to troubleshoot high CPU usage on an 1841. It has a five minute average (as you can see below) of 37% CPU usage, not no process has a CPU usage that high, so how can this be? An invisible process is tying up the CPU? This router typically has a sustained CPU of 4-5%, now its circa 40%.
Any ideas people can give to diagnose this would be greatly appreciated
That appears to be mostly interrupt CPU, which ought to be fast path packet forwarding.
What's you overall interface loading when this is happening?
08-15-2012 08:07 AM
Use CPU profiling to determine where the CPU spends its time when it increased by interrupts. You will need cisco to decode output.
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml
08-16-2012 10:01 AM
Of course, interrupt processing, mental slip I think!
Although the router was only pushing around 5Mbps of traffic (in one interface and out the other, and it only has these two Ethernet interfaces. There are 2 WIC-1ADSL cards which weren't doing anything, a few Kbps or bps).
Due to the wonders of CEF shouldn't CPU processing only occur for control packets like ICMP, or BGP and IGP updates etc. Despite the low traffic volume, do you think (as a side question) the type of traffic passing through was causing the high CPU load as apposed to the volume of traffic?
Many thanks,
James.
08-16-2012 05:12 PM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
CEF is just an optimal fast path routing lookup, still uses CPU, and I believe it would be in the "interrupt" category.
The 1841 is rated at 75 Kpps (fast path), so 5 Mbps duplex shouldn't need much more than about 25% CPU, but much depends on your traffic characteristics and router configuration.
08-17-2012 09:47 AM
Great, thanks for the info!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide