High CPU Utilization

[yahyasharif]

Sometimes my C-3900 router behaves abnormally. I am showing my last 5 second sorted utilization of the router.

It shows my highest utilization for a process id is .31 % for 5 seconds and 1 min it is 0.33% but why the cpu utilization is 91 % for 5 minutes (It was 99% for a few minutes ago )

DC-Pri#show processes cpu | inc CPU
CPU utilization for five seconds: 87%/84%; one minute: 88%; five minutes: 91%

DC-Pri#show processes cpu sorted 5sec
CPU utilization for five seconds: 91%/89%; one minute: 93%; five minutes: 97%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
138 102189776 903689724 113 0.31% 0.33% 0.35% 0 IP Input
16 21162696 17113266 1236 0.31% 0.05% 0.02% 0 Environmental mo
310 410495156 81843786 5015 0.31% 0.41% 0.99% 0 Multicast Replic
331 259935276 321995169 807 0.31% 0.48% 1.29% 0 OSPF-1 Router
145 6060 1373 4413 0.15% 0.05% 0.01% 645 SSH Process
2 906128 3436296 263 0.15% 0.10% 0.14% 0 Load Meter
109 5120636 2142891642 2 0.15% 0.14% 0.08% 0 Ethernet Msec Ti
87 8422300 68483268 122 0.15% 0.14% 0.15% 0 Netclock Backgro
69 6545288 17236839 379 0.15% 0.10% 0.09% 0 Per-Second Jobs
333 75371352 774831635 97 0.15% 0.19% 0.22% 0 OSPF-1 Hello
320 4102072 1055599726 3 0.07% 0.07% 0.05% 0 HSRP Common
--More--

[Leo Laohoo · Accepted solution]

Please post the output in a text file.  Post the output to the following commands: 

sh version
sh proc cpu h
sh proc cpu sort | ex 0.00
sh interface <WAN port>

[yahyasharif]

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*0 C3900-SPE150/K9 FOC19433ZW9

Technology Package License Information for Module:'c3900'

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
ipbase ipbasek9 Permanent ipbasek9
security securityk9 Permanent securityk9
uc None None None
data None None None

777777777777788888888888888888888888887777788888888888888888
888444447777722222111111111133333333337777799999111112222233
100
90 *****
80 *** ****************************************************
70 ************************************************************
60 ************************************************************
50 ************************************************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

DC-Pri#sh int GigabitEthernet0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 84b2.61c9.f480 (bia 84b2.61c9.f480)
Description: BVI-1-Member
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 7/255, rxload 16/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 63990000 bits/sec, 11354 packets/sec
5 minute output rate 28097000 bits/sec, 10567 packets/sec
33626926783 packets input, 22117511554204 bytes, 0 no buffer
Received 8894219 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
2177528 input errors, 0 CRC, 0 frame, 2177528 overrun, 0 ignored
0 watchdog, 8865525 multicast, 0 pause input
30045734357 packets output, 8621610418918 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
3525877704 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

DC-Pri#sh int GigabitEthernet0/1
GigabitEthernet0/1 is down, line protocol is down
Hardware is PQ3_TSEC, address is 84b2.61c9.f481 (bia 84b2.61c9.f481)
Description: BVI
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto Duplex, Auto Speed, media type is unknown media type
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Georg Pauwen]

Hello,

in addition to Leo's remarks, also post the output of 'sh run' (your full running configuration).

[yahyasharif]

That show running-config is huge and it's a property of our office ..It is confidential

[Georg Pauwen]

Hello,

check for access lists with the 'log' keyword, these would punt traffic to the CPU.

[Leo Laohoo]

---

@yahyasharif wrote:

DC-Pri#sh int GigabitEthernet0/0
GigabitEthernet0/0 is up, line protocol is up 
 Full Duplex, 1Gbps, media type is RJ45
 2177528 input errors, 0 CRC, 0 frame, 2177528 overrun, 0 ignored

---

I think I found an issue:  Self-inflicted line errors

Look at Gi 0/0.  The interface is a COPPER LINK. 

Yet the speed and duplex settings are hard-coded. 

The line errors are due to speed/duplex mismatch.  The "sh logs" will probably be filled with it.

This means that the router is dropping packets and is doing a lot of retries.

[yahyasharif]

So how should I configure this? Should I change the speed settings or what? 

[Leo Laohoo]

---

@yahyasharif wrote:

That show running-config is huge and it's a property of our office ..It is confidential

---

Uhhhh ... what a lame excuse. 

Passwords and IP addresses can be removed. 
If that is still not acceptable, then raise a TAC Case.

Post the complete output to the following commands: 

sh version
sh proc cpu h
sh proc cpu sort | ex 0.00

[yahyasharif]

Alright I am sharing this tomorrow.. Work hour is over now

[StillMaKiniT]

Uhhh... you may be right, or you may be wrong, with regard to an, "excuse." But that was kinda of a rude reaction to his inquiry. Especially considering if he's got even a single NAT'd global in that config. If so, and he sends it, he might as well post the text version of the config in a Wuhan magazine ad in terms of risk. If it really needs to be sent key it, attatch and email. DAO.

[Leo Laohoo]

When I first started with CSC there were a lot of people who "refuse" to share the config because they were "highly confidential" (but are not government or intelligence) -- They just want to feel "special".  

The more senior guys in the forum gave them an option:  Share your config or go and raise a TAC Case. 

[yahyasharif]

aa

[balaji.bandi]

based on the current output i can only see IP input process taking high CPU, with this we can not say what is wrong. 

CPU high may cause due to any reason 

1. mis-configuration.

2. routing loops

3.acl with log enable in the end

4. maybe bug

what kind of bandwidth it serving ? or expecting to serve here?

so make a correct suggestion, can we have your configuration, show process CPU sort  | ex 0.00 and show version for the bug.

[Joseph W. Doherty]

Your other CPU usage, was likely "interrupt" CPU usage, at least was in the last five seconds.  I.e. "CPU utilization for five seconds: 91%/89%".  BTW, ideally we like to see interrupt usage within 2 to 3% of total usage.  Also, the sum of all the process CPU percentages should equal the delta between the overall usage (first number) and interrupt usage (second number).

Interrupt usage is not shown as any process CPU, and generally represents "best case" traffic forwarding.  It's usage should correlate with the amount of traffic passing through the router.