topic Re: High CPU Utilization in Routing and SD-WAN

High CPU Utilization

yahyasharif — Sun, 27 Sep 2020 05:09:54 GMT

Sometimes my C-3900 router behaves abnormally. I am showing my last 5 second sorted utilization of the router.

It shows my highest utilization for a process id is .31 % for 5 seconds and 1 min it is 0.33% but why the cpu utilization is 91 % for 5 minutes (It was 99% for a few minutes ago )

DC-Pri#show processes cpu | inc CPU
CPU utilization for five seconds: 87%/84%; one minute: 88%; five minutes: 91%

DC-Pri#show processes cpu sorted 5sec
CPU utilization for five seconds: 91%/89%; one minute: 93%; five minutes: 97%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
138 102189776 903689724 113 0.31% 0.33% 0.35% 0 IP Input
16 21162696 17113266 1236 0.31% 0.05% 0.02% 0 Environmental mo
310 410495156 81843786 5015 0.31% 0.41% 0.99% 0 Multicast Replic
331 259935276 321995169 807 0.31% 0.48% 1.29% 0 OSPF-1 Router
145 6060 1373 4413 0.15% 0.05% 0.01% 645 SSH Process
2 906128 3436296 263 0.15% 0.10% 0.14% 0 Load Meter
109 5120636 2142891642 2 0.15% 0.14% 0.08% 0 Ethernet Msec Ti
87 8422300 68483268 122 0.15% 0.14% 0.15% 0 Netclock Backgro
69 6545288 17236839 379 0.15% 0.10% 0.09% 0 Per-Second Jobs
333 75371352 774831635 97 0.15% 0.19% 0.22% 0 OSPF-1 Hello
320 4102072 1055599726 3 0.07% 0.07% 0.05% 0 HSRP Common
--More--

Re: High CPU Utilization

Leo Laohoo — Sun, 27 Sep 2020 06:15:38 GMT

Please post the output in a text file. Post the output to the following commands:

sh version
sh proc cpu h
sh proc cpu sort | ex 0.00
sh interface <WAN port>

Re: High CPU Utilization

Georg Pauwen — Sun, 27 Sep 2020 06:50:03 GMT

Hello,

in addition to Leo's remarks, also post the output of 'sh run' (your full running configuration).

Re: High CPU Utilization

balaji.bandi — Sun, 27 Sep 2020 06:54:23 GMT

based on the current output i can only see IP input process taking high CPU, with this we can not say what is wrong.

CPU high may cause due to any reason

1. mis-configuration.

2. routing loops

3.acl with log enable in the end

4. maybe bug

what kind of bandwidth it serving ? or expecting to serve here?

so make a correct suggestion, can we have your configuration, show process CPU sort | ex 0.00 and show version for the bug.

Re: High CPU Utilization

Joseph W. Doherty — Sun, 27 Sep 2020 09:12:27 GMT

Your other CPU usage, was likely "interrupt" CPU usage, at least was in the last five seconds. I.e. "CPU utilization for five seconds: 91%/89%". BTW, ideally we like to see interrupt usage within 2 to 3% of total usage. Also, the sum of all the process CPU percentages should equal the delta between the overall usage (first number) and interrupt usage (second number).

Interrupt usage is not shown as any process CPU, and generally represents "best case" traffic forwarding. It's usage should correlate with the amount of traffic passing through the router.

Re: High CPU Utilization

yahyasharif — Sun, 27 Sep 2020 11:48:35 GMT

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*0 C3900-SPE150/K9 FOC19433ZW9

Technology Package License Information for Module:'c3900'

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
ipbase ipbasek9 Permanent ipbasek9
security securityk9 Permanent securityk9
uc None None None
data None None None

777777777777788888888888888888888888887777788888888888888888
888444447777722222111111111133333333337777799999111112222233
100
90 *****
80 *** ****************************************************
70 ************************************************************
60 ************************************************************
50 ************************************************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

DC-Pri#sh int GigabitEthernet0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 84b2.61c9.f480 (bia 84b2.61c9.f480)
Description: BVI-1-Member
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 7/255, rxload 16/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 63990000 bits/sec, 11354 packets/sec
5 minute output rate 28097000 bits/sec, 10567 packets/sec
33626926783 packets input, 22117511554204 bytes, 0 no buffer
Received 8894219 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
2177528 input errors, 0 CRC, 0 frame, 2177528 overrun, 0 ignored
0 watchdog, 8865525 multicast, 0 pause input
30045734357 packets output, 8621610418918 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
3525877704 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

DC-Pri#sh int GigabitEthernet0/1
GigabitEthernet0/1 is down, line protocol is down
Hardware is PQ3_TSEC, address is 84b2.61c9.f481 (bia 84b2.61c9.f481)
Description: BVI
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto Duplex, Auto Speed, media type is unknown media type
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Re: High CPU Utilization

yahyasharif — Sun, 27 Sep 2020 11:49:28 GMT

That show running-config is huge and it's a property of our office ..It is confidential

Re: High CPU Utilization

Georg Pauwen — Sun, 27 Sep 2020 12:11:14 GMT

Hello,

check for access lists with the 'log' keyword, these would punt traffic to the CPU.

Re: High CPU Utilization

Leo Laohoo — Sun, 27 Sep 2020 12:17:01 GMT

@yahyasharif wrote:DC-Pri#sh int GigabitEthernet0/0 GigabitEthernet0/0 is up, line protocol is up Full Duplex, 1Gbps, media type is RJ45 2177528 input errors, 0 CRC, 0 frame, 2177528 overrun, 0 ignored

I think I found an issue: Self-inflicted line errors

Look at Gi 0/0. The interface is a COPPER LINK.

Yet the speed and duplex settings are hard-coded.

The line errors are due to speed/duplex mismatch. The "sh logs" will probably be filled with it.

This means that the router is dropping packets and is doing a lot of retries.

Re: High CPU Utilization

Leo Laohoo — Sun, 27 Sep 2020 12:23:43 GMT

@yahyasharif wrote:

That show running-config is huge and it's a property of our office ..It is confidential

Uhhhh ... what a lame excuse.

Passwords and IP addresses can be removed.
If that is still not acceptable, then raise a TAC Case.

Post the complete output to the following commands:

sh version sh proc cpu h sh proc cpu sort | ex 0.00

Re: High CPU Utilization

yahyasharif — Sun, 27 Sep 2020 12:38:15 GMT

Alright I am sharing this tomorrow.. Work hour is over now

Re: High CPU Utilization

yahyasharif — Sun, 27 Sep 2020 12:41:48 GMT

So how should I configure this? Should I change the speed settings or what?

Re: High CPU Utilizatio

StillMaKiniT — Sun, 27 Sep 2020 17:38:26 GMT

Uhhh... you may be right, or you may be wrong, with regard to an, "excuse." But that was kinda of a rude reaction to his inquiry. Especially considering if he's got even a single NAT'd global in that config. If so, and he sends it, he might as well post the text version of the config in a Wuhan magazine ad in terms of risk. If it really needs to be sent key it, attatch and email. DAO.

Re: High CPU Utilizatio

Leo Laohoo — Sun, 27 Sep 2020 22:49:22 GMT

When I first started with CSC there were a lot of people who "refuse" to share the config because they were "highly confidential" (but are not government or intelligence) -- They just want to feel "special".

The more senior guys in the forum gave them an option: Share your config or go and raise a TAC Case.

Re: High CPU Utilization

yahyasharif — Tue, 29 Sep 2020 05:33:14 GMT