HIGH CPU UTILIZATION ON C2921 ROUTER

[Sam94]

Hi Guys,

I can do with some help right now.

I am experiencing severe traffic degradation through my c2921 internet router. I have noticed high CPU utilization in the past few weeks which is beginning to impact performance.

This router only did static routing and NAT for internal users. I  only recently added ACLs just to troubleshoot this issue to no avail.

Port Mirror captures on the both interfaces of the routers shows little information except few torrent users which macs have long been blocked.

Please see running config and show process cpu sorted | ex 0.00 output below.

Current configuration : 7455 bytes
!
! Last configuration change at 10:24:29 UTC Thu Jun 23 2022 by xxxxxx
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname XXXX
!
boot-start-marker
boot-end-marker
!
!
logging buffered 8000
!
no aaa new-model
!
no ip source-route
no ip gratuitous-arps
ip cef
!
!
!
!
!
!
no ip bootp server
no ip domain lookup
ip domain name yourdomain.com
ip name-server XXXX
ip name-server XXXX
ip name-server 8.8.8.8
no ipv6 cef
multilink bundle-name authenticated
!
!
!
crypto pki trustpoint TP-self-signed-1175820851
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-1175820851
revocation-check none
rsakeypair TP-self-signed-1175820851
!
!
crypto pki certificate chain TP-self-signed-1175820851
certificate self-signed 01
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 31313735 38323038 3531301E 170D3133 30393139 30393036
31345A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 31373538
32303835 3130819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100C190 5B0E9435 5D7FEB5B 33E36C39 FF0537CD 008D45B2 C886AAF8 FA1D399E
AE24D3BA 9D06BED7 B7777F14 EAFEAA4F C7D9FAAA D5711D8C 7BE72431 B5E05BF6
5888B83D 05280CF3 721ABEA7 C12605E4 B55CBC53 680808EB A275D8F8 649B52E6
C41DAF54 D3676780 272567DC 381EF5EF 91B5ACA7 8D94BE18 F1FDB777 A2A47635
A74F0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
551D2304 18301680 14505B43 4C7C4A1A DE580173 DE40ACD3 9CD3C448 36301D06
03551D0E 04160414 505B434C 7C4A1ADE 580173DE 40ACD39C D3C44836 300D0609
2A864886 F70D0101 05050003 818100B7 B96DB8A6 940CAD70 65938FE4 FFAB55DA
ECBD18D5 AB5346B8 F869D6FE 290BB483 FF642756 04FB3140 4426F1FA E999DECB
00EF3F5B C09F300C 83143502 59721DD2 85B665AA 79844EA4 855F22A5 5003AA9E
3D1E531F 1ED3E14C C12A7F15 2732FE1B EF18916C 805EE0C8 349475FB A930C3AE
B10BECCD 70C8359F 23FCF126 97F797
quit
license udi pid CISCO2921/K9 sn FCZ17386103
!
!
username XXXX privilege 15 secret 4 XXXX
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description "WAN Interface"
ip address My_Public_IP
ip access-group Address in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly in max-reassemblies 64
duplex full
speed auto
no cdp enable
no mop enabled
!
interface GigabitEthernet0/1
no ip address
no ip proxy-arp
ip virtual-reassembly in max-reassemblies 64
duplex full
speed auto
no cdp enable
!
interface GigabitEthernet0/1.70
encapsulation dot1Q 70
ip address 10.64.0.50 255.255.248.0
no ip proxy-arp
ip nat inside
ip virtual-reassembly in max-reassemblies 64
no cdp enable
!
interface GigabitEthernet0/2
ip address 10.18.18.1 255.255.255.240
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip nat pool GUEST_DED Public IP Public IP prefix-length 24
ip nat inside source list 18 pool GUEST_DED overload
ip nat inside source list NAT interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 Gateway
ip route yyyyy GigabitEthernet0/1
!
ip access-list standard NAT
permit 10.64.0.0 0.0.7.255
!
ip access-list extended Address
deny icmp host 104.83.208.99 host My_Public_IP
deny ip host 195.158.248.218 host My_Public_IP
deny ip host 213.232.87.127 host My_Public_IP
deny icmp 23.0.0.0 0.255.255.255 host My_Public_IP
deny tcp any any range 6881 6882
permit icmp any any
permit ip any any
ip access-list extended LAN
deny udp any any range 6881 6999
deny tcp any any range 6881 6999
permit icmp any any
permit ip any any
ip access-list extended Spoof
deny ip 100.64.0.0 0.63.255.255 any
deny ip 127.0.0.0 0.255.255.255 any
deny ip 169.254.0.0 0.0.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.0.0.0 0.0.0.255 any
deny ip 192.0.2.0 0.0.0.255 any
deny ip 192.168.0.0 0.0.255.255 any
deny ip 198.18.0.0 0.1.255.255 any
deny ip 198.51.100.0 0.0.0.255 any
deny ip 203.0.113.0 0.0.0.255 any
deny ip 224.0.0.0 31.255.255.255 any
permit ip any any
!
kron occurrence NAT_SCH in 30 recurring
policy-list clearNAT
!
kron policy-list clearNAT
cli clear ip nat translation *
!
logging host 192.168.103.87
access-list 18 permit 10.18.18.0 0.0.0.15
access-list 23 permit 10.10.10.0 0.0.0.7
!
!
snmp-server community public RO
snmp-server enable traps entity-sensor threshold
!
control-plane
!
!
banner exec ^CCCCC
#############################################################################
# WARNING!!! #
# This system is solely for the use of authorized users for official #
# purposes. You have no expectation of privacy in its use and to ensure #
# that the system is functioning properly, individuals using this #
# computer system are subject to having all of their activities monitored #
# and recorded by system personnel. Use of this system evidences an #
# express consent to such monitoring and agreement that if such #
# monitoring reveals evidence of possible abuse or criminal activity, #
# system personnel may provide the results of such monitoring to #
# appropriate officials. #
# XXXXXXXXXX, IT Teams. #
#############################################################################
^C
banner login ^CCCCC
#############################################################################
# WARNING!!! #
# This system is solely for the use of authorized users for official #
# purposes. You have no expectation of privacy in its use and to ensure #
# that the system is functioning properly, individuals using this #
# computer system are subject to having all of their activities monitored #
# and recorded by system personnel. Use of this system evidences an #
# express consent to such monitoring and agreement that if such #
# monitoring reveals evidence of possible abuse or criminal activity, #
# system personnel may provide the results of such monitoring to #
# appropriate officials. #
# XXXXXXXXXX, IT Teams. #
#############################################################################
^C
!
line con 0
login local
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class 23 in
privilege level 15
login local
length 0
transport input telnet ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

CPU utilization for five seconds: 99%/94%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
118 3962911512 2729083990 0 2.85% 2.90% 3.02% 0 IP Input
296 719596728 115240336 6244 1.55% 1.62% 1.52% 0 IP NAT Ager
6 209921588 13835093 15173 0.32% 0.13% 0.08% 0 Check heaps
2 14203860 11351781 1251 0.16% 0.10% 0.11% 0 Load Meter
32 188926716 862465073 219 0.16% 0.12% 0.13% 0 ARP Input
312 148997620 56405304 2641 0.16% 0.19% 0.21% 0 CFT Timer Proces
125 114047692 2526372583 0 0.08% 0.07% 0.07% 0 Ethernet Msec Ti
102 148387844 226561421 654 0.08% 0.11% 0.12% 0 Netclock Backgro
83 27979144 57024663 490 0.08% 0.03% 0.02% 0 Per-Second Jobs
115 54337952 113287150 479 0.08% 0.03% 0.02% 0 BPSM stat Proces

[balaji.bandi]

what kind of bandwidth it serving ?

can you post show interface GigabitEthernet0/0  and GigabitEthernet0/1  (outside and inside interface)

show proc cpu sorted

show interfaces switching

show process cpu |  ex 0.00

[Georg Pauwen]

Hello,

can you post the output of:

show buffers

[Leo Laohoo]

What is the WAN bandwidth?

Post the complete output to the command "sh proc cpu sort | ex 0.00".

[Sam94]

Hi Guys,

Thanks for the replies.

The output of "show proc cpu sorted" was massive so I filtered 0.00 processes. Please find below

#sh proc cpu sort | ex 0.00

CPU utilization for five seconds: 88%/83%; one minute: 90%; five minutes: 96%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
118 3975423804 2734815339 0 2.39% 2.61% 2.68% 0 IP Input
6 210728132 13876584 15185 0.79% 0.13% 0.11% 0 Check heaps
296 723314880 115353810 6270 0.55% 0.65% 0.61% 0 IP NAT Ager
125 114710268 2531643488 0 0.15% 0.22% 0.21% 0 Ethernet Msec Ti
102 149295260 226783035 658 0.15% 0.12% 0.13% 0 Netclock Backgro
32 189486656 863244388 219 0.15% 0.16% 0.15% 0 ARP Input
2 14281420 11362946 1256 0.07% 0.07% 0.07% 0 Load Meter
312 149751064 56461307 2652 0.07% 0.12% 0.12% 0 CFT Timer Proces
162 31637232 1712857442 18 0.07% 0.05% 0.05% 0 IPAM Manager
115 54515920 113397524 480 0.07% 0.03% 0.02% 0 BPSM stat Proces

Output of show interfaces switching

#show interfaces switching

GigabitEthernet0/0 "WAN Interface"
Throttle count 17887
Drops RP 32122 SP 0
SPD Flushes Fast 11929180 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 930411524 Drops 197

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 617486 1100192042 1341458222 3414440991
Cache misses 0 - - -
Fast 3234135465 3054061262 2319160325 2929194355
Auton/SSE 0 0 0 0

Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 84817 6530909
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 912955224 3237706394 830019 49801140
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2597658 208485301 5670407 340224420
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
GigabitEthernet0/1
Throttle count 430
Drops RP 1473 SP 0
SPD Flushes Fast 14865230 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 940266878 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 793773402 157059482 3025661084 795563588
Cache misses 0 - - -
Fast 2363203593 3375404250 3184175351 4123298682
Auton/SSE 0 0 0 0

Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 94503 7276731
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2899221 173953260 1901393 121689152
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 1 413
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2840764 619866302 5670941 340256460
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
GigabitEthernet0/2
Throttle count 100
Drops RP 13 SP 0
SPD Flushes Fast 16 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 48970323 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 1346195 383100388 122 9184
Cache misses 0 - - -
Fast 1390 137153 7689 1223470
Auton/SSE 0 0 0 0

Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 94504 7276808
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 48970307 2938218420 3642 218520
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 946814 528941022 1052021 426067923
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 6951 849599 5670955 340257300
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
NVI0

All statistics for this interface are zero.

Output of show buffers

#show buffers

Buffer elements:
1506 in free list
602575563 hits, 0 misses, 1851 created

Public buffer pools:
Small buffers, 104 bytes (total 264, permanent 50, peak 442 @ 7w0d):
256 in free list (20 min, 150 max allowed)
402824052 hits, 351633 misses, 176491 trims, 176705 created
157993 failures (0 no memory)
Middle buffers, 600 bytes (total 55, permanent 25, peak 172 @ 7w0d):
52 in free list (10 min, 150 max allowed)
4204859625 hits, 155298 misses, 97117 trims, 97147 created
67840 failures (0 no memory)
Big buffers, 1536 bytes (total 107, permanent 50, peak 158 @ 7w0d):
107 in free list (5 min, 150 max allowed)
247899157 hits, 24807 misses, 16505 trims, 16562 created
13816 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 15 @ 7w0d):
10 in free list (0 min, 100 max allowed)
281654 hits, 5510 misses, 427 trims, 427 created
5510 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 4 @ 7w0d):
1 in free list (0 min, 10 max allowed)
542 hits, 5023 misses, 6131 trims, 6132 created
5023 failures (0 no memory)
Huge buffers, 18024 bytes (total 5, permanent 0, peak 17 @ 7w0d):
5 in free list (4 min, 10 max allowed)
11071 hits, 4146 misses, 12844 trims, 12849 created
3260 failures (0 no memory)

Interface buffer pools:
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 5868 trims, 5869 created
0 failures (0 no memory)
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 5868 trims, 5869 created
0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 7w0d):
101 in free list (132 min, 132 max allowed)
7110 hits, 0 misses
IPMUX SF buffers, 1500 bytes (total 500, permanent 500):
500 in free list (0 min, 1000 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 7w0d):
26 in free list (25 min, 50 max allowed)
0 hits, 0 misses, 5868 trims, 5869 created
0 failures (0 no memory)
IPC buffers, 4096 bytes (total 2, permanent 2):
1 in free list (1 min, 8 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 7w0d):
3 in free list (2 min, 4 max allowed)
0 hits, 0 misses, 5868 trims, 5869 created
0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 7w0d):
2 in free list (1 min, 2 max allowed)
0 hits, 0 misses, 5868 trims, 5869 created
0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 7w0d):
17 in free list (16 min, 16 max allowed)
0 hits, 0 misses, 946914 trims, 946915 created
0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 768, permanent 768):
256 in free list (128 min, 1024 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
180289985 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 1664 bytes (total 768, permanent 768):
256 in free list (128 min, 1024 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 1676 bytes (total 3840, permanent 3840):
3840 in free list (128 min, 4096 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)

Private particle pools:
HQF Particle buffers, 0 bytes (total 2000, permanent 2000):
2000 in free list (500 min, 2000 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
IDS SM buffers, 240 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
128 hits, 0 fallbacks
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache
IPMUX particle pool buffers, 512 bytes (total 500, permanent 500):
0 in free list (0 min, 1000 max allowed)
500 hits, 1 misses
1000 max cache size, 500 in cache
0 hits in cache, 0 misses in cache
GigabitEthernet0/0 buffers, 1664 bytes (total 1024, permanent 1024):
0 in free list (0 min, 1024 max allowed)
1024 hits, 0 fallbacks
1024 max cache size, 768 in cache
4199871349 hits in cache, 0 misses in cache
GigabitEthernet0/1 buffers, 1664 bytes (total 1024, permanent 1024):
0 in free list (0 min, 1024 max allowed)
1024 hits, 0 fallbacks
1024 max cache size, 768 in cache
3363890825 hits in cache, 0 misses in cache
GigabitEthernet0/2 buffers, 1664 bytes (total 1024, permanent 1024):
0 in free list (0 min, 1024 max allowed)
1024 hits, 0 fallbacks
1024 max cache size, 768 in cache
80414637 hits in cache, 0 misses in cache

[Georg Pauwen]

Hello,

you have a huge amount of buffer misses. Try and increase the buffer size and monitor if the amount if misses and trims decreases. Start out with the values below (in config mode):

buffers small permanent 100
buffers small max-free 300
buffers small min-free 40
!
buffers middle permanent 50
buffers middle max-free 300
buffers middle min-free 20
!
buffers big permanent 100
buffers big max-free 300
buffers big min-free 10
!
buffers verybig permanent 20
buffers verybig max-free 200
buffers verybig min-free 10
!
buffers large permanent 10
buffers large max-free 20
buffers large min-free 10
!
buffers huge permanent 10
buffers huge max-free 20
buffers huge min-free 8