High CPU Utilization Cisco 1941 Router

[jtillman11]

The CPU utilization has been averaging about 65% which is way up from its usual 3-5%. When I do a "show processes cpu sorted" I do not see anything using a high amount of CPU. What could it be/where else should I look?

[Georg Pauwen]

Hello,

post the output of:

show ver

sh run

[jtillman11]

CEMC-Adel1941#show ver
Cisco IOS Software, C1900 Software (C1900-UNIVERSALK9-M), Version 15.1(4)M4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 20-Mar-12 17:58 by prod_rel_team

ROM: System Bootstrap, Version 15.0(1r)M15, RELEASE SOFTWARE (fc1)

CEMC-Adel1941 uptime is 23 weeks, 4 days, 9 hours, 6 minutes
System returned to ROM by reload at 12:37:02 UTC Mon Feb 26 2018
System restarted at 08:01:02 EASTERN Mon Feb 26 2018
System image file is "flash0:c1900-universalk9-mz.SPA.151-4.M4.bin"
Last reload type: Normal Reload
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco CISCO1941/K9 (revision 1.0) with 487424K/36864K bytes of memory.
Processor board ID FGL162722M6
6 Gigabit Ethernet interfaces
1 terminal line
DRAM configuration is 64 bits wide with parity disabled.
255K bytes of non-volatile configuration memory.
250880K bytes of ATA System CompactFlash 0 (Read/Write)

License Info:

License UDI:

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*0 CISCO1941/K9 FGL162722M6

Technology Package License Information for Module:'c1900'

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
ipbase ipbasek9 Permanent ipbasek9
security None None None
data None None None

Configuration register is 0x2102

CEMC-Adel1941#show run
Building configuration...

Current configuration : 6876 bytes
!
! Last configuration change at 09:03:41 DST Thu Jul 12 2018 by admin
! NVRAM config last updated at 09:03:33 DST Thu Jul 12 2018 by admin
! NVRAM config last updated at 09:03:33 DST Thu Jul 12 2018 by admin
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname CEMC-Adel1941
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
enable secret 5 $1$G2Mp$HprsNiCj0niaz3QhM7BSh.
enable password 7 070C24414D441A0C041104
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
clock timezone EASTERN -5 0
clock summer-time DST date Mar 10 2018 21:00 Nov 3 2018 21:00
!
no ipv6 cef
ip source-route
ip cef
!
!
!
ip multicast-routing
ip multicast auto-enable
no ip dhcp use vrf connected
ip dhcp excluded-address 10.0.3.1 10.0.3.100
ip dhcp excluded-address 10.0.3.200 10.0.3.254
!
ip dhcp pool Adel
import all
network 10.0.3.0 255.255.255.0
dns-server 10.0.0.19
default-router 10.0.3.1
!
!
ip domain name colquitt.local
ip name-server 10.0.0.19
ip dhcp-server 10.0.3.1
multilink bundle-name authenticated
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-3600394647
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3600394647
revocation-check none
rsakeypair TP-self-signed-3600394647
!
!
crypto pki certificate chain TP-self-signed-3600394647
certificate self-signed 01
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33363030 33393436 3437301E 170D3132 30373033 32323233
30375A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 36303033
39343634 3730819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100ABC1 7853AB99 0CCAA945 E9A04EA1 85A6E47A 337FCE33 437ACFE8 0D40FF5C
1A25A822 6BF21388 FF5D3CA2 6D89C111 F661BAED 279CECF7 AC304840 3EDA0C5B
CF485DD7 E7DE1813 6051030A 90B6164C 595877C3 F438D27C E559F1E4 1A10DF78
73DF9BBB 94E431E8 55C1D759 607CF456 F1F609CD 48F64851 34EDB59D 04E01D3E
7B0B0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
551D2304 18301680 14439FB2 7C49E540 297C863B 86ECD803 361213D5 C1301D06
03551D0E 04160414 439FB27C 49E54029 7C863B86 ECD80336 1213D5C1 300D0609
2A864886 F70D0101 05050003 8181007D 382CBE77 BCB9DED8 68D609DE 9F2160F1
A60F065A 28E217EE 7CA626DB 341AF6B9 ED581DBE 8F740295 BFEE966A FD76C68E
B9105339 4673EA4C EE024837 4BFE56C9 4E3E3F16 5AB164A9 BC699919 303DCD45
E47D2866 A795D15B 535B72CB 1B40C96A 0BDF5E3D CEFE6732 C4D2DF97 21D59737
64D09F3F 961012E3 CE71CAA2 CD770E
quit
license udi pid CISCO1941/K9 sn FGL162722M6
!
!
archive
log config
hidekeys
username admin privilege 15 secret 4 DdflTgbrpjev2rAg20ZNwM6SNj3k3jpJ5amqux5YeJw
!
!
!
class-map match-all voice
description Voice Class Traffic
match access-group 102
!
!
policy-map VoiceTraffic
class voice
priority 768
class class-default
fair-queue
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description G0/0 LAN Interface
ip address 10.0.3.1 255.255.255.0
ip pim dense-mode
duplex auto
speed auto
!
interface GigabitEthernet0/0.1
ip pim dense-mode
!
interface GigabitEthernet0/0.2
encapsulation dot1Q 2
ip address 10.0.103.254 255.255.255.0
ip pim dense-mode
!
interface GigabitEthernet0/0.3
encapsulation dot1Q 17
ip address 10.0.43.254 255.255.255.0
ip pim dense-mode
!
interface GigabitEthernet0/1
description G0/1 WAN Interface
ip address 192.168.150.3 255.255.255.248
ip nbar protocol-discovery
ip pim dense-mode
ip igmp join-group 235.0.1.24
ip igmp join-group 235.0.1.20
ip igmp join-group 235.0.1.16
ip igmp join-group 235.0.1.10
ip igmp join-group 235.0.1.4
duplex auto
speed auto
!
interface GigabitEthernet0/0/0
switchport access vlan 3
no ip address
!
interface GigabitEthernet0/0/1
no ip address
!
interface GigabitEthernet0/0/2
no ip address
!
interface GigabitEthernet0/0/3
no ip address
shutdown
!
interface Vlan1
no ip address
!
interface Vlan3
ip address 192.168.100.2 255.255.255.224
ip pim dense-mode
ip igmp join-group 235.0.1.24
ip igmp join-group 235.0.1.20
ip igmp join-group 235.0.1.16
ip igmp join-group 235.0.1.10
ip igmp join-group 235.0.1.4
!
!
router eigrp 100
variance 2
network 10.0.3.0 0.0.0.255
network 10.0.43.0 0.0.0.255
network 10.0.103.0 0.0.0.255
network 192.168.100.0 0.0.0.31
network 192.168.150.0 0.0.0.15
network 192.168.150.0 0.0.0.7
passive-interface GigabitEthernet0/0
passive-interface GigabitEthernet0/0.2
passive-interface GigabitEthernet0/0.3
!
ip default-gateway 10.0.0.254
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 0.0.0.0 0.0.0.0 10.0.0.254
ip route 10.0.103.0 255.255.255.0 10.0.3.253
!
ip access-list standard SECURE_SSH
permit 10.0.0.79
permit 10.0.100.250
permit 10.0.2.0 0.0.0.255
!
logging 10.0.0.79
access-list 23 permit 10.10.10.0 0.0.0.7
access-list 23 permit 10.0.0.0 0.0.0.255
access-list 23 permit 10.0.3.0 0.0.0.255
access-list 23 permit 10.0.4.0 0.0.0.255
access-list 23 permit 10.0.6.0 0.0.0.255
access-list 23 permit 10.0.5.0 0.0.0.255
access-list 23 permit 10.0.7.0 0.0.0.255
access-list 23 permit 192.168.100.0 0.0.0.255
access-list 23 permit any
access-list 102 permit ip 10.0.100.0 0.0.0.255 any
access-list 102 permit ip 10.0.103.0 0.0.0.255 any
access-list 102 permit ip 10.0.104.0 0.0.0.255 any
access-list 102 permit ip 10.0.105.0 0.0.0.255 any
access-list 102 permit ip 10.0.106.0 0.0.0.255 any
access-list 102 permit ip 10.0.107.0 0.0.0.255 any
!
!
!
!
!
control-plane
!

!
line con 0
line aux 0
password 7 10621D1D0C04062703093A29303C31
modem Dialin
modem autoconfigure discovery
transport input all
autoselect ppp
speed 1200
flowcontrol hardware
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class SECURE_SSH in
privilege level 15
password 7 123511131B1818282526342A272120
transport input ssh
line vty 5 15
access-class SECURE_SSH in
privilege level 15
password 7 123511131B1818282526342A272120
transport input ssh
line vty 16 1370
password 7 123511131B1818282526342A272120
transport input all
!
scheduler allocate 20000 1000
ntp server 10.0.0.1
end

CEMC-Adel1941#

[Leo Laohoo]

What is the WAN speed?

[jtillman11]

The WAN speed is 50 Mbps.

[johnlloyd_13]

hi,

you can also check if there's a debug currently running using the show debug.

just issue a undebug all to disable it.

[jtillman11]

After checking, there are no debugs running. Utilization is now at 73% and there are no users at that office at this time so it's strange.

[jtillman11]

Here are the results from "show interface stats". ip cef is enabled, is this abnormally high packets being processed by the CPU?

[jtillman11]

Here is output of "sh cef drop"...it could be helpful.

[jtillman11]

Results from sh int switching for LAN and WAN interfaces.

[Georg Pauwen · Accepted solution]

Hello,

a few things in your configuration seem obsolete, since I don't know what your entire topology looks like, they might have a use but I doubt it. Can you also post the output of 'show ip route' ?

Try and remove and change the items I marked in bold:

CEMC-Adel1941#show run
Building configuration...

Current configuration : 6876 bytes
!
! Last configuration change at 09:03:41 DST Thu Jul 12 2018 by admin
! NVRAM config last updated at 09:03:33 DST Thu Jul 12 2018 by admin
! NVRAM config last updated at 09:03:33 DST Thu Jul 12 2018 by admin
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname CEMC-Adel1941
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 warnings
enable secret 5 $1$G2Mp$HprsNiCj0niaz3QhM7BSh.
enable password 7 070C24414D441A0C041104
!
aaa new-model
!
aaa session-id common
clock timezone EASTERN -5 0
clock summer-time DST date Mar 10 2018 21:00 Nov 3 2018 21:00
!
no ipv6 cef
--> no ip source-route
ip cef
!
ip multicast-routing
ip multicast auto-enable
no ip dhcp use vrf connected
ip dhcp excluded-address 10.0.3.1 10.0.3.100
ip dhcp excluded-address 10.0.3.200 10.0.3.254
!
ip dhcp pool Adel
import all
network 10.0.3.0 255.255.255.0
dns-server 10.0.0.19
default-router 10.0.3.1
!
ip domain name colquitt.local
ip name-server 10.0.0.19
ip dhcp-server 10.0.3.1
multilink bundle-name authenticated
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-3600394647
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3600394647
revocation-check none
rsakeypair TP-self-signed-3600394647
!
crypto pki certificate chain TP-self-signed-3600394647
certificate self-signed 01
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33363030 33393436 3437301E 170D3132 30373033 32323233
30375A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 36303033
39343634 3730819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100ABC1 7853AB99 0CCAA945 E9A04EA1 85A6E47A 337FCE33 437ACFE8 0D40FF5C
1A25A822 6BF21388 FF5D3CA2 6D89C111 F661BAED 279CECF7 AC304840 3EDA0C5B
CF485DD7 E7DE1813 6051030A 90B6164C 595877C3 F438D27C E559F1E4 1A10DF78
73DF9BBB 94E431E8 55C1D759 607CF456 F1F609CD 48F64851 34EDB59D 04E01D3E
7B0B0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
551D2304 18301680 14439FB2 7C49E540 297C863B 86ECD803 361213D5 C1301D06
03551D0E 04160414 439FB27C 49E54029 7C863B86 ECD80336 1213D5C1 300D0609
2A864886 F70D0101 05050003 8181007D 382CBE77 BCB9DED8 68D609DE 9F2160F1
A60F065A 28E217EE 7CA626DB 341AF6B9 ED581DBE 8F740295 BFEE966A FD76C68E
B9105339 4673EA4C EE024837 4BFE56C9 4E3E3F16 5AB164A9 BC699919 303DCD45
E47D2866 A795D15B 535B72CB 1B40C96A 0BDF5E3D CEFE6732 C4D2DF97 21D59737
64D09F3F 961012E3 CE71CAA2 CD770E
quit
license udi pid CISCO1941/K9 sn FGL162722M6
!
archive
log config
hidekeys
username admin privilege 15 secret 4 DdflTgbrpjev2rAg20ZNwM6SNj3k3jpJ5amqux5YeJw
!
--> no class-map match-all voice
description Voice Class Traffic
match access-group 102
!
--> no policy-map VoiceTraffic
class voice
priority 768
class class-default
fair-queue
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description G0/0 LAN Interface
ip address 10.0.3.1 255.255.255.0
ip pim dense-mode
duplex auto
speed auto
!
interface GigabitEthernet0/0.1
ip pim dense-mode
!
interface GigabitEthernet0/0.2
encapsulation dot1Q 2
ip address 10.0.103.254 255.255.255.0
ip pim dense-mode
!
interface GigabitEthernet0/0.3
encapsulation dot1Q 17
ip address 10.0.43.254 255.255.255.0
ip pim dense-mode
!
interface GigabitEthernet0/1
description G0/1 WAN Interface
ip address 192.168.150.3 255.255.255.248
--> no ip nbar protocol-discovery
ip pim dense-mode
ip igmp join-group 235.0.1.24
ip igmp join-group 235.0.1.20
ip igmp join-group 235.0.1.16
ip igmp join-group 235.0.1.10
ip igmp join-group 235.0.1.4
duplex auto
speed auto
!
interface GigabitEthernet0/0/0
switchport access vlan 3
no ip address
!
interface GigabitEthernet0/0/1
no ip address
!
interface GigabitEthernet0/0/2
no ip address
!
interface GigabitEthernet0/0/3
no ip address
shutdown
!
interface Vlan1
no ip address
!
interface Vlan3
ip address 192.168.100.2 255.255.255.224
ip pim dense-mode
ip igmp join-group 235.0.1.24
ip igmp join-group 235.0.1.20
ip igmp join-group 235.0.1.16
ip igmp join-group 235.0.1.10
ip igmp join-group 235.0.1.4
!
router eigrp 100
variance 2
network 10.0.3.0 0.0.0.255
network 10.0.43.0 0.0.0.255
network 10.0.103.0 0.0.0.255
network 192.168.100.0 0.0.0.31
network 192.168.150.0 0.0.0.15
network 192.168.150.0 0.0.0.7
passive-interface GigabitEthernet0/0
passive-interface GigabitEthernet0/0.2
passive-interface GigabitEthernet0/0.3
!
--> no ip default-gateway 10.0.0.254
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 0.0.0.0 0.0.0.0 10.0.0.254
ip route 10.0.103.0 255.255.255.0 10.0.3.253
!
ip access-list standard SECURE_SSH
permit 10.0.0.79
permit 10.0.100.250
permit 10.0.2.0 0.0.0.255
!
logging 10.0.0.79
access-list 23 permit 10.10.10.0 0.0.0.7
access-list 23 permit 10.0.0.0 0.0.0.255
access-list 23 permit 10.0.3.0 0.0.0.255
access-list 23 permit 10.0.4.0 0.0.0.255
access-list 23 permit 10.0.6.0 0.0.0.255
access-list 23 permit 10.0.5.0 0.0.0.255
access-list 23 permit 10.0.7.0 0.0.0.255
access-list 23 permit 192.168.100.0 0.0.0.255
access-list 23 permit any

--> no access-list 102
access-list 102 permit ip 10.0.100.0 0.0.0.255 any
access-list 102 permit ip 10.0.103.0 0.0.0.255 any
access-list 102 permit ip 10.0.104.0 0.0.0.255 any
access-list 102 permit ip 10.0.105.0 0.0.0.255 any
access-list 102 permit ip 10.0.106.0 0.0.0.255 any
access-list 102 permit ip 10.0.107.0 0.0.0.255 any
!
control-plane
!
line con 0
line aux 0
password 7 10621D1D0C04062703093A29303C31
modem Dialin
modem autoconfigure discovery
transport input all
autoselect ppp
speed 1200
flowcontrol hardware
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class SECURE_SSH in
privilege level 15
password 7 123511131B1818282526342A272120
transport input ssh
line vty 5 15
access-class SECURE_SSH in
privilege level 15
password 7 123511131B1818282526342A272120
transport input ssh
line vty 16 1370
password 7 123511131B1818282526342A272120
transport input all
!
scheduler allocate 20000 1000
ntp server 10.0.0.1
end

[jtillman11]

Just to make sure - removing the voice policy-map and class-map won't affect Voice QoS? (I am a CCENT so I am not very knowledgeable in regards to these configurations). Are those policies not applied to an interface or something?

Here is "show ip route"

CEMC-Adel1941#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is 10.0.0.254 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 10.0.0.254
10.0.0.0/8 is variably subnetted, 22 subnets, 2 masks
D 10.0.0.0/24 [90/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
D 10.0.2.0/24 [90/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
C 10.0.3.0/24 is directly connected, GigabitEthernet0/0
L 10.0.3.1/32 is directly connected, GigabitEthernet0/0
D EX 10.0.4.0/24 [170/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
D 10.0.5.0/24 [90/3072] via 192.168.150.5, 3d11h, GigabitEthernet0/1
D 10.0.6.0/24 [90/28416] via 192.168.150.2, 3d11h, GigabitEthernet0/1
[90/30720] via 192.168.100.3, 3d11h, Vlan3
D 10.0.7.0/24 [90/3072] via 192.168.150.4, 3d10h, GigabitEthernet0/1
D 10.0.40.0/24 [90/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
C 10.0.43.0/24 is directly connected, GigabitEthernet0/0.3
L 10.0.43.254/32 is directly connected, GigabitEthernet0/0.3
D EX 10.0.44.0/24 [170/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
D 10.0.45.0/24 [90/3072] via 192.168.150.5, 3d11h, GigabitEthernet0/1
D 10.0.46.0/24 [90/26112] via 192.168.150.2, 3d11h, GigabitEthernet0/1
[90/28416] via 192.168.100.3, 3d11h, Vlan3
D 10.0.47.0/24 [90/3072] via 192.168.150.4, 3d10h, GigabitEthernet0/1
D 10.0.100.0/24 [90/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
C 10.0.103.0/24 is directly connected, GigabitEthernet0/0.2
L 10.0.103.254/32 is directly connected, GigabitEthernet0/0.2
D EX 10.0.104.0/24 [170/3072] via 192.168.150.1, 3d10h, GigabitEthernet0/1
D 10.0.105.0/24 [90/3072] via 192.168.150.5, 3d11h, GigabitEthernet0/1
D 10.0.106.0/24 [90/26112] via 192.168.150.2, 3d11h, GigabitEthernet0/1
[90/28416] via 192.168.100.3, 3d11h, Vlan3
D 10.0.107.0/24 [90/3072] via 192.168.150.4, 3d10h, GigabitEthernet0/1
192.168.100.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.100.0/27 is directly connected, Vlan3
L 192.168.100.2/32 is directly connected, Vlan3
192.168.150.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.150.0/29 is directly connected, GigabitEthernet0/1
L 192.168.150.3/32 is directly connected, GigabitEthernet0/1
192.168.200.0/30 is subnetted, 1 subnets
D 192.168.200.16
[90/2170112] via 192.168.150.2, 3d11h, GigabitEthernet0/1
[90/2172416] via 192.168.100.3, 3d11h, Vlan3
CEMC-Adel1941#

[jtillman11]

I have applied your suggestions. The "no ip nbar protocol-discovery" was not a recognized command on this router.

[jtillman11]

Ah i see the "no ip nbar protocol-discovery" was an interface subcommand for Gig0/1. I have applied that command.

[jtillman11]

HOLY COW! CPU just dropped to 9%/7%