01-26-2018 10:03 AM - edited 03-08-2019 01:34 PM
Hello,
I've high CPU utilization on Cisco 3900 Series router. Every, roughly 20 mins CPU utilization is 100% and there is packet loss for 1-2 minutes.
There is not a lot of traffic, around 150Mbps on gigabit interface, so it should not be capacity issue.
IOS version is:
Cisco IOS Software, C3900 Software (C3900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)
Output of cpu history:
111119992221159999122122199922111599992222321999222215999921
723216871115783697803890998721889427763385038777322272499909
100 *#* *#* *#* *#* *#* *#*
90 *#* ###* *#* ###* *#* ###*
80 *#* ###* *#* ###* *#* ###*
70 *#* ###* *#* ###* *#* ###*
60 *#* *###* *#* ###* *#* ###*
50 *#* *###* ##* *###* ##* *###*
40 ##* *###* ### *#### ### *####
30 ### *#### * ### *#### *** ### *####
20 * ######**#####*############**###########################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
Output of the processes:
sh processes cpu sorted 5sec | ex 0.00
CPU utilization for five seconds: 96%/96%; one minute: 64%; five minutes: 38%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
2 23028 6165 3735 0.23% 0.14% 0.06% 0 Load Meter
113 220264 122904 1792 0.15% 0.16% 0.15% 0 Netclock Backgro
94 150068 30928 4852 0.07% 0.10% 0.09% 0 Per-Second Jobs
206 18564 3726083 4 0.07% 0.09% 0.09% 0 Ethernet Msec Ti
In normal condition between spikes:
sh processes cpu sorted 5sec | ex 0.00
CPU utilization for five seconds: 12%/11%; one minute: 41%; five minutes: 46%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
17 91180 31476 2896 0.55% 0.08% 0.05% 0 Environmental mo
113 226036 126090 1792 0.23% 0.16% 0.14% 0 Netclock Backgro
206 19064 3823336 4 0.15% 0.12% 0.10% 0 Ethernet Msec Ti
259 50380 428209 117 0.07% 0.05% 0.07% 0 ADJ resolve proc
59 15780 498502 31 0.07% 0.01% 0.02% 0 Net Background
94 154028 31727 4854 0.07% 0.10% 0.09% 0 Per-Second Jobs
Any help would be very appreciated!
Thanks
Salja
01-26-2018 11:46 AM
Hello,
there are numerous bugs related to high CPU on the 3900. Which one might apply depends on your configuration, can you post that ?
01-26-2018 02:53 PM
Hi Georg and Leo,
thank you very much for the answers!
The config is pretty simple. Both Ipv4 and Ipv6 are running and there is BGP for both families.
Some simple ACLs and thats it. The issue started suddenly to happen since today.
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname xxxx
!
boot-start-marker
boot-end-marker
!
!
logging buffered informational
!
no aaa new-model
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip domain name xxxx.com
ip cef
ipv6 unicast-routing
ipv6 cef
!
!
multilink bundle-name authenticated
!
!
cts logging verbose
license udi pid C3900-SPE100/K9 sn FOC19036QGW
!
!
!
redundancy
!
!
!
!
!
ip ssh version 1
!
class-map match-all CLASS_kvm_43252
match access-group name kvm_43252
!
policy-map POLICY_TRAFFIC_LIMIT
class CLASS_kvm_43252
police 15000000 61250 61250 conform-action transmit exceed-action drop
!
!
!
!
!
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
ip address x.x.x.x 255.255.255.240 secondary
ip address x.x.x.x 255.255.255.252
ip access-group ISP_inbound in
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip flow egress
duplex full
speed 1000
ipv6 address x.x.x.x/126
service-policy input POLICY_TRAFFIC_LIMIT
!
interface GigabitEthernet0/1
ip address x.x.x.x 255.255.255.0 secondary
ip address x.x.x.x 255.255.255.0
ip access-group acl_HK_inbound in
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip flow egress
duplex auto
speed auto
ipv6 address x.x.x.x/48
!
interface GigabitEthernet0/2
no ip address
shutdown
duplex auto
speed auto
!
router bgp xxxx
bgp log-neighbor-changes
neighbor x:x:x:.. remote-as xxxx
neighbor x:x:x:x:.. description xxx
neighbor x.x.x.y remote-as xxx
neighbor x.x.x.y description xxxx
neighbor x.x.x.y ebgp-multihop 5
!
address-family ipv4
network x.x.x.0
network x.x.x.0
redistribute connected
redistribute static
no neighbor x:x:x:x: activate
neighbor x.x.x.y activate
neighbor x.x.x.y send-community
neighbor x.x.x.y remove-private-as
neighbor x.x.x.y prefix-list bgp_default_route in
neighbor x.x.x.y prefix-list xxx4HK-out out
neighbor x.x.x.y route-map xxx-in in
neighbor x.x.x.y route-map xxx-out out
exit-address-family
!
address-family ipv6
redistribute connected
redistribute static
network x:x::/48
neighbor x:x::x activate
neighbor x:x::x remove-private-as
neighbor x:x::x prefix-list HK6-out out
exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip flow-top-talkers
top 25
sort-by packets
!
ip route 0.0.0.0 0.0.0.0 218.213.248.193
ip route x.x.x.0 255.255.255.0 Null0 250
ip route x.x.x.0 255.255.255.0 Null0 250
!
ip access-list extended ISP_inbound
permit ip host x.x.x.x any
.
.
.
permit ip any any
ip access-list extended acl_HK_inbound
permit ip x.x.x.0 0.0.0.255 any
permit ip x.x.x.0 0.0.0.255 any
deny ip any any
ip access-list extended acl_vty_in
.
.
.
permit tcp host x.x.x.x any eq 22
permit tcp host x.x.x.x any eq 22
permit tcp host x.x.x.x any eq 22
.
.
.
deny ip any any log
ip access-list extended kvm_43252
permit ip any host x.x.x.x
deny ip any any
!
!
ip prefix-list bgp_default_route seq 5 permit 0.0.0.0/0
!
ip prefix-list 4HK-out seq 10 permit x.x.x.0/24
ip prefix-list 4HK-out seq 20 permit x.x.x.0/24
ipv6 route x:x:x::/48 Null0 250
!
!
ipv6 prefix-list xxx6-out seq 10 permit x:x:x::/48
ipv6 prefix-list xxx6-out seq 20 deny ::/0 le 128
route-map xxx-out permit 100
!
route-map xxx-in permit 100
!
!
snmp-server community xxxx RO 50
snmp-server ifindex persist
access-list 50 permit x.x.x.x
access-list 50 permit x.x.x.x
!
ipv6 access-list acl_vty_ipv6_in
permit ipv6 host x:x:x::x host x:x:x::x
permit tcp host x:x:x::x host x:x:x::x eq 22
deny ipv6 any any
!
control-plane
!
!
!
line con 0
login local
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class acl_vty_in in
ipv6 access-class acl_vty_ipv6_in in
login local
transport input all
line vty 5 15
access-class acl_vty_in in
ipv6 access-class acl_vty_ipv6_in in
login local
transport input ssh
!
scheduler allocate 20000 1000
!
end
01-26-2018 02:44 PM
01-27-2018 04:01 AM
From the output, it appears you are not filtering the ipv6 routes being advertised to you by your bgp neighbors. If this is true, your router's processor may not be able to handle the routing table of ipv6 as being advertised by your peers. show ipv6 route might help you know if you are receiving ipv6 routes from your neighbors. Check here for more https://www.timigate.com/2017/08/basic-ebgp-setup-on-cisco-router.html#more
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide