High CPU Utilization on Cisco 3900 roughly every 20 minutes 100%

[Salja]

Hello,

I've high CPU utilization on Cisco 3900 Series router. Every, roughly 20 mins CPU utilization is 100% and there is packet loss for 1-2 minutes.

There is not a lot of traffic, around 150Mbps on gigabit interface, so it should not be capacity issue.

IOS version is:

Cisco IOS Software, C3900 Software (C3900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)

Output of cpu history:

      111119992221159999122122199922111599992222321999222215999921
      723216871115783697803890998721889427763385038777322272499909
  100      *#*       *#*       *#*       *#*       *#*       *#*  
   90      *#*      ###*       *#*      ###*       *#*      ###*  
   80      *#*      ###*       *#*      ###*       *#*      ###*  
   70      *#*      ###*       *#*      ###*       *#*      ###*  
   60      *#*     *###*       *#*      ###*       *#*      ###*  
   50      *#*     *###*       ##*     *###*       ##*     *###*  
   40      ##*     *###*       ###     *####       ###     *####  
   30      ###     *####    *  ###     *####  ***  ###     *####  
   20 *    ######**#####*############**###########################
   10 ############################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

Output of the processes:

sh processes cpu sorted 5sec | ex 0.00
CPU utilization for five seconds: 96%/96%; one minute: 64%; five minutes: 38%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
   2       23028        6165       3735  0.23%  0.14%  0.06%   0 Load Meter       
 113      220264      122904       1792  0.15%  0.16%  0.15%   0 Netclock Backgro
  94      150068       30928       4852  0.07%  0.10%  0.09%   0 Per-Second Jobs  
 206       18564     3726083          4  0.07%  0.09%  0.09%   0 Ethernet Msec Ti

In normal condition between spikes:

sh processes cpu sorted 5sec | ex 0.00
CPU utilization for five seconds: 12%/11%; one minute: 41%; five minutes: 46%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  17       91180       31476       2896  0.55%  0.08%  0.05%   0 Environmental mo
 113      226036      126090       1792  0.23%  0.16%  0.14%   0 Netclock Backgro
 206       19064     3823336          4  0.15%  0.12%  0.10%   0 Ethernet Msec Ti
 259       50380      428209        117  0.07%  0.05%  0.07%   0 ADJ resolve proc
  59       15780      498502         31  0.07%  0.01%  0.02%   0 Net Background   
  94      154028       31727       4854  0.07%  0.10%  0.09%   0 Per-Second Jobs 

Any help would be very appreciated!

Thanks

Salja

[Georg Pauwen]

Hello,

there are numerous bugs related to high CPU on the 3900. Which one might apply depends on your configuration, can you post that ?

[Salja]

Hi Georg and Leo,

thank you very much for the answers!

The config is pretty simple. Both Ipv4 and Ipv6 are running and there is BGP for both families.

Some simple ACLs and thats it. The issue started suddenly to happen since today.

!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname xxxx
!
boot-start-marker
boot-end-marker
!
!
logging buffered informational
!
no aaa new-model
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip domain name xxxx.com
ip cef
ipv6 unicast-routing
ipv6 cef
!
!
multilink bundle-name authenticated
!
!
cts logging verbose
license udi pid C3900-SPE100/K9 sn FOC19036QGW
!
!
!
redundancy
!
!
!
!
!
ip ssh version 1
!
class-map match-all CLASS_kvm_43252
 match access-group name kvm_43252
!
policy-map POLICY_TRAFFIC_LIMIT
 class CLASS_kvm_43252
  police 15000000 61250 61250 conform-action transmit  exceed-action drop
!
!
!
!
!
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 ip address x.x.x.x 255.255.255.240 secondary
 ip address x.x.x.x 255.255.255.252
 ip access-group ISP_inbound in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip flow ingress
 ip flow egress
 duplex full
 speed 1000
 ipv6 address x.x.x.x/126
 service-policy input POLICY_TRAFFIC_LIMIT
!
interface GigabitEthernet0/1
 ip address x.x.x.x 255.255.255.0 secondary
 ip address x.x.x.x 255.255.255.0
 ip access-group acl_HK_inbound in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip flow ingress
 ip flow egress
 duplex auto
 speed auto
 ipv6 address x.x.x.x/48
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
router bgp xxxx
 bgp log-neighbor-changes
 neighbor x:x:x:.. remote-as xxxx
 neighbor x:x:x:x:.. description xxx
 neighbor x.x.x.y remote-as xxx
 neighbor x.x.x.y description xxxx
 neighbor x.x.x.y ebgp-multihop 5
 !        
 address-family ipv4
  network x.x.x.0
  network x.x.x.0
  redistribute connected
  redistribute static
  no neighbor x:x:x:x: activate
  neighbor x.x.x.y activate
  neighbor x.x.x.y send-community
  neighbor x.x.x.y remove-private-as
  neighbor x.x.x.y prefix-list bgp_default_route in
  neighbor x.x.x.y prefix-list xxx4HK-out out
  neighbor x.x.x.y route-map xxx-in in
  neighbor x.x.x.y route-map xxx-out out
 exit-address-family
 !
 address-family ipv6
  redistribute connected
  redistribute static
  network x:x::/48
  neighbor x:x::x activate
  neighbor x:x::x remove-private-as
  neighbor x:x::x prefix-list HK6-out out
 exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip flow-top-talkers
 top 25
 sort-by packets
!
ip route 0.0.0.0 0.0.0.0 218.213.248.193
ip route x.x.x.0 255.255.255.0 Null0 250
ip route x.x.x.0 255.255.255.0 Null0 250
!
ip access-list extended ISP_inbound
 permit ip host x.x.x.x any
 .
 .
 .
 permit ip any any
ip access-list extended acl_HK_inbound
 permit ip x.x.x.0 0.0.0.255 any
 permit ip x.x.x.0 0.0.0.255 any
 deny   ip any any
ip access-list extended acl_vty_in
 .
 .
 .
 permit tcp host x.x.x.x any eq 22
 permit tcp host x.x.x.x any eq 22
 permit tcp host x.x.x.x any eq 22
 .
 .
 .
 deny   ip any any log
ip access-list extended kvm_43252
 permit ip any host x.x.x.x
 deny   ip any any
!
!
ip prefix-list bgp_default_route seq 5 permit 0.0.0.0/0
!
ip prefix-list 4HK-out seq 10 permit x.x.x.0/24
ip prefix-list 4HK-out seq 20 permit x.x.x.0/24
ipv6 route x:x:x::/48 Null0 250
!
!
ipv6 prefix-list xxx6-out seq 10 permit x:x:x::/48
ipv6 prefix-list xxx6-out seq 20 deny ::/0 le 128
route-map xxx-out permit 100
!
route-map xxx-in permit 100
!
!
snmp-server community xxxx RO 50
snmp-server ifindex persist

access-list 50 permit x.x.x.x
access-list 50 permit x.x.x.x
!
ipv6 access-list acl_vty_ipv6_in
 permit ipv6 host x:x:x::x host x:x:x::x
 permit tcp host x:x:x::x host x:x:x::x eq 22
 deny ipv6 any any
!
control-plane
!
!
!
line con 0
 login local
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class acl_vty_in in
 ipv6 access-class acl_vty_ipv6_in in
 login local
 transport input all
line vty 5 15
 access-class acl_vty_in in
 ipv6 access-class acl_vty_ipv6_in in
 login local
 transport input ssh
!
scheduler allocate 20000 1000
!
end

[Leo Laohoo]

Upgrade the IOS.

[ashioma1982]

From the output, it appears you are not filtering the ipv6 routes being advertised to you by your bgp neighbors. If this is true, your router's processor may not be able to handle the routing table of ipv6 as being advertised by your peers. show ipv6 route might help you know if you are receiving ipv6 routes from your neighbors. Check here for more https://www.timigate.com/2017/08/basic-ebgp-setup-on-cisco-router.html#more