High CPU load on Cisco 2921

[wggfilonenko1]

Hi.

We are seeing a high CPU load on the router (Cisco 2921). All other problems are a consequence of this.
At the same time, there is only one high-load process in the show proc cpu command output:
CPU utilization for five seconds: 99%/47%; one minute: 95%; five minutes: 95%
172 57333076 87035101 658 42.46% 41.33% 40.21% 0 IP Input

This is a lot, but it is far from 100%. How can we determine the cause of the CPU load?

[Leo Laohoo]

What is the speed of the WAN link?

[wggfilonenko1]

G0/0 (link to firewall)
5 minute input rate 28040000 bits/sec, 9652 packets/sec
5 minute output rate 65538000 bits/sec, 9225 packets/sec

G0/1 (link to ISPs)
30 second input rate 65183000 bits/sec, 9122 packets/sec
30 second output rate 36672000 bits/sec, 10896 packets/sec

G0/2 (link to VPN, 25-50% with IPSec)
30 second input rate 95423000 bits/sec, 18687 packets/sec
30 second output rate 93583000 bits/sec, 16208 packets/sec

[Georg Pauwen]

Hello,

the IP INPUT process uses a lot of CPU, that indicates a lot of process-switches packets. Post the running config of your 2921...

[wggfilonenko1]

version 15.7
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname scbk-router1
!
boot-start-marker
boot system flash:c2900-universalk9-mz.SPA.157-3.M8.bin
boot-end-marker
!
!
logging buffered 51200 warnings
enable secret 5 bla-bla-bla
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
!
!
!
aaa session-id common
!
!
!
!
!
!
no ip source-route
!
!
!
ip vrf SCBK_Internet
rd 1000:1
!
ip vrf SCBK_L3VPN
rd 1000:2
!
!
!
!
ip domain name scbk.ru
ip name-server 8.8.8.8
ip name-server 77.88.8.8
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
cts logging verbose
!
crypto pki trustpoint TP-self-signed-3190922259
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3190922259
revocation-check none
rsakeypair TP-self-signed-3190922259
!
!
crypto pki certificate chain TP-self-signed-3190922259
certificate self-signed 01
quit
license udi pid CISCO2921/K9 sn FCZ143572EP
license boot module c2900 technology-package securityk9
!
!
file privilege 1
username admin privilege 15 password 7 bla-bla-bla
!
redundancy
!
!
!
!
!
track 1 interface GigabitEthernet0/1 line-protocol
!
!
!
crypto isakmp policy 40
encr aes 256
authentication pre-share
group 5
crypto isakmp key bla-bla-bla address 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10 5 periodic
!
!
crypto ipsec transform-set ESP-AES-SHA-TRANSPORT esp-aes esp-sha-hmac
mode transport
!
crypto ipsec profile IPSEC
set transform-set ESP-AES-SHA-TRANSPORT
set pfs group5
!
!
!
!
!
!
!
interface Tunnel1
description ---L3VPN DMVPN---
ip address 10.255.250.34 255.255.255.0
no ip redirects
ip nhrp authentication ILP111
ip nhrp map 10.255.250.1 172.20.0.2
ip nhrp map multicast 172.20.0.2
ip nhrp network-id 111
ip nhrp holdtime 60
ip nhrp nhs 10.255.250.1
ip nhrp registration timeout 10
ip tcp adjust-mss 1376
tunnel source GigabitEthernet0/2.25
tunnel mode gre multipoint
tunnel key bla-bla-bla
!
interface Tunnel2
description ---DMVPN to MSK DC---
ip address 10.255.255.19 255.255.255.0
no ip redirects
ip mtu 1416
no ip split-horizon eigrp 1
ip nhrp authentication ILP100
ip nhrp map 10.255.255.1 92.242.43.68
ip nhrp map multicast 92.242.43.68
ip nhrp network-id 100
ip nhrp nhs 10.255.255.1
ip tcp adjust-mss 1376
tunnel source GigabitEthernet0/0.32
tunnel mode gre multipoint
tunnel key bla-bla-bla
tunnel protection ipsec profile IPSEC
!
interface Tunnel1004
bandwidth 10000000
ip vrf forwarding SCBK_Internet
ip address 10.255.110.14 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication nhrp110
ip nhrp map 10.255.110.1 10.255.200.17
ip nhrp map multicast 10.255.200.17
ip nhrp network-id 110
ip nhrp nhs 10.255.110.1
delay 100
tunnel source GigabitEthernet0/2.52
tunnel mode gre multipoint
tunnel key bla-bla-bla
tunnel vrf SCBK_Internet
!
interface Tunnel2104
bandwidth 1000000
ip vrf forwarding SCBK_L3VPN
ip address 10.255.111.114 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication nhrp111
ip nhrp map 10.255.111.1 10.255.200.25
ip nhrp map multicast 10.255.200.25
ip nhrp network-id 111
ip nhrp nhs 10.255.111.1
delay 100
tunnel source GigabitEthernet0/2.53
tunnel mode gre multipoint
tunnel key bla-bla-bla
tunnel vrf SCBK_L3VPN
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description Channels to LAN SCBK
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0/0.4
description Link to wireless
bandwidth 100000
encapsulation dot1Q 4
ip address 91.216.175.253 255.255.255.240
standby 2 ip 91.216.175.254
standby 2 priority 105
standby 2 preempt delay minimum 60
standby 2 track 1 decrement 10
!
interface GigabitEthernet0/0.32
description Link to firewall
bandwidth 100000
encapsulation dot1Q 32
ip address 91.216.175.1 255.255.255.240
standby 1 ip 91.216.175.3
standby 1 priority 105
standby 1 preempt delay minimum 60
standby 1 track 1 decrement 10
!
interface GigabitEthernet0/0.51
encapsulation dot1Q 51
ip vrf forwarding SCBK_L3VPN
ip address 10.10.128.201 255.255.255.240
standby 0 authentication SCBK_L3V
standby 6 ip 10.10.128.203
standby 6 priority 105
standby 6 preempt delay minimum 60
standby 6 track 1 decrement 10
!
interface GigabitEthernet0/1
description Channels to Internet
no ip address
load-interval 30
duplex auto
speed auto
!
interface GigabitEthernet0/1.31
description MTS channel to Internet
bandwidth 100000
encapsulation dot1Q 31
ip address 84.17.4.150 255.255.255.252
!
interface GigabitEthernet0/1.33
description NWT channel to Internet
bandwidth 2000
encapsulation dot1Q 33
ip address 212.48.214.170 255.255.255.248
!
interface GigabitEthernet0/1.34
description Rostelecom channel to Internet
bandwidth 10000
encapsulation dot1Q 34
ip address 95.167.190.98 255.255.255.248
!
interface GigabitEthernet0/2
description Leased channels
no ip address
load-interval 30
duplex auto
speed auto
!
interface GigabitEthernet0/2.25
description MTS VPN channel
encapsulation dot1Q 25
ip address 172.20.0.34 255.255.255.248
!
interface GigabitEthernet0/2.51
encapsulation dot1Q 51
ip vrf forwarding SCBK_Internet
ip address 10.10.128.193 255.255.255.240
standby 0 authentication bla-bla-bla
standby 5 ip 10.10.128.195
standby 5 priority 105
standby 5 preempt delay minimum 60
standby 5 track 1 decrement 10
!
interface GigabitEthernet0/2.52
encapsulation dot1Q 52
ip vrf forwarding SCBK_Internet
ip address 10.10.128.209 255.255.255.248
standby 3 ip 10.10.128.211
standby 3 priority 105
standby 3 preempt delay minimum 60
standby 3 track 1 decrement 10
!
interface GigabitEthernet0/2.53
encapsulation dot1Q 53
ip vrf forwarding SCBK_L3VPN
ip address 10.10.128.217 255.255.255.248
standby 4 ip 10.10.128.219
standby 4 priority 105
standby 4 preempt delay minimum 60
standby 4 track 1 decrement 10
!
!
router eigrp 11
distribute-list prefix REDISTRIBUTE_STATIC out Tunnel1
network 10.255.250.0 0.0.0.255
network 91.216.175.0 0.0.0.15
redistribute static route-map STATIC_TO_EIGRP
passive-interface default
no passive-interface Tunnel1
no passive-interface GigabitEthernet0/0.32
eigrp router-id 10.255.250.34
!
!
router eigrp 1
distribute-list prefix REDISTRIBUTE_STATIC_SCBK out Tunnel2
network 10.255.255.0 0.0.0.255
network 91.216.175.0 0.0.0.15
redistribute static route-map STATIC_TO_EIGRP_SCBK
passive-interface default
no passive-interface Tunnel2
no passive-interface GigabitEthernet0/0.32
eigrp router-id 10.255.250.19
!
!
router eigrp 110
!
address-family ipv4 vrf SCBK_Internet
redistribute ospf 3 route-map OSPF->EIGRP
network 10.255.110.0 0.0.0.255
passive-interface default
no passive-interface Tunnel1004
distribute-list route-map DenyLocal in
autonomous-system 110
eigrp router-id 10.255.240.28
exit-address-family
!
!
router eigrp 111
!
address-family ipv4 vrf SCBK_L3VPN
redistribute ospf 4 route-map OSPF->EIGRP_B
network 10.255.111.0 0.0.0.255
passive-interface default
no passive-interface Tunnel2104
distribute-list route-map DenyLocal in
autonomous-system 111
eigrp router-id 10.255.240.30
exit-address-family
!
router ospf 3 vrf SCBK_Internet
router-id 10.255.240.28
redistribute eigrp 110 subnets route-map EIGRP110->OSPF3
passive-interface default
no passive-interface GigabitEthernet0/2.51
network 10.10.128.192 0.0.0.15 area 0
default-information originate metric 10
distance ospf external 180
!
router ospf 4 vrf SCBK_L3VPN
router-id 10.255.240.30
redistribute eigrp 111 subnets route-map EIGRP111->OSPF4
passive-interface default
no passive-interface GigabitEthernet0/0.51
network 10.10.128.192 0.0.0.15 area 0
default-information originate metric 220
distance ospf external 180
!
router bgp 51165
bgp log-neighbor-changes
network 91.216.175.0 mask 255.255.255.0
aggregate-address 91.216.175.0 255.255.255.0 summary-only
redistribute connected
neighbor 84.17.4.149 remote-as 8359
neighbor 84.17.4.149 prefix-list no-default-route in
neighbor 84.17.4.149 prefix-list pi_network out
neighbor 84.17.4.149 route-map set_med out
neighbor 91.216.175.2 remote-as 51165
neighbor 91.216.175.2 next-hop-self
neighbor 95.167.190.97 remote-as 12389
neighbor 95.167.190.97 prefix-list no-default-route in
neighbor 95.167.190.97 prefix-list pi_network out
neighbor 95.167.190.97 route-map set_med out
neighbor 212.48.214.169 remote-as 8997
neighbor 212.48.214.169 prefix-list pi_network out
neighbor 212.48.214.169 route-map set_med out
!
ip forward-protocol nd
!
ip bgp-community new-format
ip community-list 10 permit 12389:2800
ip http server
ip http access-class 1
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 10.10.128.0 255.255.192.0 91.216.175.4
ip route 10.10.190.0 255.255.255.0 91.216.175.4
ip route 91.216.175.16 255.255.255.240 91.216.175.4
ip route 91.216.175.16 255.255.255.255 91.216.175.5
ip route 91.216.175.17 255.255.255.255 91.216.175.5
ip route 91.216.175.18 255.255.255.255 91.216.175.5
ip route 91.216.175.19 255.255.255.255 91.216.175.5
ip route 91.216.175.20 255.255.255.255 91.216.175.5
ip route 91.216.175.21 255.255.255.255 91.216.175.5
ip route 91.216.175.22 255.255.255.255 91.216.175.5
ip route 91.216.175.24 255.255.255.255 91.216.175.5
ip route 91.216.175.25 255.255.255.255 91.216.175.5
ip route 91.216.175.26 255.255.255.255 91.216.175.5
ip route 91.216.175.27 255.255.255.255 91.216.175.5
ip route 91.216.175.28 255.255.255.255 91.216.175.5
ip route 91.216.175.29 255.255.255.255 91.216.175.5
ip route 91.216.175.30 255.255.255.255 91.216.175.5
ip route 91.216.175.31 255.255.255.255 91.216.175.5
ip route 91.216.175.32 255.255.255.240 91.216.175.5
ip route 91.216.175.48 255.255.255.240 91.216.175.4
ip route 91.216.175.50 255.255.255.255 91.216.175.5
ip route 172.20.0.0 255.255.255.0 172.20.0.33
ip route vrf SCBK_Internet 10.255.200.0 255.255.255.0 10.10.128.214
ip route vrf SCBK_L3VPN 10.255.200.0 255.255.255.0 10.10.128.222
ip ssh version 2
!
!
ip prefix-list L3VPN_Nets seq 10 permit 192.168.22.0/24 le 32
ip prefix-list L3VPN_Nets seq 15 permit 192.168.100.0/24 le 32
ip prefix-list L3VPN_Nets seq 20 permit 192.168.104.0/24 le 32
ip prefix-list L3VPN_Nets seq 30 permit 192.168.249.0/24 le 32
ip prefix-list L3VPN_Nets seq 40 permit 10.10.176.0/22 le 32
ip prefix-list L3VPN_Nets seq 50 permit 10.10.180.0/22 le 32
ip prefix-list L3VPN_Nets seq 60 permit 10.10.156.0/24 le 32
ip prefix-list L3VPN_Nets seq 70 permit 10.10.190.0/24 le 32
!
ip prefix-list NoServiceNets seq 10 permit 10.255.110.0/24 le 32
ip prefix-list NoServiceNets seq 20 permit 10.255.111.0/24 le 32
ip prefix-list NoServiceNets seq 30 permit 10.255.200.0/24 le 32
ip prefix-list NoServiceNets seq 40 permit 10.10.128.192/27 le 32
!
ip prefix-list REDISTRIBUTE_FAKE seq 5 permit 10.0.0.0/8
ip prefix-list REDISTRIBUTE_FAKE seq 10 permit 172.16.0.0/12
ip prefix-list REDISTRIBUTE_FAKE seq 15 permit 192.168.0.0/16
!
ip prefix-list REDISTRIBUTE_STATIC seq 5 permit 10.10.190.0/24
!
ip prefix-list REDISTRIBUTE_STATIC_SCBK seq 10 permit 10.10.128.0/18
!
ip prefix-list no-default-route seq 5 deny 0.0.0.0/0
ip prefix-list no-default-route seq 10 permit 0.0.0.0/0 le 32
!
ip prefix-list pi_network seq 5 permit 91.216.175.0/24
ip prefix-list pi_network seq 10 deny 0.0.0.0/0
!
route-map STATIC_TO_EIGRP_SCBK permit 10
match ip address prefix-list REDISTRIBUTE_STATIC_SCBK
!
route-map CONNECTED_TO_EIGRP permit 10
match ip address prefix-list REDISTRIBUTE_FAKE
!
route-map set_rt_community permit 10
set community 12389:2800
!
route-map STATIC_TO_EIGRP permit 10
match ip address prefix-list REDISTRIBUTE_STATIC
!
route-map OSPF->EIGRP_B deny 10
match ip address prefix-list NoServiceNets
!
route-map OSPF->EIGRP_B permit 20
match tag 1014
set metric 100000 100 255 1 1500
!
route-map OSPF->EIGRP deny 10
match ip address prefix-list NoServiceNets
!
route-map OSPF->EIGRP permit 20
match tag 1014
set metric 1000000 100 255 1 1500
!
route-map set_med permit 10
set metric +10
!
route-map EIGRP110->OSPF3 deny 10
match ip address prefix-list NoServiceNets
!
route-map EIGRP110->OSPF3 deny 20
match tag 1014
!
route-map EIGRP110->OSPF3 permit 30
match ip address prefix-list L3VPN_Nets
set metric 20
!
route-map EIGRP110->OSPF3 permit 40
set metric 10
!
route-map EIGRP111->OSPF4 deny 10
match ip address prefix-list NoServiceNets
!
route-map EIGRP111->OSPF4 deny 20
match tag 1014
!
route-map EIGRP111->OSPF4 permit 30
match ip address prefix-list L3VPN_Nets
set metric 15
!
route-map EIGRP111->OSPF4 permit 40
set metric 25
!
route-map DenyLocal deny 10
match tag 1014 3 4
!
route-map DenyLocal permit 20
!
!
snmp-server community bla-bla-bla RW 2
snmp-server ifindex persist
snmp-server host 10.10.136.20 version 2c bla-bla-bla
access-list 1 permit 78.36.44.202
access-list 1 permit 212.109.12.45
access-list 1 permit 192.168.239.44
access-list 1 permit 91.216.175.0 0.0.0.255
access-list 1 permit 10.10.136.0 0.0.0.255
access-list 1 permit 10.10.138.0 0.0.0.255
access-list 2 permit 91.216.175.22
access-list 2 permit 10.10.136.31
access-list 2 permit 10.10.136.20
access-list 10 deny 0.0.0.0 255.255.255.240
access-list 10 permit any
access-list 11 deny 0.0.0.0 255.255.255.240
access-list 11 permit any
!
!
!
control-plane
!
privilege exec level 5 terminal length
privilege exec level 5 terminal
privilege exec level 5 show ip route
privilege exec level 5 show ip
privilege exec level 5 show configuration
privilege exec level 5 show
!
line con 0
exec-timeout 30 0
logging synchronous
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class 1 in
exec-timeout 30 0
logging synchronous
transport input telnet ssh
line vty 5 15
access-class 1 in
exec-timeout 30 0
logging synchronous
transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

[Georg Pauwen]

Hello,

you have a rather massive config for a relatively small (2921) router. What are the specs of the router (sh ver) ? My guess is that the router is just overloaded in general, and the only remedy is to upgrade to a more powerful one...

[wggfilonenko1]

Yes the config looks big. Perhaps even intimidating. But it is not difficult in terms of load. There is no NAT, firewall, QoS. There's just a few VRFs and just routing (with dynamic protocols).

[Joseph W. Doherty]

"This is a lot, but it is far from 100%."

"CPU utilization for five seconds: 99%/47%; one minute: 95%; five minutes: 95%"

Far from?

"How can we determine the cause of the CPU load?"

"CPU utilization for five seconds: 99%/47%; one minute: 95%; five minutes: 95%"

172 57333076 87035101 658 42.46% 41.33% 40.21% 0 IP Input

47% + 42.46% = 89.46%

The other "missing" 9.54% (99% - 42.46%) should be found in the sum of all the other software processes.

That said, your IP Input percentage is very high.  Ideally, you want to see interrupt percentage within 2 or 3% of total usage percentage.  A high IP Input is something you, ideally, want to mitigate.  For example, see: https://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41160-highcpu-ip-input.html

[Richard Burts]

It would be helpful if we could see the entire config (after disguising sensitive information). It does suggest that there is a lot of process switching. This might be caused by interface configuration that disables cef/fast switching. Or it might be caused by an access list applied that contains the log parameter. Or it might be caused by something else. Seeing the config would help us identify the issue.

[wggfilonenko1]

 version  15.7
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname scbk-router1
!
boot-start-marker
boot system flash:c2900-universalk9-mz.SPA.157-3.M8.bin
boot-end-marker
!
!
logging buffered 51200 warnings
enable secret 5 bla-bla-bla
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
!
!
!
aaa session-id common
!
!
!
!
!
!
no ip source-route
!
!
!
ip vrf SCBK_Internet
rd 1000:1
!
ip vrf SCBK_L3VPN
rd 1000:2
!
!
!
!
ip domain name scbk.ru
ip name-server 8.8.8.8
ip name-server 77.88.8.8
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
cts logging verbose
!
crypto pki trustpoint TP-self-signed-3190922259
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3190922259
revocation-check none
rsakeypair TP-self-signed-3190922259
!
!
crypto pki certificate chain TP-self-signed-3190922259
certificate self-signed 01
quit
license udi pid CISCO2921/K9 sn FCZ143572EP
license boot module c2900 technology-package securityk9
!
!
file privilege 1
username admin privilege 15 password 7 bla-bla-bla
!
redundancy
!
!
!
!
!
track 1 interface GigabitEthernet0/1 line-protocol
!
!
!
crypto isakmp policy 40
encr aes 256
authentication pre-share
group 5
crypto isakmp key bla-bla-bla address 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10 5 periodic
!
!
crypto ipsec transform-set ESP-AES-SHA-TRANSPORT esp-aes esp-sha-hmac
mode transport
!
crypto ipsec profile IPSEC
set transform-set ESP-AES-SHA-TRANSPORT
set pfs group5
!
!
!
!
!
!
!
interface Tunnel1
description ---L3VPN DMVPN---
ip address 10.255.250.34 255.255.255.0
no ip redirects
ip nhrp authentication ILP111
ip nhrp map 10.255.250.1 172.20.0.2
ip nhrp map multicast 172.20.0.2
ip nhrp network-id 111
ip nhrp holdtime 60
ip nhrp nhs 10.255.250.1
ip nhrp registration timeout 10
ip tcp adjust-mss 1376
tunnel source GigabitEthernet0/2.25
tunnel mode gre multipoint
tunnel key bla-bla-bla
!
interface Tunnel2
description ---DMVPN to MSK DC---
ip address 10.255.255.19 255.255.255.0
no ip redirects
ip mtu 1416
no ip split-horizon eigrp 1
ip nhrp authentication ILP100
ip nhrp map 10.255.255.1 92.242.43.68
ip nhrp map multicast 92.242.43.68
ip nhrp network-id 100
ip nhrp nhs 10.255.255.1
ip tcp adjust-mss 1376
tunnel source GigabitEthernet0/0.32
tunnel mode gre multipoint
tunnel key bla-bla-bla
tunnel protection ipsec profile IPSEC
!
interface Tunnel1004
bandwidth 10000000
ip vrf forwarding SCBK_Internet
ip address 10.255.110.14 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication nhrp110
ip nhrp map 10.255.110.1 10.255.200.17
ip nhrp map multicast 10.255.200.17
ip nhrp network-id 110
ip nhrp nhs 10.255.110.1
delay 100
tunnel source GigabitEthernet0/2.52
tunnel mode gre multipoint
tunnel key bla-bla-bla
tunnel vrf SCBK_Internet
!
interface Tunnel2104
bandwidth 1000000
ip vrf forwarding SCBK_L3VPN
ip address 10.255.111.114 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication nhrp111
ip nhrp map 10.255.111.1 10.255.200.25
ip nhrp map multicast 10.255.200.25
ip nhrp network-id 111
ip nhrp nhs 10.255.111.1
delay 100
tunnel source GigabitEthernet0/2.53
tunnel mode gre multipoint
tunnel key bla-bla-bla
tunnel vrf SCBK_L3VPN
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description Channels to LAN SCBK
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0/0.4
description Link to wireless
bandwidth 100000
encapsulation dot1Q 4
ip address 91.216.175.253 255.255.255.240
standby 2 ip 91.216.175.254
standby 2 priority 105
standby 2 preempt delay minimum 60
standby 2 track 1 decrement 10
!
interface GigabitEthernet0/0.32
description Link to firewall
bandwidth 100000
encapsulation dot1Q 32
ip address 91.216.175.1 255.255.255.240
standby 1 ip 91.216.175.3
standby 1 priority 105
standby 1 preempt delay minimum 60
standby 1 track 1 decrement 10
!
interface GigabitEthernet0/0.51
encapsulation dot1Q 51
ip vrf forwarding SCBK_L3VPN
ip address 10.10.128.201 255.255.255.240
standby 0 authentication SCBK_L3V
standby 6 ip 10.10.128.203
standby 6 priority 105
standby 6 preempt delay minimum 60
standby 6 track 1 decrement 10
!
interface GigabitEthernet0/1
description Channels to Internet
no ip address
load-interval 30
duplex auto
speed auto
!
interface GigabitEthernet0/1.31
description MTS channel to Internet
bandwidth 100000
encapsulation dot1Q 31
ip address 84.17.4.150 255.255.255.252
!
interface GigabitEthernet0/1.33
description NWT channel to Internet
bandwidth 2000
encapsulation dot1Q 33
ip address 212.48.214.170 255.255.255.248
!
interface GigabitEthernet0/1.34
description Rostelecom channel to Internet
bandwidth 10000
encapsulation dot1Q 34
ip address 95.167.190.98 255.255.255.248
!
interface GigabitEthernet0/2
description Leased channels
no ip address
load-interval 30
duplex auto
speed auto
!
interface GigabitEthernet0/2.25
description MTS VPN channel
encapsulation dot1Q 25
ip address 172.20.0.34 255.255.255.248
!
interface GigabitEthernet0/2.51
encapsulation dot1Q 51
ip vrf forwarding SCBK_Internet
ip address 10.10.128.193 255.255.255.240
standby 0 authentication bla-bla-bla
standby 5 ip 10.10.128.195
standby 5 priority 105
standby 5 preempt delay minimum 60
standby 5 track 1 decrement 10
!
interface GigabitEthernet0/2.52
encapsulation dot1Q 52
ip vrf forwarding SCBK_Internet
ip address 10.10.128.209 255.255.255.248
standby 3 ip 10.10.128.211
standby 3 priority 105
standby 3 preempt delay minimum 60
standby 3 track 1 decrement 10
!
interface GigabitEthernet0/2.53
encapsulation dot1Q 53
ip vrf forwarding SCBK_L3VPN
ip address 10.10.128.217 255.255.255.248
standby 4 ip 10.10.128.219
standby 4 priority 105
standby 4 preempt delay minimum 60
standby 4 track 1 decrement 10
!
!
router eigrp 11
distribute-list prefix REDISTRIBUTE_STATIC out Tunnel1
network 10.255.250.0 0.0.0.255
network 91.216.175.0 0.0.0.15
redistribute static route-map STATIC_TO_EIGRP
passive-interface default
no passive-interface Tunnel1
no passive-interface GigabitEthernet0/0.32
eigrp router-id 10.255.250.34
!
!
router eigrp 1
distribute-list prefix REDISTRIBUTE_STATIC_SCBK out Tunnel2
network 10.255.255.0 0.0.0.255
network 91.216.175.0 0.0.0.15
redistribute static route-map STATIC_TO_EIGRP_SCBK
passive-interface default
no passive-interface Tunnel2
no passive-interface GigabitEthernet0/0.32
eigrp router-id 10.255.250.19
!
!
router eigrp 110
!
address-family ipv4 vrf SCBK_Internet
redistribute ospf 3 route-map OSPF->EIGRP
network 10.255.110.0 0.0.0.255
passive-interface default
no passive-interface Tunnel1004
distribute-list route-map DenyLocal in
autonomous-system 110
eigrp router-id 10.255.240.28
exit-address-family
!
!
router eigrp 111
!
address-family ipv4 vrf SCBK_L3VPN
redistribute ospf 4 route-map OSPF->EIGRP_B
network 10.255.111.0 0.0.0.255
passive-interface default
no passive-interface Tunnel2104
distribute-list route-map DenyLocal in
autonomous-system 111
eigrp router-id 10.255.240.30
exit-address-family
!
router ospf 3 vrf SCBK_Internet
router-id 10.255.240.28
redistribute eigrp 110 subnets route-map EIGRP110->OSPF3
passive-interface default
no passive-interface GigabitEthernet0/2.51
network 10.10.128.192 0.0.0.15 area 0
default-information originate metric 10
distance ospf external 180
!
router ospf 4 vrf SCBK_L3VPN
router-id 10.255.240.30
redistribute eigrp 111 subnets route-map EIGRP111->OSPF4
passive-interface default
no passive-interface GigabitEthernet0/0.51
network 10.10.128.192 0.0.0.15 area 0
default-information originate metric 220
distance ospf external 180
!
router bgp 51165
bgp log-neighbor-changes
network 91.216.175.0 mask 255.255.255.0
aggregate-address 91.216.175.0 255.255.255.0 summary-only
redistribute connected
neighbor 84.17.4.149 remote-as 8359
neighbor 84.17.4.149 prefix-list no-default-route in
neighbor 84.17.4.149 prefix-list pi_network out
neighbor 84.17.4.149 route-map set_med out
neighbor 91.216.175.2 remote-as 51165
neighbor 91.216.175.2 next-hop-self
neighbor 95.167.190.97 remote-as 12389
neighbor 95.167.190.97 prefix-list no-default-route in
neighbor 95.167.190.97 prefix-list pi_network out
neighbor 95.167.190.97 route-map set_med out
neighbor 212.48.214.169 remote-as 8997
neighbor 212.48.214.169 prefix-list pi_network out
neighbor 212.48.214.169 route-map set_med out
!
ip forward-protocol nd
!
ip bgp-community new-format
ip community-list 10 permit 12389:2800
ip http server
ip http access-class 1
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 10.10.128.0 255.255.192.0 91.216.175.4
ip route 10.10.190.0 255.255.255.0 91.216.175.4
ip route 91.216.175.16 255.255.255.240 91.216.175.4
ip route 91.216.175.16 255.255.255.255 91.216.175.5
ip route 91.216.175.17 255.255.255.255 91.216.175.5
ip route 91.216.175.18 255.255.255.255 91.216.175.5
ip route 91.216.175.19 255.255.255.255 91.216.175.5
ip route 91.216.175.20 255.255.255.255 91.216.175.5
ip route 91.216.175.21 255.255.255.255 91.216.175.5
ip route 91.216.175.22 255.255.255.255 91.216.175.5
ip route 91.216.175.24 255.255.255.255 91.216.175.5
ip route 91.216.175.25 255.255.255.255 91.216.175.5
ip route 91.216.175.26 255.255.255.255 91.216.175.5
ip route 91.216.175.27 255.255.255.255 91.216.175.5
ip route 91.216.175.28 255.255.255.255 91.216.175.5
ip route 91.216.175.29 255.255.255.255 91.216.175.5
ip route 91.216.175.30 255.255.255.255 91.216.175.5
ip route 91.216.175.31 255.255.255.255 91.216.175.5
ip route 91.216.175.32 255.255.255.240 91.216.175.5
ip route 91.216.175.48 255.255.255.240 91.216.175.4
ip route 91.216.175.50 255.255.255.255 91.216.175.5
ip route 172.20.0.0 255.255.255.0 172.20.0.33
ip route vrf SCBK_Internet 10.255.200.0 255.255.255.0 10.10.128.214
ip route vrf SCBK_L3VPN 10.255.200.0 255.255.255.0 10.10.128.222
ip ssh version 2
!
!
ip prefix-list L3VPN_Nets seq 10 permit 192.168.22.0/24 le 32
ip prefix-list L3VPN_Nets seq 15 permit 192.168.100.0/24 le 32
ip prefix-list L3VPN_Nets seq 20 permit 192.168.104.0/24 le 32
ip prefix-list L3VPN_Nets seq 30 permit 192.168.249.0/24 le 32
ip prefix-list L3VPN_Nets seq 40 permit 10.10.176.0/22 le 32
ip prefix-list L3VPN_Nets seq 50 permit 10.10.180.0/22 le 32
ip prefix-list L3VPN_Nets seq 60 permit 10.10.156.0/24 le 32
ip prefix-list L3VPN_Nets seq 70 permit 10.10.190.0/24 le 32
!
ip prefix-list NoServiceNets seq 10 permit 10.255.110.0/24 le 32
ip prefix-list NoServiceNets seq 20 permit 10.255.111.0/24 le 32
ip prefix-list NoServiceNets seq 30 permit 10.255.200.0/24 le 32
ip prefix-list NoServiceNets seq 40 permit 10.10.128.192/27 le 32
!
ip prefix-list REDISTRIBUTE_FAKE seq 5 permit 10.0.0.0/8
ip prefix-list REDISTRIBUTE_FAKE seq 10 permit 172.16.0.0/12
ip prefix-list REDISTRIBUTE_FAKE seq 15 permit 192.168.0.0/16
!
ip prefix-list REDISTRIBUTE_STATIC seq 5 permit 10.10.190.0/24
!
ip prefix-list REDISTRIBUTE_STATIC_SCBK seq 10 permit 10.10.128.0/18
!
ip prefix-list no-default-route seq 5 deny 0.0.0.0/0
ip prefix-list no-default-route seq 10 permit 0.0.0.0/0 le 32
!
ip prefix-list pi_network seq 5 permit 91.216.175.0/24
ip prefix-list pi_network seq 10 deny 0.0.0.0/0
!
route-map STATIC_TO_EIGRP_SCBK permit 10
match ip address prefix-list REDISTRIBUTE_STATIC_SCBK
!
route-map CONNECTED_TO_EIGRP permit 10
match ip address prefix-list REDISTRIBUTE_FAKE
!
route-map set_rt_community permit 10
set community 12389:2800
!
route-map STATIC_TO_EIGRP permit 10
match ip address prefix-list REDISTRIBUTE_STATIC
!
route-map OSPF->EIGRP_B deny 10
match ip address prefix-list NoServiceNets
!
route-map OSPF->EIGRP_B permit 20
match tag 1014
set metric 100000 100 255 1 1500
!
route-map OSPF->EIGRP deny 10
match ip address prefix-list NoServiceNets
!
route-map OSPF->EIGRP permit 20
match tag 1014
set metric 1000000 100 255 1 1500
!
route-map set_med permit 10
set metric +10
!
route-map EIGRP110->OSPF3 deny 10
match ip address prefix-list NoServiceNets
!
route-map EIGRP110->OSPF3 deny 20
match tag 1014
!
route-map EIGRP110->OSPF3 permit 30
match ip address prefix-list L3VPN_Nets
set metric 20
!
route-map EIGRP110->OSPF3 permit 40
set metric 10
!
route-map EIGRP111->OSPF4 deny 10
match ip address prefix-list NoServiceNets
!
route-map EIGRP111->OSPF4 deny 20
match tag 1014
!
route-map EIGRP111->OSPF4 permit 30
match ip address prefix-list L3VPN_Nets
set metric 15
!
route-map EIGRP111->OSPF4 permit 40
set metric 25
!
route-map DenyLocal deny 10
match tag 1014 3 4
!
route-map DenyLocal permit 20
!
!
snmp-server community bla-bla-bla RW 2
snmp-server ifindex persist
snmp-server host 10.10.136.20 version 2c bla-bla-bla
access-list 1 permit 78.36.44.202
access-list 1 permit 212.109.12.45
access-list 1 permit 192.168.239.44
access-list 1 permit 91.216.175.0 0.0.0.255
access-list 1 permit 10.10.136.0 0.0.0.255
access-list 1 permit 10.10.138.0 0.0.0.255
access-list 2 permit 91.216.175.22
access-list 2 permit 10.10.136.31
access-list 2 permit 10.10.136.20
access-list 10 deny 0.0.0.0 255.255.255.240
access-list 10 permit any
access-list 11 deny 0.0.0.0 255.255.255.240
access-list 11 permit any
!
!
!
control-plane
!
privilege exec level 5 terminal length
privilege exec level 5 terminal
privilege exec level 5 show ip route
privilege exec level 5 show ip
privilege exec level 5 show configuration
privilege exec level 5 show
!
line con 0
exec-timeout 30 0
logging synchronous
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class 1 in
exec-timeout 30 0
logging synchronous
transport input telnet ssh
line vty 5 15
access-class 1 in
exec-timeout 30 0
logging synchronous
transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

[wggfilonenko1]

42% is traffic load and 47% is internal processes (interrupt handling). Why can there be such a high internal load?

Total traffic ~ 200M. cef statistics are present i.e. fast switching works

G0/0 (link to firewall)
5 minute input rate 28040000 bits/sec, 9652 packets/sec
5 minute output rate 65538000 bits/sec, 9225 packets/sec

G0/1 (link to ISPs)
30 second input rate 65183000 bits/sec, 9122 packets/sec
30 second output rate 36672000 bits/sec, 10896 packets/sec

G0/2 (link to VPN, 25-50% with IPSec)
30 second input rate 95423000 bits/sec, 18687 packets/sec
30 second output rate 93583000 bits/sec, 16208 packets/sec

[Leo Laohoo]

200 Mbps link?  On a 2921 router????

[wggfilonenko1]

According to Cisco documentation, the performance of 2921 is up to 479 kpps or 3.5 Gbps on large packets. A typical packet mix should be around 1 Gbps.
An indirect confirmation is that the encryption performance is up to 207 Mbps.
40+% in IP Input confirms both these calculations and the availability of free performance.
Moreover, the current setup has been working for a long time with this load and the problems started recently.
The problem arose when the router began to spend a lot of CPU on some internal processes. And we cannot understand exactly which processes, because diasgnostics does not indicate this. And I don’t know the commands that could help us.
We cannot just turn off some channels or services because they are in use. And when they are not used, they do not give a load and we will not see the reason.

But I understood your idea. We have two routers, the main and the backup, I will try to split the services into two devices and reduce the load. This will solve the problem and allow planning and replacement of equipment.
But now I don't even know how to choose a model if cisco is lying de-facto in its specifications.

[Joseph W. Doherty]

". . .the current setup has been working for a long time with this load and the problems started recently."

Possibly a change in what's been routine traffic, and/or downstream MTU considerations?

I notice you're using tunnels, and although you using "ip tcp adjust-mss 1376", that has no effect on UDP traffic, like video streams, which can create fragmentation CPU performance issues, which might be accounted as IP Input.  (NB: I've seen such newly created streams really chew up a small ISR's CPU.)

Further, although you're using "ip tcp adjust-mss 1376", there's more, I believe (NB: I haven't carefully studied your config) to optimize tunnels that impact MTU (NB: often, though, all the techniques don't have much control over typical UDP stream fragmentation).