Processor at 99% on Cisco DMVPN HUB router

[qsscisco]

Hello,

We have DMVPN network with 2 HUB and 100+ spokes. The router is 3900 with 15.7 ios image. Everything was worked till yesterday when we notice that we have a lot of packets dropped. When we access device : show processess cpu we saw that procesor is 99%.When we shutdown outside interface to the ISP, processor imiditaly comes on 1-2%.we used shoe processes cpu sorted we see about 1,2 % top processes regarding IP input, 0.95 crypto IKAMP, 0.23 EIGRP EIGRP-IPV4.We are getting message also EIGRP is down, peer restarted, eigrp id down hold timer expired, interface peer termination recieved and so on.

Does anyone have any idea to    recommend how to find and solve this problem?

[balaji.bandi]

what kind of Links you have what throughput of that links ? check on the interface is there any over utilization (if you have NMS that give you more information)

How long was the uptime of the devices - try reboot one hub at a time and see any improvements.

can you able to share the show process cpu | ex 0.00 - also what logs you see from yesterday to today on both the Hub Logs ?

Also check some suggestion CEF

https://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html

[MHM Cisco World]

if Spokes 100+ is use Hub to access internet then yes the hub face this issue BUT BUT

IP input <- this need to check 
I will update you with some steps to check why IP Input is High in router 

MHM

[MHM Cisco World]

High CPU – IP Input on Cisco Router | tukang-tukang oprek (wordpress.com)

check this 

focus on 
show ip traffic part, you need to run this command many times to see why high rate traffic send to CPU 

MHM

[qsscisco]

@MHM Cisco World @balaji.bandi 

Hello,

We faces maybe about 50mbps  mbps traffic on iside and outside interface. Encryption licence is 85mbps. When we deny some traffic going from hub network to the spokes on hub inside interface traffic goes down and processor goes about 20-30%. The question is why router cant handle about 80mbps traffic normaly? 

[MHM Cisco World]

I mention that in first comment' if router can not handle all traffic then you will face issue with hub router 

Did you check 

Show ip traffic?

MHM

[qsscisco]

@MHM Cisco World Sorry i didnt see you reply because in the meanwhile i went to location and do the action that i describe above. So when router is on encryption licence 80%, it can cause cpu showing 99% or is it anything else that i can do now?

[MHM Cisco World]

Show platform hardware throughput crypto 

Show ip traffic  <- do this two or three times 

Share this to understand it is crypto issue of there is multicast traffic push to cpu 

MHM

[qsscisco]

The first command doesnt exist on this router. i run show platform cerm-information and there is output:

Crypto Export Restrictions Manager(CERM) Information:
CERM functionality: ENABLED

----------------------------------------------------------------
Resource Maximum Limit Available
----------------------------------------------------------------
Tx Bandwidth(in kbps) 85000 85000
Rx Bandwidth(in kbps) 85000 85000
Number of tunnels 225 144
Number of TLS sessions 1000 1000

Resource reservation information:
D - Dynamic
-----------------------------------------------------------------------
Client Tx Bandwidth Rx Bandwidth Tunnels TLS Sessions
(in kbps) (in kbps)
-----------------------------------------------------------------------
VOICE 0 0 0 0
IPSEC D D 81 N/A
SSLVPN D D 0 N/A

Statistics information:
Failed tunnels : 0
Failed sessions : 0
Failed tx bandwidth: 0
Failed rx bandwidth: 0
Failed encrypt pkts: 3749
Failed decrypt pkts: 1779
Failed encrypt pkt bytes: 4716576
Failed decrypt pkt bytes: 1971684
Passed encrypt pkts: 47454251
Passed decrypt pkts: 43903598
Passed encrypt pkt bytes: 33366272792
Passed decrypt pkt bytes: 17583170912

i attach output from show ip traffic, but keep in mind that now acl blocking traffic and now it is not working hours.

VPN-router2#show ip traffic
IP statistics:
Rcvd: 2220724 total, 1371897 local destination
0 format errors, 0 checksum errors, 10174 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
1427446 fragmented, 2854892 fragments, 0 couldn't fragment
0 invalid hole
Bcast: 1448 received, 0 sent
Mcast: 668266 received, 30130 sent
Sent: 4384205 generated, 132315780 forwarded
Drop: 95 encapsulation failed, 0 unresolved, 0 no adjacency
1 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 7 unreachable
6 echo, 1400 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
6 time exceeded, 0 info replies
Sent: 0 redirects, 67988 unreachable, 1400 echo, 6 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 10174 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 644353 total, 0 checksum errors, 1484 no port 0 finput
Sent: 3510417 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

TCP statistics:
Rcvd: 3982 total, 0 checksum errors, 1 no port
Sent: 4234 total

EIGRP-IPv4 statistics:
Rcvd: 722021 total
Sent: 36510 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 48018 requests, 8 replies, 0 reverse, 0 other
Sent: 5391 requests, 89 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
VPN-router2#show ip traffic
IP statistics:
Rcvd: 2221785 total, 1372586 local destination
0 format errors, 0 checksum errors, 10174 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
1427535 fragmented, 2855070 fragments, 0 couldn't fragment
0 invalid hole
Bcast: 1448 received, 0 sent
Mcast: 668615 received, 30145 sent
Sent: 4385109 generated, 132324662 forwarded
Drop: 95 encapsulation failed, 0 unresolved, 0 no adjacency
1 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 7 unreachable
6 echo, 1400 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
6 time exceeded, 0 info replies
Sent: 0 redirects, 68027 unreachable, 1400 echo, 6 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 10174 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 644692 total, 0 checksum errors, 1484 no port 0 finput
Sent: 3510935 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

TCP statistics:
Rcvd: 3990 total, 0 checksum errors, 1 no port
Sent: 4243 total

EIGRP-IPv4 statistics:
Rcvd: 722363 total
Sent: 36518 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 48038 requests, 8 replies, 0 reverse, 0 other
Sent: 5391 requests, 89 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
VPN-router2#show ip traffic
IP statistics:
Rcvd: 2221981 total, 1372692 local destination
0 format errors, 0 checksum errors, 10174 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
1427549 fragmented, 2855098 fragments, 0 couldn't fragment
0 invalid hole
Bcast: 1448 received, 0 sent
Mcast: 668670 received, 30148 sent
Sent: 4385282 generated, 132325655 forwarded
Drop: 95 encapsulation failed, 0 unresolved, 0 no adjacency
1 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 7 unreachable
6 echo, 1400 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
6 time exceeded, 0 info replies
Sent: 0 redirects, 68033 unreachable, 1400 echo, 6 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 10174 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 644735 total, 0 checksum errors, 1484 no port 0 finput
Sent: 3511005 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

TCP statistics:
Rcvd: 4000 total, 0 checksum errors, 1 no port
Sent: 4254 total

EIGRP-IPv4 statistics:
Rcvd: 722416 total
Sent: 36520 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 48041 requests, 8 replies, 0 reverse, 0 other
Sent: 5391 requests, 89 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
VPN-router2#

[MHM Cisco World]

Sent: 4385282 generated, 132325655 forwarded

Sent: 4384205 generated, 132315780 forwarded

From show to show the route forward 10000 packets, sure it is face high cpu utilize.

if you do 
show ip interface <tunnel>
show ip interface <tunnel source> 
and you see the CEF is enable 

then I think you need New More powerful router,  and to be more sure contact cisco.
MHM

[balaji.bandi]

The Router  can support upto 300+MB - but what License you have on the router?

check PPS calculation :

https://sec.cloudapps.cisco.com/security/center/resources/network_performance_metrics.html

what is your Link speed from ISP. as per your ip traffic i see some drops, may be clear the counter and run again see how quick the drops accumulate.

on the interface can you check any drop out and inside interfaces and what load you see on the interface ? 

show ip traffic interface gig x/x

[tvotna]

@balaji.bandi, how do you know that the router supports 300+ MB if you don't know what kind of router this is, whether it's 3925 or 3945 or 3925E or 3945E and which hardware crypto accelerator it has?

@qsscisco It's very common for old ISR platforms that real performance numbers don't match what is officially published. Every feature configured on the router decreases performance. Also, if you don't see "IP Input" process on top of the "show proc cpu sorted 5sec" output, you shouldn't worry about process-switched traffic and it's unlikely you can do anything here. If you see it, check how many packets are punted to the process switching path over time with the "show ip cef switching statistics" and "show ip cef switching statistics feature".

[balaji.bandi]

@tvotna  The model 3900 support  upto 300+ MB 

• The Cisco 3900 Series enables deployment in high-speed WAN environments with concurrent services enabled up to 350 Mbps.

which hardware crypto accelerator it has?  - this question should go to OP 

[tvotna]

Of course. Just one slide.