cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1687
Views
0
Helpful
6
Replies

DMVPN Tunnels causing 99% CPU on 2951

the-lebowski
Level 4
Level 4

Seeing this issue today with a spoke in India.  We have dual hub dual cloud and if either tunnel is up the CPU spikes to 99% and the router starts dropping packets.  If I shut both tunnels down everything returns to normal and I have no idea what could be causing it?  Something pushing a lot of data across the DMVPN tunnels?  Once I bring the tunnels backup I see EIGRP constantly flapping and the CPU spikes immediately but nothing showing me what is causing the problem.   If it is someone trying to push traffic between sites it would show regardless of which tunnel was up as they are redundant but I am at a loss.  

No changes made to this router.  Any ideas?  

DMVPN tunnel shutdown:

CPU utilization for five seconds: 1%/0%; one minute: 1%; five minutes: 18%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
88 46924 1768 26540 0.15% 0.11% 0.10% 0 Per-Second Jobs
107 48420 5072 9546 0.23% 0.19% 0.18% 0 Netclock Backgro
145 12028 116244 103 0.07% 0.10% 0.08% 0 Ethernet Msec Ti
458 11244 194180 57 0.39% 0.39% 0.35% 0 IP SLAs XOS Even

1 or both DVMPN tunnels up:

CPU utilization for five seconds: 99%/97%; one minute: 51%; five minutes: 53%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
88 46856 1418 33043 0.15% 0.11% 0.10% 0 Per-Second Jobs
107 48164 3691 13049 0.07% 0.15% 0.16% 0 Netclock Backgro
117 23784 2362 10069 0.15% 0.16% 0.11% 644 SSH Process
128 7040 33962 207 0.07% 0.06% 0.05% 0 SEC BATCH
145 12024 72842 165 0.07% 0.08% 0.08% 0 Ethernet Msec Ti
455 184752 1484 124495 0.38% 0.13% 0.24% 0 CFT Timer Proces
456 862876 2097 411481 0.54% 0.29% 0.40% 0 FNF Cache Ager P
458 11168 108909 102 0.07% 0.25% 0.23% 0 IP SLAs XOS Even
]
1 Accepted Solution

Accepted Solutions

sarabsin
Level 1
Level 1

you have interrupt high that is causing the CPU high. The traffic is going to CPU..

Make sure  CEF switching is enabled  on all interfaces....

check this link for possible causes of high interrupts..

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html

Thanks,

Sarabjit

View solution in original post

6 Replies 6

Philip D'Ath
VIP Alumni
VIP Alumni

If you do a "show interface tunnel xxx" after a minute of running does it show a lot of traffic?

And to clarify, this is an existing working configuration that has just recently developed the issue.  Correct?

Its been resolved with TAC help.  Enabled netflow top talkers and saw a lab host pushing a ton of traffic across this router.  Blocked with an ACL and the problem went away.  

Thanks for the update.

just curious what kind of traffic it was?  probably unreachable destination causing generation of icmp unreachable packets hence causing high CPU ..or some traffic which was not cef switched?

I would believe so although at the time I wasn't able to verify what it was doing. 

I do know the offending host was generating 4000+ flows to single IP on the far side of the DMVPN tunnels. So much that it caused an issue on the DMVPN hub router as well. Once I blocked that host the flows dropped to 200 give or take.  

The destination IP doesn't resolve today so I am not even sure it ever was valid, if it wasn't that would match up with your theory on ICMP unreachable packets.  

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 E648 0035 960
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 87EE 0035 941
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 8DEB 0035 929
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 BBF2 0035 917
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 BBC7 0035 914
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 B028 0035 904
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 C8F8 0035 860
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 B2C2 0035 814
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 E458 0035 746
Gi0/0.2 10.187.99.200 Tu2* 10.222.36.10 11 AC6E 0035 745
10 of 10 top talkers shown. 4090 flows processed.

ok thanks

sarabsin
Level 1
Level 1

you have interrupt high that is causing the CPU high. The traffic is going to CPU..

Make sure  CEF switching is enabled  on all interfaces....

check this link for possible causes of high interrupts..

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html

Thanks,

Sarabjit