cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
326
Views
0
Helpful
1
Replies

DMVPN & eigrp CPU-HOG and hang

Shawn Lebbon
Level 1
Level 1

I have 3 1721 routers with VPN modules and IOS 12.3(10) on them.

There is a simple DMVPN setup on these 3. We're planning on rolling out more sites in the future, but right now we want to get these 3 working (1 hub, 2 spokes). Eventually we want to have 2 hubs and 6 spokes.

Everything works great as soon as the routers are turned on, and they seem to operate just fine. The routes are correct, nhrp shows it's working, the tunnels are active, the crypto engine shows that everything is correct, and dynamic spoke-to-spoke tunnels come up and go down without issue when we generate traffic between them. However, (there's always one of those!), After everthing sits for a few days we start to have lots of problems. The EIGRP HELLO process starts to use ~30-40% average CPU time without stop. It comes in bursts so that like 1 in 3 seconds the router will completely lock up, then resume processing.

I've turned on logging and there are CPU-HOG messages generated for the eigrp process every few seconds...

Also at this time one of the tunnels always seems to go down, while the other stays up. After awhile the 'down' tunnel will come back up, but eigrp will continue to hang until the router is rebooted. After that everything comes back up just great...until a few more days go by and it hangs again.

I'm not sure of the exact time of the problem starting, because I'm only here every other day to look at it.

It might also be relivant that the hub router is ALSO running as a L2TP VPN Gateway for Windows users to connect from the Internet, but I don't think they're conflicting, as I've tested the DMVPN alongside a user connecting to the L2tp VPN and the spokes kept working fine. They just seem to go down at various points overnight, or after a day or two...

The configs are attached. As I said the Hub has L2tp dial-in on it along with the DMVPN. The Spokes are each identicle except for IP addresses, so only one sample is attached.

The Hub is shown with ip xx.xx.xx.66

SpokeA is shown with ip xxx.xxx.xxx.50

1 Reply 1

Shawn Lebbon
Level 1
Level 1

It turns out that the problem was Cisco Bug:

CSCef77648 "Excessive replication of locally generated multicast on mGRE"

Bug Toolkit

http://www.cisco.com/cgi-bin/Support/Bugtool/home.pl

Release Notes

Symptom: Over time there is increasing CPU utilization, dropped packets and instability in the routing protocol on the DMVPN network. This may also effect the physical network.

Condition: Running IOS Version 12.3(9.11) or 12.3(9.11)T or later on a router that is the hub for a DMVPN network.

Cause: Overtime the NHRP list of tunnel destinations for multicast packets

increases. Each spoke (tunnel destination) is listed in this list

multiple times and the number of entries per spoke increase over

time. You can test this by doing the following:

If you are running (EIGRP, OSPF or RIP) over the DMVPN

then ping [224.0.0.10 (for EIGRP)] or [224.0.0.5 (for OSPF)]

or [224.0.0.9 (for RIP)].

If you get back more then 1 ping reply per spoke router then you

have this problem.

Workaround: Configure static neighbors, which use unicast, and

passive-interface on the mGRE Tunnel under the routing

protocol configuration and remove the command

'ip nhrp map multicast dynamic' from the tunnel

configureation.

Downgrade to 12.3(9.10) or 12.3(9.10)T or earlier.

In general we recommend 12.3(9)b and 12.3(8)T4