EIGRP Issue

[majsa]

Background first:

I am at the tail end of deploying a dual cloud DMVPN similar to the following:

http://www.cisco.com/warp/public/105/dmvpn_05.gif

although my 2 hub routers are at disparate locations.

My main data center has an MLPPP connection with 8 T1's while my backup location has an MLPPP with 3 T1's.

The VPN works fine for the most part but I have started noticing that as I add more remote sites (at 89), the eigrp adjacencies do not stay up on ONE of the tunnel interfaces. Both tunnels (198:main/199:backup)never drop and are created exactly the same other than the tunnel keys and subnets, etc.

The adjacency is created, but then drops as the holding time expires.

019604: *Apr 28 06:43:57.707 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is down: holding time expired

019605: *Apr 28 06:44:21.603 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is up: new adjacency

019606: *Apr 28 06:45:07.287 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is down: holding time expired

019607: *Apr 28 06:45:11.507 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is up: new adjacency

019608: *Apr 28 06:45:30.639 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is down: holding time expired

019609: *Apr 28 06:45:31.023 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is up: new adjacency

019610: *Apr 28 06:45:52.567 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is down: holding time expired

019611: *Apr 28 06:45:54.227 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is up: new adjacency

I did notice on a hello packet debug on the hub router that is dropping, Interface Goodbyes received (odd that its spelled wrong "Inteface goodbye received")

Remote routers: 2801 12.3(8r)T8

Hub routers: 3825 12.4(3b)

[vmoopeung]

If no hellos are received within the hold time, which is 15 seconds by default on most links, the router informs the neighbor that the neighbor relationship has been torn down and logs a syslog message.

http://www.cisco.com/en/US/tech/tk870/tk451/tk374/technologies_tech_note09186a00800947a5.shtml

[boblutton]

I have a similar problem with Eigrp. It loses it's neighbors for unknown reason.

Until we find out what's wrong with Eigrp, I'm considering using backup static routes...but have not tried this yet.

For the affected Eigrp neighbor, I would use a "floating static" route, with a admin distance of 91, which takes over when Eigrp fails...

ip route x.x.x.x y.y.y.y z.z.z.z 91

Has anyone ever used "Floating static routes" and what were your results ?

Any input would be appreciated

[mlinsemier]

I am having the exact same problem that you are, with Cisco 871 and 877 routers deployed to my teleworkers. My config has them dual attached to two hubs (one at corporate and one at my collocation site). I had thought about increasing the EIGRP hello interval to see if this fixes the problem but I wanted to see if you had received a solution as of yet?

The problem seems to exist on both 12.4(4)T2 and 12.(6)T I think.

Matt

[sundar.palaniappan]

You guys seem to be hitting this bug. Does the description sound about right?

Headline EIGRP flap over GRE tunnel with high traffic load - packet loss

Product IOS

Feature OTHERS Components Duplicate of

Severity 2 Severity help Status Assigned Status help

First Found-in Version 12.4(2.2) All affected versions First Fixed-in Version Version help

Release Notes

Symptoms: EIGRP adjacencies that pass through a GRE tunnel may flap.

Conditions: This symptom is observed when there is a high traffic load

through the GRE tunnel. The EIGRP neighbor flaps about every 10 or 20

minutes.

Workaround: There is no workaround.

HTH,

Sundar

* Please rate all helpful posts.

[majsa]

I don't know that this is the issue I'm having. The tunnels that are dropping are the "backup" tunnels that have almost zero traffic.

Granted there are 2 tunnels and the other is used, the problem happens no matter if there is traffic passing through or not. In addition, the flap is much more often than 10-20 minutes.

[mlinsemier]

Any word on this? I am still running into this exact same problem (similar setup to yours). I get EIGRP flaps on my remote routers.

Matt

[majsa]

I've been so busy I haven't tried working on it much. I did try to bring up a separate EIGRP session and run it exclusively between the backup site and the remote router but ran into the same problem.

Im going to open a TAC case.

[Richard Burts]

Is it always the same neighbor that becomes unstable? (the output in the original post suggests that it is always the same neighbor) Is that neighbor perhaps the one most recently configured - or the last one mentioned in the config file? I am wondering if it reflects a scaling issue in DMVPN? Can DMVPN send 89 hellos and keep up with 89 neighbors?

What are you seeing at the remote router? Is it reporting its EIGRP neighbor down because of lost hellos?

I had a situation not quite the same but perhaps similar running EIGRP over VPN/GRE tunnels and was experiencing some instability in EIGRP neighbor relationsips. We changed the timers (lengthened them to 15 and 45) and it did seem to improve the stability.

If you do open a TAC case and learn anything please post back to the forum with your results.

HTH

Rick

[majsa]

The remote routers have 2 tunnels, the odd thing is only one of the tunnels has the problem. I would think it is a scaling problem if BOTH tunnels lost their adjacencies.

The remote just gives a holding time expired message:

059186: *May 12 07:33:43.829 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is down: holding time expired

059187: *May 12 07:33:45.017 DST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 10: Neighbor 10

.8.199.254 (Tunnel199) is up: new adjacency

I may change the timers but I find it odd that 1 tunnel has an issue while the other doesnt.

[Richard Burts]

I agree that it is odd that one tunnel has the problem and the other tunnel does not.

HTH

Rick

[majsa]

Problem Solved. Unfortunately its something that will probably not satisfy others with similar issues.

Long winded explanation:

Our 2 main hub locations are running through AT&T managed routers (how I let them talk me into that I don't know).

In the course of management, I requested Read-Only SNMP strings be placed on the routers so that I can monitor the interfaces. Immediately after setting up the node on my tool (Solarwinds Orion), I noticed around 10,000 discards every 5 minutes (polling interval).

I opened a ticket with AT&T and spent around 24 hours explaining that they need to check their Multi-Link bundle. After being told there was no issue with their equipment or circuits, I started sniffing on my end to see if I was leaking private IP's.

An AT&T manager called to tell me they were closing the ticket and that my tool must have a bug when he happend to notice that Weighted-Fair Queueing was on the router with FIFO on the backbone router.

Long Story short: AT&T changed both ends to FIFO and my adjacencies have been stable ever since.

Quite honestly Im suprised the GRE tunnels were not affected by the massive amounts of dropped packets.

[Richard Burts]

Nick

Thanks for posting back to the forum indicating the solution to the problem. It makes the forum more useful when we can read about a problem and then see what solved the problem.

I agree that many others may not experience quite the symptoms or problem that you did. But it is quite helpful to consider this as a problem that might happen in other networks.

As far as affecting the GRE tunnel, I am not clear whether you have configured GRE keepalives on the tunnel or not. Without the keepalives configured (and they are a fairly recent feature in IOS) the tunnel will come up and stay up/up so long as the router has a valid route to the tunnel endpoint. The traditional tunnel is not impacted by dropped packets.

HTH

Rick