EIGRP flapping between directly connect neighbors

[cheeseburger]

Hello,

I'm actually having two issues with EIGRP flapping, one I attribute to internet.

The first one that is really driving me crazy.... Just recently, at three locations, we are having EIGRP flapping every so often (no specific intervals that I can see) between directly connected neighbors. Logging shows that one neighbor sends a goodbye because holding time expired. I find it very hard to believe there might be a bottle neck between a directly connected neighbor on two TenGig interfaces in a port channel. Upon review, we see that CPU spikes to around 99% during these times. My theory is something is causing a spike and the EIGRP packets come in but resources in the control plane are unable to process them. I have zero experience with troubleshooting CPU spikes. Is there anything someone can recommend?

One extra piece in here, I would think we would see CPU spikes of around 90 seconds but they only last for approximately 10 seconds according to the "show processes cpu history" command. If this truly is a CPU spike issue, why wouldn't later packets be processed after the CPU spike goes away?

The second issue is using DMVPN with internet. This is only happening at one location right now. The SPOKE router is in fact sending HELLO packets at the appropriate interval but the HUB router is not receiving them according to debug. There is no CPU spike on either of these devices and I attribute this to internet. Is there any other troubleshooting I can do for this? All the other SPOKES seem to be stable...

I only have 4 hairs left that I can pull out... please help me keep them!

[freshcalendars]

I have this problem too  

[cheeseburger]

So we've narrowed this down... 

The high CPU is most likely from constant EIGRP routing updates. I'm looking at a specific route across a tunnel. When I do show IP route 10.48.0.0 at one point the two neighbors point to each other for that specific route. That's no good.....

Here are some outputs....

This comes up first:

sh ip route 10.48.0.0
Routing entry for 10.48.0.0/19
Known via "bgp 64558", distance 20, metric 1601280
Tag 64570, type external
Redistributing via eigrp 37
Advertised by eigrp 37 metric 1000 1000 255 1 1500
Last update from 10.100.4.1 00:00:00 ago
Routing Descriptor Blocks:
* 10.100.4.1, from 10.100.4.1, 00:00:00 ago
Route metric is 1601280, traffic share count is 1
AS Hops 1
Route tag 64570
MPLS label: none

After 30 seconds of this route being up I get this....

sh ip route 10.48.0.0
Routing entry for 10.0.0.0/8
Known via "bgp 64558", distance 20, metric 10240
Tag 65000, type external
Redistributing via eigrp 37
Advertised by eigrp 37 metric 1000 1000 255 1 1500
Last update from 10.100.2.1 2w6d ago
Routing Descriptor Blocks:
* 10.100.2.1, from 10.100.2.1, 2w6d ago
Route metric is 10240, traffic share count is 1
AS Hops 1
Route tag 65000
MPLS label: none

That route lasts for about 1 second then this route disappears and I get this.... until the first route comes back and the cycle repeats.

sh ip route 10.48.0.0
Routing entry for 10.48.0.0/19
Known via "eigrp 37", distance 90, metric 3072, type internal
Redistributing via eigrp 37, bgp 64558
Advertised by bgp 64558
Routing Descriptor Blocks:
* directly connected, via Null0
Route metric is 3072, traffic share count is 1
Total delay is 20 microseconds, minimum bandwidth is 1000000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 0

I guess I don't really understand the Null0

More specific routes route properly, they go over the correct circuit and are having no issues. Perhaps there is an auto summary somewhere that shouldn't be?

[Julio E. Moisa]

Hi

Do you have a topology or a picture to understand your infraestructure, also is possible to know the configuration of each location? In some situations CPU issues are difficult to identify , High CPU could affect the control plane so the routing protocols could be affected. Now DMVPN flapings could be generated if the destination address is also known through the tunnel.

:-)

[cheeseburger]

Hey Julio,

To begin to address this issue, there was a routing loop at the point I originally posted. We have two DMVPN hubs and a spoke. The spoke has a BGP peering with each hub and an EIGRP peering with each hub. I believe the 10.48.0.0/19 route was circling the network and then upon returning back to the spoke, the spoke said, "oh hey, this is better than null0 I'll install this!" The router installed it and withdrew its route from the EIGRP and BGP neighbors then the withdrawn route circled the network and got withdrawn from the spoke so then it had to go back to its summarized route with with null0 as next hop and then the process started all over again. 

Wow, that was long winded.... This issue has been resolved. We removed the EIGRP neighborship and just left the eBGP between HUB and SPOKE.

We still seem to have an issue with EIGRP HELLOs not reaching a directly connected neighbor. I can provide you with a simple topology in the next day or two along with any config if you need it. I can show you a bit of what we're seeing right now as well that may or may not contribute to the problem. I've attached an output of "show ip eigrp events" for DS1. I've also attached "show logging" for DS1 which is showing HSRP flapping as well. I assume that if the router cannot get EIGRP HELLOs it cannot get HSRP HELLOs either. Finally, I've included CPU history graphs from "show processes cpu history". Hopefully you recognize something I do not see. If you do, please point me in the right direction so I can continue to attempt to solve this, it helps me learn when I finally have that "AH-HA!" moment.

Let me know how detailed and how far upstream you need me to go with a topology and I can try to throw something together in Visio.

I sincerely appreciate your thoughts again man! Thank you so much!