Random EIGRP Peer Termination Received Msgs

[jeremy.irvine]

I am having trouble narrowing down what is causing these messages.  About every 4-5 weeks we're seeing our VPLS network go crazy for about 15-30 seconds.  They seem to be getting shorter as time goes on, but I would like to find the cause. 

Sep 27 2012 14:47:21.194 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.14 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.194 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.11 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.194 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.15 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.196 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.16 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.196 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.12 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.198 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.13 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.198 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.1 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:21.198 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.8 (GigabitEthernet0/2) is down: Peer Termination received

Sep 27 2012 14:47:25.452 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.11 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.456 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.1 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.456 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.8 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.456 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.16 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.458 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.12 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.458 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.13 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.460 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.14 (GigabitEthernet0/2) is up: new adjacency

Sep 27 2012 14:47:25.460 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.15 (GigabitEthernet0/2) is up: new adjacency

We've been in contact with our VPLS provider and they are not seeing any L2 errors on their end, and we aren't either.  No Interface Errors & no Interface Drops.  We've also had them move us to a different switch in their network as well as run a new cable for us.  This seemed to change the behavior from high latency during the outage to the EIGRP drops above, yet still no errors.  We are also running an MPLS network at this location as a backup link and redistributing this into EIGRP, but MPLS has no issues during this time.  The problem with this is the outage isn't long enough for the failover to kick in and it still ends up dropping all of our IP Phone calls & Contact Center agents as our primary VOIP servers are at this location using VPLS as primary.  We are running this type of setup at 2 other locations and they are not experiencing these issues.

I am unfamiliar where to start looking, I still think its possibly an issue with the Carrier's VPLS network but I have no proof.  It could also be a routing issue, but I can't understand why this isn't happening more often or at other locations.  Does anyone have any documentation on what could be causing 'Peer Termination received' messages?  It sounds like every other VPLS site is telling this facility they're going offline, yet those remote facilities can still talk to each other without any issues during the outage. 

[Raju Sekharan]

Peer Termination received - Indicates that the other neighbor initiates the reset of the neighbor relationship. The router getting the message is not the one resetting the neighbor

So we need to look at  neighbors logs to see why is it restting the EIGRP session with this neighbor. We need to start from there

What is the model of router you are using

[jeremy.irvine]

We're using 3925E's at the 3 core locations and a mix of 2911's and 2921's at the other locations, all running IOS 15.1(4)M2.

When I view the logs for that time period across the neighbors all I see are holding time expired messages:

Sep 27 2012 14:47:21.186 EDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.6 (GigabitEthernet0/2) is down: holding time expired

Sep 27 2012 14:47:25.443 EDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4 811: Neighbor x.x.x.6 (GigabitEthernet0/2) is up: new adjacency

I have also made sure EIGRP is part of our QoS policies, even though the circuit going to the location displaying the 'Peer Termination Received' messages is barely at 2-3% load.  The other locations maybe reach 50-60% load on their circuit.  CPU load across all routers reach maybe 10-20% at maximum.

[Raju Sekharan]

The other end has reeset the neighbor relation becasue it didn't receive eigrp hellp within the hold-time

This is a packet loss from headednd to neighbors. If you are not seeing any interface drops, then you need to work with VPLS provider.

[mbroberson1]

Hi Jeremy,

Just curious did you ever find out what was causing the PEER Terminations in your case. I am actually starting to work on a similar issue for a new 10MB ethernet WAN circuit. I have seen some posts where the issue was at layer 1 with the circuit and not clean enough.

Regards,

Brandon

[jeremy.irvine]

No we never actually narrowed down what was causing the issue, however the Carrier I was working with was installing a new switch for another customer at the same data center we were having our issue in.  They asked us if we would be interested in moving to that brand new switch along with the other customer since their other switches (including the one we were on) were near or at capacity.

That was about 6 weeks ago and we haven't had any other oddities since. (knock on wood)  I keep an eye on the logs but so far we haven't had any new occurrances since the move on the carrier side.  I am guessing if that move did indeed resolve our issue, that it is something inside the carrier equipment that is causing a delay in certain packets arriving or leaving their equipment.

[jpl861]

We are experiencing this EIGRP peer termination issue but it is because of SSH and we are still trying to nail down why.

[EU UC Support]

We had similar issue where 2 sites were connected via Layer2 and as backup via MPLS. We found that on 6509 interface QOS settings were causing this issue. By removing wrr-queue bandwidth and wrr-queue queue-limit resolved this issue.

I know someone mentioned that it's not the issue on the router that received Termination messages but on the peer router with whom it establishes adjacency with.

Thanks.

MK