EIGRP flapping on vlan between cores

[Chad Parish]

Question, during business hours, I am seeing a continuous flapping of one of my vlan's between my two core 6500' (Prod1 and Prod2) and the same vlan flapping between Prod 2 and the upstream ASA. 

Does this look like a Layer 2 utilization issue, in other words, the buffers gets flooded and EIGRP hellos get ignored till the buffers reset?

Does this look like a Layer 3 EIGRP config issue?  Is it because it is advertised via RIP which has that network set passive?  We have route maps that adjust the AS for EIGRP so that certain traffic that needs to use RIP prefer that protocol.A show route reveals that they know they are directly connected and that it is being redistributed into EIGRP and RIP (the EIGRP AD is set 210/210, so it is actually RIP advertising the directly connected route and in RIP it is set passive)

I ran a wireshark and it looks like Prod2 sends out the EIGRP hello packets and Prod1 seems to receive it, but Prod2 never gets back the SYN/ACK

Excerpt of Show log from Prod2 (Subnet has been changed from actual one)

Dec 20 09:08:32.001 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.12 (Vlan200) is down: holding time expired
Dec 20 09:08:35.793 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.12 (Vlan200) is up: new adjacency
Dec 20 09:13:18.031 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.110 (Vlan200) is down: holding time expired
Dec 20 09:13:21.931 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.110 (Vlan200) is up: new adjacency
Dec 20 09:17:08.456 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.110 (Vlan200) is down: holding time expired
Dec 20 09:17:12.948 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.110 (Vlan200) is up: new adjacency
Dec 20 09:22:16.210 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.20 (Vlan200) is down: holding time expired
Dec 20 09:22:19.962 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.220.20 (Vlan200) is up: new adjacency

Excerpt of Show log from Prod1 (Subnet has been changed from actual one)

Dec 20 08:42:25.650 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is down: Peer Termination received
Dec 20 08:42:28.678 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is up: new adjacency
Dec 20 08:43:16.815 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is down: Peer Termination received
Dec 20 08:43:20.211 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is up: new adjacency
Dec 20 08:44:12.911 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is down: Peer Termination received
Dec 20 08:44:15.887 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is up: new adjacency
Dec 20 08:46:05.836 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is down: Peer Termination received
Dec 20 08:46:09.460 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 150: Neighbor 10.1.200.3 (Vlan200) is up: new adjacency

Prod2#sho int vlan 200
Vlan101 is up, line protocol is up
Hardware is EtherSVI, address is 0017.dff8.1800 (bia 0017.dff8.1800)
Internet address is 10.1.200.3/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 15/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 12w5d
Input queue: 52/75/133392337/1811900 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 59307000 bits/sec, 10782 packets/sec
5 minute output rate 4178000 bits/sec, 3041 packets/sec
L2 Switched: ucast: 52369754098 pkt, 40757139145981 bytes - mcast: 25752905 pkt, 3296117986 bytes
L3 in Switched: ucast: 45468423285 pkt, 28778594685045 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 16871292972 pkt, 5264412494305 bytes mcast: 0 pkt, 0 bytes
45857248145 packets input, 29029538161575 bytes, 0 no buffer
Received 26450963 broadcasts (96465 IP multicasts)
0 runts, 0 giants, 1327403 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
17375955382 packets output, 5364471459178 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out

Prod1#sho int vlan200

Vlan101 is up, line protocol is up
Hardware is EtherSVI, address is 0017.dff4.9000 (bia 0017.dff4.9000)
Internet address is 10.1.200.12/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 8/255, rxload 15/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 12w5d
Input queue: 0/75/25/25 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 59434000 bits/sec, 8467 packets/sec
5 minute output rate 33890000 bits/sec, 8316 packets/sec
L2 Switched: ucast: 10124742 pkt, 713250832 bytes - mcast: 25433021 pkt, 3287151708 bytes
L3 in Switched: ucast: 21258971325 pkt, 15086644842473 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 39135987368 pkt, 27750226183444 bytes mcast: 154 pkt, 17146 bytes
21297925150 packets input, 15090928329414 bytes, 0 no buffer
Received 26265968 broadcasts (85423 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
39169349929 packets output, 27757273190878 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out

Prod2#sho ip route 10.1.200.12
Routing entry for 10.1.200.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via eigrp 150, rip
Advertised by rip
Routing Descriptor Blocks:
* directly connected, via Vlan200
Route metric is 0, traffic share count is 1

Prod1#sho ip route 10.1.200.3
Routing entry for 10.1.200.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via eigrp 150, rip
Advertised by rip
Routing Descriptor Blocks:
* directly connected, via Vlan200
Route metric is 0, traffic share count is 1

Prod2#sho run | b router eigrp

 network 10.1.200.0 0.0.0.255

router rip

version 2

passive-interface Vlan200

[Paul Chapman]

Hi -

You seem to have a high number of "throttles" on the Prod2 switch.  I would look at CPU utilization on Prod2 throughout the day.  Based on the multiple protocols in use it's possible that you have a routing loop.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/12027-53.html

PSC

[chrihussey]

Why does the output of the "sh int vlan 200" command show it as vlan 101?

Prod1#sho int vlan200

Vlan101 is up, line protocol is up

[Richard Burts · Accepted solution]

You describe multiple things about your network. But you provide output only about the relationship between prod 1 and prod 2, which appears to be on vlan 200. So that is what I can comment about.

First I am very confused about this

Prod2#sho int vlan 200
Vlan101 is up, line protocol is up

The command asks for vlan 200 and the output is for vlan 101. How did this happen? It suggests that you have edited and cut and pasted output and a way that makes me wonder if I can trust it since it seems you have introduced inconsistency in what is reported.

Based on the output that you posted here is what I believe is happening.

- Prod 1 sends EIGRP hello to Prod 2 and Prod 2 initializes EIGRP neighbor relationship

- Prod 2 sends EIGRP packets to Prod 1 but does not receive any response.

- Prod 2 retries the EIGRP packets to Prod 1 and still receives no response.

- Based on timeout of the EIGRP hello packets Prod 2 terminates the EIGRP neighbor relationship

- This cycle repeats over and over

Part of the output supports this interpretation. Here is the output from Prod 1

L3 in Switched: ucast: 21258971325 pkt, 15086644842473 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 39135987368 pkt, 27750226183444 bytes mcast: 154 pkt, 17146 bytes

Note that there are multicast packets sent but none received

Here is the output from Prod 2

L3 in Switched: ucast: 45468423285 pkt, 28778594685045 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 16871292972 pkt, 5264412494305 bytes mcast: 0 pkt, 0 bytes

I find this puzzling. The fact that Prod 2 establishes a new neighbor adjacency indicates that it received an EIGRP hello. But this output indicates no multicast was received. Since the interface output is inconsistent with the observed behavior I can only assume that something is not right in the interface output.

We do not have enough information to determine why there appears to be failure of the EIGRP traffic.

HTH

Rick

[Chad Parish]

Thank you for the feedback and sorry, I thought I had changed the IP for 101 to 200, my apparently vain attempt to not reveal the network IP scheme.  I missed that one.