EIGRP neighbour ACK / UPDATE queued NOT sending

[Paul Morgan]

Hello all,

I have a Cisco L3 3650 switch with two L3 ports attached to two 2950 routers. The two switchports are configured with NO SWITCHPORT and an IP address command.

The two routers are configured with IP addresses accordingly to the subnets of the attached interfaces.

Both connections can ping both ways. The ARP tables and CEF tables both show the correct links for the remote end IPs.

All transmission occurs normally. Except for EIGRP packets. The HELLO packets are sent and received in both directions but the ACKs or UPDATEs are queued on the return path on one or the other connection. This means that the neighbourship isnt completed and is torn down after it times out. It is then re-established when the next HELLO is seen.

Ive run DEBUG EIGRP PACKETS on the switch and shown the following:

003064: *Jan 18 10:37:38.178: EIGRP: Sending HELLO on GigabitEthernet1/0/24
003065: *Jan 18 10:37:38.178:   AS 6001, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
003066: *Jan 18 10:37:38.370: EIGRP: Sending HELLO on GigabitEthernet1/0/22
003067: *Jan 18 10:37:38.370:   AS 6001, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
003068: *Jan 18 10:37:38.378: %DUAL-5-NBRCHANGE: EIGRP-IPv4 6001: Neighbor 192.168.179.50 (GigabitEthernet1/0/22) is down: retry limit exceeded
003069: *Jan 18 10:37:38.403: EIGRP: Received ACK on GigabitEthernet1/0/24 nbr 192.168.178.50
003070: *Jan 18 10:37:38.403:   AS 6001, Flags 0x0:(NULL), Seq 0/6866 interfaceQ 0/0
003071: *Jan 18 10:37:39.409: EIGRP: Received HELLO on GigabitEthernet1/0/22 nbr 192.168.179.50
003072: *Jan 18 10:37:39.409:   AS 6001, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0
003073: *Jan 18 10:37:39.409: %DUAL-5-NBRCHANGE: EIGRP-IPv4 6001: Neighbor 192.168.179.50 (GigabitEthernet1/0/22) is up: new adjacency
003074: *Jan 18 10:37:39.409: EIGRP: Enqueueing UPDATE on GigabitEthernet1/0/22 nbr 192.168.179.50 tid 0 iidbQ un/rely 0/1 peerQ un/rely 0/0
003075: *Jan 18 10:37:39.418: EIGRP: Sending HELLO on GigabitEthernet1/0/22
003076: *Jan 18 10:37:40.269: EIGRP: Sending UPDATE on GigabitEthernet1/0/22 nbr 192.168.179.50, retry 4, RTO 505 tid 0
003077: *Jan 18 10:37:40.269:   AS 6001, Flags 0x0:(NULL), Seq 6869/5163 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 1-4302
003078: *Jan 18 10:37:40.780: EIGRP: Sending UPDATE on GigabitEthernet1/0/22
003079: *Jan 18 10:37:41.479: EIGRP: Received HELLO on GigabitEthernet1/0/24 nbr 192.168.178.50
003080: *Jan 18 10:37:41.479:   AS 6001, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
003081: *Jan 18 10:37:41.540: EIGRP: Sending UPDATE on GigabitEthernet1/0/22 nbr 192.168.179.50, retry 6, RTO 1135 tid 0
003082: *Jan 18 10:37:42.679: EIGRP: Sending UPDATE on GigabitEthernet1/0/22 nbr 192.168.179.50, retry 7, RTO 1702 tid 0
003083: *Jan 18 10:37:42.679:   AS 6001, Flags 0x0:(NULL), Seq 6869/5163 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 1-4302
003084: *Jan 18 10:37:44.390: EIGRP: Sending UPDATE on GigabitEthernet1/0/22 nbr 192.168.179.50, retry 8, RTO 2553 tid 0
003085: *Jan 18 10:37:44.390:   AS 6001, Flags 0x0:(NULL), Seq 6869/5163 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 1-4302
003086: *Jan 18 10:37:44.433: EIGRP: Received HELLO on GigabitEthernet1/0/22
003087: *Jan 18 10:37:46.051: EIGRP: Received HELLO on GigabitEthernet1/0/24 nbr 192.168.178.50
003088: *Jan 18 10:37:46.051:   AS 6001, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
003089: *Jan 18 10:37:46.948: EIGRP: Sending UPDATE on GigabitEthernet1/0/22 nbr 192.168.179.50, retry 9, RTO 3829 tid 0
003090: *Jan 18 10:37:46.948:   AS 6001, Flags 0x0:(NULL), Seq 6869/5163 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 1-4302

So the successful peer comes up on Gi1/0/24 with nbr 178.50

But the peer on Gi1/0/22 with nbr 179.50 doesnt. It just queues the interface.

Ive tried packet capture. This just verifies the packets shown in the debug.

So...

Does anyone know why the interface doesnt send the packets?

[Vasilii Mikhailovskii]

Hello.

Do you see EIGRP neighbours on both peers (over G1/0/22)? Do you seen non-zero Q count on both peers? Please provide "show ip eigrp ne det" from both of them.

If you see EIGRP nei on locl side only - then it's one way multicast connection.

If you see EIGRP nei on both peers, and both have non-zero Q-count - it's unicast issue (like ACL, PBR, rate-limiter and etc.).

[Paul Morgan]

The Q count on Gi1/0/22 is always 1 on the switch

SW1#sh ip eig n
EIGRP-IPv4 Neighbors for AS(6001)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                                            (sec)        (ms)       Cnt Num
0   192.168.179.50          Gi1/0/22                 14 00:00:54    1  5000  1  7144
1   192.168.178.50          Gi1/0/24                 11 1d03h        1    100  0  534923

All the other counts are 0

SW1#sh ip eig n det
EIGRP-IPv4 Neighbors for AS(6001)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   192.168.179.50          Gi1/0/22                 10 00:00:16    1  5000  1  7162
   Version 14.0/2.0, Retrans: 4, Retries: 3, Prefixes: 38
   Topology-ids from peer - 0
    UPDATE seq 10852 ser 1-7742 Sent 14130 Sequenced
1   192.168.178.50          Gi1/0/24                 12 1d03h       1   100  0  535526
   Version 5.0/3.0, Retrans: 1, Retries: 0, Prefixes: 37
   Topology-ids from peer - 0

[Vasilii Mikhailovskii]

Hello, Paul.

Please provide the same output from 192.168.179.50.

[Paul Morgan]

CORERTR1#sh ip eig n de
EIGRP-IPv4 Neighbors for AS(6001)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
1   192.168.179.53          Gi0/0                    12 00:00:33    8   100  0  14940
   Version 7.0/3.0, Retrans: 1, Retries: 0
   Topology-ids from peer - 0

*This is the 179.50 router*

For completeness, the interface configs are here but are not difficult.

interface GigabitEthernet0/0
 ip address 192.168.179.50 255.255.255.0
 ip access-group 100 in
 ip nbar protocol-discovery
 ip nat inside

CORERTR1#sh access-l 100
Extended IP access list 100
    10 deny ip host 192.168.1.21 host 88.**.**.**
    20 deny ip host 192.168.1.21 host 88.**.**.**
    30 permit tcp host 192.168.177.41 eq smtp any
    40 permit tcp host 192.168.177.41 any eq smtp
    50 deny tcp any eq smtp any
    60 deny tcp any any eq smtp
    70 permit ip any any

SW1#sh run int g1/0/22
Building configuration...

Current configuration : 126 bytes
!
interface GigabitEthernet1/0/22
 description ConnectToCoreRTR1
 no switchport
 ip address 192.168.179.53 255.255.255.0

[Vasilii Mikhailovskii]

Hello.

Looks like one way unicast issue.

Please check if you have any L2 ACLs in between or MTU issue (SW1's MTU is lower than RTR1's, or just some other issue on the path).

[Paul Morgan]

No ACLs but I thought MTU wasn't used in EIGRP neighbourships?

The core switches use MTU 9000 for large file transfers.

----------------------------------------------------------------

Looking at the interface counters on 179.50, you could be onto something with that.

5 minute input rate 3000 bits/sec, 2 packets/sec
  5 minute output rate 2000 bits/sec, 2 packets/sec
     693608 packets input, 404547663 bytes, 0 no buffer
     Received 35551 broadcasts (33123 IP multicasts)
     0 runts, 42637 giants, 0 throttles
     42637 input errors, 0 CRC, 42637 frame, 0 overrun, 0 ignored
     0 watchdog, 35538 multicast, 0 pause input
 

[Vasilii Mikhailovskii · Accepted solution]

Please make sure ip MTU is equal on both L3 interfaces (and underlying path MTU is consistent).

MTU might be an issue if one side is sending packet of 2000 bytes, while other side can't receive it.

[Paul Morgan]

Can I change the port MTU without turning off jumbo frame support on the switch?

The IP MTU command is accepted but doesn't change it.

#Sh ver

Switch Ports Model              SW Version        SW Image             
------ ----- -----              ----------        ----------            ----
*    1 28    WS-C3650-24TD      03.03.01SE        cat3k_caa-universalk9

int g1/0/22

  ip mtu 1500

SW1#sh int g1/0/22
GigabitEthernet1/0/22 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 24e9.**.** (bia 24e9.**.**)
  Description: ConnectToCoreRTR1
  Internet address is 192.168.179.53/24
  MTU 9000 bytes,

[Vasilii Mikhailovskii]

can you ping over the link with size 1500 and df-bit set?

Please provide "show ip int G..." from both sides.

[Paul Morgan]

So,

Can I change the port MTU without turning off jumbo frame support on the switch?

The IP MTU command is accepted but doesn't change it...

That PING was a brilliant idea!

SW1#ping 192.168.179.50  df-bit size 2000
Type escape sequence to abort.
Sending 5, 2000-byte ICMP Echos to 192.168.179.50, timeout is 2 seconds:
Packet sent with the DF bit set
.....
Success rate is 0 percent (0/5)
SW1#ping 192.168.179.50  df-bit size 1500
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.179.50, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/10 ms

SW1#sh ip int g1/0/22
GigabitEthernet1/0/22 is up, line protocol is up
  Internet address is 192.168.179.53/24
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 9000 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Multicast reserved groups joined: 224.0.0.10 224.0.0.251

CORERTR1#sh ip int g0/0
GigabitEthernet0/0 is up, line protocol is up
  Internet address is 192.168.179.50/24
  Broadcast address is 255.255.255.255
  Address determined by non-volatile memory
  MTU is 1500 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Multicast reserved groups joined: 224.0.0.251 224.0.0.10

[Vasilii Mikhailovskii]

Hello.

If "ip mtu " did not help, please provide "show runn int g1/0/22", "show ip int g1/0/22" and show ver.

[Paul Morgan]

The MTU was the problem - Ive replied to the answer higher up the thread. Thanks again. Help was so valuable

[Cisco Freak]

Paul,

Can you please give the interface level configuration and EIGRP configuration from both the ends?

CF

[Paul Morgan]

Hold the phone guys and dolls!

That is it. Ive added IP MTU 1500 to both routed interfaces on the switch. Whilst the MTU size is still showing as 9000 (which is global jumbo frame support), the transfer packets are now being received as standard frames.

And now,

SW1#sh ip eig n
EIGRP-IPv4 Neighbors for AS(6001)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                                           (sec)        (ms)        Cnt Num
0   192.168.179.50          Gi1/0/22                13 00:26:34    1     100  0     10987
1   192.168.178.50          Gi1/0/24                12 1d18h        1     100  0     588191

Many many thanks Vasilii