High RTO with low SSRT values on EIGRP

[spfister336]

We are running several remote sites connecting back to a central site using EIGRP as the routing protocol over AT&T's ASE on demand switched ethernet service. It's been mostly fine, but occasionally for 25 or 30 minutes we get a lot of flapping that seems to go away on its own. During this time, 'show ip eigrp neigh' shows lines like:

31 10.210.77.2 Vl277 13 00:00:40 1 5000 1 806
26 10.210.43.2 Vl243 13 00:00:40 1 100 0 124
20 10.210.15.2 Vl215 11 00:00:40 1 100 0 413
9 10.210.81.2 Vl281 13 00:00:40 2 1702 0 100
4 10.210.17.2 Vl217 14 00:00:40 1 5000 1 1072
29 10.210.66.2 Vl266 13 00:00:46 1 5000 1 462
25 10.210.71.2 Vl271 13 00:00:46 1 5000 1 2129
24 10.210.22.2 Vl322 14 00:00:46 17 771 1 1493

The SRTT values seem relatively OK, but the RTOs go into the thousands. Is it possible that this is packet loss, or multicast rate limiting on AT&T's part?

[spfister336]

No, not all sites have had static neighbor configured yet. So far, only the ones where the third octet is:

15, 17, 22, 23, 36, 37, 43, 44, 47, 74, 75, 76, 77

That's about half the remote sites

[MHM Cisco World]

so this static router ?

Stub Peer Advertising (CONNECTED SUMMARY ) Routes
Suppressing queries
19 10.210.77.2 Vl277 12 01:42:25 1 225 0 1077
Version 20.0/2.0, Retrans: 33, Retries: 0, Prefixes: 4
Topology-ids from peer - 0
Topologies advertised to peer: base

[MHM Cisco World]

31 10.210.77.2 Vl277 13 00:00:40 1 5000 1 806<<- we start here Qcnt have one message and RTO around 5000
19 10.210.77.2 Vl277 12 01:42:25 1 225 0 1077 <<- here there is no message in Qcnnt and RTO normal around 255 
the Retrans:33 but I think this value when the peer flapping 
still one point why the eigrp not show that this router config as static neigbor. <<- I will check this point 

please can you share the log of eigrp flapping appear after static 

[spfister336]

Yes, at least it's supposed to be. It's one of the one I add the neighbor statements to (central site and remote site)

[spfister336]

Here are the lines that pertain to that peer after the static neighbor statements were put into place:

Apr 13 13:05:21.269: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.210.77.2 (Vlan277) is down: Interface PEER-TERMINATION received
Apr 13 13:05:24.749: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.210.77.2 (Vlan277) is up: new adjacency
Apr 13 14:27:21.344: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.210.77.2 (Vlan277) is down: Interface PEER-TERMINATION received
Apr 13 14:27:25.888: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.210.77.2 (Vlan277) is up: new adjacency
Apr 13 18:14:55.252: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.210.77.2 (Vlan277) is down: Interface PEER-TERMINATION received
Apr 13 18:14:59.468: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.210.77.2 (Vlan277) is up: new adjacency

[MHM Cisco World]

can check the remote site 
show ip eigrp neighbor detail <<- are there any waiting message ??

[spfister336]

Stub Peer Advertising (CONNECTED SUMMARY ) Routes
Suppressing queries
19 10.210.77.2 Vl277 11 01:58:39 1 100 0 1081
Version 20.0/2.0, Retrans: 5, Retries: 0, Prefixes: 4
Topology-ids from peer - 0
Topologies advertised to peer: base

[MHM Cisco World]

show ip eigrp interface detail 

[spfister336]

99-c95-clan1#show ip eigrp interface detail
EIGRP-IPv4 Interfaces for AS(1)
Xmit Queue PeerQ Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable Un/Reliable SRTT Un/Reliable Flow Timer Routes
Vl88 0 0/0 0/0 0 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 68154/14590
Hello's sent/expedited: 9173080/17190
Un/reliable mcasts: 0/58978 Un/reliable ucasts: 193302/415804
Mcast exceptions: 66733 CR packets: 30920 ACKs suppressed: 559
Retransmissions sent: 213824 Out-of-sequence rcvd: 4426
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl89 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 80809/19138
Hello's sent/expedited: 9172298/21035
Un/reliable mcasts: 0/49939 Un/reliable ucasts: 135870/538517
Mcast exceptions: 94242 CR packets: 19212 ACKs suppressed: 1952
Retransmissions sent: 207131 Out-of-sequence rcvd: 7538
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl99 0 0/0 0/0 0 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 1389/2
Hello's sent/expedited: 9174024/3
Un/reliable mcasts: 0/941 Un/reliable ucasts: 5/5
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl428 0 0/0 0/0 0 0/0 0 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 0/0
Hello's sent/expedited: 9174260/1
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Twe1/0/5 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 54218/1150
Hello's sent/expedited: 9160251/4
Un/reliable mcasts: 0/52623 Un/reliable ucasts: 49017/8932
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 3 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl92 0 0/0 0/0 0 0/0 840 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 38452/105
Hello's sent/expedited: 9030940/878
Un/reliable mcasts: 0/38552 Un/reliable ucasts: 7135/24698
Mcast exceptions: 2 CR packets: 3 ACKs suppressed: 7
Retransmissions sent: 18958 Out-of-sequence rcvd: 187
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl93 0 0/0 0/0 0 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 37499/82
Hello's sent/expedited: 9031066/870
Un/reliable mcasts: 0/37867 Un/reliable ucasts: 7066/24559
Mcast exceptions: 2 CR packets: 2 ACKs suppressed: 2
Retransmissions sent: 18873 Out-of-sequence rcvd: 234
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl94 0 0/0 0/0 0 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 5441/2
Hello's sent/expedited: 9040238/2
Un/reliable mcasts: 0/3031 Un/reliable ucasts: 2443/1484
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 7 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl200 0 0/0 0/0 0 0/0 0 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 0/0
Hello's sent/expedited: 8521654/1
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl438 0 0/0 0/0 0 0/0 0 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 0/0
Hello's sent/expedited: 4875732/0
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl248 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3618/0
Hello's sent/expedited: 916040/84
Un/reliable mcasts: 0/3186 Un/reliable ucasts: 104/1842
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1766 Out-of-sequence rcvd: 4
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl24 0 0/0 0/0 0 0/0 0 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 0/0
Hello's sent/expedited: 805037/1
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl212 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2532/0
Hello's sent/expedited: 153969/73
Un/reliable mcasts: 0/3395 Un/reliable ucasts: 1165/2208
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2136 Out-of-sequence rcvd: 13
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl213 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3111/2
Hello's sent/expedited: 154110/82
Un/reliable mcasts: 0/2793 Un/reliable ucasts: 129/1808
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1734 Out-of-sequence rcvd: 28
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl214 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3169/0
Hello's sent/expedited: 154100/76
Un/reliable mcasts: 0/2777 Un/reliable ucasts: 123/1723
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1656 Out-of-sequence rcvd: 19
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl215 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3145/1
Hello's sent/expedited: 154076/76
Un/reliable mcasts: 0/2749 Un/reliable ucasts: 123/1733
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1673 Out-of-sequence rcvd: 29
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl217 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3110/1
Hello's sent/expedited: 154055/82
Un/reliable mcasts: 0/2798 Un/reliable ucasts: 129/1754
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1684 Out-of-sequence rcvd: 28
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl221 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3208/4
Hello's sent/expedited: 151171/95
Un/reliable mcasts: 0/2857 Un/reliable ucasts: 173/1857
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1774 Out-of-sequence rcvd: 39
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl223 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3140/1
Hello's sent/expedited: 151162/89
Un/reliable mcasts: 0/2917 Un/reliable ucasts: 115/1867
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1784 Out-of-sequence rcvd: 3
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl225 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3179/0
Hello's sent/expedited: 151079/77
Un/reliable mcasts: 0/2773 Un/reliable ucasts: 116/1797
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1728 Out-of-sequence rcvd: 12
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl227 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3173/1
Hello's sent/expedited: 151139/71
Un/reliable mcasts: 0/2672 Un/reliable ucasts: 114/1767
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1708 Out-of-sequence rcvd: 23
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl231 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2403/0
Hello's sent/expedited: 149332/63
Un/reliable mcasts: 0/3144 Un/reliable ucasts: 1126/2151
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2088 Out-of-sequence rcvd: 17
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl236 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3135/0
Hello's sent/expedited: 149479/77
Un/reliable mcasts: 0/2781 Un/reliable ucasts: 125/1783
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1715 Out-of-sequence rcvd: 11
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl237 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2998/0
Hello's sent/expedited: 149499/87
Un/reliable mcasts: 0/2746 Un/reliable ucasts: 135/1812
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1737 Out-of-sequence rcvd: 24
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl241 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2980/0
Hello's sent/expedited: 149516/88
Un/reliable mcasts: 0/2736 Un/reliable ucasts: 138/1776
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1696 Out-of-sequence rcvd: 29
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl243 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3012/1
Hello's sent/expedited: 149517/87
Un/reliable mcasts: 0/2764 Un/reliable ucasts: 141/1777
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1698 Out-of-sequence rcvd: 26
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl244 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2326/0
Hello's sent/expedited: 135671/76
Un/reliable mcasts: 0/3204 Un/reliable ucasts: 1114/2137
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2062 Out-of-sequence rcvd: 15
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl245 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3058/1
Hello's sent/expedited: 135813/79
Un/reliable mcasts: 0/2737 Un/reliable ucasts: 120/1808
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1733 Out-of-sequence rcvd: 18
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl246 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2294/0
Hello's sent/expedited: 135735/69
Un/reliable mcasts: 0/3103 Un/reliable ucasts: 1079/2111
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2045 Out-of-sequence rcvd: 20
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl247 1 0/0 0/0 2 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2344/0
Hello's sent/expedited: 135650/55
Un/reliable mcasts: 0/3054 Un/reliable ucasts: 1115/2095
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2041 Out-of-sequence rcvd: 22
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl255 1 0/0 0/0 2 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2972/1
Hello's sent/expedited: 135821/71
Un/reliable mcasts: 0/2591 Un/reliable ucasts: 116/1698
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1634 Out-of-sequence rcvd: 10
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl256 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2840/2
Hello's sent/expedited: 135060/81
Un/reliable mcasts: 0/2596 Un/reliable ucasts: 124/1757
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1685 Out-of-sequence rcvd: 32
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl261 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2856/0
Hello's sent/expedited: 135011/74
Un/reliable mcasts: 0/2507 Un/reliable ucasts: 113/1743
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1677 Out-of-sequence rcvd: 29
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl264 0 0/0 0/0 0 0/0 0 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 0/0
Hello's sent/expedited: 135755/1
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl266 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2906/0
Hello's sent/expedited: 135008/86
Un/reliable mcasts: 0/2685 Un/reliable ucasts: 145/1785
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1714 Out-of-sequence rcvd: 39
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl271 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 3029/1
Hello's sent/expedited: 135004/74
Un/reliable mcasts: 0/2666 Un/reliable ucasts: 118/1781
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1717 Out-of-sequence rcvd: 12
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl274 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2191/0
Hello's sent/expedited: 134138/72
Un/reliable mcasts: 0/2990 Un/reliable ucasts: 1077/2064
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1994 Out-of-sequence rcvd: 21
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl275 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2892/2
Hello's sent/expedited: 134243/76
Un/reliable mcasts: 0/2595 Un/reliable ucasts: 122/1767
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1700 Out-of-sequence rcvd: 31
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl276 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2286/0
Hello's sent/expedited: 134125/55
Un/reliable mcasts: 0/3022 Un/reliable ucasts: 1080/2054
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2002 Out-of-sequence rcvd: 12
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl277 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2836/1
Hello's sent/expedited: 134297/81
Un/reliable mcasts: 0/2611 Un/reliable ucasts: 127/1742
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1672 Out-of-sequence rcvd: 19
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl279 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2909/1
Hello's sent/expedited: 134260/66
Un/reliable mcasts: 0/2492 Un/reliable ucasts: 106/1696
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1643 Out-of-sequence rcvd: 25
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl281 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2854/0
Hello's sent/expedited: 132123/82
Un/reliable mcasts: 0/2585 Un/reliable ucasts: 129/1752
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1678 Out-of-sequence rcvd: 24
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl297 0 0/0 0/0 0 0/0 0 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 0/0
Hello's sent/expedited: 132815/1
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl280 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2928/0
Hello's sent/expedited: 131973/86
Un/reliable mcasts: 0/2650 Un/reliable ucasts: 106/1765
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 1685 Out-of-sequence rcvd: 9
Interface has all stub peers
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

Vl322 1 0/0 0/0 1 0/0 50 0
Hello-interval is 5, Hold-time is 15
Split-horizon is enabled
Next xmit serial <none>
Packetized sent/expedited: 2334/0
Hello's sent/expedited: 131426/65
Un/reliable mcasts: 0/3108 Un/reliable ucasts: 1137/2118
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 2053 Out-of-sequence rcvd: 14
Topology-ids on interface - 0
Authentication mode is not set
Topologies advertised on this interface: base
Topologies not advertised on this interface:

99-c95-clan1#

[Kanan Huseynli]

Hi,

does problem (flap) happen at the same time for all remotes? If yes, I think it is unicast underlay issue (transport issue) at some time by provider.

Enable ip sla between main switch (9500) and three (choose randomly, better with different type remote switch eigrp devices) remote switches and track them. Below can be template config (frequency 5 sec as in eigrp, but timeout and threshould 1000 so immediate reaction happens). Track should log when sla goes down.

ip sla 1
icmp-echo [remote_sw_ip] source-ip [local_ip]
threshold 1000
frequency 5
ip sla schedule 1 life forever start-time now

track 1 ip sla 1

If eigrp and track goes down at the same time and comes back at the same time, then it is most probably unicast issue. Because, you have too many retransmission and CR messages. This indicates, no ack received from remote sites.

[MHM Cisco World]

Yes but both side send hello/update and get ack. 

Other thing peer receive terminate eigrp

The peer receive terminate eigrp dont have issue' issue in router send it.

So last thing in my mind that router interface is flapping not the eigrp.

[Kanan Huseynli]

@MHM Cisco World 

it can be also due to mtu issue. A sends update to B, B can't receive because underlay MTU is lower, A declares B as lost neighbors (because B does not receive update, can not response with ack). A sends peer down to B (B receives this message and logs "Interface PEER-TERMINATION received",  because packet length is lower than underlay MTU).

@spfister336 ,

could you ask provider what is MTU in there network? You may also check with ping [from_9500] [remote-branch] size 1500 df

if no response is coming, play with size number and find what is MTU between 9500 and remote branches.

[MHM Cisco World]

It can be jumbo frame but that not explain up the peer then after 30 min down' mtu mismatch must detect immediately.

But why not let check mtu also' this L3SW and mtu can be more than defualt 1500 bytes 

[Kanan Huseynli]

We don't know details, normally for such cases (abnormal, something works and after some time issue happens, then again works) you need to sit, monitor even sometimes debug to understand what happens really.

[spfister336]

On the ping test... 1500 goes through, but anything larger fails

When the problem is really bad, EIGRP sessions do not stay up for more than 5 minutes and this happens to all sites. This normally lasts maybe 20-30 minutes and goes away on its own. On a normal day, most sessions stay up, but maybe one or two flap once. I've seen sessions stay up 5 or 6 weeks for some sites. Yesterday, the first full day after making half of the sites use static neighbor statements, one site and just one would flap every several minutes. This was happening mostly during the daytime when all remote sites were sending a lot of traffic. As I type this, the sites aren't up to their normal traffic levels for the day yet, and sessions have been up 17+ hours