High RTO with low SSRT values on EIGRP

[spfister336]

We are running several remote sites connecting back to a central site using EIGRP as the routing protocol over AT&T's ASE on demand switched ethernet service. It's been mostly fine, but occasionally for 25 or 30 minutes we get a lot of flapping that seems to go away on its own. During this time, 'show ip eigrp neigh' shows lines like:

31 10.210.77.2 Vl277 13 00:00:40 1 5000 1 806
26 10.210.43.2 Vl243 13 00:00:40 1 100 0 124
20 10.210.15.2 Vl215 11 00:00:40 1 100 0 413
9 10.210.81.2 Vl281 13 00:00:40 2 1702 0 100
4 10.210.17.2 Vl217 14 00:00:40 1 5000 1 1072
29 10.210.66.2 Vl266 13 00:00:46 1 5000 1 462
25 10.210.71.2 Vl271 13 00:00:46 1 5000 1 2129
24 10.210.22.2 Vl322 14 00:00:46 17 771 1 1493

The SRTT values seem relatively OK, but the RTOs go into the thousands. Is it possible that this is packet loss, or multicast rate limiting on AT&T's part?

[MHM Cisco World]

the time must match in both side, so Yes EIGRP can if timer is mismatch.

[spfister336]

Waiting on approval for the timers change. In the meantime, should I be setting the rest of the sites to static neighbors?

[spfister336]

The timers were changed yesterday afternoon. We ended up using 5 sites as test sites

[MHM Cisco World]

Hi friend, 
so from yesterday until now are the peer stable ?

[spfister336]

They are, but so have most of the other sites too. I think traffic levels may be a little lower than usual today. We might need a couple of days before we see results.

[spfister336]

At least one of the 5 sites that got modified EIGRP timers has flapped today

[MHM Cisco World]

from Yesterday to know what is the UP time?
can you share show ip eigrp neighbor 

[spfister336]

See below. Note: a power glitch at the central side brought the ATT switch offline briefly 2 days ago (Saturday afternoon)

EIGRP-IPv4 Neighbors for AS(1)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
31 10.210.74.2 Vl274 12 00:23:08 1 100 0 6267
18 10.210.71.2 Vl271 10 00:54:22 1 100 0 2384
10 10.210.21.2 Vl221 28 01:12:51 1 100 0 932
20 10.210.55.2 Vl255 14 01:36:54 1 100 0 1364
13 10.210.22.2 Vl322 13 01:38:30 1 100 0 3279
5 10.210.17.2 Vl217 10 02:22:29 1 100 0 1387
24 10.210.47.2 Vl247 10 03:32:59 1 100 0 56445
25 10.210.80.2 Vl280 12 05:18:24 1 100 0 393
28 10.210.76.2 Vl276 11 05:57:08 1 100 0 3090
1 10.210.25.2 Vl225 11 1d22h 1 100 0 7
30 10.210.23.2 Vl223 12 2d00h 1 100 0 45
29 10.210.66.2 Vl266 28 2d00h 2 100 0 720
27 10.210.15.2 Vl215 26 2d00h 1 100 0 54
26 10.210.14.2 Vl214 21 2d00h 1 100 0 1374
23 10.210.56.2 Vl256 14 2d00h 1 100 0 45894
22 10.210.36.2 Vl236 14 2d00h 2 100 0 21614
21 10.210.79.2 Vl279 14 2d00h 1 100 0 403
19 10.210.46.2 Vl246 11 2d00h 1 100 0 3468
17 10.210.77.2 Vl277 13 2d00h 1 100 0 1136
16 10.210.61.2 Vl261 12 2d00h 1 100 0 35321
15 10.210.31.2 Vl231 13 2d00h 1 100 0 2881
14 10.210.48.2 Vl248 15 2d00h 1 100 0 693
12 10.210.75.2 Vl275 11 2d00h 1 100 0 1475
11 10.210.12.2 Vl212 27 2d00h 1 100 0 34197
9 10.210.81.2 Vl281 10 2d00h 1 100 0 379
8 10.210.13.2 Vl213 12 2d00h 1 100 0 11736
7 10.210.45.2 Vl245 11 2d00h 1 100 0 3927
6 10.210.41.2 Vl241 14 2d00h 1 100 0 477
4 10.210.37.2 Vl237 14 2d00h 1 100 0 1406
3 10.210.27.2 Vl227 10 2d00h 1 100 0 1358
2 10.210.44.2 Vl244 11 2d00h 1 100 0 2080
0 10.210.43.2 Vl243 12 2d00h 1 100 0 382
32 172.18.0.2 Twe1/0/5 11 6w2d 1 100 0 6882
38 10.99.89.98 Vl89 13 1y18w 1 100 0 323755

[MHM Cisco World]

I run Lab to check issue here
NOW 
EIGRP have two types of message reliable unreliable 
UPDATE is reliable and 
Hello is unreliable 

the show ip eigrp neighbor detail show status of reliable message 
Hello message which is unreliable we can not see it status in show ip eigrp neighbor 

we can check by show ip eigrp traffic and see hello count is increase or not. 

you use VLAN not router-port to connect to neighbor, we try change the multi to unicast and same issue 
we also increase the hello and hold timer and same issue. 

so there is something in path make Hello message net receive correctly 

in Lab what m make NSK 1 know that this hello message for correct SVI ?
the destination is same (multicast in my lab), so the NSK need correct VLAN tag to receive hello for correct SVI. 

so please make double check the VLAN tag in your path also share 
show ip eigrp traffic when the flapping happened (do show command three times and share here)
Thanks 
MHM

 

[Yousif Ahmed]

spfister is out on Vacation.

I checked out the VLAN tags they are set correctly. 

I was able to turn on EIGRP Debugging on the neighboring 9300, but it has not flapped yet. Once it flaps it might tell us what the issue is. spfister also got a TAC in before leaving, we are going to do Packet Capture and stop it instantly when EIGRP Flaps using a script. I'm leaning towards ISP Issue, I think they have us put into a maxed out hub, causing oversubscription. 

[MHM Cisco World]

Happy vacation  @spfister336 

I check at&t on-demand service' abd check some video about add vlan to circuit' 

Assume you config correct bw whcih in end the bw of all vlan equal to bw you get from at&t? Just make check.

[Yousif Ahmed]

All remote sites have a bw of 1Gbps with the central site having 10Gbps.

It is all equal to the bw we get from AT&T. I even checked Solarwinds and the port usage does not even exceed 50% or 500 mbps or lower.

[Yousif Ahmed]

Seems like the sites are now flapping 1 at a time. We have had 3 sites flap out of 34. the latest one was 32 seconds ago, below is the output you requested:

#show ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(1)
Hellos sent/received: 102956946/337557357
Updates sent/received: 1694505/334606
Queries sent/received: 176516/97717
Replies sent/received: 55932/204179
Acks sent/received: 472987/2241081
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 591
PDM Process ID: 589
Socket Queue: 0/10000/1387/0 (current/max/highest/drops)
Input Queue: 0/10000/1387/0 (current/max/highest/drops)

#show ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(1)
Hellos sent/received: 102956975/337557368
Updates sent/received: 1694505/334606
Queries sent/received: 176516/97717
Replies sent/received: 55932/204179
Acks sent/received: 472987/2241081
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 591
PDM Process ID: 589
Socket Queue: 0/10000/1387/0 (current/max/highest/drops)
Input Queue: 0/10000/1387/0 (current/max/highest/drops)

#show ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(1)
Hellos sent/received: 102956983/337557371
Updates sent/received: 1694505/334606
Queries sent/received: 176516/97717
Replies sent/received: 55932/204179
Acks sent/received: 472987/2241081
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 591
PDM Process ID: 589
Socket Queue: 0/10000/1387/0 (current/max/highest/drops)
Input Queue: 0/10000/1387/0 (current/max/highest/drops)

[MHM Cisco World]

Just make time between two show ip eigrp traffic equal to 2xhello timer of eigrp

I need to see how much hello message send receive 

Thanks 

MHM

[Yousif Ahmed]

Check this out:

21 x.x.x.x Vlxxx 11 00:31:21 268 1608 0

RTO  is 1608 the rest are 150-225

Also, here is a delayed version of show ip eigrp traffic:

99-c95-clan1#show ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(1)
Hellos sent/received: 102989851/337581843
Updates sent/received: 1694824/334620
Queries sent/received: 176556/97719
Replies sent/received: 55934/204217
Acks sent/received: 473038/2241373
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 591
PDM Process ID: 589
Socket Queue: 0/10000/1387/0 (current/max/highest/drops)
Input Queue: 0/10000/1387/0 (current/max/highest/drops)

99-c95-clan1#show ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(1)
Hellos sent/received: 102989930/337581898
Updates sent/received: 1694824/334620
Queries sent/received: 176556/97719
Replies sent/received: 55934/204217
Acks sent/received: 473038/2241373
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 591
PDM Process ID: 589
Socket Queue: 0/10000/1387/0 (current/max/highest/drops)
Input Queue: 0/10000/1387/0 (current/max/highest/drops)

99-c95-clan1#show ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(1)
Hellos sent/received: 102990010/337581955
Updates sent/received: 1694824/334620
Queries sent/received: 176556/97719
Replies sent/received: 55934/204217
Acks sent/received: 473038/2241373
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 591
PDM Process ID: 589
Socket Queue: 0/10000/1387/0 (current/max/highest/drops)
Input Queue: 0/10000/1387/0 (current/max/highest/drops)