EIGRP equal load balancing problem stuck with single link with Drop !

[Dr.X]

Hello Team ,

We have been running eigrp equal load balancing instead of EtherChannel between :

Cisco ASR ---> Cisco 3560.

we are balancing 4G traffic on six links of 1G speed.

But recently, with no recent change , we began to see that one of the Eigrp links is 1G while others are 400Mbps. which causes network drop all the time ...

From Eigrp, all routes have the same AD/Metric and equal loads of six links on the routing table.

Tried to clear BGP nei, Routing tables, and flush everything; one link always stays 1G while the others are 500 Mbps.

If i drop one link from the group, another link is filled with 1G and the same issue.

I added the 7th link to the eigrp balance, and the issue was solved all links get equal balance now.

Based on the search, i don't think the issue has to occur as Eigrp is much better than Etherchannel L3 solution.

My eigrp is simple :

router eigrp 4
maximum-paths 10
variance 128

Here is sample of the routing table :

D EX 10.14.2.152/32
[170/2560000768] via 172.26.40.2, 00:07:35, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 00:07:35, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 00:07:35, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 00:07:35, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 00:07:35, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 00:07:35, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 00:07:35, GigabitEthernet0/0/1

here is my router info :

Gateway-ASR1002#sh version
Cisco IOS XE Software, Version 03.13.00.S - Extended Support Release
Cisco IOS Software, ASR1000 Software (PPC_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.4(3)S, RELEASE SOFTWARE (fc11)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 28-Jul-14 04:11 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

Gateway-ASR1002 uptime is 24 weeks, 4 days, 17 hours, 39 minutes
Uptime for this control processor is 24 weeks, 4 days, 17 hours, 43 minutes
System returned to ROM by reload at 18:39:44 UTC Sun Aug 6 2017
System image file is "bootflash:asr1000rp1-adventerprisek9.03.13.00.S.154-3.S-ext.bin"
Last reload reason: PowerOn

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco ASR1002 (2RU) processor (revision 2RU) with 1650497K/6147K bytes of memory.
Processor board ID FOX1807GBZW
12 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
7757823K bytes of eUSB flash at bootflash:.

Configuration register is 0x2102

Any one can help if this could be a hardware issue or a bug?

Thanks 

[Joseph W. Doherty]

"If there is a question about whether EIGRP or Etherchannel would be preferred I would prefer EIGRP."

Actually, my question is, doesn't EIGRP also have a (optional) dynamic load routing capability?  If so, have you ever used it and/or might you recommend trying it to improve the OP's load sharing balance?

[MHM Cisco World]

RouterB#show ip cef  x.x.x.x internal

please share this,  

[Dr.X]

Gateway-ASR1002#sh ip cef 10.12.5.0/24 internal
10.12.5.0/24, epoch 2, RIB[I], refcnt 6, per-destination sharing
sources: RIB
feature space:
IPRM: 0x00028000
Broker: linked, distributed at 4th priority
ifnums:
GigabitEthernet0/0/1(9): 172.20.40.2
GigabitEthernet0/0/2(10): 172.21.40.2
GigabitEthernet0/1/1(13): 172.22.40.2
GigabitEthernet0/1/2(14): 172.23.40.2
GigabitEthernet0/1/3(15): 172.26.40.2
GigabitEthernet0/1/5(17): 172.24.40.2
GigabitEthernet0/1/6(18): 172.25.40.2
path list 44444C00, 5863 locks, per-destination, flags 0x4D [shble, hvsh, rif, hwcn]
path 449D8F54, share 1/1, type attached nexthop, for IPv4
nexthop 172.20.40.2 GigabitEthernet0/0/1, IP adj out of GigabitEthernet0/0/1, addr 172.20.40.2 38C581C0
path 449D8E84, share 1/1, type attached nexthop, for IPv4
nexthop 172.21.40.2 GigabitEthernet0/0/2, IP adj out of GigabitEthernet0/0/2, addr 172.21.40.2 38C58EC0
path 449D8ADC, share 1/1, type attached nexthop, for IPv4
nexthop 172.22.40.2 GigabitEthernet0/1/1, IP adj out of GigabitEthernet0/1/1, addr 172.22.40.2 38C59BC0
path 449D9364, share 0/1, type attached nexthop, for IPv4
nexthop 172.23.40.2 GigabitEthernet0/1/2, IP adj out of GigabitEthernet0/1/2, addr 172.23.40.2 38C58D20
path 449D775C, share 1/1, type attached nexthop, for IPv4
nexthop 172.24.40.2 GigabitEthernet0/1/5, IP adj out of GigabitEthernet0/1/5, addr 172.24.40.2 38C589E0
path 449D85FC, share 1/1, type attached nexthop, for IPv4
nexthop 172.25.40.2 GigabitEthernet0/1/6, IP adj out of GigabitEthernet0/1/6, addr 172.25.40.2 38C58500
path 449DA064, share 1/1, type attached nexthop, for IPv4
nexthop 172.26.40.2 GigabitEthernet0/1/3, IP adj out of GigabitEthernet0/1/3, addr 172.26.40.2 38C596E0
output chain:
loadinfo 39516D0C, per-session, 7 choices, flags 0003, 2935 locks
flags [Per-session, for-rx-IPv4]
14 hash buckets
< 0 > IP adj out of GigabitEthernet0/0/1, addr 172.20.40.2 38C581C0
< 1 > IP adj out of GigabitEthernet0/0/2, addr 172.21.40.2 38C58EC0
< 2 > IP adj out of GigabitEthernet0/1/1, addr 172.22.40.2 38C59BC0
< 3 > IP adj out of GigabitEthernet0/1/2, addr 172.23.40.2 38C58D20
< 4 > IP adj out of GigabitEthernet0/1/5, addr 172.24.40.2 38C589E0
< 5 > IP adj out of GigabitEthernet0/1/6, addr 172.25.40.2 38C58500
< 6 > IP adj out of GigabitEthernet0/1/3, addr 172.26.40.2 38C596E0
< 7 > IP adj out of GigabitEthernet0/0/1, addr 172.20.40.2 38C581C0
< 8 > IP adj out of GigabitEthernet0/0/2, addr 172.21.40.2 38C58EC0
< 9 > IP adj out of GigabitEthernet0/1/1, addr 172.22.40.2 38C59BC0
<10 > IP adj out of GigabitEthernet0/1/2, addr 172.23.40.2 38C58D20
<11 > IP adj out of GigabitEthernet0/1/5, addr 172.24.40.2 38C589E0
<12 > IP adj out of GigabitEthernet0/1/6, addr 172.25.40.2 38C58500
<13 > IP adj out of GigabitEthernet0/1/3, addr 172.26.40.2 38C596E0
Subblocks:
None

[MHM Cisco World]

< 0 > IP adj out of GigabitEthernet0/0/1, addr 172.20.40.2 38C581C0
< 1 > IP adj out of GigabitEthernet0/0/2, addr 172.21.40.2 38C58EC0
< 2 > IP adj out of GigabitEthernet0/1/1, addr 172.22.40.2 38C59BC0
< 3 > IP adj out of GigabitEthernet0/1/2, addr 172.23.40.2 38C58D20
< 4 > IP adj out of GigabitEthernet0/1/5, addr 172.24.40.2 38C589E0
< 5 > IP adj out of GigabitEthernet0/1/6, addr 172.25.40.2 38C58500
< 6 > IP adj out of GigabitEthernet0/1/3, addr 172.26.40.2 38C596E0
< 7 > IP adj out of GigabitEthernet0/0/1, addr 172.20.40.2 38C581C0
< 8 > IP adj out of GigabitEthernet0/0/2, addr 172.21.40.2 38C58EC0
< 9 > IP adj out of GigabitEthernet0/1/1, addr 172.22.40.2 38C59BC0
<10 > IP adj out of GigabitEthernet0/1/2, addr 172.23.40.2 38C58D20
<11 > IP adj out of GigabitEthernet0/1/5, addr 172.24.40.2 38C589E0
<12 > IP adj out of GigabitEthernet0/1/6, addr 172.25.40.2 38C58500
<13 > IP adj out of GigabitEthernet0/1/3, addr 172.26.40.2 38C596E0

this what I mention before the CEF is use same hash and load the traffic to all port.  

what I think you face that CEF is make some port forward packet more than other. 
so please can you check my idea next time you face issue 

< 0 > IP adj out of GigabitEthernet0/0/1, addr 172.20.40.2 38C581C0
< 1 > IP adj out of GigabitEthernet0/0/2, addr 172.21.40.2 38C58EC0
< 2 > IP adj out of GigabitEthernet0/1/1, addr 172.22.40.2 38C59BC0
< 3 > IP adj out of GigabitEthernet0/1/2, addr 172.23.40.2 38C58D20
< 4 > IP adj out of GigabitEthernet0/0/1, addr 172.24.40.2 38C589E0
< 5 > IP adj out of GigabitEthernet0/0/2, addr 172.25.40.2 38C58500
< 6 > IP adj out of GigabitEthernet0/1/1, addr 172.26.40.2 38C596E0
< 7 > IP adj out of GigabitEthernet0/1/2, addr 172.20.40.2 38C581C0
< 8 > IP adj out of GigabitEthernet0/0/1, addr 172.21.40.2 38C58EC0
< 9 > IP adj out of GigabitEthernet0/0/2, addr 172.22.40.2 38C59BC0
<10 > IP adj out of GigabitEthernet0/1/1, addr 172.23.40.2 38C58D20
<11 > IP adj out of GigabitEthernet0/1/2, addr 172.24.40.2 38C589E0
<12 > IP adj out of GigabitEthernet0/0/1, addr 172.25.40.2 38C58500 <<- 
<13 > IP adj out of GigabitEthernet0/0/2, addr 172.26.40.2 38C596E0 <<- each port use three times except the 0/0/1 and 0/0/2 they use four times. 

[Dr.X]

ok im still not answered. Could it be a bug ? or it's a normal EIGRP behavior?

[Joseph W. Doherty]

"or it's a normal EIGRP behavior?"

As noted in my original posting, I believe it's likely "normal" behavior.

[Richard Burts]

You are not yet answered because we do not understand your issue. I agree with @Georg Pauwen " I am still not clear on what you mean when you say 'we see 400Mbps'. What do you see exactly, what output are you looking at ?".

Also you say "ok but im using here EIGRP, not EtherChannel, Do you mean its for Eigrp too?". There was a question about Etherchannel or EIGRP. I supplied some information about Etherchannel on ASR1002. That information applies to Etherchannel and does not apply to EIGRP.

Also @Joseph W. Doherty I am not aware of EIGRP having  a (optional) dynamic load routing capability.

[Richard Burts]

As I read through the discussion again I have a few more comments:

You tell us you use EIGRP to forward traffic from router to switch and use static default routes to forward traffic from switch to router. If the switch is running EIGRP with the router I wonder why you do not have the switch use a default route advertised by the router instead of static default routes? I also wonder if there is any possibility that there is some mismatch between the routes used by the router to get to the switch and the routes used by the switch to get to the router?

I am also wondering about this "we only started to see the issue recently after our Upstream provider made some maintenance that put the upstream links down and then up again". I have difficulty understanding how maintenance on upstream links would have impact on routing logic for links inside your network?

I still do not really understand what the problem really is, but as best I understand at this point it seems to be that some links work better than other links. If that is the case then there is little reason to look for issues with EIGRP. EIGRP is responsible for finding paths through the network to destinations and getting those paths into the routing table. After that EIGRP does not influence anything. The decision about a packet and which interface to use to forward it is made by other processes, not by EIGRP.

[Joseph W. Doherty]

"I am not aware of EIGRP having  a (optional) dynamic load routing capability."

Okay, just did a quick search for what I recalled (?) about EIGRP having a "load" metric - it does - also known as the K2 metric, and off by default.

However, it appears EIGRP doesn't (?) update this metric as the interface TX rate changes - i.e. it appears it's not dynamically recomputed.

[Richard Burts]

@Joseph W. Doherty Thanks for the clarification. Yes Indeed there is an element in the EIGRP metric for load. My understanding was that was included in the original specification of the protocol thinking that at some time there might be a way to use this factor in making routing decisions. But nothing was ever developed to do this. So there is a load parameter and it is used when the prefix is originally placed into the database/routing table and is not recalculated if there were to be any change in load of the actual interface.  So not a factor in the present discussion. But good for you +5 for remembering that it might have been a factor in this issue.

[Dr.X]

Hello Richard , let me explain .
i mean the links all of them are 1G Ethernet . 

i mean with 400Mbps that certian links utilized with 400Mbps and another single  is link utilized with 999Mbps.
im trying here to say that there is no equal distribution on the links although all the routes are in routing table with same AD/Metric .

so EIGRP already counted the metrcis and put all in routing table , all links for all routes have same AD/Metric with 6 records in the routing table .
Now im using 7 , so will be as below :
D EX 10.59.0.0/24
[170/2560000768] via 172.26.40.2, 2d14h, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 2d14h, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 2d14h, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 2d14h, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 2d14h, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 2d14h, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 2d14h, GigabitEthernet0/0/1
D EX 10.60.60.0/24 [170/3328] via 172.26.40.2, 1w0d, GigabitEthernet0/1/3
[170/3328] via 172.25.40.2, 1w0d, GigabitEthernet0/1/6
[170/3328] via 172.24.40.2, 1w0d, GigabitEthernet0/1/5
[170/3328] via 172.23.40.2, 1w0d, GigabitEthernet0/1/2
[170/3328] via 172.22.40.2, 1w0d, GigabitEthernet0/1/1
[170/3328] via 172.21.40.2, 1w0d, GigabitEthernet0/0/2
[170/3328] via 172.20.40.2, 1w0d, GigabitEthernet0/0/1
D EX 10.72.99.0/24
[170/2560000768] via 172.26.40.2, 2d14h, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 2d14h, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 2d14h, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 2d14h, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 2d14h, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 2d14h, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 2d14h, GigabitEthernet0/0/1
S 10.74.64.253/32 [1/0] via 10.75.60.17
C 10.75.60.16/30 is directly connected, Port-channel2
L 10.75.60.18/32 is directly connected, Port-channel2
D EX 10.75.62.24/29
[170/2560000768] via 172.26.40.2, 1w0d, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 1w0d, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 1w0d, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 1w0d, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 1w0d, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 1w0d, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 1w0d, GigabitEthernet0/0/1
D EX 10.77.78.0/24 [170/3072] via 172.26.40.2, 1w0d, GigabitEthernet0/1/3
[170/3072] via 172.25.40.2, 1w0d, GigabitEthernet0/1/6
[170/3072] via 172.24.40.2, 1w0d, GigabitEthernet0/1/5
[170/3072] via 172.23.40.2, 1w0d, GigabitEthernet0/1/2
[170/3072] via 172.22.40.2, 1w0d, GigabitEthernet0/1/1
[170/3072] via 172.21.40.2, 1w0d, GigabitEthernet0/0/2
[170/3072] via 172.20.40.2, 1w0d, GigabitEthernet0/0/1
D EX 10.80.0.0/30
[170/2560000768] via 172.26.40.2, 1w0d, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 1w0d, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 1w0d, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 1w0d, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 1w0d, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 1w0d, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 1w0d, GigabitEthernet0/0/1
D EX 10.80.0.4/30
[170/2560000768] via 172.26.40.2, 1w0d, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 1w0d, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 1w0d, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 1w0d, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 1w0d, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 1w0d, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 1w0d, GigabitEthernet0/0/1
D EX 10.80.0.8/30
[170/26880256] via 172.26.40.2, 1w0d, GigabitEthernet0/1/3
[170/26880256] via 172.25.40.2, 1w0d, GigabitEthernet0/1/6
[170/26880256] via 172.24.40.2, 1w0d, GigabitEthernet0/1/5
[170/26880256] via 172.23.40.2, 1w0d, GigabitEthernet0/1/2
[170/26880256] via 172.22.40.2, 1w0d, GigabitEthernet0/1/1
[170/26880256] via 172.21.40.2, 1w0d, GigabitEthernet0/0/2
[170/26880256] via 172.20.40.2, 1w0d, GigabitEthernet0/0/1

once i disconnect one , another link is utilized with 999Mbps and other links are  almost 300-400Mbps  while single link is full close to 1G utlilization .

the issue was solved after i added more links in the EIGRP group .

it was 5 links , then i put 6 , then i put 7 while real usage is 4Gbps max .

Im trying to say the above , but seems my style of writting is not clear and i do apologize .

i pasted the ASR IOS version so you can track if there is a certain bug based on what i provided here .

Thank you again for your time here .

[MHM Cisco World]

I run small lab, I want to share with you result, 
R1 have four equal cost to R2, now the R3 send 1000 ping to LO in R2 
I capture the traffic and see that all 10000 packet is pass through one link, because as I mention before this is behave of CEF not routing protocol. 
I try play with CEF load balance algorithm but no way the best case the R1 select only two path. 
so I want to make load equal how, I can do that and you can see that all four path capture ping paacket.
I can do that with under interface 
ip load-sharing per-packet.

hope this solve your issue. 

[Joseph W. Doherty]

"ip load-sharing per-packet."

Usually NOT recommended (as it typically causes TCP retransmissions).

[David Ruess]

Hello,

I see a lot of suggestions and discussion by lots of great minds. Unless I missed it can you provide the output of a single route learned form EIGRP in the routing table with the below command:

show ip route 10.14.2.152 <mask>   - go ahead and include another subnet learned through EIGRP just for good measure. 

-Thanks

David

[Dr.X]

Gateway-ASR1002#Sh ip route 10.14.212.164 255.255.255.255
Routing entry for 10.14.212.164/32
Known via "eigrp 30", distance 170, metric 2560000768, type external
Redistributing via eigrp 30
Last update from 172.23.40.2 on GigabitEthernet0/1/2, 11:17:32 ago
Routing Descriptor Blocks:
172.26.40.2, from 172.26.40.2, 11:17:32 ago, via GigabitEthernet0/1/3
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 106/255, Hops 2
172.25.40.2, from 172.25.40.2, 11:17:32 ago, via GigabitEthernet0/1/6
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 138/255, Hops 2
172.24.40.2, from 172.24.40.2, 11:17:32 ago, via GigabitEthernet0/1/5
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 94/255, Hops 2
172.23.40.2, from 172.23.40.2, 11:17:32 ago, via GigabitEthernet0/1/2
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 104/255, Hops 2
* 172.22.40.2, from 172.22.40.2, 11:17:32 ago, via GigabitEthernet0/1/1
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 103/255, Hops 2
172.21.40.2, from 172.21.40.2, 11:17:32 ago, via GigabitEthernet0/0/2
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 101/255, Hops 2
172.20.40.2, from 172.20.40.2, 11:17:32 ago, via GigabitEthernet0/0/1
Route metric is 2560000768, traffic share count is 1
Total delay is 30 microseconds, minimum bandwidth is 1 Kbit
Reliability 1/255, minimum MTU 1 bytes
Loading 113/255, Hops 2