EIGRP equal load balancing problem stuck with single link with Drop !

[Dr.X]

Hello Team ,

We have been running eigrp equal load balancing instead of EtherChannel between :

Cisco ASR ---> Cisco 3560.

we are balancing 4G traffic on six links of 1G speed.

But recently, with no recent change , we began to see that one of the Eigrp links is 1G while others are 400Mbps. which causes network drop all the time ...

From Eigrp, all routes have the same AD/Metric and equal loads of six links on the routing table.

Tried to clear BGP nei, Routing tables, and flush everything; one link always stays 1G while the others are 500 Mbps.

If i drop one link from the group, another link is filled with 1G and the same issue.

I added the 7th link to the eigrp balance, and the issue was solved all links get equal balance now.

Based on the search, i don't think the issue has to occur as Eigrp is much better than Etherchannel L3 solution.

My eigrp is simple :

router eigrp 4
maximum-paths 10
variance 128

Here is sample of the routing table :

D EX 10.14.2.152/32
[170/2560000768] via 172.26.40.2, 00:07:35, GigabitEthernet0/1/3
[170/2560000768] via 172.25.40.2, 00:07:35, GigabitEthernet0/1/6
[170/2560000768] via 172.24.40.2, 00:07:35, GigabitEthernet0/1/5
[170/2560000768] via 172.23.40.2, 00:07:35, GigabitEthernet0/1/2
[170/2560000768] via 172.22.40.2, 00:07:35, GigabitEthernet0/1/1
[170/2560000768] via 172.21.40.2, 00:07:35, GigabitEthernet0/0/2
[170/2560000768] via 172.20.40.2, 00:07:35, GigabitEthernet0/0/1

here is my router info :

Gateway-ASR1002#sh version
Cisco IOS XE Software, Version 03.13.00.S - Extended Support Release
Cisco IOS Software, ASR1000 Software (PPC_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.4(3)S, RELEASE SOFTWARE (fc11)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 28-Jul-14 04:11 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

Gateway-ASR1002 uptime is 24 weeks, 4 days, 17 hours, 39 minutes
Uptime for this control processor is 24 weeks, 4 days, 17 hours, 43 minutes
System returned to ROM by reload at 18:39:44 UTC Sun Aug 6 2017
System image file is "bootflash:asr1000rp1-adventerprisek9.03.13.00.S.154-3.S-ext.bin"
Last reload reason: PowerOn

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco ASR1002 (2RU) processor (revision 2RU) with 1650497K/6147K bytes of memory.
Processor board ID FOX1807GBZW
12 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
7757823K bytes of eUSB flash at bootflash:.

Configuration register is 0x2102

Any one can help if this could be a hardware issue or a bug?

Thanks 

[MHM Cisco World]

as I know 
the equal cost multi path of routing protocol is only allow router to add additional path. 
the hard work is in CEF which make the load sharing between the all path. 

[MHM Cisco World]

Routing entry for 1.1.1.1/32
Known via "eigrp 100", distance 90, metric 2690560, type internal
Redistributing via eigrp 100
Last update from 110.0.0.1 on FastEthernet3/0, 00:00:12 ago
Routing Descriptor Blocks:
110.0.0.1, from 110.0.0.1, 00:00:12 ago, via FastEthernet3/0
Route metric is 5250560, traffic share count is 41
Total delay is 5100 microseconds, minimum bandwidth is 500 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1
* 100.0.0.1, from 100.0.0.1, 00:00:12 ago, via FastEthernet2/0
Route metric is 2690560, traffic share count is 80
Total delay is 5100 microseconds, minimum bandwidth is 1000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1

you can see both entry add for same route using variance 2 in EIGRP but still each one have it value of sharing load.

[Georg Pauwen]

Hello,

the output is somewhat confusing. In your original post, you are talking about 6 GigabitEthernet links, this output shows FastEthernet links ? Also, did you set the bandwidth manually ? 1000kbit = 1Mbps (and 500Kbits = 0.5Mpbs), which does not look right.

I would suggest to manually set the bandwidth on all participating interfaces to 1000000Kbit (which equals 1G).

[Dr.X]

Hello Georg ,

Well, i have not posted any fast ethernet, i only said that i have 6 links, and i used 7 links to solve the problem.

The problem is random; sometimes, its solved with 4 , sometimes 5 , sometimes 6 - 7 links, and so on.

so i dont think the problem is logical can you help me if there is a similar bug on routers?

[Georg Pauwen]

Hello,

my bad, I did not see that it was not your output. Either way, can you show that output (show iproute x.x.x.x) from your actual devices ? Seeing both running configs would be even better...

EDIT: I am still not clear on what you mean when you say 'we see 400Mbps'. What do you see exactly, what output are you looking at ?

[Dr.X]

But they are equal routes in the routing table, so I belive the routing will happen there based on packets.
I'm not sure if I understood your answer.

Thanks 

[Georg Pauwen]

Hello,

I am not clear on the issue: the throughput of the links drops ? Can you explain what exactly you mean by the below ?

--> But recently, with no recent change , we began to see that one of the Eigrp links is 1G while others are 400Mbps. which causes network drop all the time ...

Also, post the full running configs of the ASR and the 3560.

[Joseph W. Doherty]

"Based on the search, i don't think the issue has to occur as Eigrp is much better than Etherchannel L3 solution."

Yes, perhaps better, but not perfect.

From what you've described, I would expect this to be "normal" behavior.

Likely, you have a couple of bandwidth intense flows that end of congesting the same link with 5 or 6 links but not 7.

I recall (?) EIGRP can, in theory, dynamically load balance.  If so, not something I've ever tried with it, nor could comment on how well it works.

PfR has the capability to dynamically load balance links (often doing a fairly good job at it).

Otherwise, the simplest solution might be the one you've already found, i.e. vary number of member links.  Of course, two major issues using this approach, first whether you have "extra" links you might use for this purpose and second, problem may reappear.

BTW, once you get in the neighborhood of more than 4 links, you might want to consider it it makes sense to upgrade to the next bandwidth step, 10g in your case.(?)

Also BTW, @Richard Burts might be able to comment on this too.  I know he is well acquainted with EIGRP and we recently were going back and forth on the merits, or lack there of, of load balancing using ECMP if you're trying to achieve real/true load balancing.

[Dr.X]

Hello Joseph , i underatsnd the best would be 10g link .

But we have hardware limitations at the moment.

I want to tell you that the solution has been stable for more than four years, and we only started to see the issue recently after our Upstream provider made some maintenance that put the upstream links down and then up again.


For the config its simple at the two routers :

router eigrp 30
maximum-paths 10
variance 128
network 10.30.30.1 0.0.0.0
network 10.30.30.2 0.0.0.0
network 172.20.40.1 0.0.0.0
network 172.21.40.1 0.0.0.0
network 172.22.40.1 0.0.0.0
network 172.23.40.1 0.0.0.0
network 172.24.40.1 0.0.0.0
network 172.25.40.1 0.0.0.0
network 172.26.40.1 0.0.0.0
passive-interface default
no passive-interface GigabitEthernet0/0/1
no passive-interface GigabitEthernet0/0/2
no passive-interface GigabitEthernet0/1/1
no passive-interface GigabitEthernet0/1/6
no passive-interface GigabitEthernet0/1/5
no passive-interface GigabitEthernet0/1/7
no passive-interface GigabitEthernet0/1/2
no passive-interface GigabitEthernet0/1/3


from ASR--->3560 it user EIGRP for download traffic .

but from 3560---> ASR we use default route load balance static which loads equally for upload traffic .

The only issue here is the EIGRP issue that i dont see any logical for it to happen .

Here is my router info :

NAME: "Chassis", DESCR: "Cisco ASR1002 Chassis"
PID: ASR1002 , VID: V06, SN: FOXxxxxx

NAME: "module F0", DESCR: "Cisco ASR1000 Embedded Services Processor, 5Gbps"
PID: ASR1000-ESP5 , VID: V04,

[Richard Burts]

@Joseph W. Doherty Thanks for mentioning me and getting me involved in this discussion. If there is a question about whether EIGRP or Etherchannel would be preferred I would prefer EIGRP. One factor I consider is that the original post describes using 6 (or 7) links. My understanding of the documentation is that ASR1002 can support up to 4 links.

@Georg Pauwen I agree with you that we need a better understanding of the issue. Where/how was it observed 1G while others are 400Mbps?

As far as "you are talking about 6 GigabitEthernet links, this output shows FastEthernet links" I believe that the original poster has been consistent in referring to Gig links. I believe that the reference to FastEther is from a lab experiment set up by @MHM Cisco World 

[Dr.X]

Hi richard ,

One factor I consider is that the original post describes using 6 (or 7) links. My understanding of the documentation is that ASR1002 can support up to 4 links.

is it ASR1002 limitation 4 links ? can you share the doc link?

Thanks 

[Richard Burts]

Here is the link. Look at the first paragraph which says "A port channel bundles individual Ethernet links into a single logical link that provides an aggregated bandwidth of up to four physical links."

https://www.cisco.com/c/en/us/td/docs/routers/asr1000/configuration/guide/chassis/asr1000-software-config-guide/evc-port-chann-asr.html

[Dr.X]

ok but im using here EIGRP, not EtherChannel, Do you mean its for Eigrp too?

[Georg Pauwen]

@Richard Burts I see, my bad, I thought this was the output from OP. I don't really see what that output has to do with the issue...