OSPF ECMP between Nexus 7k and ASR9K

[aozhanbin.champ1]

I have 5 10G links between Nexus 7k and ASR9K with OSPF. But those links is not doing ECMP. One of the links is getting 10G BW usage and other links are less than 5G BW usage. I checked the ASR9K has max-path 32 configs under router OSPF configs. And the Nexus 7k device doesn't have this config. And it shows the " Maximum paths to destination 8".

I wonder what should I do to make the ECMP works for all 5 10gig links?

The cost are the same on all those 5 links on both ends.

[balaji.bandi]

Most of the best practice of Link will be even most of the time like  2,4,6,8 ( 5 is bit odd to look)

is this nexus in vPC ?  and config both the sides ?

what kind of reference bandwith on OSPF ?

show ip load-sharing  - will give what kind of LB configured in LACP ? - this need to check both the sides

[aozhanbin.champ1]

No VPC and LAG on these 5 links.

On the Nexus 7K switch the "Reference Bandwidth is 40000 Mbps" and 400000 on ASR 9K side.

[Richard Burts]

It would be helpful if you would post the output of the command show ip route ospf from both routers and identify which prefixes should be doing ECMP. Do both outputs show 5 entries for those prefixes? If so then you have ECMP. If both outputs do not show 5 entries then we need additional information about your environment so that we can figure out why there is not ECMP.

Bear in mind that ECMP is intended to provide load sharing but not necessarily load balancing. Seeing not equal load on the links is not necessarily a problem. Each source/destination pair will always take the same path. So with 5 links if one of the links is carrying traffic that is doing data backup from a major server while the other 4 links are carrying user inquiries into a data base on that server then we would expect to see significantly heavier load on that one link.

[aozhanbin.champ1]

Richard, I saw all 5 links were as next hops for single particular route on the Nexus 7K. The traffic is the CDN video stream. The CDN server is connecting to the Nexus 7K. And the ASR9000 is the downstream router connecting to Nexus 7000 switch. I got your point that the load-balancing was per flow instead of per packet. But the current BW usage on the fifth link is 10G and other link is only getting 3-5G BW usage.

[Richard Burts · Accepted solution]

Thanks for the additional information confirming that all 5 links do show up in the IP routing table. This confirms that OSPF ECMP is working.

I am a bit puzzled about this statement " I got your point that the load-balancing was per flow instead of per packet. But the current BW usage on the fifth link is 10G and other link is only getting 3-5G BW usage." If you get the point that traffic is balanced per flow then it is expected behavior that some link may have more traffic while other links have less traffic. But you seem to feel that there is some problem. I understand that you would want the traffic on all links to be equal. But that is not what OSPF ECMP is intended to do.

[Joseph W. Doherty]

As Rick has explained, it's common to see unequal load balancing using ECMP (or Etherchannel) unless something like per packet load balancing is being used (which may be an option with your CEF - although if it is, highly recommended not to use it).

If better load balancing is desired, if the platform supports one of the later PfR variants (your ASR may, not so sure about NEXUS), that might be used, but even it, generally, won't achieve 100% equal multi-link utilization.

What it can do, though, if one flow can saturate 100% of a link's capacity, it will insure all other flows are distributed across the other links not being used at 100%.

[aozhanbin.champ1]

Thanks Joseph and Richard, I understand it much better now.

[Richard Burts]

I am glad that our explanations have been helpful and that you now understand it better now. Thank you for marking this question as solved. This will help other participants in the community to identify discussions which have helpful information. This community is an excellent place to ask questions and to learn about networking. I hope to see you continue to be active in the community.

[MHM Cisco World]

....

[MHM Cisco World]

OSPF ECMP
ospf support only equal cost,
Cost can be equal for multi path through :-
A- change the BW
B- config cost

Your case as I think is config cost so it not change BW
one link is 10G 
other is 5G 
but both Cost is equal.

now why BW is different, BW with auto negation change the BW to be equal in both side, so 
5G is auto negation between ASR and NSK and end with use 5G.
please check these points.