ECMP between different Colour

[XinniX]

Dear Cisco Experts,

I am wondering if anyone could demystify SDWAN ECMP mechanism and point me to the right direction. I have been reading up for weeks now but am still very much confused on this topic and would really appreciate any guidance on the following.

The Issue:

I have noticed on many of my sites that different "colour" WAN link are not getting load balanced efficiently on the vEdges.

Example, on a site with a single vEdge and 2 WAN link with different colour:

Ge0/4 - 100Mbps, colour: Private1 (MPLS)

Ge0/5 – 100Mbps, colour: Custom1 (DIA)

Data Traffic seems to favor the Custom1 link and would often overutilized it (~90% utilization) while the available Private1 is mostly idle (~5% utilization). Upon further investigation on the pcap and Solarwind's netflow, I am able to observed that the ingress traffic on Custom1 are from different sources with different IP address. So I am abit stumbled now as I had thought that ECMP would do a session based load balancing and divert different traffic sessions from different sources out both WAN links equally. Digging through the configuration, I can observe the following and hopes it could help with the diagnosis:

+ "Preference" are the same at 100

+ "Weight" is not configured

+ "Restrict" is not configured 

+ no PBR has been configured

+ no Application-Aware Routing policy has been configured

+ Enabling Enhance ECMP does not change the interface statistics shown

While I do agree that "TRUE" redundancy is also in question here because each of those WAN link should be able to cater to the entire traffic during failover, its hard to persuade the business for an upgrade when a good 100Mbps MPLS link is not getting fully utilized... 

[Flavio Miranda]

Hi

 Each company has its policies but the way I see it, based on the company I´ve working, what is happening there is the expected behavior.

MPLS is a expensive link and should be used only for data that needs to go to the Data Center with more security and reliability DIA is the opposite. You use cheaper internet link to send no critical data to the internet locally.  You dont need to use the Data Center for that.

 DIA means Direct Internet Access for a reason.

[XinniX]

Hi Flavio,

thanks for the response really appreciate the input. Agree with the fact that the MPLS is used for the datacenter but now it makes me wonder if SDWAN have any features which would allow it to direct the traffic out the MPLS link when the DIA is maxing out so packet loss can be mitigated. A PBR could probably achieve this but it will be a tedious and a non-standardized routine to force a particular traffic out the MPLS so am wondering if theres other options out there

[Flavio Miranda · Accepted solution]

I´m pretty sure it is possible but If you are using DIA in the way I´ve seen it being used this is not recommended.  The traffic sent out to DIA is meant to be the traffic you wont in your data center which means you probably will not have all the tools to treat it properly. Of course, you can create a shadow infrastructure locally and on the Data Center for DIA traffic but this means more costs.

  For the porpuse you are looking for, I´ve seen companies using the MPLS and biz-internet or public-internet to load balance traffic and create redundancy when necessary. The ipsec tunnel has two path and you can load balance or prioritize one of them according to your internal policies.

But the concept of  DIA is a bit different as fas as I´ve seen so far.

[XinniX]

Hi Flavio, 

thanks again for the insight on this matter. Really appreciate the inputs. Indeed the MPLS link should be set aside for DC traffic. I suppose my next step will be to persuade the business for a higher DIA BW or add a redundancy vEdge with a 2nd DIA for load sharing.

[Flavio Miranda]

Yeah, that´s correct.

You can use the internet link to achieve the redundancy you are looking for but on this case, you are going to stablish a ipsec tunnel and finish the ipsec tunnel on the data center. Althought the traffic goes through the internet, it will represent a redundancy for MPLS and you can apply the best load balance policy you want.

But when it comes do  DIA the idea change a little bit. On this case, you just send out the traffic toward the internet using some NAT and Firewall so that you can avoid using your tunnel to the Data Center. DIA is in my opinion, what makes Cisco SDWAN defensible from the money point of view.