SD-WAN Controller - OMP best path selection

[iores]

Hi,

I am trying to understand how the Controller determines the best path(s) which then sends to WAN Edges.

On WAN edge ecmp-limit is set to 4.

When Controller is using Number of Paths Advertised per Prefix = 4, then WAN Edge chooses 2x OMP routes with C,I,R status:

TENANT    VPN    PREFIX              FROM PEER        ID     LABEL    STATUS    TYPE       TLOC IP          COLOR            ENCAP  PREFERENCE  NUMBER      REGION ID   REGION PATH      
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0         10     0.0.0.0/0           100.0.0.101      3      1003     C,I,R     installed  1.1.1.1          mpls             ipsec  -           None        None        -                
                                     100.0.0.101      4      1003     C,I,R     installed  1.1.1.1          biz-internet     ipsec  -           None        None        -                
                                     100.0.0.101      7      1004     R         installed  1.1.1.2          mpls             ipsec  -           None        None        -                
                                     100.0.0.101      8      1004     R         installed  1.1.1.2          biz-internet     ipsec  -           None        None        -                

When Controller is using Number of Paths Advertised per Prefix = 8, then WAN Edge chooses 4x OMP routes with C,I,R status:

TENANT    VPN    PREFIX              FROM PEER        ID     LABEL    STATUS    TYPE       TLOC IP          COLOR            ENCAP  PREFERENCE  NUMBER      REGION ID   REGION PATH      
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0         10     0.0.0.0/0           100.0.0.101      1      1004     R         installed  2.2.2.2          mpls             ipsec  -           None        None        -                
                                     100.0.0.101      2      1004     R         installed  2.2.2.2          biz-internet     ipsec  -           None        None        -                
                                     100.0.0.101      3      1003     C,I,R     installed  1.1.1.1          mpls             ipsec  -           None        None        -                
                                     100.0.0.101      4      1003     C,I,R     installed  1.1.1.1          biz-internet     ipsec  -           None        None        -                
                                     100.0.0.101      5      1004     C,I,R     installed  2.2.2.1          mpls             ipsec  -           None        None        -                
                                     100.0.0.101      6      1004     C,I,R     installed  2.2.2.1          biz-internet     ipsec  -           None        None        -                
                                     100.0.0.101      7      1004     R         installed  1.1.1.2          mpls             ipsec  -           None        None        -                
                                     100.0.0.101      8      1004     R         installed  1.1.1.2          biz-internet     ipsec  -           None        None        -                

Please note that routes from 1.1.1.1 and 2.2.2.1 receive TLOC preference of 500, and routes from 1.1.1.2 and 2.2.2.2 TLOC preference of 400:

tloc entries for 1.1.1.1
     preference        500
tloc entries for 1.1.1.1
     preference        500
tloc entries for 1.1.1.2
     preference        400
tloc entries for 1.1.1.2
     preference        400
tloc entries for 2.2.2.1
     preference        500
tloc entries for 2.2.2.1
     preference        500
tloc entries for 2.2.2.2
     preference        400
tloc entries for 2.2.2.2
     preference        400

Question: shouldn't WAN Edge in both cases have 4 OMP default routes? Why it has only 2 when Controller is advertising 4 paths per prefix?

[M02@rt37]

Hello @iores 

It is expected because the wan edge only installs ecmp paths from the best TLOC preference group, not from all advertised paths.

When the controller advertises 4 paths, it does not include enough equal-best (preference 500) TLOCs, so the wan edge only see one top-tier TLOC and install 2 paths (one per color), whereas with 8 advertised paths both top-preference TLOCs (1.1.1.1 and 2.2.2.1) are included, allowing ecmp across them and resulting in four installed routes...

[iores]

M02@rt37 

I thought the controller selects the best path using OMP criteria and advertises it to WAN Edges. Does it actually send multiple paths (N routes), leaving WAN Edges to pick the best? What criteria does it use to decide which routes to advertise? It seems that in my case TLOC preference is not being considered, and only the System IP is used? On the controller, I can see all 8 routes with status C and R, ordered by System IP.

Since the controller applies tie-breakers and can always determine a single best path at any given time, would it be more accurate to say that the send-path limit defines the maximum number of equal-cost paths, or the maximum number of best paths that get advertised?

[M02@rt37]

@iores 

vsmart does not advertise only a single “best” path...it advertise up to N best candidates per prefix, and the WAN edge then run its own best path + ECMP selection...

[iores]

M02@rt37 

But which criteria does the controller use to sort out the routes before deciding which N routes to send?

[M02@rt37]

vsmart does not randomly pick N paths...it runs full OMP best-path selection first, ranks all paths, and then advertises only the top N paths.

So, vsmart first run the full OMP best-path selection process—ranking all available routes using attributes in order such as OMP route preference, then TLOC preference, followed by origin, metric, and tie-breakers—and only after this ranking does it apply the “number of paths advertised per prefix” limit by selecting the top n path.

Therefore, when set to 4, it sends only the highest-ranked paths (all from TLOCs with preference 500), meaning the WAN Edge receives a less diverse set of paths that are not all equal for ECMP, resulting in fewer installed routes, whereas increasing to 8 includes lower-preference TLOCs (400), giving the wan egde more candidate that can qualify for ecmp up to its limit...

Hope it is clear...

[iores]

M02@rt37 

But the thing is when set to send 4 routes, it doesn't send only 4 routes with TLOC preference 500 (from 1.1.1.1 and 2.2.2.1). It sends 4 routes from 1.1.1.1 and 1.1.1.2.

The TLOC preference is set with inbound control policy.

[iores]

When usin OMP instead TLOC preference, controller seems to select best paths correctly. Four routes (with OMP preference 500) get 'C, R' status. Other routes have 'R' status. 

[Royalty · Accepted solution]

Hi @iores,

Hope you've been doing well!

We are clear so far that there are two best-path selection processes occuring in SD-WAN, which are these:

• SD-WAN Controller (formerly vSmart) best-path selection
• WAN Edge (vEdge/cEdge) best-path selection

A routing update follows these steps:

1. WAN Edges (vEdges/cEdges) advertise vRoutes (OMP Routes) to the SD-WAN Controller(s) (vSmart(s))
2. The SD-WAN Controller(s) go through best-path selection and further advertise vRoute prefixes based on the OMP Best-Path Algorithm.
3. The WAN Edges (vEdges/cEdges) receive the vRoutes from the SD-WAN Controller(s) and use the OMP Best-Path Algorithm to determine which routes should be chosen and installed, or ultimately, which ones are candidate to be installed in the RIB

---

Question: shouldn't WAN Edge in both cases have 4 OMP default routes? Why it has only 2 when Controller is advertising 4 paths per prefix?

---

The SD-WAN Controller (vSmart) is allowed to advertise a maximum of 4 equal-cost paths by default. In this case/situation, the SD-WAN Controller is advertising 4 prefixes because because that is the default limit that enforced by the 'send-path limit'. The WAN Edge, however, only believes that 2 of the vRoute prefixes are equal-cost...

Why is there a discrepancy between what the SD-WAN Controller believes and what the WAN Edge believes? The reason is because the SD-WAN Controller and the WAN Edge use different steps of the OMP Best-Path Algorithm. Specifically, the whole issue here is that the SD-WAN Controller does not care about TLOC Preference. However, the WAN Edges do care, and they consider the TLOC Preference when determining if routes are equal-cost. In this document: OMP Best-Path Algorithm it explains that TLOC Preference is only considered and applied to 'Edge devices'.

So this is what is happening in the first case:

1. The SD-WAN Controllers (vSmarts) receive vRoutes for 0.0.0.0/0 in VPN 10. The SD-WAN Controllers run the best-path selection process and realise that the vRoutes are all considered equal (because the SD-WAN Controllers do not look at TLOC Preference for their best-path selection). However, there is a problem. The send-path limit is set to 4 by default, meaning the SD-WAN Controllers can only send 4 equal-cost routes. It selects the 4x 0.0.0.0/0 prefixes according to the tiebreaker criteria (in this case, lowest System IP address) and advertises them to the other respective WAN Edges.
2. The WAN Edges now receive 4 vRoutes for 0.0.0.0/0 in VPN 10. The WAN Edge goes through the OMP Best-Path Selection process and finds that the routes are not equal-cost and therefore cannot be considered for ECMP. This is because the WAN Edge DOES consider the TLOC Preference, unlike the Controllers which do not. Therefore, it can only install the 2 0.0.0.0/0 prefixes - those that have the highest TLOC Preference.

In the second case, this is the behaviour:

1. The SD-WAN Controller begins sending 8 prefixes. This now allows the additional two vRoutes which have a TLOC Preference of 500 to be received by the WAN Edge. This now means that the WAN Edge has received two new additional equal-cost routes that also have a TLOC Preference of 500 like the original two that were received. It can now install 4 ECMP routes - those that all have a TLOC Preference of 500.

Just to add some summary answers to these below:

When using OMP instead TLOC preference, controller seems to select best paths correctly. Four routes (with OMP preference 500) get 'C, R' status. Other routes have 'R' status.

---

Yes that's spot on, the OMP Preference is an OMP attribute that is considered by both the SD-WAN Controller and WAN Edges.

But which criteria does the controller use to sort out the routes before deciding which N routes to send?

Any step here under the 'Applied to' column that says "Cisco Catalyst SD-WAN Controller"

I should also note that the documentation of the link I provided is not a full comprehensive operation of the algorithm. It does not mention an explicit step for choosing the lowest System IP address (but is mentioned as part of step 18), even though that is one of the tiebreaker steps at the end. It also does not mention which of the steps are tiebreakers and which aren't. From my understanding, step 17 and beyond are tiebreakers (and the hidden step of the System IP address which is merged with step 18).

Hope that helps. Let me know if anything is confusing!

[iores]

Hi @Royalty ,

Thank you very much for you detailed response, I appreciate it! This explains very well my scenario but I couldn't find any document saying that controller doesn't take into account TLOC prefference, and started to think it's a bug or something. Thank you for the linked document, as well, I can now see which step of OMP best path selection algorithm is applied at Edge, and which at Controller.