MPLS TLOC Control Connections

[JakSmith27]

I am currently working on a Cisco SD-WAN deployment with cEdges.  Each site has a single router with a MPLS TLOC and a Biz-Internet TLOC.  I am attempting to get control connections from my MPLS Tlocs up.  I have advertised a default route from the main site towards the MPLS carrier, so that default route exists within the carrier network/customer vrf on the carrier's mpls l3 backbone.  On the remote sites I have configured dual static routes, one for biz-internet and one for mpls.  I went ahead and configured the static route for MPLS because the BGP route was not making into the rib due to the internet route being static.  

At the main site I configured an additional sub-interface within VPN0 that connects to a DMZ on that site's firewall.  I have had the firewall engineer check rules/routing on the firewall and those appear good.  As a test I configured the new sub-interface on the router as a tunnel-interface w/restrict turned on and it was able to successfully build control-connections.

I think my control connections from the MPLS TLOCs at the remote sites should at least be getting to the main site cEdge.  My confusion comes from what happens once the control connection attempts get to the main site's cEdge?  There are two static default routes configured there.  One pointing out the new DMZ sub-interface and one pointing out the biz-internet connection.  Do I need to assign a traffic-data policy to this site that directs traffic from the MPLS tlocs out the new DMZ?  

As a test I also tried configuring the Biz-Internet interface at the main site with NAT to try and allow the MPLS TLOCS from the other sites to reach the controllers from the BIZ-Internet connection at main and that did not seem to work either.

Hopefully I am over complicating this and there is a simpler solution that what I am trying to do.  From my understanding if the remote sites were to lose their biz-internet connection for more than twelve hours then when the MPLS tunnel ipsec-keys needed rekeyed the MPLS connection would drop as well and the site would be down?  This makes me want to ensure that I can establish control-connections through the MPLS tlocs as well.

Thanks!

[JakSmith27]

 Here is a basic diagram of what I am trying to describe.

[Kanan Huseynli · Accepted solution]

Hi,

in SD-WAN if you have 2 static routes in VPN 0, both are used using respective interface. Hence, if router sources control traffic with MPLS TLOC IP, then it can send data only to its gateway, it can not send traffic to another gateway via other interface. That's why 2 static routes work, and routing dont do ECMP logic on underlay. Basically, if you have 2 interfaces (mpls, biz-internet) , but only one default route (via biz-internet), then mpls interface can not establish connection to other networks (because it is not allowed to use biz-internet default route).

So, MPLS TLOC uses its gateway, biz-internet TLOC uses its own gateway to reach controllers.

Above is general logic used in SD-WAN (a bit different what we know from legacy-traditional networking)

Let's return to your case.

As I understand, your ISR4K router at main site runs SD-WAN. In this case, if you have configured tunnel-interface on MPLS TLOC, then remote branches can't reach controllers, even you have routing viz DMZ interface. This is because, "tunnel-interface" configured interfaces is hardcoded only for SD-WAN, transit dataplane can't path over it (basically, branch traffic can't reach tunnel-interface configured interface and is routed over another interface like DMZ in your case).

For these cases you have 2 option:

A) use another device like CE equipment. For example, your FW can be. Your main router still need MPLS TLOC, it can be achieved creating FW-MAIN_ROUTER interconnection (like you did for DMZ). Basically, FW will be CE equipment (main site device for termination remote MPLS traffic and communicating in CE-PE routing). Main router need just FW-ROUTER interconnection and MPLS TLOC with tunnel-interface configured.

But for security reasons or may be you don't run routing (CE-PE) on firewall and still want to place main site's router inline, then you can do the 2nd option , so called "loopback interface tunnels".

B) In two words: tunnel-interface is removed from physical interface, then loopback interface is created with tunnel-interface option (it is mandatory to run control plane over that interface) and bounded to physical interface.

So, you definitely need to remove tunnel-interface configuration from physical (mpls facing) interface to allow transit packets from remote branches. Then you need to create loopback interface in VPN0 and bind it to respective (mpls) interface. In order to enable SD-WAN functionality, you need to do tunnel-interface configuration on that loopback interface. For this case, of course, you still need FW-ROUTER interconnection (dmz) like you did, but dont enable tunnel-interface on that interconnection link, leave it for pure routing.

I highly recommend to read SD-WAN design guide (very well written, thanks to author(s) ). Loopback method is described in this section:

https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html#LoopbackInterfaceTunnels

Figure 71 is similar to your case, even though case in reality describes on-prem deployment.

HTH,

[JakSmith27]

Thank you for reading the whole post and giving clear concise answers to all of my questions.  It is rare to get a response like that!

The solution worked perfectly.  I re-configured the physical interface that terminates the internet circuit to be a non-tunnel interface, then configured a new loopback that is bound to that interface.  I then re-configured the DMZ interface to be a non-tunnel interface and configured a new loopback with a public color and restrict turned on to give the main site a redundant connection as well.  After all of that was configured at the main site I turned "max-control-connections 0" off on all of the remote site's MPLS interface and control connections popped right up on the MPLS TLOCs at all of my sites.  

Thanks again!

[svemulap@cisco.com]

The other option is to have your carrier carry your MPLS traffic across their backbone and they reach controllers in the internet through their exist point.
aka. all your sites (branch and headend) will have their MPLS-IP mapped to MSP provided public-IP, so control connections get established across MPLS.
This has been done and used by customers. It assumes, MSP will carry the MPLS traffic across their backbone.