Cisco SD-WAN vEdge MPLS Color Connectivity to Cloud Hosted Controllers

[JakSmith27]

In most of our current sites we use 2 Cisco DMVPN clouds, one with a public internet underlay and one with a metro-ethernet underlay.  DMVPN does not have any requirement to talk to vSmarts out on the internet so this works fine. 

We are trying to find a best practice for this scenario with SD-WAN and vEdges.  We have no problem with the internet transport link, that will natively have reachability to the Cisco hosted controllers.  Our question comes in with the metro-ethernet transport links, they will not natively have reachability to the hosted controllers.  They will only have reachability to each other because the only things these networks have today is layer two reachability between the vEdges. 

Question 1, what is best practice to give metro-ethernet links access to the internet?  

Question 2, if we were to move forward with an installation like this where the metro-ethernet transport links did not have access to the internet without using our internet transport circuits and those internet transport circuits were to go down, would that site go down?

[elesani · Accepted solution]

In Cisco SD-WAN, a given WAN interface can become a "Transport" interface if it has a route back to SD-WAN Controller nodes by sourcing that interface - this is the default behaviour unless you turn off "Control Connections" through the relevant "Feature Template". 

Now let's try to cover your questions.

Q1. You might want to consider introducing a GW within your Metro-Ethernet segment to advertise route towards your controller nodes. In this way, your metro-ethernet network assigned "color" can have an independent route back to your controller nodes.

Q2. The quick answer is YES! in an ideal situation where your Internet circuits are up, everything works fine as expected however, in a failure scenario, Edge router will lose access to all controller nodes. As a result that given site will:

• show offline on vManage
• lose OMP links to vSmart nodes

In that given scenario, you can expect surprises and unexpected behaviour on established TLOCs through metro-ethernet color transports.

Hope I covered your questions. 

[JakSmith27]

That is definitely a great start to answering my question!  

Q1. You might want to consider introducing a GW within your Metro-Ethernet segment to advertise route towards your controller nodes. In this way, your metro-ethernet network assigned "color" can have an independent route back to your controller nodes.

, 

1) What I have in mind when you say this is terminating the metro-ethernet connection into the main site's switch, and extending that metro-ethernet connection to the vEdge and to a firewall DMZ.  That new firewall DMZ would have an IP that is layer 2 adjacent to all of the vEdge's on the metro-ethernet network, and I would add a default route in VPN 0 of all of the vEdges that points to the layer 2 adjacent firewall.  The firewall would PAT the new DMZ out towards the internet and that should get connectivity from the vEdge metro-ethernet color to the cloud hosted controllers.

That seems pretty straight forward, I am not sure if I am thinking about this too traditionally/old school or if there is a better way to do this in SD-WAN terminology.  We only have a few months under our belt in the SD-WAN World

2) If there were a need to add a new interface in VPN 0 that is a non-transport interface, say for instance we had a MPLS color transport at a main site that is only a /30, one ip address for us and one for the carrier, I would need to add a new interface that is not actually part of the MPLS transport.  This new interface would have a default route to a firewall...how would I ensure that the actual MPLS color interface is the one that is seen as the source of the route?  I suppose all of the remote sites would receive that route on their MPLS color interfaces so they would be fine.  Just not sure how the transport interface at the main site would work because the new non-transport interface would be the source of the route and not the actual interface that terminates MPLS connectivity.

I may be over-complicating that last one!

Thank you for your answers.

[elesani]

1) you are following the right approach. Just bear in mind to pick a "public color" for your metro-ethernet segment if that will seat behind a NAT. - you can find details on below link:

https://explore.cisco.com/sd-wan-adopt/cvd-sd-wan-design-colors#page=7

2) It really depends on how you are designing your VPN0 routing as part of your underlay design. I can give you below two tips as an example:

1. default static route: Make sure you will have the default route pointing to the Next Hope (NH) reachable from a given Transport Interface. In nutshell, you will need to have 1x Default Route per Transport Interface. So, if you have 3x Transport Interface, I would expect you to have 3x Default Routes within your VPN0 and each is referring to underlying transport interface NH independently.
2. BGP: You should have at least one neighbor per Transport Interface and each of these neighbors should either advertise a default route or more specific routes of controller nodes.

When it comes to design your underlay network (all above is about underlay) I recommend you to stick with simplicity and bring all routing decisions to your Virtual Overlay (SD-WAN Fabric) Control-Plane.

In the above scenario, consider deploying an ECMP situation and use Traffic Steering and/or Application-Aware routing to manipulate traffic path.

[JakSmith27]

1) you are following the right approach. Just bear in mind to pick a "public color" for your metro-ethernet segment if that will seat behind a NAT. - you can find details on below link:

I have read about public and private colors but I find the explanation below a bit confusing:
"If you are using a private color and need NAT to communicate to another private color, the carrier setting in the configuration dictates whether you use the private or public IP address"

In this instance our metro-ethernet transport interfaces would use NAT to establish control-connections to controllers but they would not use NAT to establish ipsec tunnels between themselves.  Those circumstances would make it ok to use a private color......I think???

Again, thank you for your answers!

[elesani]

In simple terms, A transport interface using a private "color" can't seat behind NAT while public "color" doesn't have such a limitation. 

Design-wise, there would be no downside to use public "color" over a metro-ethernet or MPLS infrastructure. However, as clearly mentioned in CVD, there should.be no NAT in between endpoints Transport interfaces in case of using private "color".

So, "private color" should work in your scenario  :) 

[JakSmith27]

 Great information!  Thanks Elesani for the insight and prompt responses