topic Re: Ask the Expert- SD-WAN fundamentals and implementation in SD-WAN and Cloud Networking

Ask the Expert- SD-WAN fundamentals and implementation

Cisco Moderador — Wed, 21 Aug 2019 00:16:20 GMT

This topic is a chance to discuss more about SD-WAN, it's foundations and inner mechanisms as well as its correct design and implementation to achieve desired business outcomes. Software-Defined WAN (SD-WAN), is a popular technology and this event is aimed to help engineers/customers/partners understand the benefits and possible advantages that its implementation can bring.

To participate in this event, please use thebutton below to ask your questions

Ask questions from Monday 19th to Friday 30th of August, 2019

Featured expert

David Samuel Peñaloza Seijas works as a Senior Network Consulting Engineer at Verizon Enterprise Solutions in the Czech Republic. Previously, he worked as a Network Support Specialist in the IBM Client Innovation Center in the Czech Republic. David is an expert interested in all topics related to networks. However, he focuses mainly on data centers, enterprise networks, and network design, including software-defined networking (SDN). David has a long relationship with Cisco. He has been a Cisco Instructor for the Cisco Academy and was recognized as a Cisco Champion and a Cisco Designated VIP for 2017, 2018 and 2019. David holds a CCNP R&S, CCDP, CCNA Security, CCNA CyberOps and a CCNA SP certification. Currently, he is preparing for a CCDE.

David might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the SD-WAN community.

Find other events https://community.cisco.com/t5/custom/page/page-id/Events?categoryId=technology-support

**Helpful votes Encourage Participation! **
Please be sure to rate the Answers to Questions

Re: Ask the Expert- SD-WAN fundamentals and implementation

Seth Beauchamp — Tue, 20 Aug 2019 19:43:23 GMT

Are there any best practices regarding MSP design specifically with access into the SP cloud for privately addressed services, such as a cucm cluster within a customer vrf? Is it possible to use something such as a vedge 5000 to aggregate multiple customers on the public side and split them into vrfs back toward the SP mpls while still keeping a separate control plane instance per customer (vmanage)?

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Tue, 27 Aug 2019 10:57:15 GMT

Hello Seth,

The common SD-WAN deployment in SP networks can be divided in the following 2 groups:

Customer dedicated vManage
Shared vManage - Multitenancy

Each one of them with their advantages and drawbacks. One of the advantages of the shared vManage is to have a centralized platform with all the customers properly segmented within the controller (i.e. multitenancy). This translates into less devices/servers/controllers deployed, thus, less resources used. It does fit into low-scale deployments with several customers having a handful of sites to support. Might be even ideal for PoCs as well.

The main drawback of it being scalability: Each vSmart controller supports a limit of around 5400 control connections (and those are shared when deployed in multitenancy mode), please note that each TLOC will establish a control connection. Furthermore, doing the math by increasing the number of TLOCs in each vEdge will cut down that limit substantially:

One TLOC - 5400 vEdges
Two TLOCs - 2700 vEdges
Three TLOCs - 1800 vEdges

Each vManage controller supports a limit of around 2700 vEdges routers.

Regarding the services: Usually this is done via L3VPNs in the underlay, leaked into the transport VPN (0) via a vEdge deployed as a CE device in the MPLS network in a management tenant, also could be called control center (vManage, vSmart, vBond and Management vEdge). It could be implemented specifically for a customer (like the one you have mentioned - only existing services in a customer VPN) or shared services offered to all the customers (a "services" VPN to host services like DNS or NTP, for instance).

Is it possible to use something such as a vedge 5000 to aggregate multiple customers on the public side and split them into vrfs back toward the SP mpls while still keeping a separate control plane instance per customer (vmanage)?

This would imply deploying a device to work in the overlay to aggregate that traffic, similar to what a PE does currently in the underlay. In all honesty, I have not seen this deployment, usually the overlay - either per customer or per group of customers - is deployed to operate as separate/segmented as possible from the underlay (MPLS/DIA transport). The goal is to provide each of the customers with their own/dedicated environment, using vSmart, vManage and vEdges as your segmentation tools (vBond can be shared between customers in a multitenancy deployment), while using the underlay (which is already shared between customers) purely as a transport, and leaking prefixes to the transport VPN (using BGP) as needed to provide a service or a set of services to the customer requesting them.

Hope this is useful!

David

Re: Ask the Expert- SD-WAN fundamentals and implementation

Hilda Arteaga — Thu, 22 Aug 2019 00:23:24 GMT

Hi @David Samuel Penaloza Seijas, thanks for sharing your knowledge on this session.

Could you please help to answer the following question from @jackfarrer

Official post link.

Re: Ask the Expert- SD-WAN fundamentals and implementation

rbncarvalho — Thu, 22 Aug 2019 09:40:10 GMT

Hi @David Samuel Penaloza Seijas,

I'm trying to figure out how to use service insertion, and the main difficulty here is the redirection policy itself, as well as how the FW should be placed.

I've have the Hub router announcing two different services, netsvc1 and netsvc2 which are inside and outside interfaces of the firewall. I'm doing it like this because you cannot have multiple "FW" type service on the same VPN, so only one IP can be announced, and I confess that I don't have the knowledge on how to have the FW inspecting ou applying rules on one interface and send the traffic back on the same interface.

The problem is that this policy is a Custom Control Policy, and such policy needs to coexist with the Custom Topology policies in place, for Hub and Spoke topology.

Do you have some information or experience in implementations like this, and how should the FW work, routed or transparent.

I've watched David Klebanov videos on this, but it's not clear how the FW is placed here.

Also even though the vSmart receives the services, they are not reflected back to the routers participating on that service VPN, you can see the label towards the destination remains the same, even though on the vSmart the label for that specific service is different.

Thank you for any help you may provide,

I salute you for this initiative

Best Regards,

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Thu, 22 Aug 2019 11:28:08 GMT

Hello @jackfarrer - not a docker expert here.

Seems to me that despite the fact you have created the 20G drive, the storage driver is not able to see it and therefore it cannot mount it when you try to install the image. I could be blatantly wrong on this (the special device raises some questions), apologies in advance.

Are you able to provide more logs? Is there anything else observed by you?

I will try to replicate your intent and come back with my results.

Thanks.

Re: Ask the Expert- SD-WAN fundamentals and implementation

jackfarrer — Thu, 22 Aug 2019 14:14:59 GMT

David,

I'm not a Docker expert either, so I'm not sure where to go to get any additional logs or information. I did have issues with the second drive on the vManage server I spun up. I used qemu-img to create the image, but the vManage server was not able to find a partition and would constantly reboot. I ended up attaching the image to a basic Linux VM and using fdisk to partition it and then attached it back to the vManage and it was then able to format it. I tried the same process with the vContainer, but the result was the same, the image install failed. When I drop to vshell on the vContainer and try to run any of the normal Docker commands, I get the message that it cannot connect to the Docker daemon even though the messages log shows the daemon was started.

local7.info: Aug 16 17:55:45 vcontainer SYSMGR[329]: %Viptela-vcontainer-SYSMGR-6-INFO-200017: Started daemon docker @ pid 725 in vpn 0

Permissions on the path for the second drive are locked down to root with execute for everyone else, so I'm not sure if that's an issue or not.

drwx--x--x 11 root root 4096 Aug 16 17:58 docker

I'm also not sure if the standard Docker commands are applicable here, since the image install is done via the Viptela CLI

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Thu, 22 Aug 2019 19:30:26 GMT

Hello @rbncarvalho

The approach is right, you differentiate the interfaces by advertising them in separate services. You would then link those services with a site list, so you can advertise the respective service (untrusted interface/zone) to the remote offices or towards the DC or HQ (trusted interface/zone). That way, remote offices will only see the service pointing to the right interface so the traffic does not get dropped. Please note that the site lists are supposed to be planned properly for this to be in place.

The ability to advertise services via OMP allows you to place the firewall at any point in the network as long as its within the SD-WAN fabric. That being said, the advantage of this approach is to designate any specific place (being a regional hub or a colo facility - so delay and jitter are reduced and application performance impact minimized/improved/optimized) to place/deploy/install the firewall appliances or any other device providing the service you are advertising, reducing then expenses for dedicated firewalls in remote offices. Its an sweet spot between centralized FW in a DC - consuming DC's computing and BW resources - and dedicated FW per remote office (high expenses involved for appliances).

For the case of the Hub-and-Spoke topology, would be advantageous to place the FW in the hub, and its the natural place where the traffic flow is headed and does not incur in awkward/confusing/complex redirecting.

The firewall can be either transparent or routed. I personally have bad experiences with transparent firewalls, they are stateful devices anyway (regardless of the mode they operate) and its important to know where they are so you are able to pinpoint problems and troubleshoot efficiently.

Could you provide outputs? What can you see when you issue "show omp services" command?

Thanks!

David

Re: Ask the Expert- SD-WAN fundamentals and implementation

daniel.dib — Fri, 23 Aug 2019 05:13:03 GMT

To add to @David Samuel Penaloza Seijas excellent explanation, here are some more considerations for you @rbncarvalho

As David mentioned, the firewall could be placed anywhere. However, remember that service chaining is a special form of forwarding where we bypass the normal forwarding logic. For this reason, the firewall, or the service that is provided, needs to be "directly connected" to a vEdge. What does that mean? It doesn't mean that the service needs to be physically connected to the vEdge but that the vEdge must be in the same subnet as the service. There can be no L3 hops between the vEdge and the service. You can either stretch L2 by having a switch in between or you can tunnel traffic, for example by using IPSec towards the FW. I recommend you read my blog if you want to see an example of a service chaining topology: http://lostintransit.se/2019/08/20/the-tale-of-the-mysterious-traceroute/

When you advertise a service from a vEdge, it gets advertised to the vSmart(s). Services are NOT reflected to the other vEdges in the network. What you do is to create a control or data policy that based on your match criteria modifies vRoutes that get advertised from the vSmart to the vEdges. Keep in mind that the service VPNs used are actually also services. VPNs are identified by labels and packets sent in data plane are encapsulated with MPLS label. Let's say that we have a route 10.0.0.0/24 in VPN 10 and it has label 1003. Now, when you enforce the service for traffic destined to 10.0.0.0/24, what happens is that the label changes to say 1005. This will also mean that the TLOC (next-hop) changes to be for example a DC instead of a branch.

How does this all work under the covers? When the traffic flow starts, normal forwarding is used where the service label has been applied. When traffic reaches vEdge where FW is located, we can't use normal forwarding as that would just send the traffic onwards. The vEdge looks at the label and sends traffic towards the FW, when traffic returns back, normal forwarding, according to IP route lookup occurs.

Can you hairpin traffic on a FW? Absolutely yes depending on the FW of course. You don't necessarily need two interfaces. It depends on the design. Are you just filtering traffic between branches or between branches and say a DC? If you have two interfaces, you need to advertise to services. Like you said yourself, you could do this by advertising netsvc1 and netsvc2.

If you have access to the SD-WAN mastery course, I recommend you check David Klebanov's videos on service chaining. I also recommend you read through the following links:

https://sdwan-docs.cisco.com/Product_Documentation/Software_Features/Release_18.4/07Policy_Applications/02Service_Chaining

https://sdwan-docs.cisco.com/Product_Documentation/Software_Features/Release_18.4/07Policy_Applications/02Service_Chaining/Service_Chaining_Configuration_Examples

I hope this helps.

Best regards,

Daniel

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Fri, 23 Aug 2019 06:37:15 GMT

Daniel,

Thanks a ton for complementing! Fantastic contribution!

Re: Ask the Expert- SD-WAN fundamentals and implementation

Hilda Arteaga — Fri, 23 Aug 2019 15:58:25 GMT

Hi @daniel.dib and @David Samuel Penaloza Seijas

Thanks for helping to solve this question

Re: Ask the Expert- SD-WAN fundamentals and implementation

juraj.papic — Fri, 23 Aug 2019 16:00:20 GMT

HI all,

I have a multi-tenant lab, and I'm trying to work with Ansible but I'm getting errors because of the multi-tenant, I would like to know if someone was able to work with Ansible and multi-tenant.

Thanks.

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Fri, 23 Aug 2019 16:36:11 GMT

Hello @juraj.papic

Could you please elaborate more your statement? what kind of error? what are you trying to do/achieve? are you able to provide logs?

Thanks!

Re: Ask the Expert- SD-WAN fundamentals and implementation

rbncarvalho — Mon, 26 Aug 2019 09:08:39 GMT

Hi @David Samuel Penaloza Seijas and @daniel.dib.

Thank you for your feedback.

I've managed to get the service insertion to work, but I had to do it without the custom control policy because I have a custom topology in place and the Site ID were in use.

However I used a data policy for this, and thanks to your inputs this started to work.

Thank you guys for your support.

Best Regards,

Rúben

Re: Ask the Expert- SD-WAN fundamentals and implementation

kenneth.meyers — Mon, 26 Aug 2019 12:30:05 GMT

Question around control machine limits and ztp:

We’re looking at a fairly large SD-WAN rollout and I was wondering on limits of the controllers around bfd sessions, control connections etc. We’ll most likely have a hub and spoke type configuration as “branches” do not need connectivity between each other. What I’m trying to find information on is how many bfd type sessions a “hub vEdge” device can accommodate in additional to the capacity of vSmarts around control connections to begin to look at sizing things appropriately (including failover of one “Hub” or vSmart device and how this plays into overall design of the overlay control plane).

With respect to ztp, if we would like to deploy our own certificates (in house CA) would we need to “touch” each vEdge before shipping to remote site (or have on-site personnel install a certificate on the device) before the vEdge contacts vBond? Is ztp possible with self-signed certificate requirement?

Thanks,

Re: Ask the Expert- SD-WAN fundamentals and implementation

Seth Beauchamp — Mon, 26 Aug 2019 22:36:20 GMT

We are a SP looking to use a compute cluster where we deploy one vedge cloud per customer. We would like to put all customers in a shared underlay for the transport interface, that is a single vlan with a /24 and each vedge gets an IP in that subnet. We also would like our customers to have access to their own vManage to make changes. A danger I see here is a customer changing their vedge cloud transport IP to an IP that overlaps with another customer, allowing customer A to bring down customer B. What could I do to prevent that?

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Tue, 27 Aug 2019 09:28:23 GMT

Those are fantastic new! Glad we could help somehow :D

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Tue, 27 Aug 2019 09:50:04 GMT

Hello @Seth Beauchamp

We would like to put all customers in a shared underlay for the transport interface, that is a single vlan with a /24 and each vedge gets an IP in that subnet. We also would like our customers to have access to their own vManage to make changes.

Being an MSP where your business is about offering transport and sharing the same infrastructure with all your customers, this is always a risk. That being said, there are techniques (mostly relying ion virtualization) to segment your customers so their failure domain is contained and separated, hence, not affecting other customers sharing the same infrastructure.

Are you trying to save IP addresses? Allowing the customers to share the same broadcast domain is dangerous, involves fate sharing. Unless you can enforce it somewhere else in the infrastructure (many access lists or similar tools) can only be cumbersome and posse as a highly complex operational model. Is there a hard constraint? is there any other reason behind this request? cant you simply segment them through subnetting? maybe even PVLANs come to my mind if you need/must go down this road, alas, it would not prevent a customer from using an unauthorized IP address and affecting some other customer's operation. The best is always to keep them "together but not scrambled" - with their own playground.

David

Re: Ask the Expert- SD-WAN fundamentals and implementation

David Samuel Penaloza Seijas — Tue, 27 Aug 2019 14:47:25 GMT

Hello @kenneth.meyers

Effectively, as you have mentioned, one of the ways to scale the solution is to rely on a hierarchical model to restrict the tunnels between sites - the solution works in an any-to-any fashion which taxes scalability as the state is held in the network even if those tunnels are not needed.

Quoting a previous reply in this thread:

The main drawback of it being scalability: as each vSmart controller supports a limit of around 5400 control connections (and those are shared when deployed in multitenancy mode), please note that each TLOC will establish a control connection. Furthermore, doing the math by increasing the number of TLOCs in each vEdge will cut down that limit substantially:

One TLOC - 5400 vEdges
Two TLOCs - 2700 vEdges
Three TLOCs - 1800 vEdges

Regarding the vEdge BFD session limits:

As far as I know, the ZTP process relies on certificates signed by a CA, being Symantec or your enterprise root CA chain, which is then installed in vManage (and all vEdges would need to have the root certificate as well - which means touching them). Have not seen this being accomplished with a self-signed certificate.

Hope that helps.

David

Re: Ask the Expert- SD-WAN fundamentals and implementation

kenneth.meyers — Tue, 27 Aug 2019 11:40:38 GMT

Thanks David,

Would it be safe to assume that a "HUB" type vEdge device would have the same scaling limitations as the vSmart controllers previously mentioned.

One TLOC - 5400 vEdges
Two TLOCs - 2700 vEdges
Three TLOCs - 1800 vEdges

In the hierarchical model we're wondering how many "spoke vEdges" can connect to the "HUB vEdge" before we start taxing the capabilities of the Hub device with respect to BFD and IPSEC sessions.

Thanks again.

Regards