cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

SD-WAN Controller Setup Guide (On-Prem, Non Cloud-Managed)

12705
Views
20
Helpful
9
Comments

Benefits

Cisco’s software-defined wide area network (SD-WAN) solution allows user to quickly and seamlessly establish an overlay fabric to connect an enterprise’s data centers, branch and campus locations, as well as colocation facilities in order to improve the network’s speed, security, and efficiency.

Documentation

This configuration example is designed to be used in combination with the official Cisco SD-WAN End-to-End Deployment Guide.

Prerequisites

This configuration guide assumes that the following has already been setup and configured:

  • UCS server with at least the following hardware resources available:
    • vManage
      • 2 CPU
      • 32GB Memory
      • Disk 1: 30GB
      • Disk 2: 100GB
    • vSmart
      • 2 CPU
      • 4 GB Memory
      • Disk 1: 10.5GB (Approx. image size)
    • vBond
      • 4 CPU
      • 2GB Memory
      • Disk 1: 10.5GB (Approx. image size)
    • VMWare ESXI v6.0+ instance for managing VMs and VM networks:
      • One virtual network and switch for VM management and
      • One virtual network and switch for SD-WAN
      • Datastore(s) configured with at minimum 500GB
    • Workstation with network access to UCS server and controllers, with access to a certificate-signing server/software, such as xca. 

Image Download Links

vEdge (vBond), vManage, vSmart:

https://software.cisco.com/download/home/286320995/type

Limitations 

This configuration example only covers the process of installing the SD-WAN controller software images on a VMWare ESXI instance, establishing the transport and management networks for the three controllers to communicate, and ensuring that each controller has a valid certificate installed. This configuration does not go over the process of creating/applying configuration templates or adding edge devices, through vManage.

Supported Platforms

The Cisco SD-WAN Solution is compatible with all Cisco ASR 1000 Series Routers, ISR 1000 Series Routers, ISR 4000 series routers, vEdge Routers, CSR, and ISRv.

License Requirements

A Cisco DNA Essentials license is necessary in order to implement the SD-WAN solution.

Topology

This diagram describes the topology of the configuration which is covered by this setup guide.Screen Shot 2019-09-17 at 1.34.18 PM.png

Step-By-Step Configuration

Configure SD-WAN Component VMs

  1. Navigate to your VMWare ESXI management interface, and select “Virtual Machines” then click “Create/Register VM
  2. For vManage, click “Deploy a virtual machine from an OVF or an OVA file"Screen Shot 2019-09-19 at 11.18.45 AM.png
  3. Enter a name for your vManage instance, and select the downloaded file for vManageScreen Shot 2019-09-19 at 11.19.23 AM.png
  4. Select the datastore where the VM is going to be stored
  5. Select the VM management network for the VM Network, and select “Thick Provisioning” for Data Provisioning. Uncheck “Power on AutomaticallyScreen Shot 2019-09-19 at 11.19.45 AM.png
  6. Click “Finish” to register the vManage VM
  7. Navigate to the left pane and click on "Networking" to add an additional Network Adapter. Click on "Add port group"
    Screen Shot 2020-07-21 at 11.45.38 AM.png

     

  8. In the pop-up window, type "SD-WAN" as a name of the New Port Group and click on "Add"Screen Shot 2020-07-21 at 11.45.55 AM.png

     

  9. Navigate to the Virtual Machines and right click the vManage VM and click “Edit SettingsScreen Shot 2019-09-19 at 11.20.55 AM.png
  10. Click “Add Network Adapter” and select the defined SD-WAN network for Network Adapter 2Screen Shot 2019-09-19 at 11.21.25 AM.png
  11. Click "Add Hard Disk" and select "New Standard Hard Disk"
  12. Update Hard Disk 1's size to be 30GB, and update Hard Disk 2's size to be 100GB, so that vManage has sufficient space to store all controller logs
    Screen Shot 2019-09-19 at 11.21.14 AM.png
  13. Click "Save"
  14. Repeat steps 1-8 fro vSmart and vBond, using the downloaded OVA file for each. When configuring the VMs, ensure that the first network adapter is the VM management network, and the second network adapter is the SD-WAN network

 

Configure Controller IP Addresses

  1. Start all three VM instances for vManage, vBond, and vSmart
  2. Log in to each VM instance using the default username/password: admin/admin
  3. Type the following commands to setup vManage, vSmart and vBond system configurations:
  Command Purpose
Step 1 vmanage# configure terminal Enters global configuration mode
Step 2 vmanage(config)# system System parameter information
Step 2.a vmanage(config-system)# host-name name Specifies name of the device
Step 3

 

vmanage(config-system)# system-ip IP Address

An IP address used internally by the network to identify each device. This is similar to a router ID. These must be private addresses that are unused anywhere else in the network.

 

Example:

vManage: 1.1.1.1

vSmart: 1.1.1.2

vBond:1.1.1.3

Step 4

vmanage(config-system)# organization-name <Press Enter>

(<Organization name>): name

Name of your organization. It must be identical on all the devices in your overlay network, and it must match the name in the certificates for all network devices

 

Example:

(<Organization name>): Cisco systems

Step 5 vmanage(config-system)# sp-organization-name

<Press Enter>

(<Organization name>): name

Name of your service provider. must be identical on all the devices in your overlay network, and it must match the name in the certificates for all network devices

 

 

Example:

(<Organization name>): My Service Provider

Step 6 vmanage(config-system)# site-id number

Numeric identifier of the site in the overlay network. The site ID must be the same for all devices that reside in the same site (i.e. site ID remains the same for vManage, vSmart & vBond).

 

Example:

site-id 100

Step 7.a

 

 

 

 

Step 7.b

vmanage(config-system)# vbond IP address

(for vManage &vSmart only)

 

 

vmanage(config-system)# vbond IP address local vbond

(for vBond only)

IP address of the vBond orchestrator. Must be a public IP address.

 

Example:

vbond 128.x.x.x

 

 

Configure this device to act as the vBond orchestrator.

 

Example:

vbond 128.x.x.x local vbond

Step 8 vmanage(config-system)# commit  Activate the commands in the configuration

    A summary of the above commands, with an example configuration:

configure terminal
hostname vmanage
system-ip 1.1.1.1
organization-name Cisco
sp-organization-name MyServiceProvider
site-id 100
vbond 128.0.0.3
commit

4. Once the controllers are setup, configure the transport VPN on vManage, vSmart and vBond with the help of the following commands:

  Command Purpose
Step 1

vmanage# conf t

vmanage(config)# vpn 0

This is the WAN facing interface i.e. transport VPN
Step 2

vmanage(config-vpn-0)# interface eth0 

(for vManage and vSmart)

 

vbond(config-vpn-0)# interface ge0/0

(for vBond only) 

Enters the configuration mode for an Ethernet WAN interface
Step 3

vmanage(config-interface-eth0)# ip address IP address

(enter IP address with a CIDR prefix. Example: 128.0.0.5/24 )

 

 

vmanage(config-interface-eth0)# no shut

Sets the IP address for the specified Ethernet interface

 

Enables the Ethernet interface, changing its state from administratively down to administratively up

Step 4

vmanage(config-interface-eth0)# tunnel-interface

 

vmanage(config-tunnel-interface)# allow-service all

 

vmanage(config-tunnel-interface)# commit

vmanage(config-tunnel-interface)# exit

vmanage(config-interface-eth0)# exit

 

For vBond:

vbond(config-interface-ge0/0)# tunnel-interface

 

vbond(config-tunnel-interface)# encapsulation ipsec

 

vbond(config-tunnel-interface)# allow-service all

 

vbond(config-tunnel-interface)# commit

vbond(config-tunnel-interface)# exit

vbond(config-interface-ge0/0)# exit

Configure the interface to be a secure DTLS or TLS WAN transport connection

 

Configure the services that are allowed to run over the WAN connection in VPN 0 

Step 5

vmanage(config-vpn-0)# ip route 0.0.0.0/0 IP address

vmanage(config-vpn-0)# commit

Establish a static route to the gateway IP address

 

Example:

ip route 0.0.0.0/0 128.0.0.1

 

    A summary of the above commands, with an example configuration:

configure terminal for vManage and vSmart
vpn 0
interface eth0
ip address 128.0.0.5/24
no shutdown
tunnel-interface
allow-service all
commit
exit
exit
ip route 0.0.0.0/0 128.0.0.1
commit
configure terminal for vBond
vpn 0
interface ge0/0
ip address 128.0.0.3/24
no shutdown
tunnel-interface
encapsulation ipsec allow-service all commit exit exit ip route 0.0.0.0/0 128.0.0.1 commit

Configure Controller Certificates

  1. Login to the vManage web console, by navigating to “https://<vManage Public IP Address>:8443” in a web browser
  2. Once the vManage Dashboard has loaded, navigate to the Administration menu on the right, then select “Settings"1.png
  3. Configure the Organization Name to match the one that was configured on each controller through the CLI
  4. Enter the address that was configured for vBond, with the default port “12346” unless an alternate port was configured
  5. There are many certificate authorization options available, such as using a Cisco Smart Account to automatically sign certificates. For the purposes of this configuration guide, we will select “Enterprise” for Controller Certificate Authorization, to use a custom Root Certificate Authority for signing controller certificates2.png
  6. To begin the process of creating a root certificate, access the certificate-signing server/software on a compatible workstation that has network access to the three controllers

    NOTE: There are many free options for creating and signing certificates, and each will work with the SD-WAN controllers, however, this guide will cover how to do so using the xca certificate-signing software, which is also available for free.

  7. Select “File” > “New Database” and specify a name to create a new .xdb database. Click “Save"5.png
  8. Navigate to the “Certificates” tab in xca, then select “New Certificate” on the right7.png
  9. On the “Source” tab, In the field, “Template for the new certificate” select “[default] CA” then click “Apply extensions,” “Apply subject,” and “Apply all”8.png
  10. On the Subject tab, enter the name “RootCA” in the field for Internal Name. Fill out the remaining details in correspondence with your organization. The “organizationName” and “organizationalUnitName” fields must be identical and should match the organization name configured previously on each of the controllers, as well as the vManage web console9.png
  11. Click “Generate a new key” to generate the private key for the root certificate. Click “Create” to generate the keyScreen Shot 2019-08-23 at 12.14.52 PM.png
  12. Click “OK” at the bottom right corner of the window to create the root certificate
  13. After the certificate has been created, select it and click “Export” on the right to save the certificate in PEM (.crt) formatScreen Shot 2019-08-23 at 12.23.37 PM.png
  14. Using a Terminal window, navigate to the directory where the exported root certificate was stored
  15. Open the certificate using vim by entering “vim rootca.crt” in the Terminal window. Copy all of the certificate text, including the “Begin Certificate” and “End Certificate” line10.png
  16. Navigating back to the vManage console, paste the contents of the root certificate into the setting for “Controller Certificate Authorization” and click “Import & Save”11.png
  17. Open a new Terminal window, and SSH into vManage "ssh admin@vManage IP Address", using the same IP address and credentials used to login to the vManage GUI
  18. Enter the “vshell” command to enter the virtual shell on vManage. Then, navigate to the admin directory by entering the command “cd /home/admin”
  19. Enter the command “vim root.crt” to enter VIM and create a new blank file, titled “root.crt” in order to be able to install the root certificate chain on the controller
  20. Enter Insert Mode in VIM by entering the command “i” then paste the contents of the RootCA certificate, copied from Step 15, into this blank file. Save the file by hitting the escape key, and then entering the command “:wq
  21. Enter the command “exit” to exit the virtual shell. Install the root certificate chain by entering the command “request root-cert-chain install /home/admin/root.crt
  22. Repeat Steps 17-21 for establishing an SSH session with vSmart and vBond to install the root certificate on each controllerPictureA22.jpg
  23. Return to the vManage web console. Use the menu on the left to Navigate to “Configuration” > “Certificates.” Select “Controllers” at the top of the page
  24. Click the three dots at the right of the vManage line to open the options menu for the vManage certificate. Click “Generate CSR” to generate the certificate signing request for vManage14.png
  25. Click “Download” to download the CSR. Rename the downloaded CSR file to match the name of the controller the CSR is for: vManage.csr, vSmart.csr, or vBond.csrScreen Shot 2019-08-26 at 12.59.18 PM.png
  26. Return to xca to sign the CSR. To begin, select the “Certificate Signing Requests” tab at the top of the window. Then click “Import” on the right to import the CSR that was downloaded from the vManage web GUI
  27. Once imported, right click the name of the CSR inside of the xca window and click “Sign” to begin signing the certificate15.png
  28. On the “Source” tab, under the field “Signing” ensure that the RootCA certificate that was created previously is selected as the certificate to be used for signing
  29. In the field, “Template for the new certificate” select “[default] HTTPS_client” then click “Apply extensions,” “Apply subject,” and “Apply all”16.png
  30. On the “Extensions” tab, in order to ensure that the certificate becomes valid immediately once installed on the controller, the time should be updated to be the current time on the respective controller, at this time when the certificate is being signed
  31. Utilizing the “show clock” command when accessing the controller through a CLI will show the current time of the controller. Configure the time in xca, in the “Not before” field to be the same time
  32. Click “Apply” on the right to update the “Not after” field to reflect the updated time. Then click “OK” at the bottom of the window to sign the certificate17.png18.png
  33. After signing the certificate, select the “Certificates” tab at the top of the window in xca. Click the drop-down arrow to the left of the RootCA certificate that was generated earlier. The certificate that was just signed should now appear underneath the root certificate
  34. Right click the certificate whose name corresponds to the controller for which the certificate is being signed for (vManage, vSmart, vBond) and select “Export” > “Clipboard”19.png

    NOTE: In order to use a different workstation to sign certificates in the future, you must export the original RootCA certificate as a “File” and then for the Export Option select “PEM + Key” to get a single PEM file containing the certificate contents as well as the private key

  35. Returning to the Certificates page on the vManage web console, click “Install Certificate” in the top right corner of the page
  36. Paste the contents of the signed certificate that were just copied from xca and then click “Install” to install the certificate for the controller20.png
  37. After the certificate installation has finished, there should be a message indicating that it was successfulPictureA37.jpg
  38. Next, use the menu on the left of the vManage console to navigate to “Configuration” > “Devices”. Select “Controllers” at the top of the page
  39. Click “Add Controller” and select “vBond"22.png
  40. Enter vBond VPN0 IP address for vBond that was previously configured through the CLI, as well as the credentials that are used for logging into vBond
  41. Ensure that the “Generate CSR” box is NOT checked, then click “Add” to add the controlleradd vBond to vManage.png
  42. Repeat steps 39-41 to add the vSmart controller as well
  43. Repeat steps 23-37 to generate the CSRs, as well as sign and then install the certificates for both vBond and vSmart

Verification

Upon completing the controller setup and certificate installation, the vManage web console dashboard should indicate that each of the three controllers (vManage, vSmart, and vBond) are up and reachable. Additionally, the dashboard should indicate that there are no invalid certificates.Screen Shot 2019-08-26 at 1.33.46 PM.png


The Certificate Configuration page on the web console should also indicate that the certificate has been installed on vBond and that vBond has been updated for vSmart and vManage. All three controllers should have a certificate serial and their vEdge List Status should be in “Sync”.

Screen Shot 2019-08-27 at 11.20.50 AM.png

 

Using the “show control connections” command while an SSH connection to one of the controllers has been established can be useful in troubleshooting connection issues between controllers. The command should return a list of successful connections to the other controllers.

Screen Shot 2019-09-09 at 12.42.50 PM.png

 

Troubleshooting

Troubleshoot Control Connections: https://community.cisco.com/t5/networking-documents/sd-wan-routers-troubleshoot-control-connections/ta-p/3813237

 

 

 

 

 

Comments
Beginner

Hi

I already have on-Prem controllers set up and ASR joined to it. Until now my testing has been limited to local test network. Now I want to test this further and join ASR through Internet. Do you know of some best practises document on how to position different controller personas on-prem. I've read many documents and I am still not sure where in the network should I position my controllers. I assume vBond needs to be somewhere in the DMZ with a public IP address to be reachble via the internet. vManage and vSmart can probably be inside behind the NAT. Now, I suppose I need 1:1 NAT and certain ports opened.

Is there any guide/resource explaining this in detail?

Beginner

Hi

Did you find solution to this problem?

 

Regards

 

Beginner
No not yet.


Beginner
Hello hadhaliw,
Thank you for sharing this document. The deployment is done as per this document. The challenge is where to place the controllers so that they can be accessed from service provider's network. Do not want to leak Dc routes on service provider's network.
Beginner

Hello hadhaliw,

Thank you for sharing this document. The deployment is done as per this document. The challenge is where to place the controllers so that they can be accessed from  service provider’s network. Do not want to leak  Dc routes on service provider’s network.

Controllers are installed on Service Side VPN.

 

Regards

Cisco Employee

@faisal.memon @rudimocnik 

 

Although we do not have any document that describes the best practices for controller placement, the beauty of our SD-WAN solution is that it gives you a lot of flexibility in where you can place your controllers.

 

If, however, the goal is to prevent DC routes from leaking to the service provider network then we would recommend placing your controllers behind a firewall and NAT their addresses on the firewall. This would only advertise the NATed addresses to the service provider network. It's also worth mentioning that only vSmart is responsible for knowing routes and advertising them to edge devices within the network, and that vBond still needs to be reachable by an edge router that has not yet joined the SD-WAN fabric in order for the onboarding process to work. 

Beginner
Thank you
Beginner

@hadhaliw Do you have routable "system-ip" and use that for management instead of vpn512 interface? What is recommended?

 I am a bit confused on the new version (19.2.1) of controllers when you add vbond in the vmanage settings menu it says "vBond management IP address". What IP is that? System IP, VPN512 or VPN0 IP (pre-NAt, post-NAT). I am aware that system IP is under VPN0 and could be used for management but, I suppose that is not recommended. 

 

Thanks

Rudi

Beginner

Furthermore can you elaborate on the IP assignment for vBond when all controllers sit on private IPs and have 1:1 NAT. I am following design as described in one of the ciscolive presentation with controllers in two seperate private networks with all having private to public 1:1 mapping and NAT on firewall. vManage and vSmart sit on 192.168.20.0/24 with both having public color and communicate locally. Both have vBond public IP (1.1.1.1) in their system configuration. vBond sits on 192.168.10.0/24.

 

Now, the guide on cisco page says "Configure an interface in VPN 0, to connect to the Internet or other WAN transport network. In Releases 16.3 and later, the IP address can be an IPv4 or an IPv6 address. In earlier releases, it must be an IPv4 address. Ensure that the prefix you configure for the interface contains the IP address that you configure in the vbond local command."

 

My post-NAT public IP of vBond is 1.1.1.1 and it maps to 192.168.10.1/24. On the vBond the VPN0 interface would have IP 192.168.10.1/24 configured and tunnel interface command with public color. However, the system config has "vbond 1.1.1.1 local" command. vManage, vSmart and all routers point to vbond 1.1.1.1. 

Is that correct? Note that all IPs are arbitrary.

 

Thanks 

Rudi

 

Content for Community-Ad