cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
18159
Views
5
Helpful
1
Comments
imclaugh
Cisco Employee
Cisco Employee

Contributed by Nikolai Pitaev, TME, Enterprise Business

 

This document describes interconnection between Cisco SD-WAN and Azure Virtual WAN.

Enterprise customers looking for an optimal way to interconnect on-prem locations like branches and data centers with Azure cloud infrastructure. Cloud to on-prem interconnection should be seamless, automated and secure. Main benefits are segmentation and end-to-end policy for Application Quality of Experience (AppQoE) and security.

With segmentation different departments can access their own cloud infrastructure in a secure and efficient way. AppQoE will make sure, that the end users have the best cloud access.

Cisco1.pngThis blog will describe benefits of the SD-WAN and Azure Virtual WAN interconnection, technical implementation steps and provide basic troubleshooting help.

Why to interconnect SD-WAN and Virtual WAN?

With the current trend to move workloads, applications, data centers to the cloud, the question why becomes more and more irrelevant. It is now more the question on “how to interconnect”, not “why”. The question “how” is covered in the next section.

While interconnection via customer managed VNet is a viable option, interconnection via Azure Virtual WAN enables cloud native support for transitive connectivity via a cloud native hub and spoke architecture.

Main benefits of the interconnection of SD-WAN and Virtual WAN are:

  • Cloud native Transit hub architecture support.
  • Automated and easy connectivity provisioning to the most optimal cloud entry point for all datacenters and hub locations.
  • Dynamic routing, multi pathing and deterministic failover behavior using OMP and SD-WAN Secure Extensible Network (SEN) policies.
  • Regional Hubs interconnecting multiple vHubs and regions via SEN.
  • Application and data telemetry in and out of both clouds for reporting/chargeback.

What is Azure Virtual WAN?

Azure Virtual WAN is an Azure networking service that provides optimized and automated branch connectivity to, and through, Azure. Azure Virtual WAN hubs can be deployed in various Azure regions and serve as cloud interconnectivity hubs that you can choose to connect your branches, VNets and Users to.

Please refer to Virtual WAN main page for more details.

Also see  SD-WAN connectivity architecture with Azure Virtual WAN, where the various interconnect models are described.

What is Cisco SD-WAN?

Cisco SD-WAN is a secure, cloud scale architecture that is open, programmable and scalable. Managed through the Cisco vManage console users can quickly establish an SD-WAN overlay fabric to connect data centers, cloud, branches, campuses, and colocation facilities to improve network speed, security, and efficiency.

Please visit the Cisco SD-WAN landing page for more information.

Interconnecting Cisco SD-WAN with Azure.

Enterprises that are looking to interconnect their Cisco SD-WAN with Azure could do it in one of following two models. 

  1. Interconnect SD-WAN with Azure via customer managed VNet, also known as Transit VNet
  2. Interconnect SD-WAN with Azure via Azure Virtual WAN

In the following sections we describe how to connect Cisco SD-WAN with Azure using the above models. Each model in turn has several connectivity options.

1.) How to interconnect Cisco SD-WAN with Azure VNets?

In this section we describe how to connect Cisco SD-WAN with Azure using customer managed VNet as the entry point. There are 3 different cloud and on-prem connectivity options under this model

  1. Non SD-WAN Interconnection: Dynamic Multipoint VPN (DMVPN)
  2. SD-WAN Interconnection: Do-it-yourself
  3. SD-WAN Interconnection: Using Cloud On-Ramp to enable a Transit Architecture  

 

  1. Non SD-WAN Interconnection: Dynamic Multipoint VPN (DMVPN)

If the existing WAN network is not SD-WAN capable, then the following solution can be used for on-prem to cloud interconnection: Azure Transit VNET DMVPN on Cisco Cloud Services Router 1000v (CSR1000v). Cisco Transit VNet solution on Azure uses a pair of Cisco CSR1000v virtual routers acting as DMVPN Hubs in active-active mode. The spoke VNets also have a Cisco CSR1000v acting as DMVPN Spoke connecting to both the CSR1000v devices in the Hub VNet through overlay routing such as EIGRP and BGP. This solution does not require manual configuration and is fully automated. Once deployed, this solution automatically creates dynamic spoke-to-spoke IPsec tunnels in an on-demand fashion.

    2. SD-WAN Interconnection: Do-it-yourself

However, the current trend to SD-WAN networks is obvious, so the question is how to interconnect SD-WAN networks with Azure. The simplest option is “do-it-yourself model”, where each host VNet has a dedicated virtual SD-WAN Router. The picture below describes this “do-it-yourself” model:

 

Cisco2.png

In case of few host VNets, this option meets all the requirements and can be easily implemented. Obviously, it does not scale from an economic standpoint with the increasing number of host VNets.

  3. SD-WAN Interconnection: using Cloud On-Ramp

This very popular option uses Cisco SD-WAN feature Cloud onRamp for IaaS, which automatically creates a transit VNet, instantiates two SD-WAN routers on it and maps host VNets to the transit VNet. All accomplished in few minutes from vManage – SD-WAN Network Management System.

The main benefit of this option is automation – the whole process is done automatically. The picture below illustrates Cloud onRamp for IaaS solution with the Transit VNet as central point:Cisco3.png

 

b.) SD-WAN Interconnection using Azure Virtual WAN

The newest option for the SD-WAN interconnection with Azure is with Azure Virtual WAN. This option has benefits in terms of scale and can be more cost effective depending on bandwidth requirements. The key differences to the previous options are the usage of Azure Virtual WAN and Virtual WAN hub(s), which serve as central connection points for host VNets and SD-WAN routers.

There are two possible scenarios for connecting SD-WAN to Azure Virtual WAN:

  1. Born-on-Prem SD-WAN to Azure Virtual WAN
  2. Born-in-Cloud SD-WAN to Azure Virtual WAN

The picture below summarizes the main difference between two scenarios.

 

Possible Scenarios for Connecting SD-WAN to Azure Virtual WAN.png

In the first case (“born on-prem”), physical on-prem SD-WAN routers (usually located at a central location such as On-prem Data Centers) establish standard IKE-based IPSec tunnels to vHub directly, run BGP and exchange routing information between cloud infrastructure and SD-WAN network using BGP-OMP redistribution. Connectivity from remote locations to the Central location is possible using SD-WAN IPSec tunnels.

In the second use case (“born in the cloud”), virtual SD-WAN routers (located on an Azure VNet) establish standard IKE-based IPSec tunnels to vHub directly, run BGP and exchange routing information between cloud infrastructure and SD-WAN network using BGP-OMP redistribution. Likewise, connectivity from remote locations to the virtual SD-WAN router is possible using SD-WAN IPSec tunnels. This option extends all the benefits from SD-WAN as AppQoE and other SD-WAN features closest to the Cloud and consequently, is the preferred choice.

The following table summarizes main differences between two connection options.

 

Born on-prem

Born in the cloud

Endpoints

On-prem physical SD-WAN router connects to vHub directly.

On-prem physical SD-WAN router connects to virtual SD-WAN.

Transport over Internet

Standard IKE-based IPSec tunnel

SD-WAN tunnel

Use case

Many branches need direct connected to the nearest Azure location without requirements for regional/hierarchical network structure.

Extending SD-WAN fabric into Azure and using SD-WAN features like AppQoE (FEC, TCP Opt) on the way to Azure. Regional/hierarchical network structure in place.

 

Interconnecting SD-WAN with Azure Virtua WAN – step-by-step description

The following steps are required to successfully interconnect SD-WAN with vHub for the “born in the cloud” use case. The “born on-prem” option will follow similar configuration, it is a subset of the “born in the cloud” workflow described below.

  1. Bring up SD-WAN virtual router (CSR1000v or vEdge Cloud) instance(s) on Azure and prepare it for the interconnection with Virtual WAN:
  • bring up a new SD-WAN virtual router or use an existing one.
  • use a "basic" template on vManage just to have basic connectivity from vManage to this virtual SD-WAN router.
  1. If you do not have required infrastructure on Azure, please follow the step-by-step described on the Azure tutorial mentioned here. You will need to have created:
  • Virtual WAN
  • Hub
  • VPN site

Once you have sites successfully associated your VPN site and your hub, you can proceed to the next step. The following screen shot demonstrates such association:

Cisco4.jpg

 

  1. Once you have Azure portion of the configuration done, you can proceed with the IKE-based IPSec and BGP configuration in vManage. You will need information about IPSec and BGP IP addresses, password, etc., which you can download from virtual WAN Overview Section. Use the provided information for the creation of a new vManage Template, which will have IPSec and BGP configuration in the Service VPN. Here is an example with CLI equivalent:

vpn 3

   dns 8.8.8.8 primary

   router

    bgp 65000

     no shutdown

     address-family ipv4-unicast

      redistribute connected

      redistribute omp

     !

     neighbor 192.168.33.6

      no shutdown

      remote-as     65515

      update-source ipsec6

      ebgp-multihop 255

      address-family ipv4-unicast

      !

     !

    !

   !

   interface ipsec6

    ip address 192.168.3.2/30

    tunnel-source-interface ge0/0

    tunnel-destination      52.159.168.107

    ike

     version      2

     rekey        28800

     cipher-suite aes256-cbc-sha1

     group        2

     authentication-type

      pre-shared-key

       pre-shared-secret $8$x0MowET4rOG2Cps1fJVBItC3e+3lIQB1eHNDV38tfi

      !

     !

    !

    ipsec

     rekey                   3600

     replay-window           512

     cipher-suite            aes256-gcm

     perfect-forward-secrecy group-16

    !

    mtu                     1400

    no shutdown

   !

   interface loopback3

    ip address 10.121.3.1/32

    no shutdown

   !

   ip route 192.168.33.0/24 192.168.3.1

  !

Make sure you have eBGP multihop configured, otherwise BGP session will be not be established.

  1. Attach the SD-WAN Router to the new vManage template. Please refer to the full SD-WAN Edge router configuration in appendix for details.

Automation

Currently, automation is available for Cloud onRamp for IaaS using Transit VNet, but not for Azure vWAN. As of now, a simple configuration in vManage on the SD-WAN side and on Azure side is required for the interconnection. Both parts can be automated.  Azure PowerShell or open source tools like Terraform vWAN can be used to script the configuration of vWAN / vHub.

Cisco and Microsoft are working toward enhancing the existing automation in Cloud OnRamp for the interconnection with the Azure Virtual WAN. Customers will soon be able to have all networking resources in the Azure Virtual WAN up, provisioned and connected with a few clicks of a button. Please follow the Cisco and Microsoft blogs to stay up to date with the latest product announcements.

Securing the interconnection

In many cases a very common security requirement is to inspect all the traffic leaving or entering vHub with a Firewall. The example below shows SD-WAN router vEdge 1 connected to the SD-WAN fabric on the left side and to a virtual firewall on the right side. Firewall is connected to vHub. In this scenario, SD-WAN router will establish a BGP session to the firewall and the firewall will inspect all traffic to and from the SD-WAN router.

Cisco5.png

 

Pro’s:

  • Single FW instance – less cost
  • All traffic between SD-WAN Overlay and workloads on the Cloud is protected
  • All the complexity of building tunnels with Cloud provider is transferred to the FW

Con’s:

  • Traffic between VNets is not protected

Another firewall deployment option is “shared services VNet”, which will have the firewall in a separate VNet as shown in the following diagram.

This is the most scalable option and provides most of the benefits required by customers, including:

  • Protection for traffic between VNets
  • Protection for traffic between Overlay and VNets
  • No need to have a single firewall per VNet
  • Cisco SD-WAN does not handle the complexity of configuring traffic path through the firewall.

Please refer to the following document, which shows how to steer traffic from a branch (on-premises site) connected to the Virtual WAN hub to a Spoke virtual network (VNet) via a Network Virtual Appliance (NVA).

Troubleshooting

IPSec Tunnel state verification in vManage:

IKE-based IPSec tunnels: go to the appropriate SD-WAN Router's dashboard (Monitor -> Network) and review IKE-based IPSec Tunnels. In the following example, you see 3 tunnels established - scroll to the right and look for the state. It should be "IKE_UP_IPSEC_UP":

C7.jpg

 

Verify connectivity in Azure - check vWAN Overview and Health. You should see connection state (green) and traffic volume as shown below:

C8.jpg

Appendix – full SD-WAN router configuration

EXP-BR21-vEdge# sh run

system

 host-name               EXP-BR21-vEdge

 system-ip               121.1.1.1

 site-id                 121

 admin-tech-on-failure

 no route-consistency-check

 sp-organization-name    exp-lab-dmz

 organization-name       exp-lab-dmz

 clock timezone America/Los_Angeles

 vbond 173.37.56.174

 aaa

  auth-order local radius tacacs

  usergroup basic

   task system read write

   task interface read write

  !

  usergroup netadmin

  !

  usergroup operator

   task system read

   task interface read

   task policy read

   task routing read

   task security read

  !

  user admin

   password $6$....HP1

  !

  user npitaev

   password $6$...E0

   group    netadmin

  !

 !

 logging

  disk

   enable

  !

 !

!

omp

 no shutdown

 graceful-restart

 advertise bgp

 advertise connected

 advertise static

!

security

 ipsec

  authentication-type sha1-hmac ah-sha1-hmac

 !

!

vpn 0

 dns 8.8.8.8 primary

 interface ge0/0

  ip dhcp-client

  tunnel-interface

   encapsulation ipsec

   color biz-internet restrict

   no allow-service bgp

   allow-service dhcp

   allow-service dns

   allow-service icmp

   allow-service sshd

   no allow-service netconf

   no allow-service ntp

   no allow-service ospf

   no allow-service stun

   allow-service https

  !

  no shutdown

 !

!

vpn 1

 router

  bgp 65000

   address-family ipv4-unicast

    maximum-paths paths 4

    redistribute connected

    redistribute omp

   !

   neighbor 169.254.9.105

    no shutdown

    remote-as 64512

    timers

     keepalive 10

     holdtime  30

    !

    address-family ipv4-unicast

    !

   !

   neighbor 169.254.9.121

    no shutdown

    remote-as 64512

    timers

     keepalive 10

     holdtime  30

    !

    address-family ipv4-unicast

    !

   !

  !

 !

 interface ipsec4

  ip address 169.254.9.122/30

  tunnel-source-interface ge0/0

  tunnel-destination      13.56.36.197

  ike

   version      1

   mode         main

   rekey        28800

   cipher-suite aes128-cbc-sha1

   group        2

   authentication-type

    pre-shared-key

     pre-shared-secret $8$...rDRNql0MgmWE

    !

   !

  !

  ipsec

   rekey                   3600

   replay-window           128

   cipher-suite            aes256-cbc-sha1

   perfect-forward-secrecy group-2

  !

  no shutdown

 !

 interface ipsec5

  ip address 169.254.9.106/30

  tunnel-source-interface ge0/0

  tunnel-destination      54.153.70.123

  ike

   version      1

   mode         main

   rekey        28800

   cipher-suite aes128-cbc-sha1

   group        2

   authentication-type

    pre-shared-key

     pre-shared-secret $8$/...xn5DFuXoc/

    !

   !

  !

  ipsec

   rekey                   3600

   replay-window           128

   cipher-suite            aes256-cbc-sha1

   perfect-forward-secrecy group-2

  !

  no shutdown

 !

 interface loopback1

  ip address 10.121.1.1/32

  no shutdown

 !

!

vpn 2

 dns 8.8.8.8 primary

 router

  bgp 65000

   address-family ipv4-unicast

    maximum-paths paths 4

    redistribute connected

    redistribute omp

   !

   neighbor 169.254.9.153

    no shutdown

    remote-as 64512

    timers

     keepalive 10

     holdtime  30

    !

    address-family ipv4-unicast

    !

   !

   neighbor 169.254.11.45

    no shutdown

    remote-as 64512

    timers

     keepalive 10

     holdtime  30

    !

    address-family ipv4-unicast

    !

   !

  !

 !

 interface ge0/1

  ip address 10.121.2.158/24

  no shutdown

 !

 interface ipsec2

  ip address 169.254.9.154/30

  tunnel-source-interface ge0/0

  tunnel-destination      13.52.94.90

  ike

   version      1

   mode         main

   rekey        28800

   cipher-suite aes128-cbc-sha1

   group        2

   authentication-type

    pre-shared-key

     pre-shared-secret $8$yWmfJLwmu...iHAVkfbr++wtU

    !

   !

  !

  ipsec

   rekey                   3600

   replay-window           128

   cipher-suite            aes256-cbc-sha1

   perfect-forward-secrecy group-2

  !

  no shutdown

 !

 interface ipsec3

  ip address 169.254.11.46/30

  tunnel-source-interface ge0/0

  tunnel-destination      52.9.82.251

  ike

   version      1

   mode         main

   rekey        28800

   cipher-suite aes128-cbc-sha1

   group        2

   authentication-type

    pre-shared-key

     pre-shared-secret $8$sXpsUaXK/...+w1o4c595umiilWJ8Ejt

    !

   !

  !

  ipsec

   rekey                   3600

   replay-window           128

   cipher-suite            aes256-cbc-sha1

   perfect-forward-secrecy group-2

  !

  no shutdown

 !

!

vpn 3

 dns 8.8.8.8 primary

 router

  bgp 65000

   address-family ipv4-unicast

    redistribute connected

    redistribute omp

   !

   neighbor 192.168.33.6

    no shutdown

    remote-as     65515

    update-source ipsec6

    ebgp-multihop 255

    address-family ipv4-unicast

    !

   !

  !

 !

 interface ipsec6

  ip address 192.168.3.2/30

  tunnel-source-interface ge0/0

  tunnel-destination      52.159.168.107

  ike

   version      2

   rekey        28800

   cipher-suite aes256-cbc-sha1

   group        2

   authentication-type

    pre-shared-key

     pre-shared-secret $8$...Pchy/3WWus0Qv

    !

   !

  !

  ipsec

   rekey                   3600

   replay-window           512

   cipher-suite            aes256-gcm

   perfect-forward-secrecy group-16

  !

  mtu                     1400

  no shutdown

 !

 interface loopback3

  ip address 10.121.3.1/32

  no shutdown

 !

 ip route 192.168.33.0/24 192.168.3.1

!

vpn 512

 dns 8.8.8.8 primary

 interface eth0

  ip dhcp-client

  no shutdown

 !

!

policy

 app-visibility

!

EXP-BR21-vEdge# 

EXP-BR21-vEdge# show bgp neighbor vpn 3

 

                          MSG   MSG   OUT                                                     AFI               

VPN  PEER ADDR     AS     RCVD  SENT  Q    UPTIME      STATE        LAST UPTIME               ID   AFI          

-----------------------------------------------------------------------------------------------------------------

3    192.168.33.6  65515  27    28    0    0:00:22:00  established  Sat Jun 22 08:38:35 2019  0    ipv4-unicast 

 

EXP-BR21-vEdge# 

EXP-BR21-vEdge#  show ipsec ike sessions state

 

     IF                       

VPN  NAME    STATE           

------------------------------

1    ipsec4  IKE_UP_IPSEC_UP 

1    ipsec5  IKE_UP_IPSEC_UP 

2    ipsec2  IKE_UP_IPSEC_UP 

2    ipsec3  IKE_UP_IPSEC_UP 

3    ipsec6  IKE_UP_IPSEC_UP 

 

EXP-BR21-vEdge# sh ver

18.4.1

EXP-BR21-vEdge#

EXP-BR21-vEdge# sh ip ro vpn 3

Codes Proto-sub-type:

  IA -> ospf-intra-area, IE -> ospf-inter-area,

  E1 -> ospf-external1, E2 -> ospf-external2,

  N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,

  e -> bgp-external, i -> bgp-internal

Codes Status flags:

  F -> fib, S -> selected, I -> inactive,

  B -> blackhole, R -> recursive

 

                                            PROTOCOL  NEXTHOP     NEXTHOP          NEXTHOP                                                  

VPN    PREFIX              PROTOCOL         SUB TYPE  IF NAME     ADDR             VPN      TLOC IP          COLOR            ENCAP  STATUS 

---------------------------------------------------------------------------------------------------------------------------------------------

3      10.0.0.0/8          bgp              i         -           192.168.33.6     -        -                -                -      F,S,R  

3      10.116.3.1/32       omp              -         -           -                -        116.1.1.116      biz-internet     ipsec  F,S    

3      10.121.3.1/32       connected        -         loopback3   -                -        -                -                -      F,S    

3      10.151.0.0/16       bgp              i         -           192.168.33.6     -        -                -                -      F,S,R  

3      192.168.3.0/30      connected        -         ipsec6      -                -        -                -                -      F,S    

3      192.168.3.2/32      bgp              i         -           192.168.33.6     -        -                -                -      -      

3      192.168.33.0/24     bgp              i         -           192.168.33.6     -        -                -                -      I      

3      192.168.33.0/24     static           -         ipsec6      192.168.3.1      -        -                -                -      F,S    

 

EXP-BR21-vEdge#

1 Comment
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: