Contributed by Nikolai Pitaev, TME, Enterprise Business
This document describes interconnection between Cisco SD-WAN and Azure Virtual WAN.
Enterprise customers looking for an optimal way to interconnect on-prem locations like branches and data centers with Azure cloud infrastructure. Cloud to on-prem interconnection should be seamless, automated and secure. Main benefits are segmentation and end-to-end policy for Application Quality of Experience (AppQoE) and security.
With segmentation different departments can access their own cloud infrastructure in a secure and efficient way. AppQoE will make sure, that the end users have the best cloud access.
Why to interconnect SD-WAN and Virtual WAN?
With the current trend to move workloads, applications, data centers to the cloud, the question why becomes more and more irrelevant. It is now more the question on “how to interconnect”, not “why”. The question “how” is covered in the next section.
While interconnection via customer managed VNet is a viable option, interconnection via Azure Virtual WAN enables cloud native support for transitive connectivity via a cloud native hub and spoke architecture.
Main benefits of the interconnection of SD-WAN and Virtual WAN are:
- Cloud native Transit hub architecture support.
- Automated and easy connectivity provisioning to the most optimal cloud entry point for all datacenters and hub locations.
- Dynamic routing, multi pathing and deterministic failover behavior using OMP and SD-WAN Secure Extensible Network (SEN) policies.
- Regional Hubs interconnecting multiple vHubs and regions via SEN.
- Application and data telemetry in and out of both clouds for reporting/chargeback.
What is Azure Virtual WAN?
Azure Virtual WAN is an Azure networking service that provides optimized and automated branch connectivity to, and through, Azure. Azure Virtual WAN hubs can be deployed in various Azure regions and serve as cloud interconnectivity hubs that you can choose to connect your branches, VNets and Users to.
Please refer to Virtual WAN main page for more details.
Also see SD-WAN connectivity architecture with Azure Virtual WAN, where the various interconnect models are described.
What is Cisco SD-WAN?
Cisco SD-WAN is a secure, cloud scale architecture that is open, programmable and scalable. Managed through the Cisco vManage console users can quickly establish an SD-WAN overlay fabric to connect data centers, cloud, branches, campuses, and colocation facilities to improve network speed, security, and efficiency.
Please visit the Cisco SD-WAN landing page for more information.
Interconnecting Cisco SD-WAN with Azure.
Enterprises that are looking to interconnect their Cisco SD-WAN with Azure could do it in one of following two models.
- Interconnect SD-WAN with Azure via customer managed VNet, also known as Transit VNet
- Interconnect SD-WAN with Azure via Azure Virtual WAN
In the following sections we describe how to connect Cisco SD-WAN with Azure using the above models. Each model in turn has several connectivity options.
1.) How to interconnect Cisco SD-WAN with Azure VNets?
In this section we describe how to connect Cisco SD-WAN with Azure using customer managed VNet as the entry point. There are 3 different cloud and on-prem connectivity options under this model
- Non SD-WAN Interconnection: Dynamic Multipoint VPN (DMVPN)
- SD-WAN Interconnection: Do-it-yourself
- SD-WAN Interconnection: Using Cloud On-Ramp to enable a Transit Architecture
- Non SD-WAN Interconnection: Dynamic Multipoint VPN (DMVPN)
If the existing WAN network is not SD-WAN capable, then the following solution can be used for on-prem to cloud interconnection: Azure Transit VNET DMVPN on Cisco Cloud Services Router 1000v (CSR1000v). Cisco Transit VNet solution on Azure uses a pair of Cisco CSR1000v virtual routers acting as DMVPN Hubs in active-active mode. The spoke VNets also have a Cisco CSR1000v acting as DMVPN Spoke connecting to both the CSR1000v devices in the Hub VNet through overlay routing such as EIGRP and BGP. This solution does not require manual configuration and is fully automated. Once deployed, this solution automatically creates dynamic spoke-to-spoke IPsec tunnels in an on-demand fashion.
2. SD-WAN Interconnection: Do-it-yourself
However, the current trend to SD-WAN networks is obvious, so the question is how to interconnect SD-WAN networks with Azure. The simplest option is “do-it-yourself model”, where each host VNet has a dedicated virtual SD-WAN Router. The picture below describes this “do-it-yourself” model:
In case of few host VNets, this option meets all the requirements and can be easily implemented. Obviously, it does not scale from an economic standpoint with the increasing number of host VNets.
3. SD-WAN Interconnection: using Cloud On-Ramp
This very popular option uses Cisco SD-WAN feature Cloud onRamp for IaaS, which automatically creates a transit VNet, instantiates two SD-WAN routers on it and maps host VNets to the transit VNet. All accomplished in few minutes from vManage – SD-WAN Network Management System.
The main benefit of this option is automation – the whole process is done automatically. The picture below illustrates Cloud onRamp for IaaS solution with the Transit VNet as central point:
b.) SD-WAN Interconnection using Azure Virtual WAN
The newest option for the SD-WAN interconnection with Azure is with Azure Virtual WAN. This option has benefits in terms of scale and can be more cost effective depending on bandwidth requirements. The key differences to the previous options are the usage of Azure Virtual WAN and Virtual WAN hub(s), which serve as central connection points for host VNets and SD-WAN routers.
There are two possible scenarios for connecting SD-WAN to Azure Virtual WAN:
- Born-on-Prem SD-WAN to Azure Virtual WAN
- Born-in-Cloud SD-WAN to Azure Virtual WAN
The picture below summarizes the main difference between two scenarios.
In the first case (“born on-prem”), physical on-prem SD-WAN routers (usually located at a central location such as On-prem Data Centers) establish standard IKE-based IPSec tunnels to vHub directly, run BGP and exchange routing information between cloud infrastructure and SD-WAN network using BGP-OMP redistribution. Connectivity from remote locations to the Central location is possible using SD-WAN IPSec tunnels.
In the second use case (“born in the cloud”), virtual SD-WAN routers (located on an Azure VNet) establish standard IKE-based IPSec tunnels to vHub directly, run BGP and exchange routing information between cloud infrastructure and SD-WAN network using BGP-OMP redistribution. Likewise, connectivity from remote locations to the virtual SD-WAN router is possible using SD-WAN IPSec tunnels. This option extends all the benefits from SD-WAN as AppQoE and other SD-WAN features closest to the Cloud and consequently, is the preferred choice.
The following table summarizes main differences between two connection options.
|
|
Born on-prem |
Born in the cloud |
|
Endpoints |
On-prem physical SD-WAN router connects to vHub directly. |
On-prem physical SD-WAN router connects to virtual SD-WAN. |
|
Transport over Internet |
Standard IKE-based IPSec tunnel |
SD-WAN tunnel |
|
Use case |
Many branches need direct connected to the nearest Azure location without requirements for regional/hierarchical network structure. |
Extending SD-WAN fabric into Azure and using SD-WAN features like AppQoE (FEC, TCP Opt) on the way to Azure. Regional/hierarchical network structure in place. |
Interconnecting SD-WAN with Azure Virtua WAN – step-by-step description
The following steps are required to successfully interconnect SD-WAN with vHub for the “born in the cloud” use case. The “born on-prem” option will follow similar configuration, it is a subset of the “born in the cloud” workflow described below.
- Bring up SD-WAN virtual router (CSR1000v or vEdge Cloud) instance(s) on Azure and prepare it for the interconnection with Virtual WAN:
- bring up a new SD-WAN virtual router or use an existing one.
- use a "basic" template on vManage just to have basic connectivity from vManage to this virtual SD-WAN router.
- If you do not have required infrastructure on Azure, please follow the step-by-step described on the Azure tutorial mentioned here. You will need to have created:
- Virtual WAN
- Hub
- VPN site
Once you have sites successfully associated your VPN site and your hub, you can proceed to the next step. The following screen shot demonstrates such association:
- Once you have Azure portion of the configuration done, you can proceed with the IKE-based IPSec and BGP configuration in vManage. You will need information about IPSec and BGP IP addresses, password, etc., which you can download from virtual WAN Overview Section. Use the provided information for the creation of a new vManage Template, which will have IPSec and BGP configuration in the Service VPN. Here is an example with CLI equivalent:
vpn 3
dns 8.8.8.8 primary
router
bgp 65000
no shutdown
address-family ipv4-unicast
redistribute connected
redistribute omp
!
neighbor 192.168.33.6
no shutdown
remote-as 65515
update-source ipsec6
ebgp-multihop 255
address-family ipv4-unicast
!
!
!
!
interface ipsec6
ip address 192.168.3.2/30
tunnel-source-interface ge0/0
tunnel-destination 52.159.168.107
ike
version 2
rekey 28800
cipher-suite aes256-cbc-sha1
group 2
authentication-type
pre-shared-key
pre-shared-secret $8$x0MowET4rOG2Cps1fJVBItC3e+3lIQB1eHNDV38tfi
!
!
!
ipsec
rekey 3600
replay-window 512
cipher-suite aes256-gcm
perfect-forward-secrecy group-16
!
mtu 1400
no shutdown
!
interface loopback3
ip address 10.121.3.1/32
no shutdown
!
ip route 192.168.33.0/24 192.168.3.1
!
Make sure you have eBGP multihop configured, otherwise BGP session will be not be established.
- Attach the SD-WAN Router to the new vManage template. Please refer to the full SD-WAN Edge router configuration in appendix for details.
Automation
Currently, automation is available for Cloud onRamp for IaaS using Transit VNet, but not for Azure vWAN. As of now, a simple configuration in vManage on the SD-WAN side and on Azure side is required for the interconnection. Both parts can be automated. Azure PowerShell or open source tools like Terraform vWAN can be used to script the configuration of vWAN / vHub.
Cisco and Microsoft are working toward enhancing the existing automation in Cloud OnRamp for the interconnection with the Azure Virtual WAN. Customers will soon be able to have all networking resources in the Azure Virtual WAN up, provisioned and connected with a few clicks of a button. Please follow the Cisco and Microsoft blogs to stay up to date with the latest product announcements.
Securing the interconnection
In many cases a very common security requirement is to inspect all the traffic leaving or entering vHub with a Firewall. The example below shows SD-WAN router vEdge 1 connected to the SD-WAN fabric on the left side and to a virtual firewall on the right side. Firewall is connected to vHub. In this scenario, SD-WAN router will establish a BGP session to the firewall and the firewall will inspect all traffic to and from the SD-WAN router.
Pro’s:
- Single FW instance – less cost
- All traffic between SD-WAN Overlay and workloads on the Cloud is protected
- All the complexity of building tunnels with Cloud provider is transferred to the FW
Con’s:
- Traffic between VNets is not protected
Another firewall deployment option is “shared services VNet”, which will have the firewall in a separate VNet as shown in the following diagram.
This is the most scalable option and provides most of the benefits required by customers, including:
- Protection for traffic between VNets
- Protection for traffic between Overlay and VNets
- No need to have a single firewall per VNet
- Cisco SD-WAN does not handle the complexity of configuring traffic path through the firewall.
Please refer to the following document, which shows how to steer traffic from a branch (on-premises site) connected to the Virtual WAN hub to a Spoke virtual network (VNet) via a Network Virtual Appliance (NVA).
Troubleshooting
IPSec Tunnel state verification in vManage:
IKE-based IPSec tunnels: go to the appropriate SD-WAN Router's dashboard (Monitor -> Network) and review IKE-based IPSec Tunnels. In the following example, you see 3 tunnels established - scroll to the right and look for the state. It should be "IKE_UP_IPSEC_UP":
Verify connectivity in Azure - check vWAN Overview and Health. You should see connection state (green) and traffic volume as shown below:
Appendix – full SD-WAN router configuration
EXP-BR21-vEdge# sh run
system
host-name EXP-BR21-vEdge
system-ip 121.1.1.1
site-id 121
admin-tech-on-failure
no route-consistency-check
sp-organization-name exp-lab-dmz
organization-name exp-lab-dmz
clock timezone America/Los_Angeles
vbond 173.37.56.174
aaa
auth-order local radius tacacs
usergroup basic
task system read write
task interface read write
!
usergroup netadmin
!
usergroup operator
task system read
task interface read
task policy read
task routing read
task security read
!
user admin
password $6$....HP1
!
user npitaev
password $6$...E0
group netadmin
!
!
logging
disk
enable
!
!
!
omp
no shutdown
graceful-restart
advertise bgp
advertise connected
advertise static
!
security
ipsec
authentication-type sha1-hmac ah-sha1-hmac
!
!
vpn 0
dns 8.8.8.8 primary
interface ge0/0
ip dhcp-client
tunnel-interface
encapsulation ipsec
color biz-internet restrict
no allow-service bgp
allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service ospf
no allow-service stun
allow-service https
!
no shutdown
!
!
vpn 1
router
bgp 65000
address-family ipv4-unicast
maximum-paths paths 4
redistribute connected
redistribute omp
!
neighbor 169.254.9.105
no shutdown
remote-as 64512
timers
keepalive 10
holdtime 30
!
address-family ipv4-unicast
!
!
neighbor 169.254.9.121
no shutdown
remote-as 64512
timers
keepalive 10
holdtime 30
!
address-family ipv4-unicast
!
!
!
!
interface ipsec4
ip address 169.254.9.122/30
tunnel-source-interface ge0/0
tunnel-destination 13.56.36.197
ike
version 1
mode main
rekey 28800
cipher-suite aes128-cbc-sha1
group 2
authentication-type
pre-shared-key
pre-shared-secret $8$...rDRNql0MgmWE
!
!
!
ipsec
rekey 3600
replay-window 128
cipher-suite aes256-cbc-sha1
perfect-forward-secrecy group-2
!
no shutdown
!
interface ipsec5
ip address 169.254.9.106/30
tunnel-source-interface ge0/0
tunnel-destination 54.153.70.123
ike
version 1
mode main
rekey 28800
cipher-suite aes128-cbc-sha1
group 2
authentication-type
pre-shared-key
pre-shared-secret $8$/...xn5DFuXoc/
!
!
!
ipsec
rekey 3600
replay-window 128
cipher-suite aes256-cbc-sha1
perfect-forward-secrecy group-2
!
no shutdown
!
interface loopback1
ip address 10.121.1.1/32
no shutdown
!
!
vpn 2
dns 8.8.8.8 primary
router
bgp 65000
address-family ipv4-unicast
maximum-paths paths 4
redistribute connected
redistribute omp
!
neighbor 169.254.9.153
no shutdown
remote-as 64512
timers
keepalive 10
holdtime 30
!
address-family ipv4-unicast
!
!
neighbor 169.254.11.45
no shutdown
remote-as 64512
timers
keepalive 10
holdtime 30
!
address-family ipv4-unicast
!
!
!
!
interface ge0/1
ip address 10.121.2.158/24
no shutdown
!
interface ipsec2
ip address 169.254.9.154/30
tunnel-source-interface ge0/0
tunnel-destination 13.52.94.90
ike
version 1
mode main
rekey 28800
cipher-suite aes128-cbc-sha1
group 2
authentication-type
pre-shared-key
pre-shared-secret $8$yWmfJLwmu...iHAVkfbr++wtU
!
!
!
ipsec
rekey 3600
replay-window 128
cipher-suite aes256-cbc-sha1
perfect-forward-secrecy group-2
!
no shutdown
!
interface ipsec3
ip address 169.254.11.46/30
tunnel-source-interface ge0/0
tunnel-destination 52.9.82.251
ike
version 1
mode main
rekey 28800
cipher-suite aes128-cbc-sha1
group 2
authentication-type
pre-shared-key
pre-shared-secret $8$sXpsUaXK/...+w1o4c595umiilWJ8Ejt
!
!
!
ipsec
rekey 3600
replay-window 128
cipher-suite aes256-cbc-sha1
perfect-forward-secrecy group-2
!
no shutdown
!
!
vpn 3
dns 8.8.8.8 primary
router
bgp 65000
address-family ipv4-unicast
redistribute connected
redistribute omp
!
neighbor 192.168.33.6
no shutdown
remote-as 65515
update-source ipsec6
ebgp-multihop 255
address-family ipv4-unicast
!
!
!
!
interface ipsec6
ip address 192.168.3.2/30
tunnel-source-interface ge0/0
tunnel-destination 52.159.168.107
ike
version 2
rekey 28800
cipher-suite aes256-cbc-sha1
group 2
authentication-type
pre-shared-key
pre-shared-secret $8$...Pchy/3WWus0Qv
!
!
!
ipsec
rekey 3600
replay-window 512
cipher-suite aes256-gcm
perfect-forward-secrecy group-16
!
mtu 1400
no shutdown
!
interface loopback3
ip address 10.121.3.1/32
no shutdown
!
ip route 192.168.33.0/24 192.168.3.1
!
vpn 512
dns 8.8.8.8 primary
interface eth0
ip dhcp-client
no shutdown
!
!
policy
app-visibility
!
EXP-BR21-vEdge#
EXP-BR21-vEdge# show bgp neighbor vpn 3
MSG MSG OUT AFI
VPN PEER ADDR AS RCVD SENT Q UPTIME STATE LAST UPTIME ID AFI
-----------------------------------------------------------------------------------------------------------------
3 192.168.33.6 65515 27 28 0 0:00:22:00 established Sat Jun 22 08:38:35 2019 0 ipv4-unicast
EXP-BR21-vEdge#
EXP-BR21-vEdge# show ipsec ike sessions state
IF
VPN NAME STATE
------------------------------
1 ipsec4 IKE_UP_IPSEC_UP
1 ipsec5 IKE_UP_IPSEC_UP
2 ipsec2 IKE_UP_IPSEC_UP
2 ipsec3 IKE_UP_IPSEC_UP
3 ipsec6 IKE_UP_IPSEC_UP
EXP-BR21-vEdge# sh ver
18.4.1
EXP-BR21-vEdge#
EXP-BR21-vEdge# sh ip ro vpn 3
Codes Proto-sub-type:
IA -> ospf-intra-area, IE -> ospf-inter-area,
E1 -> ospf-external1, E2 -> ospf-external2,
N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,
e -> bgp-external, i -> bgp-internal
Codes Status flags:
F -> fib, S -> selected, I -> inactive,
B -> blackhole, R -> recursive
PROTOCOL NEXTHOP NEXTHOP NEXTHOP
VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS
---------------------------------------------------------------------------------------------------------------------------------------------
3 10.0.0.0/8 bgp i - 192.168.33.6 - - - - F,S,R
3 10.116.3.1/32 omp - - - - 116.1.1.116 biz-internet ipsec F,S
3 10.121.3.1/32 connected - loopback3 - - - - - F,S
3 10.151.0.0/16 bgp i - 192.168.33.6 - - - - F,S,R
3 192.168.3.0/30 connected - ipsec6 - - - - - F,S
3 192.168.3.2/32 bgp i - 192.168.33.6 - - - - -
3 192.168.33.0/24 bgp i - 192.168.33.6 - - - - I
3 192.168.33.0/24 static - ipsec6 192.168.3.1 - - - - F,S
EXP-BR21-vEdge#
1 Comment
