cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
1769
Views
25
Helpful
4
Replies
rockstepp
Beginner

Cisco ACI Multi-Pod & Multi-Site & Multi-Cloud with Azure Design

Hi guys,
I'm working on an HLD for a customer that asked for a DC network architecture for:

  • 2 sites close to each other in a common "region", cloudy speaking;
  • 1 site is distant from the other 2 that act as DR;
  • Azure cloud connectivity, I don't have many details but it should be a single region deployment.

The idea is to propose ACI because it meets most of the customer requirements, the supposed solution is a Multi-Site architecture made up of:

  • 1 Multi-Pod fabric for the 2 active-active sites;
  • 1 Single-Pod fabric for the DR site;
  • 1 Cloud APIC deployed on the Azure region to extend the ACI policy model.

I have few doubts:

  1. Considering this as a greenfield, is it right to start with the combination of Multi-Pod and Multi-Site, instead of using only Multi-Site?
  2. Is it better to use the same IP network for the IPN and ISN or keep them separated?
  3. I read on the Multi-Site white paper that the uplink interfaces for the IPN and ISN should be the same on spine nodes. Does this mean that I can't use back-to-back connectivity with dark fibers for the Multi-Pod environment and MPLS for the ISN?
  4. Could I connect the fabrics to Azure using the IPN/ISN deploying IPsec devices on that network, through ExpressRoute links?
  5. What if I would like to use 1 Express Route terminated on the IPN/ISN and 1 internet link in any fabric?
  6. Is it better to deploy the Multi-Cloud environment with IPsec tunnels between CSRv routers in the infra Vnet and ANG objects in the user Vnets, or use Vnet peerings?

Thanks a lot in advance!

1 ACCEPTED SOLUTION

Accepted Solutions
Robert Burns
Cisco Employee

Here's a sample topology that would be similar to yours.  You can ignore the WAN connection on the bottom.  The IxN devices at each Pod/Site need to run OSPF between them and the Spine switches, but within the Inter-Site/Pod-Network you can run anything.  We only need IP reachability to the other devices.   However you want to interconnect the IxN devices is up to you. isn topology.PNG

 

Yes you're correct that onPrem CSRs are needed regardless.  There's work being done with ExpressRoute/DX in the future that may remove this requirement, but for now they're required.  They also serve to encrypt your ExpressRoute traffic which wouldn't be otherwise.

Robert

View solution in original post

4 REPLIES 4
rockstepp
Beginner

Any help?

Robert Burns
Cisco Employee

I'll offer insight where I can.  Answers inline below.
[FYI - Multisite Orchestrator (previously MSO) is known now as NDO (Nexus Dashboard Orchestrator) going forward]

Regards,

Robert

 

Considering this as a greenfield, is it right to start with the combination of Multi-Pod and Multi-Site, instead of using only Multi-Site?

[Robert]  So this comes down to cost.  Running Multipod saves you from having an extra APIC cluster to manage.  Multisite is going to be needed regardless if/when you go to Azure though, and its also extremely helpful when you want to deploy consistent policies across multiple fabrics (like the Active fabric & DR fabrics and/or Cloud sites).  To use these fabrics with multisite you're already going to need the Advantage level license for every leaf at every site, so the question of using Mpod+Msite vs. Msite only comes down to cost vs. benefit:

  • With Multipod there's one less APIC clusters to factor into the cost (Capex benefit)
  • With Multipod it brings all visibility & monitoring of all Pods to a single pane of glass (Opex Benefit)
  • With Multipod, those two Active sites are managed by a single Cluster, aka Availability zone.  Though there's separate control planes per Pod, its not going to be as resilient to a completely separate fabric managed by Msite (Business Continuity Benefit).

Is it better to use the same IP network for the IPN and ISN or keep them separated?

[Robert] Most customers dual-task their physical ISN/IPN devices. 

 

I read on the Multi-Site white paper that the uplink interfaces for the IPN and ISN should be the same on spine nodes. Does this mean that I can't use back-to-back connectivity with dark fibers for the Multi-Pod environment and MPLS for the ISN?

[Robert] Correct.  It's not supported to mix back-to-back connections with ISN configuration like MPLS.  These deployments are mutually exclusive.

Could I connect the fabrics to Azure using the IPN/ISN deploying IPsec devices on that network, through ExpressRoute links?

[Robert] The point of ExpressRoute is that it gives you a secure link directly into the cloud.  Typically customers will use IPSec for ISN to Cloud connections that transit the Internet.

What if I would like to use 1 Express Route terminated on the IPN/ISN and 1 internet link in any fabric?

[Robert] This is possible.  You use the ExpressRoute as your primary cloud connection, and use IPSec as backup connection for the same site.  Routing could be configured to prefer one or the other, but ExpressRoute will offer a far better SLA.  Another option to cut costs a bit would be to use ExpressRoute or IPSec in one site and should there be a failure reaching the cloud from an OnPrem site, it can leverage the DCI to the other ISN devices and egress towards the cloud from there - which could be ExpressRoute or IPsec.  No issues mixing & matching, but you don't want to overcomplicate your design too much.  Colos (such as Equinox) are another cost-effective options for ExpressRoute/Direct Connectivity to the public cloud providers. 

Is it better to deploy the Multi-Cloud environment with IPsec tunnels between CSRv routers in the infra Vnet and ANG objects in the user Vnets, or use Vnet peerings?

[Robert] CSRs must be used for Cloud to Cloud (ie. AWS to Azure), or Cloud to OnPrem without ExpressRoute.  For anything internal to Azure vNet Peering is the way to go when lower latency & Higher bandwidth is required. For Regions that may not support vNet peering you can still use VPN/IPsec.  Keep in mind there will be a cost premium for vNet peering vs. VPN GWs. Regardless of your choice, cloud APIC/Msite will handle the management and deployment of these intra & Inter-region peerings.  There's a good compare/contrast between these two Vnet connectivity options here https://azure.microsoft.com/en-us/blog/vnet-peering-and-vpn-gateways/

Hi Robert and thanks for your support.

I still have few doubts:

 

[Robert] Most customers dual-task their physical ISN/IPN devices.

 

Could you provide an example diagram of a dual-stack IPN/ISN network?
I was looking for mixed IPN/ISN with two possible designs:

  • One is a tiered design with two core switches placed in a PoP/Colo and two IPN/ISN switches per PoD/Site.
    All the IPN/ISN switches connect to the Core switches in a partial mesh topology (IPN1 with Core1 and IPN2 with Core2), because of the costs of dark fibres for those links.
  • The other one is a Mesh design, so without Core switches and The IPN/ISN devices connecting directly to each other (IPN1 of one site with IPN1 of the other sites).

 

 

[Robert] The point of ExpressRoute is that it gives you a secure link directly into the cloud.  Typically customers will use IPSec for ISN to Cloud connections that transit the Internet.

 

I've understood that Multi-Cloud requires IPsec routers to terminate IPsec tunnels with CSR1000v regardless of the type of transport you have, because the Cloud APIC automatically configure an infra Vnet with 1 or more CSR1000v devices and IPsec connections to IPsec devices on prem.
So my idea is to connect a couple of routers to the IPN/ISN network in the same fashion described before (as the IPN/ISN devices described before), and those routers will terminate IPsec VPNs and route the traffic originate by the CSR1000v routers to the IPN/ISN.

 

 

Many thanks!

Robert Burns
Cisco Employee

Here's a sample topology that would be similar to yours.  You can ignore the WAN connection on the bottom.  The IxN devices at each Pod/Site need to run OSPF between them and the Spine switches, but within the Inter-Site/Pod-Network you can run anything.  We only need IP reachability to the other devices.   However you want to interconnect the IxN devices is up to you. isn topology.PNG

 

Yes you're correct that onPrem CSRs are needed regardless.  There's work being done with ExpressRoute/DX in the future that may remove this requirement, but for now they're required.  They also serve to encrypt your ExpressRoute traffic which wouldn't be otherwise.

Robert

View solution in original post