multi-pod vs stretched fabric

[axfalk1]

Could someone please shed the light on whether we could have identical vlans and subnets on both sides of a multipod or stetch fabric ACI deployment?

Thanks...

_ Greg....

[Robert Burns]

Stretched Fabric is a deprecated design which should be replaced with Multipod if/when possible.  Since both designs leverage a single logical fabric, there's no issue with VLANs and/subnets being leveraged on any of the Leafs.

Robert

[ecsnnsls]

Yes, in a multipod design, the same VLAN gets stretched across the DCs.

[RedNectar · Accepted solution]

Hi Greg,

There are three ways to achieve identical VLANs and subnets on both sides of a dual site ACI deployment. Well, two anyway.

1. Stretched Fabric
   • As @Robert Burns says, this is a deprecated design and should not be considered. I'll say no more about it
2. Multi-Pod Fabric
   • In this design you have two ACI pods (minimum of two spines in each) and a number of leaves in each site
   • You connect a regular OSPF router (NXOS style Nexus 9000 typically) to each site, and build a BGP network between the two OSPF routers.  
   • You then configure the spines at each site to create a MPBGPEVPN (my favourite acronym) between the two sites
   • You connect the APICs however you want - typically two APICs at the main site and one at the second site.
   • At the end of the deployment you have a SINGLE ACI FABRIC  and every but of configuration is available to both sites without even thinking about it
     • Configuration can be created and maintained on ANY of the APICs
3. Multi-Site ACI (I prefer to think of it as Multi-Fabric ACI)
   1. In this design you have two ACI Fabrics, each with their own set of APICs
   2. Like Multi-Pod 
      • You connect a regular OSPF router (NXOS style Nexus 9000 typically) to each site, and build a BGP network between the two OSPF routers.  
      • You then configure the spines at each site to create a MPBGPEVPN between the two sites
   3. Unlike Multi-Pod
      • The infrastructure and Access Policy Chains MUST be created independently on each site. This includes the VLAN Pools, Physical Domains, AAEPs, Interface Policy Groups, Interface Policies, Interface Profiles and Switch Profiles.  It's up to you to make sure these match at each site if that is what you want
      • If you use the APICs at either site to map VLANs to EPGs and Subnets at each site, they are local to that site
      • You can however create your logical structure (Tenants, Bridge Domains, EPGs, Contracts, L3Outs etc) on a separate completely independent application with a different look and feel to the APIC GUI - this application is called the Multi-Site Orchestrator.  The logical structure is created using schemas, and then you can push the schema to one or other or both sites. 
        • Note: The logical structure includes the VLAN to EPG mappings - so if you are using static mappings, you can easily map the same VLAN to the same EPG at each site within the EPG

Multi-pod vs Multi-Site

From the point of view of your assumed objective "to have identical vlans and subnets on both sides of an ACI deployment" you need to weigh up Multi-Pod vs Multi-Site.  Both have advantages and disadvantages

Multi-Pod Advantages

• Easier to implement
• Cheaper (only 3 APICs required [Tip: Buy 4] and no MSO)
• VLANs and Domains are automatically created at each site.

Multi-Pod Disadvantages

Multi-pod really only has one disadvantage, but it's HUGE

• THE SPLIT BRAIN PROBLEM - if your sites become separated, you can only manage your fabric from a site that has two or more APICs
  • It is common design to have (for two sites) two APICs at one site, and one APIC plus a spare (called a Standby APIC) at the second site
    • If there is a disconnect between the two sites, it is quite a process (MANUAL) to bring up the standby APIC - if not done properly and the first site comes back online you could be in trouble.
    • When connectivity is restored, you basically have to go through the reverse process to make the Standby APIC "standby" again.

Multi-Site Advantages

• In the case of a disconnect, each site can continue operation without the other.
  • When connectivity is restored, the MSO takes care of synchronising the policies at each site
• With multi-site, it is possibly extend your Schemas to include additional fabrics, including Virtual ACI Pods that may be running on say AWS
• Multi-site is MUCH more extensible and flexible than Multi-Pod

Multi-Site Disadvantages

• Greater up-front cost
• Two very different GUI interfaces to contend with (when I say very different, they do at least have the same colour scheme, and there is clearly a push to migrate the APIC GUI towards an iPhone friendly/PC-unfriendly GUI like the MSO)
• Double work for initial setup, and NO guarantee that Access Policy names etc are going to be the same at each site.
  • To overcome this, I would recommend using Ansible or write your own scrips or use Postman to do the Access Policy configuration for each site - but that adds another level of complexity
  • So in terms of your original objective "to have identical vlans and subnets on both sides of an ACI deployment" - it is really up to you to ensure that you keep the same conventions at each site. MSO will NOT ENFORCE THIS

The bottom line

• In spite of the extra work and cost, I would recommend leaning towards Multi-Site, simply because of the future advantages.
  • But keep in mind, Multi-Site does not really achieve your objective "to have identical vlans and subnets on both sides of an ACI deployment" without YOU enforcing it
• On the other hand, even if you build a Multi-Pod design today, it could be either incorporated into a future Multi-Site design (MSO sees a multi-pod as a single site) or even migrated to a Multi-Site design.

[weylin.piegorsch]

If I may as a follow-up (2yr later)?

The challenge with mutlipod is that you need an IP Network to connect folks, aka “GOLF” (notwithstanding the optimization for a 2-pod cluster introduced in v5.2(3)).  And near as I can tell, GOLF cannot use the same physical routers and long-haul circuits as L3out's.  That’s a FAR more expensive investment, both in terms of capital acquisition as well as in terms of complexity for engineering to setup or operations to support.

It complicates things further that we use IS-IS as our campus IGP...

In that vein, Stretched Fabric appears to be a far more straight-forward solution for what would otherwise be a 2-pod fabric. (Would be cool too if Stretched Fabric were a single pod with other remote pods in a multipod topology).

[weylin.piegorsch]

And... as soon as I post that, I find this for v5.2(3).  I'd only previously heard about it casually.
Back-to-Back Multipod 

Being able to seamlessly migrate to using an intermediary network if you grow;
having it as a set-and-forget like VSS; and,
in the meantime you end up using the same long-haul circuits/optics as stretched fabric *and nothing else* (that I can see)

I can't envision anything else that Stretched Fabric brings to the table.

[GMetric]

Interesting questions and answers here. If I may add a question/situation.

Current situation:

• CORE is a Stackwise-virtual (C9600) stretched between 2 datacenters
• CORE functions as router voor all VLANs

What we want to do:

• Connect an ACI strechted single-pod fabric to the CORE on L2. ACI fabric wil only function as L2 domain.

The reason behind this is:

• No MSO needed
• VLANs/Subnet are available on both datacenters
• Price. We are planning on purchasing 4 Spines, 8 Leafs, and 3 APICs

A way how I think this can be done is by also splitting a VPC leaf pair between the 2 datacenters. This way you can have a VPC link to the CORE (1 link in each datacenter).

Or is there another way? Preferably without using STP where links are not used.