ACI - VLAN Pool recommendations

[udo.konstantin]

Hello, 

are there any recommendation regarding the sizing of the different VLAN pools used within an ACI fabric? 

Example:

Customer with 100 Bare Metal Server and 20 ESXi hosts. Each ESXi host has about 100 VMs. Also there is one L3out and one L2out. 

The design will use four different tenants include the common. 

What will be the basis for decision? 

Thanks 

Udo 

[Marcel Zehnder]

Hi Udo

It's not about the number of endpoints, it's about the number of EPGs:

-How many EPGs with physical domains attached (static ports)

-How many EPGs are VMM integrated (and how many VMM domains in total)

-How many EPGs which use both physical domains and vmm domains?

Marcel

[Robert Burns]

Another factor that may come into play is if you're using uSeg & vDS.  If using uSeg, then you need two VLANs per unique BD. 

Robert

[RedNectar · Accepted solution]

Hi Udo,

VLAN Pool planning is a mixture of looking at:

1. How many leaf switches do I have?
2. How many Tenants share those switches
1. And are the Tenant's servers spread across multiple switches
3. How many EPGs do I have (per Tenant)

It is considered best practice to try and not have overlapping VLAN Pools on a single switch because internally each switch only has 4000 odd VLANs, and having overlapping VLANs doubles the number of internal VLANs required to map each potentially overlapping VLAN.

And, if you DO need to use the same VLAN ID for two different purposes on the same switch, you will need to use the Per-Port VLAN feature by creating a L2 Interface Policy and use that policy in the Interface Policy Groups that are assigned to those switches/VLAN Pools.

[Footnote: I have no Idea why they called the Per-Port VLAN policy a L2 Interface Policy]

Back to your problem.

If you only have four tenants plus the common tenant, then I'd suggest (assuming equal sized tenants) creating one or two VLAN pools, for each Tenant.  I'm going to make the simple assumption that each Tenant requires at this stage about 100 VLANs - say 20 static VLANs for the Bare Metal Hosts (BMHs) and 80 dynamic VLANs for the VMs.

What you haven't told us is how you intend to use the common tenant - is the common tenant just for L3 external access?  Nor have you told us whether the VMs are shared between Tenants. Anyway, here's a couple of possible solution:

Case Study 1 - BMHs and VMM shared VLAN Pool

Step 1: The simplest approach is to create one VLAN Pool per Tenant.  Include a reference to the Tenant Name in the naming of the VLAN Pool, eg TenA:VLAN.Pool

In that VLAN Pool, create two Encap Blocks

• Make one block of 20 VLANs a static allocation (for the BMHs)
• Make the other block of 80 VLANs dynamic (for the VMM)

Make sure the VLAN Pools for each Tenant don't overlap

[Note: On many Cisco switches the VLAN range 3698-4049 and VLAN 4094 are reserved, you may wish to avoid these]

Step 2: Next create a Physical Domain per Tenant. The Physical Domain will only be used for mapping BMHs to EPGs.  Include a reference to the Tenant Name in the naming of the Physical Domain, eg TenA:BMH-PhysDom

Associate the Physical Domain to the TenA:VLAN.Pool VLAN Pool you created earlier.

Step 3: Next, if this tenant is going to use L2 Outs, create a External Bridged Domain per Tenant. The External Bridged Domain will only be used for mapping existing VLANs to L2-EPGs.  Include a reference to the Tenant Name in the naming of the External Bridged Domain, eg TenA:MappedVLANs-ExtL2Dom

Associate the External Bridged Domain to the TenA:VLAN.Pool VLAN Pool you created earlier.

Step 4: Next create a VMM per Tenant for the appropriate VMM you are using. The VMM will be used for dynamically mapping VMs to EPGs.  Include a reference to the Tenant Name in the naming of the VMM, eg TenA:vCenter-VMM.Dom [Be careful, the prompt at which you name the VMM Domain asks you to name a Virtual Switch]

Associate the VMM to the TenA:VLAN.Pool VLAN Pool you created earlier.

Step 5: Next, create an Attachable Access Entity Profile (AEP) per Tenant.  Include a reference to the Tenant Name in the naming of the AEP , eg TenA:AEP

Add the TenA:BMH-PhysDom Physical Domain, the TenA:MappedVLANs-ExtL2Dom External Bridged Domain and the TenA:vCenter-VMM.Dom VMM Domain to the TenA:AEP AEP.

Case Study 2 - Separate  VLAN Pools for BMHs and VMMS

Step 1: If you want a bit more separation, you can create two VLAN Pools per Tenant, one for BMHs and one for VMMs.  Include a reference to the Tenant Name and purpose in the naming of the VLAN Pool, eg TenA:BMH-VLAN.Pool and TenA:VMM-VLAN.Pool 

In the TenA:BMH-VLAN.Pool VLAN Pool, create one Encap Block

• Make this a block of 20 static VLANs

In the TenA:VMM-VLAN.Pool VLAN Pool, create one Encap Block

• Make this a block of 80 dynamic VLANs

Make sure the VLAN Pools within and between Tenants don't overlap

[Note: On many Cisco switches the VLAN range 3698-4049 and VLAN 4094 are reserved, you may wish to avoid these]

Step 2: Next create a Physical Domain per Tenant. The Physical Domain will only be used for mapping BMHs to EPGs.  Include a reference to the Tenant Name in the naming of the Physical Domain, eg TenA:BMH-PhysDom

Associate the Physical Domain to the TenA:BMH-VLAN.Pool VLAN Pool you created earlier.

Step 3: Next, if this tenant is going to use L2 Outs, create a External Bridged Domain per Tenant. The External Bridged Domain will only be used for mapping existing VLANs to L2-EPGs.  Include a reference to the Tenant Name in the naming of theExternal Bridged Domain, eg TenA:MappedVLANs-ExtL2Dom

Associate the External Bridged Domain to the TenA:BMH-VLAN.Pool VLAN Pool you created earlier.

Step 4: Next create a VMM per Tenant for the appropriate VMM you are using. The VMM will be used for dynamically mapping VMs to EPGs.  Include a reference to the Tenant Name in the naming of the VMM, eg TenA:vCenter-VMM.Dom [Be careful, the prompt at which you name the VMM Domain asks you to name a Virtual Switch]

Associate the VMM to the TenA:vCenter-VLAN.Pool VLAN Pool you created earlier.

Step 5: Next, create an Attachable Access Entity Profile (AEP) per Tenant.  Include a reference to the Tenant Name in the naming of the AEP , eg TenA:AEP

Add the TenA:BMH-PhysDom Physical Domain, the TenA:MappedVLANs-ExtL2Dom External Bridged Domain and the TenA:vCenter-VMM.Dom VMM Domain to the TenA:AEP AEP.

The L3 Out

I have not addressed the L3 outs in the scenarios above because I am not sure if you will be doing these through the common Tenant or whether each Tenant will have their own links to the outside world, but assuming that access to the outside world is via a shared L3Out in the common tenant, you will need to create:

• Another VLAN pool for the L3 out with some static VLANs that will map to SVIs or Sub-Interfaces on your L3 out.  Call it say Shared:L3Ext-VLAN.Pool or common:L3Ext-VLAN.Pool - it will only need a few static VLANs, probably one or two per tenant at most, it may need only one depending on how you implement the sharing!
• An External Routed Domain - Call it say Shared:L3Ext-ExtL3Dom or common:L3Ext-ExtL3Dom.  Assign the VLAN Pool created above to this External Routed Domain.

• An Attachable Access Entity Profile - Call it say Shared:L3Ext-AEP or common:L3Ext-AEP. Add the External Routed Domain created above to this AEP.

Hope this gives you some pointers.

RedNectar

aka Chris Welsh

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

References:

https://rednectar.net/2015/12/28/cisco-aci-tutorial-4-all-about-access-policies-the-new-interface-range-command/

https://dpitaci.wordpress.com/2016/09/06/per-port-vlan/

https://rednectar.net/2016/12/11/cisco-aci-per-port-vlan-feature/

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices/b_ACI_Best_Practices_chapter_010.html

[Marcel Zehnder]

If there are no overlapping encap-vlans over all tenants, I would suggest to keep it as simple as possible and go with two VLAN-pools: 

one static pool for the physical and external domains

one dynamic pool for your vmm domains

Also imho I would try to use as less domains as possible - often there is just one physical domain and one l3ext domain needed (+ a vmm domain per integrated vSwitch).

[RedNectar]

I'd agree with Marcel Zehnder on that point - the simpler the better. Less is more so they say. But just remember, if you have multiple Tenants sharing the same VLAN pool(s), you need to still manage which Tenants are using which VLANs - and one way of doing that is to put each Tenant's VLANs into their own VLAN Pool.

[udo.konstantin]

Hi Chris,

thanks for your great answer. It take some time for me to read this it in detail...;-) It's a very good explanation. 

Udo

[charles.douglas]

We have one tenant for our prod environment and are standing up another tenant for a staging environment.   We will be setting up an L2OUT to the core from the staging side to access the core prod VLAN, so I will need to at least have that VLAN used in our new tenant along with several others that won't overlap.  Is this a problem, or should we simply set up a new VLAN to the core from this tenant to avoid the overlap.   This is not a huge deployment, but I don't want this VLAN to be shared between two tenants if it will affect our prod side at all.