ACI VMM domain and bare metal server in same EPG

[SandevChopra07800]

Hi Fellow networkers, I have bumped into a situation where an EPG needs both vmm-domain and a physical domain (bare metal server) associated to it. The vlan-encap on a vmm-domain assigned pool is dynamic so it has randomly picked up any vlan from that pool. The vlan-encap for the baremtal server is static on a different vlan id, and they are able to communicate with each other. How is this working if you have different encaps on the same EPG? I also verified from the Leaf CLI if it is assigning both the same extended-vlan id, to my surprise it is NOT. 

Any suggestions on how this works in the background? 

[wajidhassan · Accepted solution]

This works in ACI because the EPG is tied to a bridge domain, not to a specific VLAN. When you associate both a VMM domain (with dynamic VLAN from a pool) and a physical domain (with static VLAN) to the same EPG, ACI maps both encapsulations internally to the same policy construct using what's called an endpoint group policy.

Each VLAN (dynamic or static) is locally significant per leaf, and ACI uses GEP (Generic Endpoint) and proxying mechanisms to handle communication between endpoints in the same EPG but on different encaps. So even if the VLANs differ, ACI ensures they’re part of the same forwarding context via the bridge domain, and traffic is handled correctly by translating the VLANs internally. This is why they can communicate seamlessly despite different VLAN IDs.

[RedNectar · Accepted solution]

Hi @SandevChopra07800 ,

Just to add to @wajidhassan excellent answer, I'd say that what you've discovered is one of the great powers of ACI - that endpoints DON'T have to be on the same VLAN to communicate, and the VMM Domain + Static VLAN mapping case you mention is the model use-case for this.

I often tell people to "forget whatever you've learned about VLANs" when it comes to ACI.

In ACI, the VLAN tag is used ONLY to identify the source EPG, so when a frame arrives at a leaf switch, the switch looks at the VLAN tag/port combination to determine the source EPG and applies policy based on the policies linked to the EPG.

In fact, the source VLAN tag is discarded when the frame arrives, and the switch uses a different internal VLAN as a temporary container for that EPG. Take a look at the (very much trimmed) output below from the command 

fabric 1201-1202 show vlan extended     # my switch IDs are 1201 and 1202

where I have extracted just the information for the AppServers_EPG in the 2Tier_AP for Tenant18

apic1# fabric 1201-1202 show vlan extended
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 16   Tenant18:2Tier_AP:AppServers_EPG vlan-1181        Eth1/28
 17   Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,
                                                        Eth191/1/28, Po2
 56   Tenant18:2Tier_AP:AppServers_EPG vlan-1380        Eth1/28, Eth1/48,
                                                        Eth191/1/28, Po2

----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 15   Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,
                                                        Eth192/1/28, Po2
 48   Tenant18:2Tier_AP:AppServers_EPG vlan-1380        Eth1/28, Eth1/48,
                                                        Eth192/1/28, Po2

Here I have mapped VLAN 1181 and 1183 statically to the EPG on leaf 1201, and VLAN 1380 was mapped dynamically via the VMM Domain. I'll call these the user assigned VLANs to distinguish them from the internally assigned VLANs.

But what I really want to point out is the VLAN ID in the first column. This is the internal switch mapping for each user-assigned VLAN - and look - they are different VLANs on each switch (as you discovered on your setup)

It is this internal mapping on each switch that makes it possible for ACI to support tens of thousands of VLANs across the whole fabric - each switch is only responsible for mapping the VLANs it sees, and like ordinary L2/L3 switches, they will may out at 4095 VLANs

One final point - when assigning a VMM Domain to an EPG, it is possible to specify the VLAN ID you want to use.  So if you want consistency of VLAN numbering you can

1. Create a dynamic VLAN pool that has the VLAN IDs that you want to be the same in the Physical Domain and the VMM Domain
   1. Add a static range to the VLAN pool with the VLANs you want to use in both places
2. Use the same VLAN pool for both your Physical Domain and VMM Domain. So you need to plan this before its implemented!!!!
3. When assigning the VMM Domain to the EPG, choose the Static option and type your chosen VLAN in the format vlan-nnnn 

[Joseph Rinehart]

I hope this doesn't muddy the waters, but VXLAN does this as well. You can map an incoming 802.1q VLAN to a specific VNI and map another VLAN on the other side. I typically map the same external VLAN just to make it less confusing.