VxLAN VNI to VLAN Mapping

[visitor68]

Folks - this may seem like a pretty fundamental question, but the implementation details still escape me. While it is the case that VxLAN offers up to 16 million VNIs in principle, every time I see an implementation of VxLAN, there is a 1:1 mapping with VLANs. In that case, you're not getting more than 4,096 VNIs. So, how does one take advantage of the 16 million VNI capability? 

Before seeing these ubiquitous implementations, I always assumed a hierarchical model in which a VNI would map to multiple VLANs - up to 4096. But that is not the case it seems.

[carl.fuller]

At face value, I get you.  Who needs more than 4000 VXLANs?  Service Providers do.  It's a lot easier to visualize it.  This is just a small set of services.  In reality, a service provider could have a ton of services on top of just what's in the (crappy) attached diagram.  You may not use 16 million, but there are scenarios where the VXLAN may need to be extened to another fabric in the same site if your requirements are great enough to require multiple fabrics connected via super-spines.  Note that VLAN1112 is used on multiple switches in the environment.  That's because they are locally significant and the tags aren't carried across the fabric.  Imagine how tough automation would be if you required the tag to match on both ends...you'd have to have additional logic just to find a matching VLAN assignment on each end.

[yang yang]

since the vxlan is only add head to the L2 package can i do the following?

• VNI 5001 - VLAN 20 10.20.1.0/24
• VNI 5001 - VLAN 30 10.30.2.0/24
• VNI 5001 - VLAN 40 10.40.2.0/24
• VNI 5001 - VLAN 50 10.50.2.0/24

[craig.armstrong]

---

@yang yang wrote:

since the vxlan is only add head to the L2 package can i do the following?

• VNI 5001 - VLAN 20 10.20.1.0/24
• VNI 5001 - VLAN 30 10.30.2.0/24
• VNI 5001 - VLAN 40 10.40.2.0/24
• VNI 5001 - VLAN 50 10.50.2.0/24

---

From what i understand above, no beacuse the VTEP doesn't encapsulate the VLAN tag in the Vxlan IP packet but :

One one VTEP you'd have : (Because of the 1 to 1 mapping)

• VNI 5001 - VLAN 20 10.20.1.0/24
• VNI 5002 - VLAN 30 10.30.2.0/24
• VNI 5003 - VLAN 40 10.40.2.0/24
• VNI 5004 - VLAN 50 10.50.2.0/24

On multiple (4) VTEPs you could have :

• VTEP1 VNI 5001 - VLAN 20 10.20.1.0/24
• VTEP2 VNI 5002 - VLAN 20 10.30.1.0/24
• VTEP3 VNI 5003 - VLAN 20 10.40.1.0/24
• VTEP4 VNI 5004 - VLAN 20 10.50.1.0/24

Or, still across 4 VTEPs :

• VTEP1 VNI 5001 - VLAN 20 10.20.1.0/24
• VTEP2 VNI 5001 - VLAN 30 10.20.2.0/24
• VTEP3 VNI 5001 - VLAN 40 10.20.2.0/24
• VTEP4 VNI 5001 - VLAN 50 10.20.2.0/24

[makharsa121]

Ok so the confusion here I believe is that you are assuming that inside of the same VNI we cannot have more than 1:1 mapping to vlans so therefore only 4000 VNI can be supported.

Note that VXLANs are not necessarily mapped to vlans
there are 2 ways to identify which traffic belongs to which VNI
1- Through subinterfaces (traffic can be untagged or tagged)
2- Through VLAN (traffic must be tagged)

Well, for the second type, the answer is yes inside the same VTEP. But if we go to different VTEP then you can have additional 4000 vlan and additional 4000 VNIs.

And if we distribute the VMs across multiple VTEPs (VNIs) then surely you can have 16M VNIs. 

VLAN here is not used in the old way of segregating L2 traffic, it is used by the VTEP to only identify which VNI is connecting to which VM. 

VLAN 1 ---> VNI 1 ---> VTEP 1

VLAN 1 ---> VNI 2 ---> VTEP 2

VNI 1 can communicate to VNI 2 through L3 BDIF. And you can have up to 16M VNI.

Only 4K VNIs can exist in one VTEP. But if there is another VTEP, it can have additional VNIs up to the theoretical maximum of 16M. 

[Umesh Shetty]

Lets try to do this via an example and lets to scale to 16 million for now I will try to explain how VLAN's are insignificant across a fabric and how VNI's take over the entire fabric and VLAN's are local to a Leaf/Switch

Assume a fabric that has 100 Leafs and each Leaf has 40 VLANs Vlan1 to Vlan40  configured. Hypothetical but good for understanding 

 Each Switch has the same 40 VLAN's but when an endpoint on Leaf1 with say VLAN tag 1 sends a frame to an Endpoint on Leaf 10 to a system in VLAN 10 the Leaf 1 encapsulates it in  a VXLAN header and assigns says VNID 1000. At the Leaf 10 the VXLAN header is stripped off and the original VLAN is now insignificant cause the Leaf 10 now knows where the endpoint is connected on the Leaf and sends the frame out of that Interface 

  This was basics of VXLAN but was important to drive the concept through. The entire fabric now can use its VNID's irrespective of the local VLAN's configured. To really get to 16 million VNID's you need an insanely large network but you get how even with 4096 VLAN's and a massive network the fabric can use the 24 bit VNID's to scale 

  Fundamentally VXLAN should never be compared with VLANs. VLANs were a flooding domain whereas VXLAN is a tunneling protocol so the other end does not really bother about the VNID

Hope this helps !!! 

[usmansa2]

hi, 

I am not sure whether this topic is still relevant, what i understand from whole explanation of VXLAN is that  VLANs are locally significant to the device, for example you can use 4096 VLANs on one device and in normal trunk network these VLANs are significant across the whole layer 2 domain but with the introduction of VXLAN in the network these VLANs are now just locally significant to switch level only, so if you use 1000 vlans on switch like first 1000 (1-1000) then you don't have to use first (1-1000) on switch to get the information across, you can use 1001 to  2000 on the next switch and communication will still happen if they are part of the same VNI so here your layer 2 broadcast domain is not VLAN its VNI, VLAN has just remained one component which is used on access layer so thats why it is said your broadcast domain is not limited to now 4096 it has enhanced to 16 million (number of VNIs)   so in my view the correct statement should be "with VXLAN you can have 16 million broadcast domains rather than 4096"

[Mohamed Gaber]

I did not study VxLAN for Cisco, but I did for another vendor. I understand That VxLAN configuration on VTEP is done on two steps, map VLAN to BD and then map BD to VNI. We could consider VNI as customer, tenant, application, department and so on. any of the mentioned tenants could have multiple subnets. So, BD is a subnet and has BDIF, while VNI is an isolation for a group of subnets. VNI seems to me as the concept of VRF. VRF isolates a routing domain or tenant and VN isolates L2.

[Mohamed Gaber]

VLAN ID to VXLAN ID mappings enable a great degree of flexibility for network designers that VXLAN-based bridge domain (BD).

VXLAN-based bridge domains can be mapped to single or multiple 802.1Q VLANs as well as to dual IEEE 802.1ad VLAN tags, covering all possible design requirements.

The different available configuration models for VXLAN-based bridge domains are:

Single tag mapping, in which an outer VLAN tag (for example, a customer ID) is mapped to a VNI on an IEEE 802.1ad port and the inner VLAN tags are preserved inside the VXLAN encapsulation. This type of mapping can be used on customer facing ‘QinQ access’ interfaces.
Double tag mapping, in which an outer VLAN + inner VLAN tag pair is mapped to a VNI on an IEEE 802.1ad port (traffic is received double-tagged in ingress and is marked with two tags in egress after VXLAN decapsulation). This type of mapping can be used on multi-VLAN ports (sometimes called QinQ trunks) facing for example an external cloud provider.
Single 802.1Q tag mapping, in which a single 802.1Q VLAN (or multiple 802.1Q VLANs) are mapped to a common VNI (for example, for inter-DC communication within the same customer’s private cloud network).

One VLAN can be associated with only one BD, but one BD can be associated with multiple VLANs.

[RedNectar]

@Mohamed Gaber ,

Another response! 

Old threads like this really should be locked.  If you have a new question or topic, write a new question or write a blog post where we can have a discussion about your (interesting) ideas rather than resurrecting an old thread.

[Mohamed Gaber]

https://support.huawei.com/enterprise/en/doc/EDOC1100116686?section=j008

Sorry, for adding a link for another vendor, but I believe they are the same since Cisco is the leader.

[RedNectar]

Hi @Mohamed Gaber ,

I'm not sure what your contribution is really about (on this already ancient thread that should be closed) but I feel I need to sort out some of you misconceptions in case someone actually reads this incredibly boring thread.

---

Firstly, let's deal with the ORIGINAL question, where @visitor68 

1. Did not actually SAY he was referring to the ACI implementation of VXLAN, not did they put it in the ACI discussion forum  - and curiously seemed that they were surprised that they did not get an answer 
2. @visitor68 's observation was that (presumably in ACI) there is a VLAN allocated to every VXLAN.
   1. What they FAILED to notice is that this is a limitation PER ACI LEAF - and seems they took that limitation to mean mean that the limitation was FOR THE WHOLE FABRIC.
   2. This is not so, every switch get a whole new set of 4096 VLANs to map, so to answer @visitor68 's question
      "how does one take advantage of the 16 million VNI capability? "
      The answer is simply to use MORE THAN ONE switch.

@visitor68 If this answers your question, it is a great idea to mark the question as being answered.  This helps:

1. others with a similar problem find the correct answer
2. people who look for "unanswered" questions to answer finding this
3. prevent your question from becoming a "dead thread"

   ---

Now to @Mohamed Gaber comments

I did not study VxLAN for Cisco, but I did for another vendor. I understand That VxLAN configuration on VTEP is done on two steps, map VLAN to BD and then map BD to VNI.

You are correct that every BD is allocated a VNI and (possibly many different) VLANs. But your description is NOT the way it happens in ACI:

1. Firstly, a VNID is allocated to a BD - this is unique throughout the FABRIC
2. Then, as each Leaf Switch REQUIRES it, the BD is PUSHED to the Leaf Switch, and EACH Leaf Switch allocates a VLANID to map the VNID locally to that switch. Every Leaf Switch does this independently, so every Leaf Switch may have a different VLAN ID allocated for the same VNID
3. And finally, the allocation is not done on a "VTEP" basis, it is done on a Leaf Switch basis. I hate to be pedantic, but it matters here:
   1. A Leaf Switch is often referred to a a VTEP, but in fact a Leaf Switch may be allocated several VTEPs
   2. Some VTEPs are allocated to devices OTHER than Leaf Switches (like APICs and Spine Switches)

We could consider VNI as customer, tenant, application, department and so on. any of the mentioned tenants could have multiple subnets. So, BD is a subnet and has BDIF, while VNI is an isolation for a group of subnets.

Well, whoever "we" is can consider a VNI as whatever you like.  Here is how ACI treats VNIDs

• Every VRF gets allocated a VNI. This is unique throughout the fabric
  • Note: In Multi-site ACI, each site is a new fabric, in multi-pod ACI there is ONE fabric
  • As each Leaf Switch REQUIRES it, the VRF (and its VNID) is PUSHED to the Leaf Switch
• Every BD gets allocated a VNID. This is unique throughout the fabric
• Every EPG gets allocated a VNID. This is unique throughout the fabric, but is used ONLY for transporting Spanning Tree BDPUs
  • Every EPG also gets a PC Tag that is unique throughout the VRF if it is > 16384, or
  • unique throughout the fabric if < 16385 
    • EPGs that provide contracts to EPGs in a different VRF are allocated PC Tags < 16385

VNI seems to me as the concept of VRF. VRF isolates a routing domain or tenant and VN isolates L2.

---

OK - let's unpack that last statement

• VNI seems to me as the concept of VRF.
  • Well, in ACI a VNID is used for MORE than just identifying VRFs - Cisco has made this concept very powerful
• VRF isolates a routing domain or tenant
  • A VRF does indeed isolate a Routing Domain BUT NOT a tenant. A tenant may have MANY VRFs in ACI
• and VN isolates L2
  • A VN? - not sure if you mean VLAN or VNID.  If you mean VLAN, you are correct that a VLAN can isolate L2, but in ACI the 802.1Q VLAN ID is used to identify EPGs, and L2 isolation is maintained using a combination of VNID and mapping that VNID to a different VLAN on each switch, giving the potential to have MANY more L2 domains than what a pure VLAN based system would

[Mohamed Gaber]

Thanks so much for the details. I started reading about VxLAN shortly for an SDN-Based-VXLAN university campus project. I have several questions and try to find the answer to them. The details of the technology are not clear to me when you come to the low-level design and the configuration level.

[chetan.shivamurthy]

even i too also have the same doubt ...