VXLAN underlay in 802.1q

[mspiegelman]

While testing a non-standard VXLAN scenario on 7.0(3)I7(5), I noticed a VTEP will not send traffic to the underlay network when the leaf/spine connection is 802.1q tagged (i.e. the VTEP's L3 interface is a VLAN interface and the connection between leaf and spine is a trunk).  I was able to get this scenario working by changing the trunk's native VLAN to match the VLAN interface I used for the underlay so it seems VXLAN doesn't have a problem with the command "switchport mode trunk" (as opposed to "no switchport" with an IP directly on the interface) but it does have a problem adding an 802.1q tag to the traffic.

Why doesn't VXLAN traffic work when it is 802.1q tagged between leaf and spine?  Is this a bug?

[ADP_89]

Hello,

There are some limitations on VXLAN deployments depending on the specific platform, the NX-OS version and even the card where the uplink to spine is connected (ALE/NFE).

I normally look up on the configuration guidelines to see if I am hitting one of those limitations.

If you do not find you answer let us know what platform you are using and where the link between leaf and spine is connected(ALE/NFE)

Thanks,

ADP

[mspiegelman]

I was testing this out on NX-OSv.

[ADP_89]

I couldn't find much technical explanation during a brief search online other than 

https://www.cisco.com/c/dam/m/sl_si/events/2016/cisco_dan_inovativnih_resitev/pdf/cisco_day_slovenia_2016_vxlan_marian_klas_final.pdf

If you check page 26 they say that the spine-leaf link should be a routed interface, not switchport.

I never saw guidelines with you setup to be honest.

IMHO I see this as an enforcemente of the CLOS topology where the L2 should be left only on the leaf access ports. Having a l2 trunk between spine and leaf will vanish this principle. 

HTH,

ADP

[mspiegelman]

---

 If you check page 26 they say that the spine-leaf link should be a routed interface, not switchport.

I never saw guidelines with you setup to be honest.

IMHO I see this as an enforcemente of the CLOS topology where the L2 should be left only on the leaf access ports. Having a l2 trunk between spine and leaf will vanish this principle.

---

You are saying page 26 is a requirement not a recommendation.  While I agree L3 links are clearly desirable to L2 links since the later introduces spanning-tree, my scenario is working (i.e. I'm not using "no switchport" on the interface).  If what I described is deliberate enforcement, it shouldn't work at all.

I ran VXLAN with a VLAN interface over an L2 access port and l2 trunk port but to get the trunk scenario working I had to match the trunk's native vlan to ID of the VLAN interface.  It seems the VTEP has a problem sending VXLAN packets when the frame includes tags since the native VLAN is tag free.  I'm not sure if this is an oversight/bug or a VXLAN restriction.

In either case, this should be documented but I don't even know how to report this bug/undocumented restriction to Cisco since NX-OSv is unsupported.

[ADP_89]

Hello,

Found this:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_011.html

VTEP does not support Layer 3 subinterface uplinks that carry VxLAN encapsulated traffic.

Point to multipoint Layer 3 and SVI uplinks are not supported. Since both uplink types can only be enabled point-to-point, they cannot span across more than two switches.

HTH,

ADP