VXLAN UNDERLAY - MTU considerations

[MambaRod16]

Hello, 

In VXLAN, which are the interfaces that must be configured with an MTU value of 9216?

It must be the physical interfaces that interconnect the Leafs with the Spines?

The SVI interfaces (Ex: vrf Tenant-1) for the servers must have MTU of 9216?

The SVI interface for the L3VNI must have an MTU of 9216?

[horia.gunica · Accepted solution]

Hello Carlosperez!

The following interfaces MUST have MTU 9216 :

 - L3 VNI Interfaces (Ex: SVI for Tenant-1)

 - IGP (IS-IS or OSPF) links - this includes

      - Leaf-Spine interfaces

      - Leaf-Leaf SVI interfaces for vPC Peer Peer Links in case of vPC

For the L2VNI SVI Interfaces ( aka the service interfaces with subnets in Tenant-1 for example) - you set the MTU what you want the MTU for the service to be . For example in our production environment we set this to 9100 - in order for additional encapsulation to occur if needed - and also give servers jumbo MTU (9000) if needed . This worked fine for us until now.

Best regards!

[Sergiu.Daniluk · Accepted solution]

Hi @MambaRod16 

Just wanted to add something on top of what @horia.gunica already mentioned.

Technically speaking, the requirement for the underlay is to increase the MTU by 50 bytes, on top of the overlay MTU (service traffic MTU). Ergo, if the overlay uses 1500 byte MTU, the underlay needs to be configured with a minimum of 1550-byte MTU. This requirement come from the MAC-in-UDP encapsulation, where VXLAN introduces these 50-byte overhead.

If the servers are sending traffic with MTU higher then 1500, it is required to have jumbo MTU enabled both for the overlay and underlay.

Overlay:

+ SVI interface for the servers (servers gateway)

Underlay:

+ L3VNI (the SVI associated to the tenant VRF)

+ L3 underlay network (on all L3 interfaces on the path between the VTEPs)

Stay safe,

Sergiu