Cisco ACI MTU size - where did the other 216 bytes go?

[cowser1979]

Hello -

Going through the global fabric policies in my aci fabric (1.03n), it would appear that you cannot set a system MTU size of greater than 9000 bytes?

FABRIC -->Fabric Policies-->Global Policies-->Fabric L2 MTU Policies

My questions is, where did the other 216 bytes go?  Will ACI ONLY support frames <9000 bytes?  My understanding was that VXLAN only required an additional 50 bytes of overheard, so how/what is using the rest?  Or is that just reserved for future use?

As a rule of thumb I am telling all of my server teams to ratchet down their MTU sizes to 9000 even, before we migrate, does that sound like good logic? The majority of my hosts are UCS VMs connected to my 9ks with an Fi (that the server team manages).  Is there a best practice, as it relates to NIC and FI MTU settings that we should be adhering to, when connecting to the ACI fabric?

thank you

[Joseph Ristaino]

This is a great question and it isn't obvious.

The MTU 9000 is a configuration for the front panel ports of the Broadcom that the servers connect to.  This setting is for packets that the servers will be sending us and that size cannot be bigger then 9000.  This is fine because most OS's and adapters only support up to 9000 anyways.

The NorthStar ASIC can support larger then 9000 MTU and is defaulted like that without any config needed in the APIC.  Because of this, the 50 Byte overhead is not an issue, and the packets should not be fragmented when VXLAN comes into play by default.

I believe you opened a TAC case for this issue last week.  I can check the case, but was there actually data traffic impacted with the OS at 9000?  If there is, I would like to investigate it.

Joey

[cowser1979]

Thanks Joey!

No issue as of yet.  I just needed documentation to take to my server teams to tell them to move down their MTU sizes on the UCS FIs that they manage.  According to them they have been setting the MTU sizes on their FIs to 9216 and the actual UCS servers/VMs to 9000.  It sounds like I don't have anything to worry about, since their OS' cant support a frame larger than 9000 bytes, its hard to see how the FI would pass a frame >9k bytes to my fabric.

My only other question is regarding that menu option.  If I am reading my TAC response correctly, it sounds like that menu setting is for fabric interface MTU sizes, not host ports?  or is that not correct?

thank you for your help with this!  

[Joseph Ristaino]

Glad we are all good on the data plane front.  For the UCS part, you nailed it.  Since there won't be any frame larger then 9000 coming from the FI, the fact that the FI is at 9216 will not be an issue.

The help documentation for the "Fabric L2 MTU Policy" is a bit misleading.  It makes it seem that this is only setting the MTU on the fabric ports facing the spines.  However, if I step that down, I see the change get pushed to my front panel ports facing my FI's: see screen-caps.

I read it again, and it does make sense:  "The Layer 2 instance policy, which is used for configuring fabric-wide layer 2 settings."  They are referring to the fabric as a whole and not just the "fabric facing" ports.

If you have any recommendations for the way the help is written, please let me know and I will file a doc defect to get this added.

Joey

[cowser1979]

Awesome, thanks for your help!

[gregnewuk]

cowser1979,

Did your UCS team leave the MTU as you described? ie Best Effort at 9216 and vNIC Templates at 9000?

I'm a UCS guy and working with the ACI team!

Cheers,

Greg