Jumbo Frames, MTU 9000 or 9216?

[Boudewijn Plomp]

Hi,

There are several blogs and white papers that describe how to configure QoS and Jumbo Frames. But they differ in therms of MTU size. Most official UCS based white papers configure an MTU size of 9000. What is best-practice for a FlexPod? Configure UCS, Nexus and NetApp all with MTU 9000? And, is there is reason for this? (e.g. 6x1500=9000 for fragmentation)

I hope this is not a stupid question but I have seen I am not the only one who is confused about it.

Thanks,

Boudewijn

[davidng]

Boudewijn,

There are several places you need to set the MTU size of and the max MTU is allowed.  Within the UCS system itself, the MTU configuration needs to be done at the FI and vNIC.  The max MTU allowed is 9216 and 9000 respectively.  The FI MTU should never be less than the MTU size of the endpoints.   Just remember, the FI is a simple L2 device and doesn't do any fragementation.  Therefore, it is okay to set the MTU on the FI to 9000 but no harm at setting it at 9216 in case of any padding added by OS, intermediary devices (vFW, vRouters, etc), etc.

Hope this helps,

David Nguyen

[Boudewijn Plomp]

For you information. We have just succesfully implemented Jumbo Frames (MTU 9000) in our envornment on UCS, Nexus and NetApp. So far we have tested, it shows a big performance improvement.

[anthony.denicola]

When we setup our environment over two years ago we set it up with the MTU of 9000 in the LAN Cloud QOS System class.  We are a FlexPod reference design, running iSCSI for our VMware storage.  This worked fine up until we got new controllers for our cluster mode; FAS8060 with NetApps UTA2 ports.  We were not able to run Jumbo frame storage connections to these.  After over a month of troubleshooting, it came down to the UCS Fabric Interconnections were dropping the Jumbo packets. 

Fix: Change the MTU from 9000 to 9216 in the LAN Cloud QOS System class.  Once this was done everything connected and tested fine. 

Reason: As I understand it, our other controllers used on straight 10GB Ethernet adapters.  However, the new UTA2 adapters on the NetApp are CNA cards and act as FCoE even when running Ethernet only traffic.  Because of that there is extra packet header information.  This caused the packet to be larger than 9000 and the FI dropped it.

As stated above there is a lot of information about Jumbo Frames with the NetApp and UCS with conflicting configurations.  We found our original docs used and it had conflicting settings in it as well.  However, if you do as mentioned above use the FlexPod reference design it does state to use 9216 for your settings.  I would say if you are connecting to a NetApp, even if you are not doing a reference design, use the design as a guide for settings.