Jumbo MTU Frame ping test over 9K Nexus switches

[William Foster]

We have a Nexus 9504 in one DC and Nexus C9396 in another DC. Currently on a I am testing a new circuit between the 2 DCs with MTUs of 9000. When I ping the Highest I can get to pass is a ping with df-bit packet size 8972. This then sends a ping of 8980. I feel like this is correct but really wanted some confirmation on this. I need to be sure my circuit is fully utilizing the full 9000 MTU. I believe the nexus throws the 8 Byte header on for ICMP my question is the other 20 bytes I am not positive about.

[francisco_1 · Accepted solution]

have you tried sending more frames? you seeing packet loss?   jumbo frames can be up to 9216 bytes in size but you have to take in to consideration the addtional headers 9216 data + 8-bytes ICMP + 20-bytes IP = 9244-bytes. if you have an end system with an MTU of 9216-bytes the command you should use to test end to end jumbo frame connectivity would be ping <ip-address> df-bit packet-size 9188. That way when the 8-byte ICMP and 20-byte IP header are added, the packet size will be 9216-bytes.

The other area to watch out for is Control Plane Policing (CoPP) on the Nexus. The Nexus switches runs CoPP by default to limit traffic hitting the CPU on switch supervisor. I've seen issues in the past with ICMP as the default value used within the policer quite low, and if you're specifying large packet sizes with the ping it's possible CoPP is dropping the traffic. You can use the show policy-map interface control-plane | in "class-map|violate" command to quickly check if you have drops in any of the classes.

[William Foster]

Sorry for the late response. I want to that you for answering my question. We for reason I am not sure why, have our mtu set to 9000. Your answer of the consideration of the 8-byte ICMP + the 20-bytes IP answered my question as to why I can only send a frame size of 8972. 

Thank you very much

[francisco_1]

you welcome

[Darian O'Dirling - TCE]

Hi @William Foster ,

I see you are working with MTU configuration. I encourage anyone who is running into this issue or wants to learn more about MTU configuration on Nexus 9000 series switches to watch this video made by a colleague mind. The video outlines the need for proper MTU in the network and explains how to remedy MTU related interface errors, via interface MTU configuration.

https://www.youtube.com/watch?v=GbBeB6h9MKk

[whistleblower14]

Hi all,

please allow me also a question regarding the MTU configuration on the N9K because I´ve a massive problem in understanding the following behavior/documentation

I´ve a test enviroment where we use the default Ethernet MTU size of 1500Byte on the physical interfaces and also the default Jumbo MTU of 9216Byte in the system configuration! On the N9K there`s a FEX and on that FEX a Lab-Device (Router C7206) is connected! On the Router physical Interface which is connected to the FEX is MTU of 9216 Byte manually configured!

When I now test with ICMP to another Device e.g. ping 10.10.10.10 size 1568 df-bit - it`s working without a problem but a ping with size 1569Byte and bigger /w DF-Bit set doesn`t! -> the first question for me... how/why can this work when the N9K Interfaces, where these packets have to travel through are configured with 1500Byte?

I´ve done some research and found the following documentation... Cisco Nexus MTU troubleshooting counters - Cisco
here`s exactly the behavior I´m facing described but for me not clearly to understand...
Fragmentation and MTU mis-match

• If Path is L3, fragmentation takes place, packet will not be dropped.
• If Path is L2, no fragmentation takes place, packet will be dropped completely
• Initiate [ICMP] with packet-size 1540B & has L2 in path still the you dont see the drops, where total size becomes 1568 [1540+20+8]
• Initiate [ICMP with ]packet-size 1541B, total packet becomes 1569, and you see the drops, and drops are seen as Giants counters

I looked as well in the configuration guide Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide, Release 10.3(x) - Configuring Basic Interface Parameters [Cisco Nexus 9000 Series Switches] - Cisco

to see, if the N9K is doing something special in the back which affect the MTU and found the following:

Modifying Interface MTU Size

By default, the Cloud-Scale ASIC NX-OS system always allows an extra 166B in the MTU on top of the configured value in order to fully support/accept different types of encapsulations in the hardware.

Question 2 for me... if 166Byte are extra to 1500 = 1666Byte - correct? so why does the ping with 1569Byte not work too?

I´m really struggling to understand that behavior and would be very happy if someboy could help me! Thank you!

[Joseph W. Doherty]

BTW, just to mention, always keep in mind there's (yet) no standard for how big a MTU a jumbo Ethernet frame should be.  So, it's not uncommon with different same vendor devices, different vendor devices and/or MAN/WAN links not all supporting the same maximum jumbo.

I.e. sometimes you need to configure your jumbos for the least common denominator (if equipment allows).