Jumbo MTU confusion

[patoberli]

Hello all

I'm currently very confused about the Jumbo MTU configuration on Nexus 9336C and 31108TCV running NX-OS 9.3(7).

As with the old Nexus 5548, I've configured a network qos policy. On the interfaces with servers attached to, I see the Jumbo Frames counter increasing, while the Giants and input error counter remains at 0. But the servers don't work with jumbo frames on the new Nexus and work again once plugged into the old switch.

About the confusion, this document here states that if the Jumbo counter increases while the Giants doesn't increase, it should be correctly forwarded:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/interfaces/configuration/guide/b-cisco-nexus-9000-nx-os-interfaces-configuration-guide-93x.pdf (Page 94, or document page 72)

While this document here states, that I probably should configure it on a per interface basis:

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/118994-config-nexus-00.html

So which one is correct? For time reasons I can't currently further test this until the new maintenance window. 

[patoberli · Accepted solution]

We had now the chance to test this.

It's indeed the case with the Nexus 9000 series that the network qos policy isn't working (or not enough), the MTU must be adjusted on a per port basis. Now everything is working as it's supposed to. 

No idea why the error counters didn't count up (or I looked at the wrong place, cut through hardware and such). 

[Sergiu.Daniluk · Accepted solution]

Hi @patoberli 

There are two aspects that needs to be clarified:

1. Difference between Jumbo Packets and Giants:

• Jumbo Packets = packets with MTU > 1500 and <= 9216
• Giants = packets with MTU > 9216 (maximum configurable MTU on Nexus platforms)

2. How to configure Jumbo MTU on Nexus platforms. This can be broke into two parts:

• Layer 3 MTU (for routed traffic)

All Layer 3 ports, regardless of Nexus platform (N5K/6K/7K/9K), are configured on a per-port basis.

• Layer 2 MTU (for switched traffic)

On Nexus 5K/6K and some models of N3K, you modify the L2 MTU using a Network QoS policy

policy-map type network-qos jumbo
  class type network-qos class-default
          mtu 9216
system qos
  service-policy type network-qos jumbo

             On Nexus 7K/9K and some models of N3K, you modify the L2 MTU on per-switchport basis

Nexus(config)#interface ethernet 1/1
Nexus(config-if)#mtu 9216

Stay safe,

Sergiu

[patoberli]

Thanks for the more in depth explanation. 

I'm still not sure about the giants though, because the manual states:

Configuration	Packet Size	Incremented Counters	Traffic
L2 port – without any MTU configuration	6400 and 10000	Jumbo, giant, and input error	Dropped

So at least by my understanding, if I send a packet between 1501 and 9100 it should be dropped and the input error and the giant counter should increase by 1 per packet.

[Sergiu.Daniluk]

It might be increasing in cases as described in documentation, and myself to not pay attention to giants., but as far as I remember, I haven't seen giants increasing unless the packets where largerthen the maximum MTU. 

However, the documentation clearly has some problems, since it refers to network-qos on Nexus 9000 platforms: L2 port – with jumbo MTU 9216 in network-qos configuration

Anyway, I think this needs to be tested to be sure.

Stay safe,

Sergiu

[patoberli]

Yeah, at least in my case, on the input port, the counters did not increase (with the network qos configuration), but the packets were dropped. 

[Sergiu.Daniluk]

That would be my expectations as well. But who knows, maybe we misinterpret the documentation. ^_^

[Gabor M.]

Do you guys know any cisco documentation about what giant is?

[patoberli]

You will find some information here: https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6000-series-switches/24048-148.html

[Gabor M.]

I am not satisfied with that document because it talks about the baby giants. Are these the same as giants?

In that document it is stated that the baby giants are smaller than the jumbo, nevertheless, Sergio said here that the giants are larger than the biggest jumbo.

Nexus 3548, NXOS: version 9.3(11)

[patoberli]

Here some more info:

The Giant (Jumbo) Frame

Any frame which is received and which is greater than the maximum frame size, is called a "giant". In theory, the jabber control circuit in the transceiver should prevent any node from generating such a frame, but certain failures in the physical layer may also give rise to over-sized Ethernet frames. Like runts, giants are discarded by an Ethernet receiver.

Jumbo Frame

Some modern Gigabit Ethernet NICs support frames that are larger than the traditional 1500 bytes specified by the IEEE. This new mode requires support by both ends of the link to support Jumbo Frames. Path MTU Discovery or PLPMTUD is required to utilise this feature, since there is no other way for a router to determine that all systems on the end-to-end path will support these larger sized frames.

Source: https://erg.abdn.ac.uk/users/gorry/course/lan-pages/mac.html

And here another (old) document expaining the difference between the two:
https://community.cisco.com/t5/switching/what-cisco-considers-jumbo-and-baby-giant-frame/td-p/1491758