Jumbo frames N5k

[BastiiGee]

Hello all, Maybe someone can help me. We have several nexus N5k in use all with version 7.3. When I look at different trunk port channels I can see jumbo packets everywhere. But the MTU is not higher than 1500 on any of the interfaces. Also in the Config I don't find MTU adjustments anywhere. I know that with the N5k series the setting is controlled by a policy and that the actual setting is shown with the command sh queuing... but even here 1500 is set. I can find several Cisco community posts about this. Many have the same phenomenon but no one can answer it. Maybe one of you can. Why are "jumbo packets" (L3???) displayed where there are none. Also with trunks to another switch jumbos are shown, but the other switch has my giants or errors.

[MHM Cisco World]

Switch#show system internal ethpm info interface eth x/x | egrep -i mtu

can you share this  

[BastiiGee]

Hello MHM Cisco World,

here is the output:

medium(broadcast), snmp trap(on), MTU(1500),

[Sergiu.Daniluk]

Hi @BastiiGee 

There is a great article on how to configure and verify both Layer2 and Layer3 MTU on all Nexus platforms:

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/118994-config-nexus-00.html

For your specific Nexus platform. the L2 MTU is configured via network-qos policy and applies system wide:

policy-map type network-qos jumbo
  class type network-qos class-default
          mtu 9216
system qos
  service-policy type network-qos jumbo

To verify the MTU you must use the queuing outputs command:

Nexus# show queuing interface ethernet 1/1

Take care,

Sergiu

[BastiiGee]

Hello Sergiu,

Thank you for your feedback.
I would like to clarify this :
I don't want to change the MTU of the Nexus, I want to understand why on the trunk portchannels jumbo frames are seen, although no setting regarding the MTU has been adjusted.
Also a cat9500 connected to the Nexus does not see any incoming giants or errors, although the Nexus counts outgoing jumbos on the interface.

[MHM Cisco World]

the output you share is indicate that NSK use MTU 1500, and why you see jumbo frame, do you check server connect to NSK ? it can be the source of jumbo is Server. 

[BastiiGee]

Yeah i check, Ports conncected to ESX Hosts.
But i wonder why the Port of the Nexus accept them. I my undersanding the switch should drop it as Giant or input error.

[MHM Cisco World]

because the NSK is cut-through not store-and-forward, 
it accept the traffic forward then mention that the traffic have error. 

[BastiiGee]

When the Nexus forwards to another Switch, this switch should also know about a greater size of the frame.
But it doesn`t. I think this is something like a bug. We dont have jumbos in our network.

[BastiiGee]

Many user of N5K are expediting this issue: The nexus counts jumbos but there aren't jumbos.
https://community.cisco.com/t5/other-data-center-subjects/nexus-5000-jumbo-packet-counters-bug/td-p/1913665

[Sergiu.Daniluk]

Ah sorry. I misread your post initially. I get it now.

First, just to clarify the terms:

• Jumbo Packets = packets with MTU > 1500 and <= 9216
• Giants = packets with MTU > 9216 (this being the maximum configurable MTU on Nexus platforms)

In other words, regardless of the MTU you have configured, you will never see giants increasing unless of course the switch receives such frames.

Second, you say you see jumbo counter increasing on egress. How about ingress, do you see any increasing jumbo counters?

Third, are any of your SVIs configured with MTU > 1500?

Cheers,

Sergiu

[BastiiGee]

No worries, thank you for helping me.
To your second question: Yes I also see ingress jumbos. 
I also understand that the Nexus always forwards no matter of the packet length. 
The point which I don't understand is the following:
On the Uplink Port Nexus to a Cat9500 i also see jumbos in and egress.
The Cat9500 with also a MTU size of 1500 doesn't count jumbos or in this case i would expect counting errors. 
When i do a "Show Interface" on the Cat there is no Jumbo counter. So could it be that IOXE just doesnt count jumbos packets until they reach the max > 9216.(counted as giant) 
I would also expect issues and trouble and also error counts if there where any mismatches.
 Third question: This is all layer 2 there are no SVIs.

[MHM Cisco World · Accepted solution]

Now we agree that NSK can accept the frame and then mark it as jumbo. 
the other Q from where ?
@Sergiu.Daniluk mention the size after that the NSK count the frame as jumbo. 
there is chance that EXSi send frame that is large than 1500, it can send 1518-1522.
So it can mismatch between IEEE use between two ESXi and Nexus.