Question regarding Jumbo Frames, MTU and adjustment results

[pclarktx]

Hello,

I have a UCS M-200.  We are having some network issues.  The issue has been hard to pin down but one of the places we see a possible problem is there are 2 vNIC's that have MTU 9000 set on them for our storage network.  The network is a 10G network with MTU's set on endpoints to 1500.    My question:

1.  If my vNIC's are set to 9000 can this cause an issue?  I've been told two different things.  I've been told yes it can and then also no it will not as the setting is just allowing the vNIC to use an MTU of 9000 as it's maximum if needed. 

2.  What would the affect be of changing the MTU in the vNIC config?  I have been told two different things on this as well.  I've been told that it will just be a small outage while the Nic resets and then I've been told that the blades will reboot.

Any help would be appreciated.  Thanks.

[Kirk J]

Greetings.

For your first question about vnic being set to 9000 when overall network doesn't support it, it depends.

I've seen windows boot from iSCSI san cases where the installer started and due to windows baremetal OS reading the 9000mtu setting from the VNIC, it tried to communicate with iSCSI target LUN which of course lost communication due to frames being fragmented and dropped.   VMware is a bit different due to the VMK port, and vswitch/port-groups defaulting to 1500.

Personally I would set it back to 1500, unless you know jumbo frames are allowed end to end.

2nd question, I got a prompt to reboot the HX rack servers I tried to test that on, so I'm guessing you have to reboot to apply the change.

Also MTU violations will show up CRCs in your FIs/IOMs.

Try running the following on each of your FIs:

#connect nxos a|b

nxos#show hard internal carmel counters interrupt match mtu*

(that will work on 6200 series FIs, if they are 6300, the command is different)

The counters will be in hex, but if you have a decent number of MTU Violations, then your FIs are dropping frames that are likely larger than your QOS class "best effort" class allows by default (1500 unless you've changed it)

Thanks,

Kirk...

[robertdemay]

Question 1.

1. If my vNIC's are set to 9000 can this cause an issue? I've been told two different things. I've been told yes it can and then also no it will not as the setting is just allowing the vNIC to use an MTU of 9000 as it's maximum if needed.

This depends on the application and server requirements. If they require Jumbo Frames and it has not be set on the FI, then you will have network issues. If everything requires 1500, then traffic should be able to go through those nics with no issues. RAC DBs are the perfect example, the RAC application will not start unless there is a second NIC configured at 9000 for the heartbeat connection.



Question2.

2. What would the affect be of changing the MTU in the vNIC config? I have been told two different things on this as well. I've been told that it will just be a small outage while the Nic resets and then I've been told that the blades will reboot.

I just tested this and click on "Evaluate" and it comes back saying it does cause a reboot. Also note, once it comes back up, you will need to update the OS. That will cause a network restart.