UCS Mini jumbo frames config

eamehostedservices · ‎09-30-2018

Hi All,

Our org is quite new to UCS mini. We are having a weird issue with some of our Esxi servers on B200 M4 blades dropping the Esxi management network for a minute or so periodically daily (Every other network is ok) and I think it may be down to our jumbo frame config. Every UCS mini is configured via a UCS central server that manages all of the UCS mini units.

Each blade has a VIC 1340 which provides 7 vnics to Esxi. All vnics are set to an MTU of 1500 except the pair connecting to NFS. These 2 vNICs are set to MTU 9000 in VMWare and UCS. They also have a QoS policy called MTU assigned to them. None of the other nics have a Qos policy assigned to them. It would be great if someone could look at how we've set things up to make sure it looks ok:

The QoS system class is defined as below within UCS central and is assigned to the UCS domain. Parent type is Domain Group:

This funnels everything through the best effort class with an MTU of 9216. No other classes are enabled.

The QoS policy set on just those 2 vNICs is also as below. Parent type is Org.

I know I may be clutching at straws, but I've got a feeling something here isn't quite right despite us having verified we can indeed helping our NAS with jumbo frames. Could some thing here be causing the random dropouts we are seeing on the Esxi management network? Is this a valid config?

Any advice much appreciated.

Niko Nikas · ‎10-01-2018

Hello,

Are the vNICs that are carrying the management traffic the same vNICs that require jumbo?

From your description, it sounds like from an MTU perspective we should really only care about the NFS links, but I wanted to be sure.

How are your uplinks configured? Do you have a disjointed upstream environment from the UCS?

It could be that the designated receiver is not where you would expect it.

connect nxos <a/b>
show platform software enm internal info vlandb all

If you run this on both sides of the fabric, are the receivers where you expect?

--

Niko

eamehostedservices · ‎10-03-2018

Hi Niko ,

Sorry for the late reply. To answer your questions, the management network traffic goes down different vnics to the NFS traffic that has the QoS policy set on it. Management traffic is set to 1500 on the vnic and in ESXi with no Qos policy assigned.

RE the network, both FI are connected to Nexus 5k switches so the network is not disjointed.

Interestingly, from what the network guys tell me, all of the 5k interfaces have a global policy setting the MTU to 9216. There are no service classes set. With this in mind, is it even necessary to have the QoS class and policy in place? Can they be removed with the NFS uplinks just set to 9000 MTU to rule them out as the cause?

Niko Nikas · ‎10-08-2018

Hello,

It may not be unless you want to guarantee some bandwidth.

You could set everything to 'Best Effort' with a MTU of 9216.

Have you checked ARP and MAC entries upstream to make sure everything appears to be how you'd expect it?

--

Niko

eamehostedservices · ‎10-08-2018

Hi Niko,

Thanks for the reply. I may have just cracked the reason why this is happening. For these new units, we built VMWare using the standard VMWare esxi 6.0U3 CD and then patched it with the latest security patches using update manager.

I rebuilt one of the hosts using a Cisco custom CD and then patched it using our standard update manager process, again, same issue. It was then I suspected it was probably the patches causing the issue. I rebuilt the host again, this time installing the Cisco offline-bundle patches to bring the hosts up to date. So far, the host I've patched hasn't had the issue. I am going to rebuild a further 2 hosts with the same issue to see whether this solves the problem. If so, happy days! It is interesting to see this behaviour, as you would think that any patches that update manager applies would be compatible, but it seems not to be the case.

will report back what I find!

Niko Nikas · ‎10-08-2018

Interesting, do let us know what you find out.

I would think that you would see impact beyond just the connection to vCenter (VMs having issues as well).

Do you know what specific patch seems to break it?

--

Niko