Re: Nexus 1000V Jumbo Frames MTU 9000

felixjai · ‎01-13-2010

I have an ESX host with 4 NICs and I added all four NICs to the 1000V DVS. All of the four NICs are part of a single port-profile (system-uplink). After I did that, everything works fine except the datastores are now not accessible. The datastores is a NFS mount which uses MTU 9000 and is mounting on a vlan. Before moving to 1000V DVS, the 4 NICs were configured with MTU 9000 in the standard vSwitch0. According to the Cisco config guide, MTU can't be set on vethernet interfaces. But the four NICs (eth3/3 - 3/6) are configured with system-uplink port-profile and MTU can't be set neither. The config guide is here http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0_4_s_v_1_2/interface/configuration/guide/n1000v_if_2basic.html#wp1221750

I'd like to know if anyone came across a similar situation. I'd hate to break the 4 NICs and move 1 or 2 NICs just to get it the MTU size correctly. I'm attaching part of the config for review. VLAN 142 is the storage vlan.

version 4.0(4)SV1(2)

vem 3
host vmware id 35303738-3634-xxxxxxxxxxxxxxxxxxxx
vlan 1,142,221,223-224
vlan 101
name control
vlan 102
name packet
vlan 100
name management
port-profile type vethernet Control
vmware port-group
switchport mode access
switchport access vlan 101
no shutdown
state enabled
port-profile type vethernet Packet
vmware port-group
switchport mode access
switchport access vlan 102
no shutdown
state enabled
port-profile type ethernet Unused_Or_Quarantine_Uplink
description Port-group created for Nexus1000V internal usage. Do not use.
vmware port-group
shutdown
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
description Port-group created for Nexus1000V internal usage. Do not use.
vmware port-group
shutdown
state enabled
port-profile type vethernet data_vlan224
vmware port-group
switchport mode access
switchport access vlan 224
no shutdown
state enabled
port-profile type ethernet data-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 142,221,223-224
channel-group auto mode on sub-group cdp
no shutdown
state enabled
port-profile type vethernet data_vlan223
vmware port-group
switchport mode access
switchport access vlan 223
no shutdown
state enabled
port-profile type vethernet service_console
vmware port-group
switchport mode access
switchport access vlan 221
no shutdown
state enabled
port-profile type vethernet storage
vmware port-group
switchport mode access
switchport access vlan 142
no shutdown
state enabled
port-profile type ethernet system-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 100-102,142,221,223-224
channel-group auto mode on sub-group cdp
no shutdown
system vlan 100-102,142,221
state enabled

interface port-channel1
inherit port-profile system-uplink

interface port-channel2
inherit port-profile system-uplink

interface Ethernet3/3
inherit port-profile system-uplink

interface Ethernet3/4
inherit port-profile system-uplink

interface Ethernet3/5
inherit port-profile system-uplink

interface Ethernet3/6
inherit port-profile system-uplink

interface Vethernet1
inherit port-profile storage
description VMware VMkernel, vmk1
vmware dvport 160

interface Vethernet2
inherit port-profile mgmt_network
description VMware VMkernel, vmk0
vmware dvport 352

jean-louis.auzepy · ‎01-25-2010

Hello

did you try to set up the system MTU first?

Nexus1000v(config)# system jumbomtu 9000

Regards

Jlouis

felixjai · ‎01-25-2010

Hi Jean-louis,

I believe we have resolved the issue. I found that the MTU setting needs to be set on the Port-channel instead of to the Ethernet interface.

I have another question. I have some Nexus 1000V instance that was installed for demo purpose, but I need to uninstall it from vCenter, but I keep getting error when object was still in used. I checked all hosts were removed and couldn't figure out what else needs to remove. Do you have a link for uninstalling? Thanks.

Felix Lai | Network & Security Architect | CCIE CCSP JNCIS-FWV JNCIA-SSL | Northeast | Presidio Networked Solutions

One Penn Plaza, Suite 1924, New York NY 10119 | flai@presidio.com

D: 212.652.5717 | C: 917.805.1431 | F: 212.244.1685 www.presidio.com<>

jean-louis.auzepy · ‎01-25-2010

Funny enough, i was trying to do exactly the same thing today.

After a lot of pain i managed to get rid of them using the troubleshooting guide:

•You are logged in to the vSphere Client and the Nexus 1000V.

Step 1 From the vSphere Client, choose
Inventory
‡
Networking.

Step 2 Select the Nexus 1000V DVS and select the
Hosts
tab.

Step 3 Right-click each host, and choose
Remove from Distributed Virtual Switch.

Step 4 On the VSM, enter the
show vmware vc extension-key
command to verify the extension key.

Step 5 Delete the extension key that is present on vCenter Server using the Managed Object Browser (MOB) Unregister Extension API, as follows:

a. Go to the extension manager [https:///mob/?moid=ExtensionManager]

b. Click
Unregister Extension
[https:///mob/?moid=ExtensionManager&method=unregisterExtension]

c. Paste Cisco_Nexus_1000V_495736846 (your extension key attached to the DVS) and click
Invoke Method.

Step 6 From the Nexus 1000V Connection Configuration mode, remove the DVS from the vCenter Server as follows:.

n1000v# conf t
n1000v(config)# svs connection VC
n1000v(config-svs-conn)# no vmware dvs

If that doesn't help, may be you can post your error message

good luck

jlouis

felixjai · ‎01-25-2010

thanks for the quick reply. What if the VSM is no longer there or was deleted before removing the SVS connection?

jean-louis.auzepy · ‎01-25-2010

I might be wrong but i think your only way is to recreate a Nexus1K using the same extension key, register it again and unregister properly

I couldn't find any other way to remove it from the DVS. I m sure a lot of people will have the same problem and we should have the option on the next version(if we re lucky)

Sorry

Jlouis

ryan.lambert · ‎04-19-2010

Just as a sidenote here (and a word of caution), there's currently an issue where the VEM modules will not retain the 9000 MTU setting upon reboot/restart, and will boot themselves @ 1500 MTU until they get their instructions from the VSM. This may cause some headaches for things like mounting network storage, etc... so just a variable to consider. In addition, it can be particularly ugly if your VSM storage is network-attached.

As I understand it, it is slated to be fixed in the next version of Nexus 1000v software later this month.

sean.wang · ‎07-25-2010

There are a lot of excellent posts out there regarding MTU, yet for weeks my colleagues and I continue to be confused about MTU on Nexus100v.

The confusions were around:

-How to set jumbo on N1kv

-Relationship between "system mtu" and "mtu"

-If "system mtu" is needed

-How to set "system mtu"

I wrote a short summary, based on multiple conversations with Cisco, and finally the definitive source. Hope it helps.

http://www.seanxwang.com/2010/07/nexus1000v-when-to-use-system-mtu.html

Darian O'Dirling - TCE · ‎11-18-2022

Hi @felixjai
I see you are working with MTU configuration. I encourage anyone who is running into this issue or wants to learn more about MTU configuration on Nexus 9000 series switches to watch this video made by a colleague mind. The video outlines the need for proper MTU in the network and explains how to remedy MTU related interface errors, via interface MTU configuration.

https://www.youtube.com/watch?v=GbBeB6h9MKk