Nexus 1000V VEM Issues

[greggooch]

I am having some problems with VSM/VEM connectivity after an upgrade that I'm hoping someone can help with.

I have a 2 ESXi host cluster that I am upgrading from vSphere 5.0 to 5.5u1, and upgrading a Nexus 1000V from SV2(2.1) to SV2(2.2).  I upgraded vCenter without issue (I'm using the vCSA), but when I attempted to upgrade ESXi-1 to 5.5u1 using VUM it complained that a VIB was incompatible.  After tracing this VIB to the 1000V VEM, I created an ESXi 5.5u1 installer package containing the SV2(2.2) VEM VIB for ESXi 5.5 and attempted to use VUM again but was still unsuccessful

I removed the VEM VIB from the vDS and the host and was able to upgrade the host to 5.5u1.  I tried to add it back to the vDS and was given the error below:

vDS operation failed on host esxi1, Received SOAP response fault from [<cs p:00007fa5d778d290, TCP:esxi1.gooch.net:443>]: invokeHostTransactionCall
Received SOAP response fault from [<cs p:1f3cee20, TCP:localhost:8307>]: invokeHostTransactionCall
An error occurred during host configuration. got (vim.fault.PlatformConfigFault) exception

I installed the VEM VIB manually at the CLI with 'esxcli software vib install -d /tmp/cisco-vem-v164-4.2.1.2.2.2.0-3.2.1.zip' and I'm able to add to to the vDS, but when I connect the uplinks and migrate the L3 Control VMKernel, I get the following error where it complains about the SPROM when the module comes online, then it eventually drops the VEM.

2014 Mar 29 15:34:54 n1kv %VEM_MGR-2-VEM_MGR_DETECTED: Host esxi1 detected as module 3
2014 Mar 29 15:34:54 n1kv %VDC_MGR-2-VDC_CRITICAL: vdc_mgr has hit a critical error: SPROM data is invalid. Please reprogram your SPROM!
2014 Mar 29 15:34:54 n1kv %VEM_MGR-2-MOD_ONLINE: Module 3 is online
2014 Mar 29 15:37:14 n1kv %VEM_MGR-2-VEM_MGR_REMOVE_NO_HB: Removing VEM 3 (heartbeats lost)
2014 Mar 29 15:37:19 n1kv %STP-2-SET_PORT_STATE_FAIL: Port state change req to PIXM failed, status = 0x41e80001 [failure] vdc 1, tree id 0, num ports 1, ports  state BLK, opcode MTS_OPC_PIXM_SET_MULT_CBL_VLAN_BM_FOR_MULT_PORTS, msg id (2274781), rr_token 0x22B5DD
2014 Mar 29 15:37:21 n1kv %VEM_MGR-2-MOD_OFFLINE: Module 3 is offline

I have tried gracefully removing ESXi-1 from the vDS and cluster, reformatting it with a fresh install of ESXi 5.5u1, but when I try to join it to the N1KV it throws the same error.

[KRIS PATE]

I am seeing the same thing.  However I have hosts that were installed as 5.5U1 and upgraded from 5.1U1.   I only saw the modules disconnect when I added a 5.1 host to the 1KV.  It was not too long after the module came online that all the modules dropped.  I was only able to get them to come back up by adding the system-vlan command to the control port-profile (we are running in L2 mode).

[Joe LeBlanc]

Hi, 

The SET_PORT_STATE_FAIL message is usually thrown when there is a communication issue between the VSM and the VEM while the port-channel interface is being programmed. 

What is the uplink port profile configuration? 

Other hosts are using this uplink port profile successfully?

The upstream configuration on an affected and a working host is the same? (ie control VLAN allowed where necessary)

Per kpate's post, control VLAN needs to be a system VLAN on the uplink port profile.

The VDC SPROM message is a cosmetic defect

https://tools.cisco.com/bugsearch/bug/CSCul65853/

HTH,

Joe

[greggooch]

I determined the issue a few weeks ago, the culprit was Jumbo Frames which I had correctely turned on end to end but I learned my NIC's didn't support it.  I went back to a host that was still on a previous version of the VEM and tried MTU 9000 pings succesfully, but when I added a -d to the command it failed, indicating I never really had Jumbo Frames working.  The new version of the VEM must be more intolerant of this misconfig than the old version.  I have since turned off Jumbo Frames everywhere and am successfully keeping the VEM's connected to the VSM's.

Thanks for the KB on the SPROM cosmetic issue, I had originally thought that was the problem but now I know it's not.

/gg