Can't remove DVS configuration

[niversen]

My VEM has become completely disconnected from reality.  It is connected to the VSM, but that is all that is correct about it.  It will not sychronize with the VSM without errors.  One one resynch, it moved all the uplink ports into the "Unused and Qurantine" port group. 

I removed all vmnics from the DVS and have tried to delete the DVS using the "no vmware dvs" command.  This returns the error:

"VMware vCenter Server 4.0.0 build-208111 DVS operation failed on host 10.x.x.x, The object or item referred to could not be found.  got (vim.fault.NotFound) exception.  Cannot complete a Distributed Virtual Switch operation for one or more host members."

This is the same error that is pushed no matter what I try to do with the DVS. If I "no connect" and then "connect" I get this.

I looked in the FAQ:  https://communities.cisco.com/docs/DOC-14464#stale-vem to try to remove the config manually.

The VEM is on an ESXi, so I can only use VMA - there is no service console to execute the shell commands listed in the FAQ like "net-dvs". 

How can I get rid of the corrupted DVS?  I need to cut my losses and re-build this DVS to get on with what I need to do.

Thanks,

Nathanael

[niversen]

So I am assuming that if things go wrong on an ESXi environment, that there is really nothing to do but move the VMs, blow the host away and start again?

Is this case, or is there another way to remove a VEM that has gotten into a corrupt state in relation to the VSM?

I think any reset commands need to be available from the VMA shell for ESXi users who don't have a service console.

Thanks for any help,

Nathanael

[sachthak]

One possible option that may give you access to net-dvs etc. commands is to access the ESXi console through unsupported mode.

Here is one link I found that has a guide regarding setting this up:

http://www.techhead.co.uk/vmware-esxi-how-to-enable-ssh-connectivity

Hope this helps

thanks,

sachin

[niversen]

Thanks for that link.  I now have shell access and have been able to execute some of the commands in the FAQ.  I was able to delete the DVS.

Hhowever, it appears that some of the VEM related commands are not in the specified location:

Next unload the VEM kernel module

[root@cae-esx-179 ~]# /usr/lib/ext/cisco/nexus/vem-v110/sbin/hotswap.sh -u

and

If the above command fails with a "bad ioctl" error then remove the VEM module completely with "vem-remove -s -r"

These files do not seem to be present anywhere.  Have these files moved to a different location?  The /usr/lib/ext directory does not even have a "cisco" folder in it on my ESXi host.

I think I am very close to having this configuration deleted.  vCenter now shows the dvs as "down".  I am wondering if vCenter has something in it's database that shouldn't be there.  I can't find any paths containing "cisco, "nexus", etc., so I am wondering if the VEM is already gone from the host, but something is hanging around in vCenter's DB....

Thanks for any additional insight you can provide,

Nathanael

[sachthak]

Does vemcmd return any output on the host or does it say cmd not found? For example, what does "vemcmd show version" return?

thanks,

sachin

[niversen]

From the root prompt on my ESXi host:

~ # vemcmd

-ash: vemcmd: not found

~ # vemcmd show version

-ash: vemcmd: not found

Thanks,

Nathanael

[sachthak]

I believe this should mean that your vem has been removed from the  host. You should now be able to reinstall the VEM (or use VUM to do so)  and re-add the host to the dvs to restore your setup.

sachin

[niversen]

VUM says there are conflicts.  At this point, I will move all my VMs off the host and re-image the host, unless there are other ideas.  It appears to be in some strange state requiring me to just flush the whole install and start again.  Time-wise, starting over seems quickest at this point.