Replacing failed VEM on Nexus 1000v

[benwilliams322]

I was curious how others accomplish replacement of a failed ESXi host that is a VEM on the Nexus 1000v. I've performed this procedure once and it seemed longwinded. The goal is to make the swap out transparent to the Nexus 1000v (same VEM #, just different VMware UUID)

- Transition host back to standard vSwitches

- Remove host from distributed switch via vCenter (right-click host and remove)

- Physically swap out the chassis

- Find UUID for new host (obtained via esxcfg-info)

- Setup new VEM in the 1000v with that UUID

- Migrate replacement host back onto 1000v which assumes the VEM number that was just setup

[Robert Burns · Accepted solution]

Ben,

Note - 1000v related questions are best posted in the "Server Networking" forum.  This forum is specifically for UCS. 

https://supportforums.cisco.com/community/netpro/data-center/server-network?view=discussions

The procedure you're using is the correct one.  An alternate procedure is to gracefully remove the host from the 1000v, then issue a "no vem X" which removes the VEM record from the VSM.  Then swap your hosts and re-add it back to the 1000v.  Is there a reason you need the same UUID?

Regards,

Robert

[benwilliams322]

Robert,

My wording was confusing. I'm not using the same UUID. I'm grabbing the UUID for the new host then issuing the "vem X" command in the VSM with the new UUID. This way the VEM number doesn't change upon adding the new host.

The procedure outlined, essentially, is a graceful removal of the host from the 1000v, but in my experience the only way to be able to right-click a host in vCenter and select "Remove from virtual distributed switch" is to move it back to vSwitch first. Otherwise, errors and warnings pop up saying certain vmnics are still in use.

Thanks for the reply and pointing me to the correct forum.

Ben

[Robert Burns]

Since we're already here, we'll finish this post in this forum.

The only way you can remove a host from a DVS in vCenter is if there are no virtual ports on that host being used.  This includes Virtual Machines and VMK interfaces.   Once these are all moved to the vSwitch, then you can remove the host from the DVS.  This is a security check so you don't accidently try to remove a host with virtual interfaces still in use.  The vmnics can remain connected to the DVS in this case, they will just unbind and become freed once the host is removed.  If you're getting errors when trying to remove the host, you still have something in use.  You can double check this with "show int virtual vem X". 

If you have VEMs 3, 4 and 5 and you have to swap out 3.  You would follow your procedure above and migrate all virtual interfaces to a vSwitch, then gracefully remove the host from the DVS in vCenter.  At this point you should remove the VEM entry on the VSM with "no vem 3".  Then go ahead and swap your server chassis out, and re-add it back to the 1000v.  It will pick up the lowest available VEM # which will be 3 again.  

Of course if you happen to have gaps in your VEM numbering and require a very specific one, then you can use the UUID remapping as you've stated.

Robert