Upgrade C240 M4SX from ESXi 6.5U3 to ESXi 6.7U3 REVISITED

[alanraczek11]

Almost a year ago I asked about doing an upgrade to ESXi 6.7U3 but back burnered this for a while. I am now looking to do this soon and I have come up with this procedure. I am fairly confident I have covered the bases but I want someone well versed to check my work before I embark on this effort. Please see below:

Cisco C240 M4SX ESXi Upgrade to 6.7U3 - 4 hosts

Downloads
-Cisco HUU 4.1(2)
-Cisco ESXi 6.7U3 custom image for ESXi 6.7.3.1 GA Install CD ISO (not ZIP)
-Cisco drivers for VMware 4.1.2e

Checks
-Check C240 M4SX vSAN upgradablilty to 6.7U3

Procedure
- Place DRS in manual mode all hosts, move VM's to another host, put ESXi in maintenance mode
- Run Cisco HUU to upgrade firmware (mount iso to CIMC KVM)
- Install the Cisco custom ESXi (mount iso to CIMC KVM)
- Install Cisco VMware drivers
* Using https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vic_drivers/install/ESX/2- 0/b_Cisco_VIC_Drivers_for_ESX_Installation_Guide/Cisco_VIC_Drivers_for_ESX_Installation_Guide_chapter2.html
- Take ESXI out of maintenance mode,enable Automatic DRS (after last host is upgraded)
- Done

Note we have 4 hosts so I have some questions:
- Did I miss anything?

- Does it matter matrix is for an M4 and not an M4SX ?
- Is it okay to do one host at a time considering we run vSAN and some hosts will be
vSphere 6.5U3 and others will be 6.7U3 until all are done? That way I don't have to necessarily
take ALL VM's offline, I can take my time doing one host at a time, moving VM's.

- Drivers install- install during OS install or after ESXi is installed?

TIA.

[alanraczek11]

Can anyone commment on if this is a good procedure? Ty.

[Wes Austin]

Looks good from what I see you have outlined. I would do one at a time and make sure you are able to move VMs back to it once complete. You may need VMware to confirm on the vSAN question with having hosts on different versions. M4 and M4SX are the same on the matrix, they are just different number of disks. Once the new version of ESXi is on the host, then install the drivers.

[alanraczek11]

Wes,

Thank you. I verified with VMware upgrading one at a time with different versions of vSAN .is fine. And I know this may be more VMware's ballpark but I assume doing the ESXi upgrade with the ISO while hitting F6 will allow me to "upgrade" the current version (6.5U3 ---> 6.7U3)? Also, how do I install the vmware drivers after the ESXi install?

They (VMware) said "place ESXi into maintenance mode and the VM's will move on their own". First off maintenance mode requires VM's to be off i thought. Second just turning off the VM's causes DRS to recognize their are new resources available and starts moving VM's to the host I am working on and totally defeats the purpose, so I will pretty much stick to my procedure.

...Alan

[Wes Austin]

Yes, if you mount the new ISO and select it during boot, VMware should detect the presence of existing VMFS datastore and ask you if you want to upgrade ESXi.

You can use this document for driver install guidance:

https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116349-technote-product-00.html 

In short, you add the VIB to a datastore the host has access to and follow instructions above to run a command to update them. They usually require reboot.

When you put a host in maintenance mode, assuming you are using HA, the VM will automagically be migrated to other hosts in the HA cluster and while staying online. You do not need to power off any VM.

[alanraczek11]

Thank you. But this does not follow my experiences as far as VM's moving. First off  vCenter always tells me VM's must be powered off or moved  to put the host in maintenance mode. Secondly if I start powering off VM's or moving them I have run across VM's from other hosts migrating to the host I am working on because resources get freed up and vCenter is attempting to re-balance the load.

So the original (sort of ) question is can I put a host in maintenance mode with VM's running on that host and expect the VM's to move automatically? 

If DRS is running SHOULD i get a message saying I need to power off or move VM's? ty.

[Wes Austin]

The answer to how to do this correctly is going to vary from environment to environment. It will depend how you have things configured. In my lab vCenter, I put a host in maintenance mode and it moves VM workloads off the host. We do not power them off. We use vSphere HA and fully automated DRS.

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-8F705E83-6788-42D4-93DF-63A2B892367F.html 

Virtual machines that are running on a host entering maintenance mode need to be migrated to another host (either manually or automatically by DRS) or shut down. The host is in a state of Entering Maintenance Mode until all running virtual machines are powered down or migrated to different hosts. You cannot power on virtual machines or migrate virtual machines to a host entering maintenance mode.

If I were you, I would open a case with VMware and have them review and understand what you need to configure in vCenter to accomplish what you need for your solution. 

[alanraczek11]

No I get all that, and I really do appreciate all your input. In getting back to the drivers, I am now thoroughly confused on what drivers to install from the download I have. I downloaded: ucs-cxxx-drivers-vmware.4.1.2e.iso . If I mount that ISO on my Windows machine I see various folders:

If I search the mounted ISO for VIB's I get these:

The example in that article on how to install drivers seems (to me) to give one example of installing VIC drivers. I would have hoped one process would install all the drivers. Sorry for all the questions today, 

[Wes Austin]

For upgrading drivers manually, first make sure you understand the correct driver you should be running using our HCL:

https://ucshcltool.cloudapps.cisco.com/public/ 

This is what it looks like based on the info you provided:

From the drop down you can see whatever driver version you should be running based on the UCS firmware version. The process to install the VIC driver is the same as the other drivers as well.

If you are looking to upgrade all drivers at once or are upgrading drivers in scale, you are going to want to look at using some automation solution from like VMware Update Manager, write a script, etc.

Something like this:

https://vswitchzero.com/2018/12/11/updating-nic-drivers-with-vmware-update-manager/ 

Let me know if this makes sense or any further questions.

[alanraczek11]

Thank you, I am taking a look at update manager. Thank you IC2317 also, very helpful. I took over the VMware administration a little over a year ago and spent a year getting "down and dirty" with the nuts and bolts of it all. I have used VMware before but not with Cisco equipment.

If I look at the manual updating I assume if I download the firmware bundle and driver bundle together I am good. I would not need to check compatibility. And doing the manual updates via esxcli if I have 4 vibs that's all I need to install, correct? If so, do I reboot after each vib install?

...Alan

[IC2317]

Reboot after vib install depends on the vib. Usually, if you download it from VMware it will say if it requires reboot or not.

You can install all vibs and then reboot the server once.

My suggestion is still to use Update manager to upgrade the host because if you upgrade the firmware and ESXi using Cisco ISO then it would most likely have recommended Firmware/Driver version. I have done it many times and generally installed versions are recommended ones unless there are new versions available for some bugfixes etc.

At last, you should be comfortable with the steps you follow and if you prefer manual steps then please go with it.

[alanraczek11]

I have never used Update Manager before so it's a short learning curve, and is the time invested to learn it worth it for 4 hosts. BUT it would be nice to learn. On the drivers and firmware, again, I am downloading them in what's paired together so I am downloading firmware and drivers from the same revision on Cisco's site. In looking at the driver download I see 3 drivers pertinent to ESXi 6.7 U3: VIC drivers and fiber drivers. There is also a FusionIOmem driver but I logged into a host and performed  'esxcli software vib list | grep scsi' and the adapters shown did not include scsi-iomemory-vsl. So what I figure from all this is I just need to ionstall 2 'nic' drivers. Correct me if I am wrong.

On using Update Manager tell me if I am wrong.

- upload the custom esxi 'image'.

- create an upgrade baseline

- load driver into repository as an 'update' , upload from file

- create a group baseline including upgrade image and 'patches (drivers)'

- apply baseline to host setting maintenance mode options

Does this sound correct?

[IC2317]

There is no requirement of creating group baseline as Custom ESXi ISO image that you import to Update Manager will have all the required drivers that are required for ESXi 6.7 U3 on Cisco Devices, that's why it is better method.

However, after upgrading the ESXi using Update Manager if you feel like you don't have recommended drivers then you can create "extension" baseline with those drivers and then install it via update manager.

Also, make sure to check all perquisites and steps mentioned on VMware website on how to upgrade VSAN nodes. More you gather information, more confident you will be.

[IC2317]

Hi,

I have upgraded the production VSAN environment without any issues in past and below steps should be fine for you to perform.

1) Select one host at time and Put ESXi host in Maintenance mode by selecting "Ensure accessibility" option as when you put host to maintenance mode in VSAN it prompts you with this message. This ensures that VMs will be accessible even if you perform some task on one of the host.

2) Upgrade Firmware of the host.

3) Import Cisco Custom ISO for ESXi 6.7 U3 to Update Manager and create "Upgrade" baseline using that ISO. You can refer below links for the same.

Import 6.7 U3 Cisco ISO image to Update Manager -https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.update_manager.doc/GUID-DE8A8F0F-7211-4527-BA9C-F18907BF23EE.html 

Create Upgrade Baseline - https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.update_manager.doc/GUID-8ECA8B9A-9596-4267-B51C-63210ADDEFB9.html

4) After above steps, Remediate host using the newly created upgrade baseline. This will automatically upgrade the host with required drivers and ESXi image to 6.7 U3. This eliminates the need to upgrade the drivers manually.

5) After the host is upgraded. You can check the drivers installed and check their compatibility with firmware versions. Ideally it should be according to the recommended versions. However, you can refer https://ucshcltool.cloudapps.cisco.com/public/ and compare the Firmware/drivers from here.

6) Once above steps are done, you can proceed with same steps on next host until all are done.