Upgrade path to fix CVE on an ESXi with a Cisco Custom Image

[battagls]

Hello.

I our Organization we have Cisco UCS C240 M5S servers with these sw installed:

• Firmware Version: 4.2(3i)
• ESXi Version: 7.0 Update 3i

Currently the latest custom version of ESXi 7.0 available for download is 7.0 Update 3o (File name: "VMware-ESXi-7.0.3o-22348816-Custom-Cisco-4.3.2-a.1").

Meanwhile, vulnerabilities with high severity have been published: these vulnerabilities are fixed in the ESXi 7.0 Update 3q (no Custom) version. As written earlier the latest Cisco Custom version is Update 3o, which is earlier than the image that fixes the vulnerabilities.

I have read in other threads that it is possible to update an ESXi with Custom Image with a non-Custom image.

At this point my question is, what is the upgrade path I need to follow to fix my ESXi servers?

I was thinking of a path like this:

1. Update fw: Update firmware from version 4.2(3i) to the latest released, 4.3(2.240053) 
2. Update ESXi: Update ESXi from Cisco Custom version 7.0 Update 3i to non-Custom version 7.0 Update 3q
3. Update drivers: Update ESXi driver with Cisco Addon for ESXi 7.0 U3o

Based on your experience, do you think this path is correct?

Finally, once I have upgraded the ESXi with a non-Custom version can I upgrade the ESXi with a Custom version?

Thank you.

Regards.

Sergio

[Riaan van Niekerk]

Hi there Sergio. You refer to custom image. There are 2 custom distribution mechanisms for ESXi:

1. Vendor custom ISO, can be used for installation only
2. Vendor custom offline bundle (.zip) which should be used for upgrades.

Are you still using the (deprecated in vSphere 8.0) vLCM baselines and baseline group based, or have you switched to using vLCM Image-based updates, the Cisco vendor add-on).

[battagls]

Hello.

Thank you for your response.
No, we don't use vLCM: we have few servers in our environment, and last time I did the server upgrade by uploading the ISO via CIMC.
To install this update I was thinking to use the offline bundle (.zip file) by installing the package via CLI.

[Riaan van Niekerk]

Your upgrade plan seems sound. The only change I would make is, If you cannot upgrade server firmware at the same time as drivers and ESXi (e.g. have the firmware upgrade pending on next reboot), I would strongly recommend that you upgrade drivers first (with / without ESXi) before upgrading server firmware. 

In our experience over the past decade plus, newer drivers with older server firmware do not cause an issue. 

What you want to avoid is your scenario, running newer server firmware (e.g. 4.3.x) with drivers that have been certified with 4.2.x server firmware.

[Riaan van Niekerk]

(deleted)

[skrg]

Hello,

We are in similar scenario, our urgent requirement is to update ESXi to 7.0 U3q to remediate ESXi vulnerability and perform UCS firmware update at a later time. Below are current versions.
UCSM - 4.1 (3h)
ESXi - 7.0 U3l
Add-on used during previous ESXi update - Cisco-UCS-Addon-ESXi-70U3 4.2.1-a.

Please advise which is the interim correct path to go with for now (or not), we are planning to use the 7.0 U3q software from Broadcom.

1) Update ESXi only to 7.0 U3q ( no change in add on)

2) Update ESXi to 7.0 U3q along with latest add on "Cisco-UCS-Addon-ESXi Version 4.3.2-a.1" ( wondering if this will create any problem as we are at lower firmware version)

Thanks

[Riaan van Niekerk]

You need to check the Cisco UCS HCL https://ucshcltool.cloudapps.cisco.com/public/ to figure out which enic & fnic drivers (the most important ones) are certified/tested with your firmware version.

Unfortunately Cisco don't refresh the vendor add-on frequently enough, e.g. you could be missing out on the latest drivers if you stick to the vendor add-on. On the 4.2.3 / 4.3.x firmware that we are running, Cisco in Jun/Jul refreshed the HCL to support/certify 5.0.0.43 fnic. This driver has been out since Dec 2023 and Cisco only certified it 6+ months later, and updated vendor add-ons have not been released on the Broadcom website or via vLCM.

Our workaround (using vLCM) is to upload the 5.0.0.43 fnic and add it as a component to the vLCM cluster image. E.g. we patched (example uses PowerCLI), but the GUI method and doing it without vLCM via esxcli should be fairly self-evident):

# Define the vLCM image items
$BaseImage   = Get-LcmImage -Type BaseImage   -Version '7.0 U3q - 23794027'
$VendorAddon = Get-LcmImage -Type VendorAddOn -Name 'Cisco-UCS-Addon-ESXi' -Version '4.3.2-a.1'
$Components  = Get-LcmImage -Type Component   -Version "*nfnic_5.0.0.43*

# Define the clusters you want to target
$CLUSTERS = Get-Cluster CLUSTERNAME*

# Configure the vLCM image to use the 
Get-Cluster $CLUSTERS | Set-Cluster -BaseImage $BaseImage -VendorAddon $VendorAddon -Component $Components

# Remediate all $CLUSTERS
Get-Cluster $CLUSTERS | Set-Cluster -Remediate -AcceptEULA -RunAsync

We patch hundreds of hosts across dozens of clusters this way, in one go.

If you want to upgrade UCS firmware during the patch reboot, set the newer firmware version on UCSM/Central, and ensure that ESXi Quick Boot is disable.

NB: the enic & fnic drivers included inbox (non-custom image/bundle) and the latest customised image/bundle and vendor-addon drivers will probably work, but we have had multiple instances where we had crashes and instability, and were told "update to the latest" or "update to the version in the HCL". which means a "component" or custom baseline in vLCM or manually installing the nfnic VIB via esxcli if you don't have vLCM (formerly VUM).

[skrg]

Thanks for your detailed info.

Currently we are in a time crunch and planning to update only ESXi (keeping UCS firmware 4.1(3h)).   Wondering if we can skip the Cisco-Add-on 4.3.2-a.1 for now and install it when we are ready to update the UCSM and Blade firmware in the near future to 4.3.2 or later version.  Please advise.

[Riaan van Niekerk]

You are welcome to do anything in your environment, depending on your risk appetite. (ours is very low, so we take the extra time and effort to go with option (b))

a) Run combinations of enic/fnic versions to UCS firmware outside of the Cisco UCS HCL. This will probably work, and you may have years without issues, but if you engage with Broadcom/VMware or Cisco for support on an issue, they will most probably say "please install enic/fnic version x, and if the problem persists, we can look at it". 

b) Stick to the UCS HCL. If the driver shipped with the customised ISO or vendor add-on is not the latest recommended version for your combination of firmware, you would have to download the correct driver from Broadcom.com and install that on top. 

[skrg]

Thanks again for your input.