Cisco Standalone C-Series HUU Utilities

[Jeffrey Foster]

The Cisco C-Series Cisco Integrated Management Controller (CIMC) software  delivers many new features and capabilities including enhancements to the Host Upgrade Utility (HUU). The Host Upgrade Utility enables users to update their CIMC, BIOS, LOM, RAID Controllers, PCI adapters and Cisco VIC cards from a single interface. In previous versions of HUU, an interactive menu-driven command line interface was the only option provided to update system BIOS, CIMC, adapter firmware, and LAN on Motherboard (LOM).

Enhancements to HUU include adding an interactive graphical user interface to HUU and adding non-interactive update capabilities for single-system and multiple-system updates for no touch upgrades of Cisco C-Series Server firmware. Updates can be applied using the bootable image in the HUU ISO which can be mounted via vMedia to boot host to HUU and triggered through the KVM using an interactive update interface. Other tools offering non-interactive updates include a Python update tool and a Microsoft Windows-based tool, both of which offering single-system and multi-system update capabilities.

The Host Update Utility (HUU) is a tool that helps users to update the various levels of firmware running on their system including:

• CIMC
• BIOS
• LAN on Motherboard (LOM)
• RAID Controllers
• PCI Adapters
• Cisco Virtual Interface Cards (VIC)

For more information on using the Interactive Utility or the Python Non-Interactive Utility, check out the Getting Started with HUU Guide available here.

Links to Getting Started Videos:

• HUU Single System (Interactive) Utility:

In this video we will use the native utility to update a single system using vKVM attached media and the new HUU Graphical User Interface to update a single system.

• Python HUU Utility (Single System)

In this video we will use the Python Tool to update a single system by passing configuration flags and options through the CLI to a remote target. This is the recommended update process for a one-time update to a single system.

• Python HUU Utility (Multi-System)

In this video we will use the Python Tool to update a group of systems using a configuration file. We will walk through the encryption setup process for the target CIMC as well as the remote share and how to reference these encrypted files in an update script. This is the recommended update process for updating multiple systems concurrently.

****NOTE: For IMC 3.x and above, Python >= 2.7.9 and OpenSSL >= 1.0.1 are required. For IMC 4.1(3) and above, Python 3 is required. 

Download the Utilities:

***New IMC 4.3(3.240022) utilities are posted! Scripts are updated on an "as needed basis" and do not necessarily match the latest release.

[terattu]

One incident:

Using remote non-interactive methods (since KVM app won't run). From HUU log file we see the certificate verify failed, like below.

Error - CIMC login failed. SSL version on the server is older than 1.0 and is not compatible.

Tried 3 different releases of HUU.

Finally did it physically and it worked. 

ERROR: 2020-11-12 14:37:46,241 - MainProcess - 48759 - We failed to reach CIMC (9.0.9.180). Reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
tried 3 different releases of HUU

broken rpm packages for SSL certificate (in most cases, the broken package is 'ca-certificates'). You need to check the packages by using 'rpm -V', for example:

rpm -V ca-certificates

or check recursively all RPMs:

rpm -Va --nofiles

and reinstall it if rpm looks corrupted:

yum reinstall ca-certificates

[juburnet]

@Jeffrey Foster - The HUU-NI tool has been very beneficial for us recently for 130+ mass upgrades of standalone C-series.  Are there any plans to upgrade the code for Python 3?  looks like it's only working with Python 2.x.  We have some use cases where Python 3 support would be needed.

[YanL]

Hello we are trying to upgrade standalone C220 M4 from 4.1(1g) to 4.1(2g). We are able to upgrade using the KVM mounted ISO and using the interactive method although we have to shutdown the host, reboot the CIMC and upgrade just the CIMC first, then upgrade the remaining modules.

If we try NI-HUU (CentOS Linux release 7.9.2009 (Core), python 2.7.5, OpenSSL 1.0.2k-fips, NI-HUU 4.1.1f, multiserver_config target 1 server) we get "Error Firmware copy has failed (Error Code : 906). This is a critical error and host will reboot. Do you wish to save logs before rebooting?"

EDIT: We also tried (Windows 2012r2, python 3.9.4, OpenSSL/PyCrypto???, NI-HUU 4.2.1a, multiserver_config target 1 server) with error

Processing config file (multiserver_config) data

checking security check present

Total of 1 servers firmware to be updated.

Updating firmware.....

[Information] Needed packages "Crypto.PublicKey.RSA" not available. Cannot support password encryption feature.
[Information] Needed packages "Crypto.Random" not available. Cannot support password encryption feature.
Process Process-1:
Traceback (most recent call last):
File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "E:\FTPROOT\Cisco\CIMC\Python_Tools_4_2_1a\update_firmware_4.2.1a.py", line 3750, in HuuHandleFirmwareUpdate
responseData = HuuProcessPendingWork(logger, work)
File "E:\FTPROOT\Cisco\CIMC\Python_Tools_4_2_1a\update_firmware_4.2.1a.py", line 3715, in HuuProcessPendingWork
work.HuuUpdateSendFirmwareUpdate(logger)
File "E:\FTPROOT\Cisco\CIMC\Python_Tools_4_2_1a\update_firmware_4.2.1a.py", line 896, in HuuUpdateSendFirmwareUpdate
responseData = self.ConfigConfMo(logger)
File "E:\FTPROOT\Cisco\CIMC\Python_Tools_4_2_1a\update_firmware_4.2.1a.py", line 2083, in ConfigConfMo
configMoRequest = configMoRequest.replace('remoteIpValue', self.remoteShareIp)
TypeError: replace() argument 2 must be str, not None

We have numerous servers to upgrade, any recommendations?

[Greg Wilkinson]

Hi - for starters, Python >2.7.9 is required in either use case you mentioned. With that said, can you please try with a newer Python 2 version? Also, are you using the multiserver config file, or upgrading a single server? What openSSL version are you running?

[YanL]

@Greg Wilkinson 

thank you for the reply I edited my post to show the second test was using Windows 2012r2 and it did not work.

We also tried NI-HUU (CentOS Linux release 7.9.2009 (Core), python 3.6.8, pycrypto 2.6.1, NI-HUU 4.2.1a, multiserver_config target 1 server) we get "Error Firmware copy has failed (Error Code : 906). This is a critical error and host will reboot. Do you wish to save logs before rebooting?"

MITIGATION: We are getting this error if the remoteshareip=https://x.x.x.x parameter is using SSL in the multiserver_config file. It works if we use remoteshareip=http://x.x.x.x without the SSL.

I don't know what in Windows 2012 R2 IIS could be causing this but thank you Greg for helping me figure this one out. I can use a browser with https and download the file ok so something weird going on.

[Greg Wilkinson]

Please email me @ g-r-e-w-i-l-k-i @ cisco.com (no hyphens in email). I would like some more details. 

[SamCU]

This seems a great tool/utility. I've not seen the code yet.  I've couple of questions..

Reading the multi-system config, there is a verify option.  Does the utility do the verify as soon as the "Update All" completes or does it reboot the system and do the "Verify" ?

Does it support updating the "HDD Firmware" along-with the "Update All"  option?  If it doesn't, does it support updating the HDD firmware ( if supported / available  for the HDDs inside the box ) along with all other components  ?

Thanks

[Greg Wilkinson]

When verifying the firmware upgrade, the system boots into the HUU a second time and runs a comparison between the current versions and previously running versions, along with a status check of the updates for each component. Update-all does not upgrade HDDs – to do this you can specify “all,hdd” for everything or just “hdd” for the drives.

[paul ratinov]

Is it possible to schedule a non-interactive upgrade on the next reboot? 

[Greg Wilkinson]

Yes. The multiserver_config file has an update_type attribute, which when set to delay, will perform the upgrade upon the next host reboot.

[paul ratinov]

Greg,

Do you have an update for this script for the latest versions?

I can't get it working with ucs-c220m5-huu-4.2.2g.iso

Thank you

Paul

[Greg Wilkinson]

Feel free to try the 4.2(2g) utilities, but also please describe the issues you are facing. 

[John Blazek]

Hello @Greg Wilkinson !
I have a C220M4 with firmware 3.0(3f)

The CIMC lets me connect with Firefox but then the screen just hangs and I get an error that the server isn't responding.
SSH connections appear to work just fine.

I am wanting to update the firmware.
Can I simply download the ISO.. put it in a USB CD and boot from the cd?
Can I go directly from 3.0(3f) to 4.1(2g) ?

[Greg Wilkinson]

Hi John, 

There are a few ways to hopefully get your firmware sorted. I recommend creating a bootable HUU USB using the following scripts: https://community.cisco.com/t5/unified-computing-system-knowledge-base/cisco-standalone-c-series-host-update-utility-usb-image-utility/ta-p/3638625

Please try this first and let me know. 

[John Blazek]

@Greg Wilkinson

That page says it is for an M6

I'm somewhat familiar with HP's and almost no experience with Cisco servers.

With the CIMC not connecting, I either have to boot from a USB CDROM drive or access the server config via SSH.

The link you provided appears to be an ongoing discussion about series M6 (not M4).
I have an M4 ..

Although I've been in IT for some 40+ years, if I'm going to "build a bootable HUU USB" using scripts, it would help to have a man-page or info-page that summarizes the procedure to correctly build and populate the iso with the proper content.

It appears that the page you referred me to requires some variant of NIX to be installed as the host operating system.

The only thing I've loaded on the server is VMWARE and that can be reinstalled later. And I'm not certain that I would have hardware access if I create a nix-VM on the server.

I assumed (perhaps incorrectly) that ISO for 4.1(2g) was an 'all inclusive' update.

I downloaded:

https://software.cisco.com/download/home/286281345/type/283850974/release/4.1(2g)

UCS C220 M4 Rack Server Software

Release 4.1(2g)

Cisco UCS Host Upgrade Utility

ucs-c220m4-huu-4.1.2g.iso

16-Sep-2021

878.00 MB

It was my intention to burn that to a cd and boot it and run the update from a physical VGA console at the server.

For the purpose of this firmware update.. we can consider this server as 'unused' and any method of updating the firmware is acceptable (other than using CIMC - since it doesn't respond after logging in).

However your post suggests you believe I should do something different. ??