CIMC Supervisor firmware upgrade times out on C220 M3 servers

[dmveron01]

Hello,

I manage about three dozen UCS C220 M3 and M4 servers for our company. I'm using an evaluation copy of Cisco's IMC Supervisor to see if it's worth purchasing for centrally managing firmware upgrades. Upgrades for our M4 servers are going well, but every attempt to upgrade the M3 servers are failing with one of these errors:
FAILED - Read timed out
FAILED - Update Complete CIMC InProgress,

I'm trying to update from 3.0(4d) to 3.0(4i). Opening the Java KVM from CIMC Supervisor, attaching the ISO as virtual media, and upgrading that way works without issue.

Any ideas on how to troubleshoot this issue?

Thanks.

[Kirk J]

Greetings.

Ultimately you may to open a tac case to review the IMC sup logs and the CIMC logs on the server to see what is the problem.

Have you tried watching the KVM of the 220m3's attempting the update to see if the HUU ISO i even being mounted?

Is there anything different with the 220M3s vs the M4s in terms of the location/connection of the CIMC connections on the network?

Are you using NFS shares, or just using the IMC sup to host the firmware images?

Also, what version of IMC Sup are you running?

I'm in process of testing that specific ISO on some 220m3's in the lab now from IMC SUP 2.2.0.3

Thanks,

Kirk...

[dmveron01]

Hi Kirk,

Thanks for the reply. I did end up watching the console during the CIMC Supervisor-initiated upgrade. The upgrade itself is stopping with an error one the CIMC firmware with a message about not enough memory to run that part of the upgrade. The hosts have 256GB of RAM if that's relevant.

When running manually on these hosts again, some upgrade fine, but some report that same error again. I seem to be able to work around it by upgrading all the other components, powering off the server for a few minutes, powering back on, and running the CIMC upgrade by itself.

I saw you are using 2.2.0.3 in your lab. The eval version is 2.2.0.0. I originally tried upgrading through Administration -> Update IMCS, but I keep receiving the error "The platform IMCS 2.2 does not exist" Do you know if this is just a limitiation of the eval version?

[Kirk J]

So I ran into some errors on same image for c220m3 in lab, but because the http/https setting, which impacts the protocol path the CIMC takes to reach back to the image hosted on the IMC Sup  (login via shelladmin, and choose option 27 to enable/disable HTTP)

Your error with memory, may be an issue with CIMC memory, and the HUU instance that is actually booted is likely not having issue with the physical RAM of overall server.

You try doing hard power cycle on server (or do reset via admin,utilities, reboot cimc) to clear up CIMC memory?

Not sure about the eval message.

Thanks,

Kirk...

[Jeffrey Foster]

I would start with a Cisco IMC (BMC) Reboot. This is something that can be done without disrupting the host and will make sure that the system is in a fresh state before attempting to update firmware.

[dmveron01]

Kirk,

After installing all the updates other than the CIMC (and the server has rebooted) I completely power off the server (Power -> Power off system from the Java console OR shutdown from vCenter), wait a couple minutes, boot from the ISO, and install only the remaining CIMC update. So far this seems to be the only way to avoid the memory issues. As far as HTTP/HTTPS, we have HTTP disabled and do not seem to have any issues using only HTTPS.

Jeffrey,

Interestingly enough, there is an option on the error message to reboot the CIMC to clear the memory issue, but it does not appear to work. The connection to the console drops, but when you reconnect to the CIMC after waiting 7-10 minutes, it's still on the same screen in the upgrade menu. I can tell the CIMC upgrade to run again, but it just immediately errors out again with the same memory issue.

(So that is what happens when clicking Yes above.) However, if I click no, run all the other updates, let the host reboot, power off the host, power on, and run just the remaining CIMC update, that works. Note, if I power off at the onset, boot from ISO, and try to install everything (or just the CIMC update), I get the same error screenshoted above.

Thanks.

[Kirk J · Accepted solution]

Thanks for the additional info.

Based on the cimc mem issues that pop up during a non-IMC sup triggered install, this would appear to not be IMC sup related, but something with the current CIMC firmware version, and the process of upgrading the CIMC itself.

Think you are at the point where it would be helpful to open a TAC case based on the Rack server, and errors during upgrade.

Probably need TAC to check the CIMC status before/during upgrade attempt, and HUU log output for the upgrade attempt.

Thanks,

Kirk...

[dmveron01]

Thanks for the feedback. These M3 servers are no longer under support for us, but we're wrapping up updates this morning anyway.

[peeat]

I've been struggling with the same/similar issue as above, with about ~5 months in between attempts at getting the latest version 3.0(4j) applied.  My initial attempts were using virtual media, activating virtual devices, then mapping the ucs-c240-huu-3.0.4j.iso file to CD/DVD and once HUU loads, I'm presented with the "Not enough memory to update, Please reboot Cisco IMC."

So the solution / workaround here is to spin up a CentOS 7 VM and apply all the updates via that method, rather than using the HUU?  In case you couldn't tell from the screenshot, this is for a Cisco UCS C240-M3S rack server.  I know it's a little bit older and probably nearing (or past) EOL / Support, but it's what we've got to work with, so I'm doing the best I can (which obviously isn't much).  I had been doing a decent job of keeping up (for the most part) as I at least got up to version 3.0(4d),  however this one just doesn't wanna fly.

I thought I read somewhere that it may be possible to extract the CIMC and BIOS data specifically from the HUU iso and apply these updates directly, is that true?  If so, could someone point me towards a guide, or give me a little bit of information to get me started?  My physical media doesn't seem to do any better than mounting a virtual device and attempting to boot from it that way, in fact it seems to hang more than not.  Am I understanding correctly that the workaround would be to spin up a little CentOS VM, and apply the updates that way?  Or can i boot from a CenOS live install and try to work some magic, is that even possible?  Will hope to have more time to devote to this in the coming days.  Seeing as how my NIC and RAID Controller don't have any updates with 3.0(4j) update, I'd love to attempt to simply apply the CIMC and BIOS upgrades specifically on their own, if that is an option someone can help me get started with.  I mounted the disc and poked around through it, but only find the following two files with any significant information contained within them, but I'm unsure how to handle, or work with .enc files, or what that type would even be.

firmware.squashfs.enc
tools.squashfs.enc

Thank you for your time and attention to my little matter.

[dmveron01 · Accepted solution]

Hi peeat,

I recently did several 3.0(4i) to 3.0(4j) updates, and the situation has not changed. We have several different sub-model C220 M3 hosts, and the problems are the same as described in the original post and your experience. I have managed to complete CIMC upgrades following these steps:

1. Power off the host completely.

2. Log into the CIMC web GUI and select the option to reboot the CIMC.

3. Once that completes, open the KVM, attache the virtual media, power on the host, and boot into the HUU ISO.

4. Once fully loaded and scanned, upgrade only the CIMC (you will most likely get a warning about not upgrading the CIMC and BIOS at the same time).

5. If successful, update everything else.

6. If you still get the CIMC needs to be rebooted due to memory error, select the option to reboot. Wait 10-15 minutes and reconnect to the CIMC and KVM.

7. The HUU should still be up, and you should be able to retry the install. I find that it always works at this point if I've followed the previous steps.

8. After the CIMC updates, update everything else.

Hopefully this works for you.

[peeat]

You are an absolute gem, @dmveron01 -- going to try to run through this right now.  I think I can follow these well laid out instructions, as I tried it enough times previously, but the warning regarding installing just CIMC scared me off and I decided that Cisco probably knew best (HA!) and shouldn't mess with the routine.  I've had issues with HUU about 25% of the time i attempt an upgrade.  Wish me luck!

 
First attempt... here we go!~  Thanks again for the pro-tip(s).  The prompt response is greatly appreciated.  Let's hope I'm competent enough to push through this.

[EDIT:]  I actually only have CIMC and the BIOS to update with the jump from current; C240M3.3.0.4a.0 (Build Date: 03/06/18) to 4b, I think dated June 22nd, 2018, but don't quote me on that exact date of the build.  I freaking love you, sir.  Truly appreciate the super fast response time... I'd been at this for quite a while, not sure how long we've had this machine, but I've got a number of firmware updates going back years, it seems.  Will have to check quick.

[peeat]

Heh, you can see (refer to second/bottom image above) I downloaded the latest HUU ISO file multiple times (twice, to be exact), as I was just turned around in circles, wholly unaware of what I was, or could be doing incorrectly.  Had poured through the documentation, release notes, installation steps... anything I could think of.  Wish I had just come here to ask the experts (YOU) initially, would have saved me a not so insignificant amount of frustration & grief, and from being on that rollercoaster of emotion that results from the lengthy waiting process... wondering, worrying if it's going to properly run through the installation this time.

[peeat]

Woop-woo!!  First flippin try.  You are A-mazing.  Seriously, thank you SO so very much.  You've saved me much pain and suffering.

BIOS Version:  C240M3.3.0.4b.0 (Build Date: 06/22/18)
CIMC Version:  3.0(4j)

 C240M3.3.0.4b.0 (Build Date: 06/22/18)

Cisco Integrated Management Controller Version: 3.0(4j)

 We're in the money, i'll be damned!  Would it be too forward of me to say that I love you, seeing as how it's Valentine's Day and all?  Kind regards, kind sir!  Appreciate your time and expertise.  Not sure I would have ever gotten over that hump, as aforementioned, this has been an ongoing frustration to the point where I just simply gave up for ~5 months because I was simply banging my head against the wall.  B ravo.  Problem solved.

[peeat]

Thanks you, yet again for your time and attention to my little matter! @dmveron01 has the solution. He's a friggin genius.  Mark that resolved, woot.  Anyone else experiencing this issue, please just follow the well documented steps above and you'll be cookin in no time flat.

Waiting on a memory upgrade for this box, at which point we'll probably try the newest custom image:

VMware_ESXi_6.7.0_10302608_Custom_Cisco_6.7.1.1.iso 

Was finally released, at some point -- I'm not exactly sure when, but I only just recently downloaded it.  I had previously downloaded (but failed to apply the upgrade(s), for one reason or another, so will almost certainly just start fresh and rebuild everything from the ground up in hopes of max compatibility.  Would REALLY love to get some VDI going with the triple Quadro 2000 that are taking up all my PCI-E slots and pumping out a lot of heat.  They're not the most powerful cards, only 192 CUDA cores and 1 GB GDDR5, but they'd make for wonderful VDI setups, hoping to spin up a few Windows workstation environments.

[dmveron01]

I'm happy to hear this was helpful. I maintain a sizeable number of C220 M3 and M4 hosts, and this has been an issue on most of our M3 ones. We initially ran a trial version of CIMC Supervisor to run the updates, which were successful on the M4 hosts, but not the M3 ones due to this strange CIMC error. Too bad for Cisco; we might have actually purchased CIMC Supervisor if they could fix the issue.