topic Re: ISE CIMC software strategy in Network Access Control

ISE CIMC software strategy

Johannes Luther — Tue, 23 Oct 2018 05:06:30 GMT

Hi board,

I'm wondering how others handle the CIMC in the Cisco ISE.

In the ISE downloads, there is one BIOS and CIMC software (3.0.3a). However, there are lot's of CIMC vulnerabilities, which needs to be closed. The hardware installation guide does not state that the 3.0.3a is the only supported BIOS.

In fact the guide states:

The following procedure is for upgrading the BIOS and Cisco IMC to version 3.0(3a). However, this procedure is generic and is applicable for newer firmware releases that are posted on Cisco.com.

Does this statement apply to the firmware posted in the "Identity Services Engine" download section or is it also supported to use a newer firmware in the downloads section of the corresponding UCS server model?

I know that other firmwares also work - the question is whether this is supported as well.

How you do handle the software of the CIMC?

My question also applies to other UCS based appliances like the WLC5520 etc. 🙂

Re: ISE CIMC software strategy

Mohammed al Baqari — Tue, 23 Oct 2018 05:21:00 GMT

CIMC is completely related to UCS out of band management which is similar
to iLO in HP. It has nothing to do with ISE or WLC or any other VM.

CIMC version is completely independent from the VM. In fact the VM is a
process running in ESXi and CIMC isn't aware about it.

Re: ISE CIMC software strategy

Johannes Luther — Tue, 23 Oct 2018 05:24:17 GMT

Hey Mohammed,

yeah - I see it exactely the same way (except the fact that there is no CIMC in VM deployments because it is purely Cisco UCS related 🙂 ).

However I don't get the point why there is exactely one offered version in the ISE download section. This implies that this is the only version to use in combination with ISE. Why isn't there just a download hint to the corresponding UCS model and the statement "use the software you want" ? 🙂

Re: ISE CIMC software strategy

Arne Bier — Tue, 23 Oct 2018 05:49:41 GMT

I think @Johannes Luther is referring to the software lifecycle management of the entire system (UCS server + ISE application) because when you purchase the SNS-3595 you should consider the life cycle management of CIMC as well. Yes of course an attacker on the ISE gig0 will never reach the CIMC (if the CIMC is running on dedicated Management Eth port) but the hacker may already be on the management network - and if they get to your CIMC then they can hose the entire server. Therefore it's probably sensible to keep patching the CIMC whenever possible. If you run CIMC and ISE application on the same GigE port, then who know what might happen (from a security risk point of view).

Server appliances are a pain in that respect because of this additional compute layer. Nevertheless, I think Johannes has an excellent question and when I recently commissioned 6 SNS-3595 servers I also looked at the CIMC version and didn't dare touch it. Not much guidance around this topic. It would be nice to know from Cisco how to maintain the SNS server CIMC software if a CVE is announced.

Re: ISE CIMC software strategy

Johannes Luther — Tue, 23 Oct 2018 06:46:43 GMT

Hi Arne,

thanks for the feedback! At least someone understands me 😉

So what I do with the CIMC boards at the moment is to keep the recommended (downloadable) software, but I do a full blown configuration of the CIMC (SNMP, Syslog, LDAPs for admin auth, SSL certificates, SoL etc.).

I think this is really important in the ISE, because the ISE application and ADE-OS doesn't monitor all hardware related issues. I guess simple things like a power supply failure is recognized by ADE-OS (operating system).

But more complex situations like an HDD failure of the RAID-10 cluster are only recognized by the CIMC. Even if hardware failures of HDDs are recognized by ADE-OS, there are much more complex failure situations regarding storage, memory and CPU.....

So bottom line is, that the CIMC is a very very crucial part to fully monitor the SLA of the ISE service.

...

And no ... Configuring the CIMC is not fun 🙂 It took me some time to build a good CLI template for my use case....

However, I'm still not sure about the SW version ...

Re: ISE CIMC software strategy

hslai — Thu, 25 Oct 2018 05:05:06 GMT

If your SNS appliances are of 3515 or 3595, they need signed binaries for secure boot. CSCvj90778 or CSCvm14331 are addressed recently with a secure signed CIMC 3.0(4j) and I believe it soon available at Cisco software download.

Re: ISE CIMC software strategy

Tima_20 — Thu, 25 Oct 2018 07:26:22 GMT

Don't try corresponding UCS CIMC, I ruined 2 SNS3495 appliances this way. One time with host upgrade utility and one time with manual upgrade, it doesn't work. Probably there are some differences between CIMC on SNS Appliance and UCS

Re: ISE CIMC software strategy

Mohamed Abd Elnaser Mohamed Mohamed Ali — Thu, 25 Oct 2018 08:22:35 GMT

Hi Everyone

Just to add to this thread, I have recently deployed SNS-3515 with Cisco ISE 2.4 which comes with default CIMC firmware version 3.0 and when I'm tried to update the SNS-3515-K9 C220M4 Appliance Firmware from 3.0 to 4.0.1a (Cisco Recommended) using the HUU ISO, it won't allow me to boot with an error "Invalid signature detected. Check Secure Boot Policy in Setup".
So when I tried to disable the Secure boot option under the CIMC Utility, it shows another error "In ISE mode BIOS secure boot can not be disabled."

So I have opened a TAC case to assist into providing the special HUU iso that can work for this ISE box.

and below is their reply stating that “For 35xx servers, we do not recommend firmware upgrade unless absolutely necessary, Please let me know the reason why you would like to upgrade the firmware?”

So it seems that the CIMC firmware image for UCS servers purpose-built for Cisco ISE software is somehow secure-boot locked with special signature to prevent inappropriate boot even with physical access to the box.

However, that make it a pain as you mentioned to maintain the CIMC firmware update to date against vulnerabilities and security flaws and the only way to my knowledge to obtain and upgrade the CIMC for SNS-35xx is through special signed image given by Cisco TAC.

Re: ISE CIMC software strategy

Arne Bier — Thu, 25 Oct 2018 12:10:12 GMT

@Tima_20 / @Mohamed Abd Elnaser Mohamed Mohamed Ali - your experience reminds me of myself when I was starting out my IT career in my mid 20's - at that time I probably would have gone ahead and tried to do the same as you and also failed (because I thought I was "just doing an upgrade" - what could possibly go wrong? - and besides, I am doing the customer a big favour by putting them on the latest and greatest software, right?).

I think many of us have been in that situation too - I have been there and also made those mistakes - left with sick feeling in my stomach when the customer's server is turned into a thousand dollar brick of nothing. And then the pain of the TAC case etc.

So I can say that getting older (and more grey hairs) has some benefits after all. Last week when I sat across 6 SNS-3595 and I looked at the CIMC, I was tempted to update them. I went to UCS web site and studied all their notes. But all the release notes there mentioned that 3.0.3 was latest for M4 model. So I Ieft it alone - thank goodness. And, as I said above, my grey hairs were telling me to stay away from this stuff because it's going to end badly. ISE servers were running just fine. Leave it alone!

No more trigger-happy fingers for me when it comes to Cisco products. Best to just follow orders.

Re: ISE CIMC software strategy

Johannes Luther — Thu, 25 Oct 2018 12:42:20 GMT

Good point - however the last SNS-35XX appliances I got have "secure boot" disabled by default.

Furthermore, does the "secure boot" setting actually restrict CIMC/BIOS updates?

"The SNS 3515 and SNS 3595 appliances support the Unified Extensible Firmware Interface (UEFI) secure boot feature. This feature ensures that only a Cisco-signed ISE image can be installed on the SNS 3515 and SNS 3595 appliances, and prevents installation of any unsigned operating system even with physical access to the device. For example, generic operating systems, such as Red Hat Enterprise Linux or Microsoft Windows cannot boot on this appliance. "

Edit after I read some other topics:

>> Furthermore, does the "secure boot" setting actually restrict CIMC/BIOS updates?

==> Yes it does 🙂

Re: ISE CIMC software strategy

Johannes Luther — Thu, 25 Oct 2018 12:37:18 GMT

>> So when I tried to disable the Secure boot option under the CIMC Utility, it shows another error "In ISE mode BIOS secure boot can not be disabled."

Strange ... I have pretty new ISE SNS-35XX appliances here without enabled secure boot. So factory default was (in my case) disabled secure boot.

Did you enable it manually or were the ISE appliances delivered this way

@Kirk J outlined in another topic:

Also,once the secure boot is enabled, it cannot be disabled (by design).

https://community.cisco.com/t5/unified-computing-system/ucs-upgrade-fails-invalid-signature-detected/m-p/2983835/highlight/true#M22244

Question is: How are the ISE appliances shipped by default?

Re: ISE CIMC software strategy

Johannes Luther — Thu, 25 Oct 2018 12:40:31 GMT

First of all thank you for the very helpful replies. Let's summarize:

ISE SNS-35XX: Only the ISE certified CIMC/BIOS images (from the ISE download section) can be used, because of SecureBoot. The SNS-35XX appliance verifies if the CIMC/BIOS images are signed for the use on SNS-35XX
==> SecureBoot cannot be disabled once it has been enabled!
For other appliances it is maybe the same. It all depends on the SecureBoot feature
We have to live with "later" or no patches for vulnerable CIMC/BIOS versions on UCS based Cisco appliances (not a very good idea if you ask me - but maybe that's just me opinion 🙂 )

Re: ISE CIMC software strategy

Arne Bier — Sun, 28 Oct 2018 22:28:28 GMT

@Johannes Luther - where exactly in the ISE Download section does one find the CIMC code? I have looked at every ISE version and can only find .iso/.ova/patches/certs. But no CIMC.

If I have a SNS-3595 that was shipped from Cisco this year May, and in the version is 3.0.3a, then this is the latest available version, right? Am I also correct in saying this is a UCS-220 M4 ?

I thought firmware is kept here

Downloads Home / Servers - Unified Computing / UCS C-Series Rack-Mount Standalone Server Software / UCS C220 M4 Rack Server Software

Re: ISE CIMC software strategy

Johannes Luther — Mon, 29 Oct 2018 06:22:35 GMT

Hi Arne,

yeah - it's a little bit hidden:

https://software.cisco.com/download/home/283801620/type/283802505/release/SNS-35X5%20Firmware

Re: ISE CIMC software strategy

walwar — Mon, 29 Oct 2018 14:20:20 GMT

Our customer bought to SNS3515 couple of weeks ago and I WAS so tempted to upgrade the CIMC and thank god I didn't, other wise I'd be pulling my hairs one by one.

These came update with the latest 3.0.3S2, at first I was not sure if I need to update them or not since I couldn't find any newer version but I let it be and now I am happy that I didn't touch it. 😄

Valuable lesson learnt from this thread!

Re: ISE CIMC software strategy

Mohamed Abd Elnaser Mohamed Mohamed Ali — Tue, 30 Oct 2018 07:19:21 GMT

Hi Johannes
Sorry for late reply...
Yes, the secure boot was enabled on that new SNS-3515 out of the box. So the appliance were delivered in that way.
regards.
Mohamed Naser

Re: ISE CIMC software strategy

Mohamed Abd Elnaser Mohamed Mohamed Ali — Tue, 30 Oct 2018 07:28:32 GMT

Hi Arnie
Thanks for sharing your experience with us, but believe me when I say it was the customer who insisted into upgrading the CIMC since it was ACS to ISE migration ( Green field implementation). So they wanted all Cisco recommended and latest software in place, although I advised them not to touch the CIMC but they wanted Cisco official response to stay on the default CIMC image that came with the box.
That why I opened a TAC case under which they confirmed that "For 35xx servers, we do not recommend firmware upgrade unless absolutely necessary".
After sharing that with the customer, they got relaxed and closed the TAC case and we moved on.
But it was an interesting thing to know about CIMC complications on SNS-35xx purpose-built for Cisco ISE.

Re: ISE CIMC software strategy

Johannes Luther — Wed, 31 Oct 2018 12:04:09 GMT

Funny - just as we're talking about it, a new CIMC SW version is published for the ISE 🙂

https://software.cisco.com/download/home/283801620/type/283802505/release/SNS%2035x5

I'm wondering why they released an update

Re: ISE CIMC software strategy

hslai — Wed, 31 Oct 2018 20:38:06 GMT

CIMC 3.0(4j) and BIOS 3.0.4(b) are to address CSCvj90778 / CSCvm14331 while the other set is due to upgrade path. The associated upgrade guide is still being updated and will be posted once it ready.

Re: ISE CIMC software strategy

TommasoOlmastroni1347 — Wed, 14 Nov 2018 18:31:13 GMT

@hslai wrote:

CIMC 3.0(4j) and BIOS 3.0.4(b) are to address CSCvj90778 / CSCvm14331 while the other set is due to upgrade path. The associated upgrade guide is still being updated and will be posted once it ready.

So, what I’m supposed to do with CIMC 3.0.3(S2)?

I’m on delivery, I hardly prefer to update now in my lab then when it will be on prod.

thanks