UCS Server Configuration Utility stuck Initializing Kernel

Steven Misrack · ‎12-18-2012

Hello,

After upgrading to the 2.1(1a) firmware, I noticed that two of my 8 blades want to boot into this new Configuration Utility after the service profile has been applied. The software just sits at Initializing Kernel forever.

Does anyone know what causes this? Or why it is only happening on 2 of the 8 blades (which all have the same service profile)?

I have attached a screen shot.

Thanks in advance,

-Steve

padramas · ‎12-18-2012

Hello Steve,

Are these rack servers managed by UCSM ?

Can you please share the server model and configured boot order for these servers ?

Padma

Steven Misrack · ‎12-19-2012

Hello Padma,

These are all blades, B200M3's. The configured boot order for the service profile is all SAN. There are currently no boot LUN targets configures, so I expect the blades to boot into the error "no disk found". This is what happens with the other 6 service profiles.

Thanks,

-Steve

David McFarland · ‎12-19-2012

Steve,

The screen shot you provided is the C-series Server Configuration Utility. It's an assisted install ISO for Windows/Linux on C-series local disk only. It is not supported on any B-series blade.

The SCU ISO is not loaded onto an FI, did you map the SCU ISO to vmedia for these blades?

Dave

Steven Misrack · ‎12-19-2012

That definately explains why it just hangs. I am not mapping any ISO's at this time.

The really odd thing is I created a service template and deployed 8 copies of it. So they all should be exactly the same, but 2 of them are exhibiting this behaviour.

Thanks,

-Steve

David McFarland · ‎12-19-2012

Steve,

If you not mapping an ISO and you are still seeing the Server Configuration Utility, would you do me a favor?

1) In KVM, reboot and select F2 and screen capture the Boot Order. I'm curious if SD or USB is in the boot order

2) Remove the blade from the chassis and look for SD-flash cards inserted into the side of the blade. There are two SD slots towards the front of the blade in the side.

3) Open the blade and look for a USB stick inserted between the HDD's

With release code, the SD flash is disabled and should not be visible as a boot device. The SD cards provided for the rack C-series do have bootable images (Server Configuration Utility) is one of the bootable images.

If you have a USB stick inserted, the blade may default boot off of it. If SCU is on the UCS stick, that may be why you are seeing it.

If you find both SD and USB, would you mind removing them one at a time and let them boot up and report the results? (i.e. SD card only one test amd USB stick only one test)

Please let me know the results of your investigation.

Thanks,

Dave

Steven Misrack · ‎12-19-2012

Hi Dave,

I can confirm there are USB memory sticks in all of the blades. We had planned to install ESXi onto them before we had our SAN environment. This equipment is in a seperate building, so it will take me a little while to go and check.

The boot order/option is set to:

UNIGEN PSD16H0M3 PMAP

UEFI: Built-in EFI

I think you are on the right track that it's booting from that device. Although all of the blades have this same memory stick in them, I noticed that we were shipped two different sizes with our order.

Thanks,

-Steve

David McFarland · ‎12-19-2012

Steve,

I love a mystery. We'll get to the bottom of this.

From what I know, the USB sticks should be formated, but, blank.

Thanks for working with us on this.

Dave

Steven Misrack · ‎12-19-2012

My pleasure.

I turned to the forum because there hasn't been any progress on my TAC ticket and I was getting a bit frustrated.

Thanks,

-Steve

David McFarland · ‎12-20-2012

Steve,

Not sure how much you'll get done before Christmas, but, it really sounds like your USB sticks have been compromised. Someone used them to install SCU on a C-series.

Were these purchased through Cisco or a Partner? Can you provide me with the Sales Order Number?

If you can live without one of them for a few days, I'd like you to get back with TAC (please provide me the SR# also) and RMA the suspect USB sticks. Request them to open a FACT (Failure analysis case). If they agree, I'll send you information to send the stick directly to me. If you return them under the RMA, they'll end up in Warehouse 13 never to be seen again.

With the second suspect USB stick, you might try loading your ESXi iso via vmedia, select F6 during boot and force boot off your vmedia. When ESXi detects the Unigen USB stick try to initialize and install onto the stick.

This is what a Cisco USB stick should look like

Again, please let me know how it goes.

Thanks,

Dave

Steven Misrack · ‎12-21-2012

Hi Dave,

Here is what I have figured out after my investigation. All of the systems that are booting into this utility have a 16gig Cisco stick in them: 17-11213-01 and they have two linux partitions on them. (I assume one is the boot partion). The ones that boot to have 4 gig USB sticks with no data on them.

So this helps explain what I am seeing. After the holiday break I am going to remove all of the USB sticks from all of our blades as they are no longer needed.

My TAC SR# is: 624222711

Side note: Is there a way to disable EFI booting in the BIOS from the UCSM? Or do I have to go into the bios of each system?

David McFarland · ‎01-02-2013

Steve,

Sorry for the long delay. Please RMA the 16GB USB sticks. I've asked the TAC engineers to do so. We only sell and support the blank formated 4GB USB sticks, so, I'm not sure how you got 16GB USB sticks with SCU installed on them

Regarding the EFI boot. If you don't have any bootable media in the boot order, BIOS will drop through to the EFI loader on the M3s. Once you have created a boot order in the Service Profile and installed an OS on the 4GB USB sticks (e.g. order 1st CD ,2nd HDD). The B200M3 will boot off the USB (HDD device) stick.

On the M3's (both B and C series) we no longer enumerate devices that do not have a mapping, regardless of the boot order. So, even if you create a boot order of 1st CD/DVD, 2nd HDD, you must map the installation ISO in vmedia and reboot in order for it to be recognized. If there is no mapping, vmedia will not be visble to the boot order.

Dave