Nexus 3000 N3K-C3064PQ-10GX bricked on upgrade from 6.0.2.U6(8) to 7.0

[alfredos]

Hello,

First, I upgraded from 6.0.2.U6(6) to 6.0.2.U6(8). On success, I upgraded to 7.0.3.I7(9). The preparation went fine, with the switch reporting that the upgrade would be disruptive, and that "Power-seq upgrade needs a power-cycle".

Last thing it said was the usual:

Finishing the upgrade, switch will reboot in 10 seconds.

After reboot, the switch would not come back to life.

Connecting a cable to console and cold booting, results in only this:

(c) Copyright 2011, Cisco Systems.
N3000 BIOS v.4.5.0, Thu 11/09/2017, 05:38 PM

It will not run the boot loader or react to Ctrl-C or Ctrl-L.

I have searched for the symptoms but haven't found this same issue. I did find a few similar but different ones. It is not CSCvb64127 or any of the other similar bugs; none of them describe the switch hanging on boot before the bootloader.

Is there any way out of this, short of opening a case with TAC and probably having the switch exchanged?

TIA.

[balaji.bandi]

turn off the device, connect the console cable post complete boot process.

[alfredos]

Thanks for your answer. I did that; it is what I referred to as cold booting. It hangs as described.

[balaji.bandi]

if we can see the boot process we can suggest better, by that message, unfortunaly i do not have any advise.

[alfredos]

The message I posted is the whole boot process. Nothing else happens.

[balaji.bandi]

Time for TAC and RMA.

[marce1000]

               >...Power-seq upgrade needs a power-cycle

Act accordingly

 M.

[alfredos]

Thanks for your answer. I did; that is what I am talking about a cold boot.

It is possible to reboot and run the switch without power cycling with a power sequencer firmware upgrade pending. I have done it many times. The only thing that happens in that case, AFAICT, is that the power sequencer doesn't get updated, but otherwise the switch boots and works normally.

[marce1000]

 - It's a bit unclear for me what you mean by that I would at least follow the steps in the bug-report (Workaround) , perhaps it is not in essence related , but use the mentioned steps to try to boot an image through tftp (e.g.) , if that works follow the subsequent steps too.

 M.

[alfredos]

Unfortunately, since I can't get a bootloader prompt, it is not possible to try to boot through tftp.

I have tried to get a bootloader prompt as described by pressing Ctrl-L but the switch won't offer any.

I think that perhaps the switch has corrupted its bootloader. Flash memory failure, or software bug?. If this hypothesis is correct, then it begs the question: Is there any way to recover from a corrupted bootloader?

[marce1000 · Accepted solution]

 - Ref : https://community.cisco.com/t5/data-center-switches/stuck-at-bios-nexus-3000/td-p/4390585

              Looks like the device may need replacement and or ask advice from Cisco too.

 M.

[alfredos · Accepted solution]

Note: Upon opening the unit seeking last chance salvage, we noticed that its storage is an eUSB module from Viking Technology. It can be trivially removed. After doing so and booting, the box goes to the boot loader. From there, it will be able to tftp boot.

We haven't tried yet but chances are that the module can be hot plugged, formatted and fitted with a new image.

Anyway, I'm leaving this note here just in case anyone else finds their Nexus in a dire situation hanging on boot before being able to go to the boot loader: A few screws and you have boot loader, and your box is alive!

Thanks to everyone. Take care.

[alfredos · Accepted solution]

Update: Swapping the eUSB module with a good switch had both switches booting, although the one with the "bad" module had to be booted manually from the boot loader the first boot.

I am leaving this here for anybody with the same predicament: Just remove your eUSB and things will start going well. Swap it with a known good one and you're instantly back in business.

The eUSB module is the same used on the ISR series. I haven't tried, but it seems that part MEM-FLSH-4U8G should work, as well as many others with references starting with MEM-FLSH-. If I had to buy one, while I'm at it, I'd buy 4 or 8 gig; it would save a lot of compact fiddling with the images.

Have fun everybody.