WS-C2960X-24PS-L won't boot after IOS upgrade

[NTAUSD]

We have about 60 Catalyst switches of this model. I started upgrading 17 of them from IOS 15.0.2-EX5 to 15.2.7-E7 but have potentially run into this bug (CSCvf24186) on 2 of them so far, the other 15 have updated successfully. At this point I've stopped upgrading this model switch.

Researching online suggests this might be a hardware defect and Cisco would be able to verify via the Serial Number, can anyone confirm this?

I've attached a log of the console output when the switch is trying to bootup.

• On powering up, the fans don't spin. SYST light flashes green rapidly. Console shows Testing Memory, etc.
• Switch proceeds onto Initializing Flash and then verifying image. SYST light still flashes green.
• SYST light turns solid green and the switch proceeds to execute IOS.
• When the console shows "% Front-end Microcode IMG MGR: malformed microcode cache", fans finally turn on at full blast. It hangs for about a minute or two.
• When the console shows "POST? MA BIST : Begin", all lights turn solid green.
• Then the switch just hangs on "POST: Thermal, Fan Tests : Begin" with the SYST light flashing green and all other lights flashing.

I've tried multiple cold boots, format the flash via ROMMON, copy a fresh copy of IOS verifying /md5, boot from USB, inserting and/or removing a stacking module, etc.

I've tried to boot multiple IOS versions such as 15.0.2-EX5, 15.2.2-E9, 15.2.4-E6, 15.2.7-E7 but the results are always the same.

I've opened a case with TAC but I don't have a service contract on the 2 switches that have failed so I don't know if they'll provide troubleshooting support if they determine it's not an RMA.

[Leo Laohoo · Accepted solution]

---

@NTAUSD wrote:

% Front-end Microcode IMG MGR: address 0, PROGRAM failed
% Front-end Microcode IMG MGR: unable to program device: 0
% Front-end Microcode IMG MGR: malformed microcode cache

---

Remove the power cable of the affected switches for about 5 seconds and apply the power back.  If the switches still fail to boot, RMA the switches.  

---

@NTAUSD wrote:

I've opened a case with TAC but I don't have a service contract on the 2 switches that have failed so I don't know if they'll provide troubleshooting support if they determine it's not an RMA.

---

Enhanced Limited Lifetime Hardware Warranty are included on every 2960X purchased.  A valid Service Contract is not necessary to RMA faulty 2960X.  

[NTAUSD]

Thanks Leo, I've already tried hard reboots, even leaving the power cable unplugged over a weekend. At this moment I'm prepared to RMA these 2 switches but I'm worried how often I'm going to run into this issue with the other 43 switches I have yet to update. To your knowledge, is this a known hardware defect with certain S/Ns, versions, etc.?

Thank again

[Leo Laohoo]

---

@NTAUSD wrote:
To your knowledge, is this a known hardware defect with certain S/Ns, versions, etc.?

---

Hardware defect?  Not that I am aware of.

[NTAUSD · Accepted solution]

Thanks for the help Leo. TAC replaced both switches via RMA. Here is their response regarding the issue.

Thank you for the detailed response describing the issue you are facing.

I researched our internal database regarding the “% Front-end Microcode IMG MGR: malformed microcode cache” log.

If the issue is not encountered only once but multiple times after trying to boot the latest IOS then this can indicate a hardware failure.

---