C2960X locking up after upgrade

[cerickson]

We recently upgrades our catalyst 2960x-48FPD-L switches from 152-2.E1 to 152-7.E7 as requested from our VoIP provider.  For the most part, all upgrades went well.  However, we have two switches that didn't seem to take.  In both instances, they were part of a stack and when singled out, we can see in the CLI that they're locking up during the loading process.  The rest of their stack counterparts are working fine.  With the problem switch, I found two things to be true:

1. If I roll back to 152-2.E1, the switch will boot and work, and
2. The switches that didn't take also reflect what appears to be an older bootldr version (15.2(6r)E is on the switches that upgraded, whereas 15.2(2r)E1 is on the switches that now fail to boot with the upgraded iOS.  

While using the upgraded iOS, it always locks up at the same point:

CPU rev: B
Image passed digital signature verification
Board rev: 18
Testing DataBus...
Testing AddressBus...
Testing Memory from 0x00000000 to 0x1fffffff.../
Using driver version 4 for media type 1
Xmodem file system is available.
Base ethernet MAC Address: 00:da:55:1d:90:80
The password-recovery mechanism is enabled.
USB EHCI 1.00
USB EHCI 1.00
USB Console INIT
Initializing Flash...
mifs[5]: 12 files, 1 directories
mifs[5]: Total bytes : 1806336
mifs[5]: Bytes used : 834560
mifs[5]: Bytes available : 971776
mifs[5]: mifs fsck took 0 seconds.
mifs[6]: 2 files, 1 directories
mifs[6]: Total bytes : 3870720
mifs[6]: Bytes used : 1213440
mifs[6]: Bytes available : 2657280
mifs[6]: mifs fsck took 1 seconds.
mifs[7]: 5 files, 1 directories
mifs[7]: Total bytes : 258048
mifs[7]: Bytes used : 8192
mifs[7]: Bytes available : 249856
mifs[7]: mifs fsck took 0 seconds.
mifs[8]: 5 files, 1 directories
mifs[8]: Total bytes : 258048
mifs[8]: Bytes used : 8192
mifs[8]: Bytes available : 249856
mifs[8]: mifs fsck took 0 seconds.
mifs[9]: 354 files, 141 directories
mifs[9]: Total bytes : 122185728
mifs[9]: Bytes used : 83788288
mifs[9]: Bytes available : 38397440
mifs[9]: mifs fsck took 67 seconds.
...done Initializing Flash.
Loading "flash:/c2960x-universalk9-mz.152-7.E7/c2960x-universalk9-mz.152-7.E7.bin"...Verifying image flash:/c2960x-universalk9-mz.152-7.E7/c2960x-universalk9-mz.152-7.E7.bin.........................................................................................................................................................................................................................................................................................................................................................................................................................Image passed digital signature verification

...@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:/c2960x-universalk9-mz.152-7.E7/c2960x-universalk9-mz.152-7.E7.bin" uncompressed and installed, entry point: 0x3000
executing...
Loading "flash:/c2960x-universalk9-mz.152-7.E7/c2960x-universalk9-mz.152-7.E7.bin"...

I've confirmed the MD5 hash matches for the upgrade file.  I've gone as far as formatting Flash and reinstalling completely from ROMMON.  Redownloaded the iOS file, tried a middle version(152-6.E) that also locked up, and attempted upgrade from USB or from FTP.  I'm stumped on what it doesn't like about the upgrade.  Short of hardware failure, the only thing I have been able to find different is the bootldr version.  

Here is what displays as part of the upgrade process (using archive download-sw /overwrite /reload flash:c2960x-universalk9-tar.152-7.E7.tar):

Stacking Version Number: 1.160

System Type: 0x00000000
Ios Image File Size: 0x0198EA00
Total Image File Size: 0x027A3A00
Minimum Dram required: 0x08000000
Image Suffix: universalk9-152-7.E7
Image Directory: c2960x-universalk9-mz.152-7.E7
Image Name: c2960x-universalk9-mz.152-7.E7.bin
Image Feature: IP|LAYER_2|SSH|3DES|MIN_DRAM_MEG=128
FRU Module Version: No FRU Version Specified

Any thoughts on what I am missing or overlooking with this?  All of the other switches in the stack took to the upgrade, just not the one.  

[Dustin Anderson]

So, as far as I know, the bootloader should get upgraded by the OS if needed on first boot. Similar to programming the asics and such. I had a similar situation with 2 stacks of 3850s a few years ago that 1 switch in each failed to upgrade. 1 I was able to get into rommon and upgrade manually, the other corrupted an asic and had to be RMA'd. Since you can function on the old code, you could try to step the versions up.

Where in the boot does it halt? Is it trying to start an asic or bootloader update?

As they are EoL, I don't think the lifetime warrantee is covered anymore. If you have support you could contact TAC. If not, you could open the switch and see if there is any noticeable damage at all.

[cerickson]

It locks up shortly after passing the image verification.  Same spot each time.  Shortly after the series of @ that run the CLI for a minute or two, it'll say the image is uncompressed and loading and that is where it hangs.  I don't see it get to any bootloader update during the process.  What I copy/pasted above is what I see with each standard boot.  I have tried the upgrade from ROMMON, no change.  With the 152-6.E iOS attempt, I was hoping since that reflected the bootloader version, it might have something special to it.  But no luck.    

We have new switches budgeted for next school year, so I just need to get it to last for the current one starting.  Unfortunately, these were bought through some e-rate funding before I worked where I am now and from what I can tell, we never had access to TAC for support.  I figure my worst case scenario is to move the VoIP phones to the upgrades switches and remake this one into an dummy switch that passes all traffic to the stack.  But if there is something I can try first, I rather have it working as intended.  

[Leo Laohoo]

Contact Cisco TAC and organize for an RMA. 

Aside from the switch being faulty, another reason why the upgrade fails (but only works on a particular version) is when the switch is a counterfeit. 

And I suspect this switch is counterfeit.

[cerickson]

That's possible.  I'm trying to pull up old purchase records to see if I can get TAC support on this.  From what I can find, multiple switches were purchased through Cisco, but then we have a few that were likely purchased through Amazon. 

[Dustin Anderson]

You can call tac to check for an RMA. If it's a valid serial and still covered they will open an RMA ticket.