Re: WLC 2504 Boot loader failure

dmcgrath.ca · ‎10-18-2018

I just bought a 2504 WLC off of eBay, and of course it would appear to be dead on arrival. Well, at least partially. When I power it up, all I get is the following:

WLCNG Boot Loader Version 1.0.20 (Built on Jan 9 2014 at 19:02:44 by cisco)
Board Revision 0.0 (SN: PSZ18331DQS, Type: AIR-CT2504-K9) (P)

Verifying boot loader integrity...

##########################################
### IMPROPER SYSTEM OPERATION DETECTED ###
### ---------------------------------- ###
### System has been halted because: ###
### 1. Boot loader failed verification ###
##########################################

I tried open up the device and insert the CF card from it into my ASA to read it, and it almost looks like the guy managed to (incorrectly) put an FUS or something on it for a 5500 series device, and bricked it. I say this because the CF card had log files on it from February 17 2018 that suggest device names of "5500" at some point in it's life.

I put a copy of the .aes file on the CF card in an attempt to see if it helps, but it would appear that this is bricked to the point where not even it's "ROMMON" (if that is what the WLC calls it even) loads, and thus I can't even get at least a WLC OS image on it long enough to boot it to fix the boot loader.

I really can't afford to be out this much money on a bricked device, so I could really use any help I can get to try to repair this device! I noticed there are some points internal to the board. Is there any way to x/y/zmodem the FUS file back onto the device using some trick?

Thanks for any help!

Rasika Nayanajith · ‎10-18-2018

I do not think you will find any document on it. But worth to check with TAC as they may know some work around.

HTH

Rasika

dmcgrath.ca · ‎10-18-2018

I tried TAC real quick, but the lady I spoke to could only mention that the device is too old for an RMA and that there was no support contract, and that eBay was not an authorised 3rd party reseller, etc.

I see there are internal RJ-45 ports and com port headers etc, but without actual technical documents on how to re-flash the image onto the internal flash to restore the boot loader, there isn't a whole lot I can do, unfortunately.

Thankfully, they sent a refund and I can send the device back, but man I gotta say that I am disappointed to find out that such a situation is possible with Cisco products! I assumed that it would have been like a ROMMON where it's (afaik) impossible to brick the device. Low and behold, you can do it with a WLC though? That just isn't right! The recovery procedures should be public self-serve knowledge, right?

Anyway, I would really love how to solve this issue. I saw some documents that discussed the RTOS recovery, but I think that this is beyond that even, since the boot loader doesn't even have a menu to recover anything with over TFTP even.

dmcgrath.ca · ‎10-19-2018

A bit of follow up before the device goes back:

So I took a look at the CF card in my Linux box. Seems there are actually 4 partitions, with one of them being a ~16mb unused (filled repeatedly with some byte) that about matches the ER file. I also found that the second partition firmware file has a file that matches the error that the device throws: u-boot-wlcng.img

From what I can tell, this is the actual WLCNG boot loader that the device boots off of. Interesting part is that the data reported by file doesn't match the actual device boot up date:

WLCNG Boot Loader Version 1.0.20 (Built on Jan 9 2014 at 19:02:44 by cisco)
Board Revision 0.0 (SN: PSZ18331DQS, Type: AIR-CT2504-K9) (P)

# file part2/firmware/u-boot-wlcng.img
part2/firmware/u-boot-wlcng.img: u-boot legacy uImage, WLCNG CP BL v.1-0-20, Firmware/MIPS, Firmware Image (Not compressed), 347556 bytes, Tue Jan 14 06:11:29 2014, Load Address: 0x00000000, Entry Point: 0x00000000, Header CRC: 0x20FE41CF, Data CRC: 0x18792A8A

Notice how the file for that FUS is off by a few days. Anyway, it would appear that Cisco made this a little tricky. I was able to find a Russian page that you can translate that seems to confirm some of this design, albeit using an older firmware.

I have to say though that it appears like the device itself is just broken and unable to communicate with the CF card, almost as if a bad flash of that u-boot-wlcng.img was uploaded to the device, or the CF slot itself is physically broken, because I get the same error regardless of ram installed or CF card etc.

Granted, this "BIOS" (the WLCNG boot loader) would only be basic, if it were corrupted or there were physical inability to read the CF card, it could explain some of these errors and the lack of a boot menu (ie: it can't find the card to talk to it, or it did but is only smart enough to know it isn't correct somehow).

Here is a list of the images there as reported by file, and I will attach a manifest and checksum of the 8.2.151.0 image that should be running here:

# file part2/cavium_main.* part2/linux.*
part2/cavium_main.bak: ELF 32-bit MSB executable, MIPS, MIPS64 rel2 version 1 (SYSV), statically linked, stripped
part2/cavium_main.pri: ELF 32-bit MSB executable, MIPS, MIPS64 rel2 version 1 (SYSV), statically linked, stripped
part2/linux.bak.img:   u-boot legacy uImage, 7.6.120.0, Linux/MIPS, Binary Flat Device Tree BLOB (gzip), 36925481 bytes, Fri May  2 09:57:06 2014, Load Address: 0x00000000, Entry Point: 0x00000000, Header CRC: 0xD6447218, Data CRC: 0x551DA790
part2/linux.pri.img:   u-boot legacy uImage, 8.2.151.0, Linux/MIPS, Binary Flat Device Tree BLOB (gzip), 36899242 bytes, Wed Mar 15 07:31:56 2017, Load Address: 0x00000000, Entry Point: 0x00000000, Header CRC: 0x455E3A6E, Data CRC: 0x12B4A8A7

If anyone could possibly up/downgrade a lab unit and try pinpoint the corrupted entry, I would be most grateful.

I should also point out that the first 440 bytes of the MBR were zero's, so I guess there is no "grub" per se, at least not at the start of the MBR (I guess the WLCNG is hardcoded in this respect?), as well, the partition table had no partitions marked bootable.

Anyway I look forward to your replies! Thanks.

patoberli · ‎10-22-2018

If you're bored, you could re-try to flash the FUS. I think that one would contain those files you see. Not sure if that will work though, as you already seem to be on the current version.
I guess if somebody with an 2504 could send you a copy of the contents of the flash, you could overwrite yours.

dmcgrath.ca · ‎10-22-2018

@patoberli wrote:
If you're bored, you could re-try to flash the FUS. I think that one would contain those files you see. Not sure if that will work though, as you already seem to be on the current version.
I guess if somebody with an 2504 could send you a copy of the contents of the flash, you could overwrite yours.

The question is, how? The problem is there is no actual boot menu or loader of any sort to actually allow you to provision the system enough for it's tools to work.

Now clearly there has to be a way (think JTAG) for the factory to write the FUS back to the ROM, or similar, but I can't find any documentation on how to reflash the ROM (like EEPROM I guess). I was actually hoping to try something like this, but without documentation on the procedure, I would be guessing. Given the nature of such a process, my guess is that it wouldn't even have any sort of feedback and would be a "blind" operation that you just assume worked after you sent the last byte. Not to mention that if it operates at slow 9600bps speeds, would probably take 20 or 30 minutes.

At this point, I think it's probably easier to just send the unit back and get another one. I really would have loved to try flashing the image using the onboard ports if I could though.

patoberli · ‎10-22-2018

I'm pretty sure Cisco hasn't documented this public, because such a fault would usually mean replace the device. So yeah, I'd buy a new one. Then you could theoretically clone the compact flash card (it has one, right?) and with some luck both will work.

dmcgrath.ca · ‎10-24-2018

Why buy one when you can buy two at twice the price? ;)

No doubt it would be nice to have two for HA tests and study, but I really don't think anyone wants to spend hundreds of dollars on a bricked device that will end up in a landfill only because the recovery procedure documents aren't publicly available :(