AIR-CT5508 wont upgrade to 7.6

kyle woodhouse · ‎03-12-2014

Hi,

I wanted to see if anyone has had issues with upgrading to 7.6 images on their AIR-CT2504 or AIR-CT5508. I have one unit in the lab that refuses to update and I had a customer call me today with the exact same issue. Maybe someone from TAC has info if other users are having the same problems and can identify if this is an issue with 7.6 and to steer clear for the time being?

Logs:

(Cisco Controller) >
(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.110.0
Bootloader Version............................... 1.0.16
Field Recovery Image Version..................... 7.0.112.21
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

<output omitted>

(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >show boot
Primary Boot Image............................... 7.4.110.0 (default) (active)
Backup Boot Image................................ 7.3.101.0

(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >!loading AIR-CT5500-K9-7-6-110-0.aes in the GUI now

(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >show boot
Primary Boot Image............................... 7.6.110.0 (default)
Backup Boot Image................................ 7.4.110.0 (active)

(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >!reloading from GUI now
(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >
Updating license storage ... Done.
Restarting system.

WLCNG Boot Loader Version 1.0.16 (Built on Feb 28 2011 at 13:14:54 by cisco)
Board Revision 1.3 (SN: FCW1709L09H, Type: AIR-CT5508-K9) (G)

Verifying boot loader integrity... OK.

OCTEON CN5645-NSP pass 2.1, Core clock: 600 MHz, DDR clock: 330 MHz (660 Mhz data rate)
FPGA Revision 1.7
Env FW Revision 1.8
USB Console Revision 2.2
CPU Cores: 10
DRAM: 1024 MB
Flash: 32 MB
Clearing DRAM........ done
Network: octeth0', octeth1
' - Active interface
E - Environment MAC address override
CF Bus 0 (IDE): OK
IDE device 0:
- Model: STEC M2T CF 1.0.0 Firm: K1367MIX Ser#: STIM2MB513071013409
- Type: Hard Disk
- Capacity: 977.4 MB = 0.9 GB (2001888 x 512)

Press <ESC> now to access the Boot Menu...

Loading primary image (7.6.110.0)
100%

36110929 bytes read
Launching... <-- Right here it gives no error, just wont load 7.6 and moves to secondary image

Loading backup image (7.4.110.0)
100%

34513219 bytes read
Launching...
init started: BusyBox v1.6.0 (2010-05-13 17:50:10 EDT) multi-call binary
starting pid 821, tty '': '/etc/init.d/rcS'
/etc/init.d/rcS: line 4: /mnt/bootsys/.core: not found
route: SIOC[ADD|DEL]RT: Network is down

mount: mounting /dev/cfa1 on /mnt/bootsys failed
Set PLX switch MPS settings .............!!!!!!!
Detecting Hardware ...
set smp_affinity for irq 48
003f
DP from CGE5.0 ...
starting pid 1071, tty '/dev/ttyS0': '/usr/bin/gettyOrMwar'
Setting up ZVM
Exporting LD_LIBRARY_PATH

Cryptographic library self-test....passed!
XML config selected
Validating XML configuration
Read HA Config before validation
octeon_device_init: found 1 DPs
readCPUConfigData: cardid 0x6070001
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 7.4.110.0
Firmware Version FPGA 1.7, Env 1.8, USB console 2.2
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Services: ok

License daemon start initialization.....

License daemon running.....
Starting Statistics Service: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services: ok
Starting System Services: ok
Starting FIPS Features: ok : Not enabled
Starting Fastpath Hardware Acceleration: ok
Starting Fastpath Console redirect : ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-1.8.0, build 269. Flags-[DUTY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(10)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2040
Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok
Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Received instruction to get link status
Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok
Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok
Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok
Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok
Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok
Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Management Frame Protection: ok
Starting Certificate Database: ok
Starting VPN Services: ok
Starting Licensing Services: LICENSE_PRINT:cnfgrCslLicense:ERROR:license request Failed for units: 500 rc:User failed to accept EULA
ok
Starting Redundancy: ok
Starting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting AVC Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Power Supply and Fan Status Monitoring Service: ok
Starting Mesh Services: ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server: enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
   Web Server:    CLI: ok
   Secure Web: ok
   License Agent: ok

(Cisco Controller)

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User: admin
Password:*****
User:admin
Password:********
(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.110.0
Bootloader Version............................... 1.0.16
Field Recovery Image Version..................... 7.0.112.21
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2

<output omitted>

! Now If i try the FUS file AIR-CT5500-K9-1-9-0-0-FUS.aes it completes successfully and i reload from the GUI

(Cisco Controller) >
(Cisco Controller) >Restarting system.

WLCNG Boot Loader Version 1.0.16 (Built on Feb 28 2011 at 13:14:54 by cisco)
Board Revision 1.3 (SN: FCW1607L035, Type: AIR-CT5508-K9) (G)

Verifying boot loader integrity... OK.

OCTEON CN5645-NSP pass 2.1, Core clock: 600 MHz, DDR clock: 330 MHz (660 Mhz data rate)
FPGA Revision 1.7
Env FW Revision 1.8
USB Console Revision 2.2
CPU Cores: 10
DRAM: 1024 MB
Flash: 32 MB
Clearing DRAM........ done
Network: octeth0', octeth1
' - Active interface
E - Environment MAC address override
CF Bus 0 (IDE): OK
IDE device 0:
- Model: STEC M2T CF 1.0.0 Firm: K1367MIX Ser#: STIM2MB513071013409
- Type: Hard Disk
- Capacity: 977.4 MB = 0.9 GB (2001888 x 512)

Press <ESC> now to access the Boot Menu...

Loading primary image (7.6.101.2)
100%

6394630 bytes read
Launching...

Loading backup image (Image not found)

** Unable to read "linux.bak.img" from ide 0:2 **
** Launch failure **

============================================================
Boot Loader Menu
============================================================

1. Run primary image (7.6.101.2) - Active
2. Run backup image (Image not found)
3. Change active boot image
4. Clear configuration
5. Format FLASH Drive
6. Manually update images

------------------------------------------------------------

!Now i am stuck in the Boot Loader menu and the unit wont even boot up. I used the RTOS files and recovered the unit several times trying to upgrade multiple ways but no matter what i try the unit will not boot with 7.6. It works perfectly fine with 7.4.

Anyone have ideas?

Known issue?

Other links with what seems like the same issue:

https://supportforums.cisco.com/pt/discussion/12130001/air-ct2504-upgrade-failure

https://supportforums.cisco.com/discussion/12060686/wlc-5508-upgrade-software-76

http://omgili.com/thread/.wHSUbtEfZS6sY9oec8tRj1DNafRV3e2EmgIsn2U7n.0aLwe7IPHyQaQO5o5MalitJoQnElOrK8-/

Scott Fella · ‎03-12-2014

If you load the FUS 1.9.0.0 more than once, you will brick your WLC. I had no issue uploading any 2504 and or 5508 to v7.6.110.0.

"!Now i am stuck in the Boot Loader menu and the unit wont even boot up. I used the RTOS files and recovered the unit several times trying to upgrade multiple ways but no matter what i try the unit will not boot with 7.6. It works perfectly fine with 7.4. "

The RTOS comes from TAC and that is not the same as the image file you download from CCO. You will need to open a TAC case to see if they can fix it or it will need to be RMA'd.

-Scott
*** Please rate helpful posts ***

kyle woodhouse · ‎03-13-2014

Thanks Scott, I am already working with TAC but he is escalating it to another engineer. Will update if they figure out whats causing the problem.

I got the RTOS file from TAC so im aware its not the same as a regular image. Once i use the RTOS file the unit works fine again all the way up to 7.4 images. No issues. Just that anytime i use the FUS 1.9 or load any 7.6 image the unit does not boot.

Do you know why trying the FUS 1.9 more than once would brick the WLC?

robinwenham · ‎04-11-2014

Same here:

Loading primary image (7.6.110.0)
100%

36110929 bytes read
Launching...

Loading backup image (7.0.116.0)

kyle woodhouse · ‎04-11-2014

Hi Robin,

Thanks for the update. I dont think the FUS is faulty. I've seen this a total of 5 times now on 5 different AIR-CT5508 units. They just wont update to 7.6 software no matter if you use the FUS or not. They just skip loading it like your example.

I believe its an issue with hardware or the 7.6 software but there hasnt been enough reported cases to TAC yet for them to veirfy a fix.

I did a TAC case and even asked that they escalate it but they said the only option was to replace it. They were very helpful but did not have an answer why it works fine with 7.4 but not 7.6. Let me know if you get anything different. :)

robinwenham · ‎04-11-2014

Just to add to this, we hit the same problem with a 5508. Cisco TAC sent us the RTOS image which re-installed 7.0.116.0 firmware, however every time I run the 1.9.0.0 FUS I see the same problem: it's fully repeatable. I'm fully aware of the instruction not to run the FUS more than once and my first thought was that I had done that, however this unit has been in a data centre since before FUS 1.9.0.0 came out so I know that it hasn't had that installed. However we may well have installed FUS 1.7.0.0 - is it possible that having installed 1.7 then the 1.9 version screws it? A question for my TAC contact...

I've run the RTOS recovery now and I'm installing the 7.6.110.0 firmware, without re-running the FUS.

The currently installed Feild Recovery Version is 7.0.112.21 which looks like the pre-FUS 1.9.0.0 version...maybe the FUS is faulty.