cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1581
Views
16
Helpful
6
Replies

Upgrade WLC from 7.4.150 > 8.0.152.0 - Bricked 2602 APs - Flashing red LED

PJR_CDF
Level 1
Level 1

I recently attempted to upgrade our WLC environment (5508 and 15+ 2602 APs) and hit a few issues and was wondering if anyone else has encountered anything similar?

 

My starting point was 7.3.112.0 on our WLC with a planned upgrade path of:

 

7.3.112.0 > 7.4.150 > 8.0.152.0 > 8.3.143.0 > 8.5.131.0

 

In case of issues and to allow a swift rollback, I powered off all APs except 2 during each upgrade hop

 

I updated FUS to 1.9.0.0 first and then WLC to 7.4.145.0  which was fine.

 

I then updated the WLC to 8.0.152.0.

 

When the 2 test APs attempted to update, they bricked with a red flashing LED.

 

Console access to the devices showed the following output:

 

--------------------------------------------

 

IOS Bootloader - Starting system.

flash is writable

FLASH CHIP:  Numonyx Mirrorbit (0089)

Xmodem file system is available.

flashfs[0]: 13 files, 5 directories

flashfs[0]: 0 orphaned files, 0 orphaned directories

flashfs[0]: Total bytes: 31997952

flashfs[0]: Bytes used: 12638208

flashfs[0]: Bytes available: 19359744

flashfs[0]: flashfs fsck took 16 seconds.

Reading cookie from SEEPROM

Base Ethernet MAC address: 4c:4e:35:03:4b:fb

Ethernet speed is 1000 Mb - FULL Duplex

Loading "flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-k9w8-mx.152-2.JB6"...flash:/ap3g2-                                                                                                                                                             k9w8-mx.152-2.JB6/ap3g2-k9w8-mx.152-2.JB6: no such file or directory

 

Error loading "flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-k9w8-mx.152-2.JB6"

 

Interrupt within 5 seconds to abort boot process.

Loading "flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-mx.153-3.JA12"...####                                                                                                                                                             #####################

 

File "flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-mx.153-3.JA12" uncompressed and installed, entry point: 0x2003000

executing...

 

Secondary Bootloader - Starting system.

Tide MB - 32MB of flash

Xmodem file system is available.

flashfs[0]: 13 files, 5 directories

flashfs[0]: 0 orphaned files, 0 orphaned directories

flashfs[0]: Total bytes: 31997952

flashfs[0]: Bytes used: 12638208

flashfs[0]: Bytes available: 19359744

flashfs[0]: flashfs fsck took 9 seconds.

flashfs[1]: 0 files, 1 directories

flashfs[1]: 0 orphaned files, 0 orphaned directories

flashfs[1]: Total bytes: 12257280

flashfs[1]: Bytes used: 1024

flashfs[1]: Bytes available: 12256256

flashfs[1]: flashfs fsck took 0 seconds.

Base Ethernet MAC address: 4c:4e:35:03:4b:fb

 

2600/3600 AP, PID: 'AIR-CAP2602I-E-K9 '. Checking for BL upgrade...

BL: C2600 Boot Loader (AP3G2-BOOT-M) LoaderVersion 12.4(25e)JA1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 30-May-12 03:34 by binugopa

 

BL Build year: 12

De-compress 'flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-bl-2600' to 'ram:/ap3g2-bl'...

File not found: flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-bl-2600

Boot CMD: 'boot  flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-k9w8-xx.152-2.JB6;flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-xx.153-3.JA12'

Loading "flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-k9w8-xx.152-2.JB6"...flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-k9w8-xx.152-2.JB6: no such file or directory

 

Error loading "flash:/ap3g2-k9w8-mx.152-2.JB6/ap3g2-k9w8-xx.152-2.JB6"

 

Interrupt within 5 seconds to abort boot process.

Unable to locate IOS image with name **xx**.

Boot CMD: 'flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-xx.153-3.JA12'

Loading "flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-xx.153-3.JA12"...###################

Error loading "flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-xx.153-3.JA12"

 

Interrupt within 5 seconds to abort boot process.

Boot process failed...

 

The system is unable to boot automatically.  The BOOT

environment variable needs to be set to a bootable

image.

 

C2600 Boot Loader (AP3G2-BOOT-M) LoaderVersion 12.4(25e)JA1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 30-May-12 03:34 by binugopa

-------------------------------

 

This appears to be IOS corruption. I was eventually able to recover the AP’s by dismounting them, and factory resetting them via the mode button but, as my maintenance window had then expired, I rolled back the WLC and AP’s to 7.4.150.

 

Has anyone else encountered issues with 2602’s when going from 7.4.150.0 > 8.0.152.0?

 

I have looked into maybe pre downloading the AP image to the APs instead to see if this helps, but this isn’t supported for this upgrade hop (7.4.150 > 8.0.152.0) as per this document:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-3/config-guide/b_cg83/b_cg83_chapter_0101.html#ID1266

 

If you upgrade from a release that is prior to Release 7.5 directly to Release 7.6.X or a later release, the predownload process on Cisco AP2600 and AP3600 fails. After the Cisco WLC is upgraded to Release 7.6.X or a later release, the new image is loaded on Cisco AP2600 and AP3600. After the upgrade to a Release 7.6.X image, the predownload functionality works as expected. The predownload failure is only a one-time failure.

 

 

1 Accepted Solution

Accepted Solutions

Leo Laohoo
Hall of Fame
Hall of Fame

@PJR_CDF wrote:

Loading "flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-mx.153-3.JA12"...####  


The controller wasn't finish with the pre-download when the AP was rebooted. 

Console into the AP and post the complete output to the command "dir flash:".

View solution in original post

6 Replies 6

Leo Laohoo
Hall of Fame
Hall of Fame

@PJR_CDF wrote:

Loading "flash:/update/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-mx.153-3.JA12"...####  


The controller wasn't finish with the pre-download when the AP was rebooted. 

Console into the AP and post the complete output to the command "dir flash:".

Flash Dir below - this is post recovery of the AP

 

Directory of flash:/

2 -rwx 77777 Jul 22 2018 18:13:27 +00:00 event.log
3 -rwx 64 Mar 1 1993 00:01:05 +00:00 sensord_CSPRNG1
4 -rwx 7192 Jul 24 2018 10:12:31 +00:00 private-multiple-fs
5 -rwx 64 Mar 1 1993 00:01:03 +00:00 sensord_CSPRNG0
45 drwx 0 Mar 1 1993 00:00:25 +00:00 configs
46 -rwx 0 Jan 30 2018 09:58:40 +00:00 ce
7 drwx 960 Jul 22 2018 18:13:17 +00:00 ap3g2-k9w8-mx.152-2.JB6
51 -rwx 380 Jul 23 2018 07:30:18 +00:00 env_vars
13 drwx 192 Feb 4 2013 13:18:02 +00:00 ap3g2-rcvk9w8-mx

31739904 bytes total (13079552 bytes free)

Ok, this is extremely EASY to fix. So I propose we force the AP to boot the RCV image and then let the AP re-download the CAPWAP image from the controller.
If you're "game" for this, then console into the AP and do the following (enable mode):
1. debug cap concli
2. delete /f /r flash:ap3g2-k9w8-mx.152-2.JB6
3. Reboot the AP.

Thanks for your response - I resolved the issue by using the Mode button to reset the AP's (which presumably boots the AP into the RCV image?)

 

Now I know the cause of the issue is related to a reboot of the AP mid update rather than an upgrade path bug/incompatibility, I can schedule another window and ensure the APs are not rebooted during their upgrade.

 

Out of interest if an AP updates to 8.0.152 and for some unforeseen reason I need to rollback the WLC to 7.4.145.0, will the AP downgrade its software gracefully automatically in the same way they upgrade?

Normally yes, just don't reboot the WLC again while they are downgrading.


@PJR_CDF wrote:

Out of interest if an AP updates to 8.0.152 and for some unforeseen reason I need to rollback the WLC to 7.4.145.0, will the AP downgrade its software gracefully automatically in the same way they upgrade?


Correct.

 

Review Cisco Networking products for a $25 gift card