CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

[ABM Networking]

Does anyone know the work around for this? 

I can install any V7.x.x image and the AP works fine, install any 8.x.x image and the AP goes into the boot loop failing on the Radio FW Image download.

Tried updating the image via Rmon, is there a version I need to install to get this AP working with V8.2.151

[Thomas Obbekaer Thomsen]

(pppsssttt .. dont tell anyone. But aparently TAC has a script that will help you ID the affected APs)

(and still dont tell anyone - 8.5MR1 and the upcomming 8.3MR4 fixes this problem - aka it is not a hardware problem).

[Leo Laohoo]

LOL.  Thanks for the tip, Thomas.

[Thomas Obbekaer Thomsen]

It does not, according to the little birdie who told me, it will ID the APs for you.

There is also another way to ID affected APs, predownload new software.

If APs fail predownloading they are affected.

Im not 100% on the workaround (other then format flash), but I think someone mentioned (maybe it was a a Squirrel near by) that resetting the AP to factory defaults from the WLC might do the trick.

But the newly release 8.5MR1 should fix this flash problem (along with a couple of others) the same goes for upcoming 8.3MR4 (due out before Christmas --- Santa promised that).

[Leo Laohoo]

---

@Thomas Obbekaer Thomsen wrote:

but I think someone mentioned (maybe it was a a Squirrel near by) that resetting the AP to factory defaults from the WLC might do the trick. 

---

So forcing to perform a factory-reset by using the controller won't fix the issue because the AP won't accept any commands. 

---

@Thomas Obbekaer Thomsen wrote:

But the newly release 8.5MR1 should fix this flash problem (along with a couple of others) the same goes for upcoming 8.3MR4 (due out before Christmas --- Santa promised that).

---

I am currently running 8.5MR1 and I'm observing the situation. 

---

@Thomas Obbekaer Thomsen wrote:

It does not, according to the little birdie who told me, it will ID the APs for you.

There is also another way to ID affected APs, predownload new software.

If APs fail predownloading they are affected. 

---

Well, TAC needs to do more work.  What about the scenario when the CAPWAP IOS is corrupt and the AP goes "dumb" and won't load the next IOS from the directory.  

Another thing, this bug affects only the 2700/3700.  It is most likely the reason why this is happening is because Cisco is playing around "renaming" the IOS files from ap3g2 to c3700.  

What is the intention for this exercise? Oh, I don't know. Maybe to make sure the 2600/3600 doesn't get supported on 8.6 even though the 2600/3600 can support the features in 8.6. Maybe.

NOTE: In the Release Notes in 8.6, it is stated that the 2600/3600 won't be supported. However, I have managed to get a 3600 loaded with an 8.6.101.0 firmware.

[Thomas Obbekaer Thomsen]

Yes you are right all the "fixes" only helps pre problem encounter (or whatever we should call it). If an AP has already gone "bad" theres only physical hands on recovery :-/

Hmmm .. Im sure it should have been fixed in 8.5MR1 .. ow well . bugger :-/

PS: this issue should also affect 2600 / 3600  not just 1700/2700/3700 . something about these APs using flash in the same way , or having the same kind of flash or flash controller .. something along those lines.

[Leo Laohoo]

Odd. I've got some 2600/3600 in the network and I've been seeing this bug rear it's revolting head exclusively on the 3700.

[Thomas Obbekaer Thomsen]

Well there are a couple of bugs regarding flash that affects these APs this is one of them ... unfortunately there are more.

It seems to be about how many events that happens on the AP (DFS, CAPWAP, unstable lines resulting in CAPWAP flapping and so on). from it being switched on, to rebooting (for whatever reason).

[Freerk Terpstra]

Such events cause read/write activity with the flashcard (event.log, capwap-saved-config etc.), it is a wild guess but maybe after some time or x amount of actions the allocation of space on the flashcard goes wrong or some kind of index is lost?

Based on S1 bug CSCvf16302 it does indeed not seem to be a hardware issue. Has anyone seen these kind of issues in deployments which are running older versions of code (<8.0.131)?

[Leo Laohoo]

---

@Freerk Terpstra wrote:

Has anyone seen these kind of issues in deployments which are running older versions of code (<8.0.131)?

---

This issue only appears when loaded with 8.X (and later) and only predominantly affects 2700/3700.  I have never seen these (plural) issues appear on 3500, 2600/3600.

[Freerk Terpstra]

I found that based on the output of the commands below it can be established if a operational access-point is having communication issues with the internal flashcard. Combine this with your favorite programming language and you can perform proactive "access-point health checks".

The good news is that bug CSCvf16302 (and others related to flash problems) will be fixed in 8.2 MR7, based on the release notes related to the recently posted interim version.

AP#show file systems
File Systems:
       Size(b)       Free(b)      Type  Flags  Prefixes
*            -             -     flash     rw   flash:

AP#show logg
*Feb 18 13:22:42.603: %LWAPP-3-CLIENTERRORLOG: Save LWAPP Config: error saving config file

AP#fsck flash:
Fsck operation may take a while. Continue? [confirm]
%Error fscking flash: (Device or resource busy)

 Please rate useful posts... :-)

Update: Cisco published a document regarding the flash problems covering multiple bugs.

[powys]

Got a 2600 on my desk that's doing it. Wiped the flash and TFTP'd a recovery image onto it, all OK. Joined back to the LAN, let the controller reflash it to ap3g2-k9w8-mx.153-3.JD16 (controller is running 8.3.143.0) and the problem comes straight back again.

IOS Bootloader - Starting system.
flash is writable
FLASH CHIP:  Numonyx Mirrorbit (0089)
Xmodem file system is available.
flashfs[0]: 61 files, 7 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 31997952
flashfs[0]: Bytes used: 15283712
flashfs[0]: Bytes available: 16714240
flashfs[0]: flashfs fsck took 16 seconds.
Reading cookie from SEEPROM
Base Ethernet MAC address: 7c:69:f6:bc:6c:58
Ethernet speed is 1000 Mb - FULL Duplex
Loading "flash:/ap3g2-k9w8-mx.153-3.JD16/ap3g2-k9w8-mx.153-3.JD16"...#########################

File "flash:/ap3g2-k9w8-mx.153-3.JD16/ap3g2-k9w8-mx.153-3.JD16" uncompressed and installed, entry point: 0x2003000
executing...

Secondary Bootloader - Starting system.
Tide MB - 32MB of flash
Xmodem file system is available.
flashfs[0]: 61 files, 7 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 31997952
flashfs[0]: Bytes used: 15283712
flashfs[0]: Bytes available: 16714240
flashfs[0]: flashfs fsck took 8 seconds.
flashfs[1]: 0 files, 1 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 12257280
flashfs[1]: Bytes used: 1024
flashfs[1]: Bytes available: 12256256
flashfs[1]: flashfs fsck took 1 seconds.
Base Ethernet MAC address: 7c:69:f6:bc:6c:58

2600/3600 AP, PID: 'AIR-CAP2602I-E-K9 '. Checking for BL upgrade...
BL: C2600 Boot Loader (AP3G2-BOOT-M) LoaderVersion 12.4(25e)JAY, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Compiled Thu 03-Oct-13 03:35 by sdcunha

BL Build year: 13
Unable to locate IOS image with name **xx**.
Boot CMD: 'boot  flash:/ap3g2-k9w8-mx.153-3.JD16/ap3g2-k9w8-xx.153-3.JD16'
Loading "flash:/ap3g2-k9w8-mx.153-3.JD16/ap3g2-k9w8-xx.153-3.JD16"...##################################################
File "flash:/ap3g2-k9w8-mx.153-3.JD16/ap3g2-k9w8-xx.153-3.JD16" uncompressed and installed, entry point: 0x1003000
executing...

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, C2600 Software (AP3G2-K9W8-M), Version 15.3(3)JD16, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Tue 05-Jun-18 02:41 by prod_rel_team

Tide MB - 32MB of flash
Initializing flashfs...

flashfs[2]: 61 files, 7 directories
flashfs[2]: 0 orphaned files, 0 orphaned directories
flashfs[2]: Total bytes: 31739904
flashfs[2]: Bytes used: 15283712
flashfs[2]: Bytes available: 16456192
flashfs[2]: flashfs fsck took 9 seconds.
flashfs[2]: Initialization complete.
flashfs[4]: 0 files, 1 directories
flashfs[4]: 0 orphaned files, 0 orphaned directories
flashfs[4]: Total bytes: 11999232
flashfs[4]: Bytes used: 1024
flashfs[4]: Bytes available: 11998208
flashfs[4]: flashfs fsck took 0 seconds.
flashfs[4]: Initialization complete.
Copying radio files from flash: to ram:
Copy in progress...CCCCC
Copy in progress...CCC
Copy in progress...CCCC
Copy in progress...CCCC
Copy in progress...CC
Copy in progress...CC
Copy in progress...CCCC
Copy in progress...CC
Copy in progress...CCCCCCCC
Copy in progress...CCCC
Copy in progress...CC
Copy in progress...C
Uncompressing radio files...
...done Initializing flashfs.

Radio0  present 8764 8000 0 A8000000 A8010000 0
timeout waitin for fw ready

 nop failed

*Mar  1 00:00:14.823: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed (15)
*Mar  1 00:00:58.179: %SYS-5-RELOAD: Reload requested by Init. Reload Reason:
Radio FW image download failed ....

[Vinay Joshi]

I had the same issue, I rest the AP 

Step 1: Press the reset button, it's between POE IN and PWR port, just open the IP rated bolt.

Step 2: Then power up the AP, while holding the reset button 

Step 3: Hold the reset button for like 20 secs, until the led status is steady Red.

then

ap: reset

that's it, AP will reload 

[Leo Laohoo]

That won't work all the time. The process you've described is actually to factory-reset the AP. If the RCV image is found above the CAPWAP image, then the RCV image will boot and all will be OK. Otherwise the CAPWAP image will load and if the corrupt file is present, the AP will go into a boot-crash-loop.
The easiest way is to delete the CAPWAP folder in order to force the AP to boot the RCV IOS.
The fix to this bug is found in 8.5.140.0.

[pielunswe]

Hi!

Any one have some new info regarding this issue? Fixed in 8.5.151?

BR, Pierre

[karol.krzyzyk]

I have exactly same problem. one from my six 3702i is booting in loop.

I found new version of software on Cisco support - Release 8.5.161.0 ED

but I don't have access to download, so I can't test

Montserrat Board
40MB format
Tide XL MB - 40MB of flash
Initializing flashfs...

flashfs[2]: 75 files, 8 directories
flashfs[2]: 0 orphaned files, 0 orphaned directories
flashfs[2]: Total bytes: 40900608
flashfs[2]: Bytes used: 21854208
flashfs[2]: Bytes available: 19046400
flashfs[2]: flashfs fsck took 14 seconds.
flashfs[2]: Initialization complete.
flashfs[4]: 0 files, 1 directories
flashfs[4]: 0 orphaned files, 0 orphaned directories
flashfs[4]: Total bytes: 11999232
flashfs[4]: Bytes used: 1024
flashfs[4]: Bytes available: 11998208
flashfs[4]: flashfs fsck took 0 seconds.
flashfs[4]: Initialization complete.
Copying radio files from flash: to ram:
Copy in progress...CCCCC
Copy in progress...CCC
Copy in progress...CCCC
Copy in progress...CCCC
Copy in progress...CC
Copy in progress...CC
Copy in progress...CCCC
Copy in progress...CC
Copy in progress...CCCCCCCC
Copy in progress...CCCC
Copy in progress...CC
Copy in progress...C
Uncompressing radio files...
...done Initializing flashfs.

Radio0 present 8764 8000 0 A8000000 A8010000 0
*Mar 1 00:00:18.847: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed (15)
*Mar 1 00:00:21.539: %DOT11-2-RADIO_FAILED: Interface Dot11Radio0, failed due to the reson code 44
-Traceback= 11B72C8z 180DD04z 180E5B8z 1895E20z 18A1CA4z 181D780z 12CE114z 12CE2F4z 12CE460z 10C41C4z 10C443Cz 13483B4z 132F13Cz
*Mar 1 00:00:21.623: %SYS-5-RELOAD: Reload requested by Init. Reload Reason:
Radio FW image download failed ....
IOS Bootloader - Starting system.
flash is writable