topic Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error in Cisco Bug Discussions

CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

ABM Networking — Thu, 21 Mar 2019 04:30:19 GMT

Does anyone know the work around for this?

I can install any V7.x.x image and the AP works fine, install any 8.x.x image and the AP goes into the boot loop failing on the Radio FW Image download.

Tried updating the image via Rmon, is there a version I need to install to get this AP working with V8.2.151

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Jades Moreira — Tue, 17 Oct 2017 18:43:12 GMT

I have the same problem with the AP 2702. Now I´ll try to recover the image via rommon. If I solve it I share what was done.
========================================================
*Mar 1 00:00:28.787: %DOT11-2-RESET_RADIO: Restarting Radio interface Dot11Radio1 due to FW: hostmem badmagic=00000000, vec=33 pc=D0109470, ev=1, @01B74AA9,s@01B72A16
*Mar 1 00:00:28.787: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to
*Mar 1 00:00:29.683: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to IOS reload
*Mar 1 00:00:30.483: %SYS-5-RELOAD: Reload requested by Init. Reload Reason:
Radio FW image download failed ....
IOS Bootloader - Starting system.
========================================================

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Wed, 13 Dec 2017 19:19:48 GMT

What controller firmware version are you using? I am very familiar with this bug.

There are two ways of fixing this:

1. Force the AP to boot the RCV image by deleting the CAPWAP image.

2. Force the AP to boot into ROMmon, format the flash, and upload the RCV image.

In both methods, console access is the only way.

These are the methods I've developed myself because I was encountering this bug when I would "loose" an AP every three days. ~~I've upgraded my controller to 8.5.103.0 and this issue has stopped.~~

Issue is present in 8.5.103.0/8.5.105.0.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Jades Moreira — Wed, 18 Oct 2017 12:00:28 GMT

Hi Leo Laohoo,

Thanks for help me.

See, the controller version is 8.3.122.0 but will be update to 8.3.130.0. Now I´ll try to recover AP via console/rommon with a new image.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Sat, 21 Oct 2017 12:15:01 GMT

Cisco will be publishing new firmwares to fix KRACK WPA2 vulnerability. I recommend you go with this fix.

Fix firmware are scheduled for release on 23 October 2017 (see image below).

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Thomas Obbekaer Thomsen — Thu, 26 Oct 2017 07:45:02 GMT

Im looking forward to hearing if the newest 8.3 solved the problem. 🙂

We have multible 2700 that does this as well.

A hardreset of the AP (holding down the reset button) is the easy fix for us at this point.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

chris.brunt — Wed, 13 Dec 2017 17:29:39 GMT

8.3.133.0 still getting a problem with 1x 3702 however I have fixed about 8x 1702 by formating flash and using recovery image.

Had about 40 APs fail in a boot loop with error like below

%CAPWAP-3-ERRORLOG: Validate Msg: msg type 4 does not supported payload 215
%LWAPP-3-CLIENTERRORLOG: Validate Msg: msg type 4 does not supported payload 215
%LWAPP-3-CLIENTERRORLOG: Validate Msg: error in unknown payload (received length = 9, payload type = 215)
%CAPWAP-3-ERRORLOG: Failed to validate vendor specific message element type 215 len 9.
%CAPWAP-3-ERRORLOG: Config status: Failed to validate vendor specific message element.
%CAPWAP-3-ERRORLOG: Failed to decode join response.

Able to fix them remotely by forcing them to an 8.2 version on another controller by using a vlan that has an ACL blocking the 8.3 controller IP. Then once they have downloaded 8.2, change vlan back to normal put them back to 8.3 and they work okay.

I have lost confidence with doing software updates as imagine having to console when you have 1000 APs in a hospital!

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Wed, 13 Dec 2017 19:18:36 GMT

Well, this issue is also present in 8.5.103.0/8.5.105.0. Will observe if this is fixed in 8.5.110.0.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Thomas Obbekaer Thomsen — Mon, 18 Dec 2017 11:59:35 GMT

Did anyone notice that this bug us now status : terminated, and there is no fix.
So this is a hardware problem ?

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Mon, 18 Dec 2017 20:08:15 GMT

The bug, and other iteration, is present in 8.0 to 8.5.
The biggest question is how to find the missing APs and fix them.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Thomas Obbekaer Thomsen — Mon, 18 Dec 2017 21:03:41 GMT

(pppsssttt .. dont tell anyone. But aparently TAC has a script that will help you ID the affected APs)

(and still dont tell anyone - 8.5MR1 and the upcomming 8.3MR4 fixes this problem - aka it is not a hardware problem).

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Mon, 18 Dec 2017 21:07:28 GMT

LOL. Thanks for the tip, Thomas.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Thomas Obbekaer Thomsen — Mon, 18 Dec 2017 21:11:20 GMT

It does not, according to the little birdie who told me, it will ID the APs for you.

There is also another way to ID affected APs, predownload new software.

If APs fail predownloading they are affected.

Im not 100% on the workaround (other then format flash), but I think someone mentioned (maybe it was a a Squirrel near by) that resetting the AP to factory defaults from the WLC might do the trick.

But the newly release 8.5MR1 should fix this flash problem (along with a couple of others) the same goes for upcoming 8.3MR4 (due out before Christmas --- Santa promised that).

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Mon, 18 Dec 2017 21:41:34 GMT

@Thomas Obbekaer Thomsen wrote:

but I think someone mentioned (maybe it was a a Squirrel near by) that resetting the AP to factory defaults from the WLC might do the trick.

So forcing to perform a factory-reset by using the controller won't fix the issue because the AP won't accept any commands.

@Thomas Obbekaer Thomsen wrote:

But the newly release 8.5MR1 should fix this flash problem (along with a couple of others) the same goes for upcoming 8.3MR4 (due out before Christmas --- Santa promised that).

I am currently running 8.5MR1 and I'm observing the situation.

@Thomas Obbekaer Thomsen wrote:

It does not, according to the little birdie who told me, it will ID the APs for you.

There is also another way to ID affected APs, predownload new software.

If APs fail predownloading they are affected.

Well, TAC needs to do more work. What about the scenario when the CAPWAP IOS is corrupt and the AP goes "dumb" and won't load the next IOS from the directory.

Another thing, this bug affects only the 2700/3700. It is most likely the reason why this is happening is because Cisco is playing around "renaming" the IOS files from ap3g2 to c3700.

What is the intention for this exercise? Oh, I don't know. Maybe to make sure the 2600/3600 doesn't get supported on 8.6 even though the 2600/3600 can support the features in 8.6. Maybe.

NOTE: In the Release Notes in 8.6, it is stated that the 2600/3600 won't be supported. However, I have managed to get a 3600 loaded with an 8.6.101.0 firmware.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Thomas Obbekaer Thomsen — Mon, 18 Dec 2017 21:46:02 GMT

Yes you are right all the "fixes" only helps pre problem encounter (or whatever we should call it). If an AP has already gone "bad" theres only physical hands on recovery 😕

Hmmm .. Im sure it should have been fixed in 8.5MR1 .. ow well . bugger 😕

PS: this issue should also affect 2600 / 3600 not just 1700/2700/3700 . something about these APs using flash in the same way , or having the same kind of flash or flash controller .. something along those lines.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Mon, 18 Dec 2017 22:30:18 GMT

Odd. I've got some 2600/3600 in the network and I've been seeing this bug rear it's revolting head exclusively on the 3700.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Thomas Obbekaer Thomsen — Mon, 18 Dec 2017 22:33:36 GMT

Well there are a couple of bugs regarding flash that affects these APs this is one of them ... unfortunately there are more.

It seems to be about how many events that happens on the AP (DFS, CAPWAP, unstable lines resulting in CAPWAP flapping and so on). from it being switched on, to rebooting (for whatever reason).

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Freerk Terpstra — Mon, 01 Jan 2018 13:07:42 GMT

Such events cause read/write activity with the flashcard (event.log, capwap-saved-config etc.), it is a wild guess but maybe after some time or x amount of actions the allocation of space on the flashcard goes wrong or some kind of index is lost?

Based on S1 bug CSCvf16302 it does indeed not seem to be a hardware issue. Has anyone seen these kind of issues in deployments which are running older versions of code (<8.0.131)?

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Leo Laohoo — Mon, 01 Jan 2018 20:19:55 GMT

@Freerk Terpstra wrote:

Has anyone seen these kind of issues in deployments which are running older versions of code (<8.0.131)?

This issue only appears when loaded with 8.X (and later) and only predominantly affects 2700/3700. I have never seen these (plural) issues appear on 3500, 2600/3600.

Re: CSCvb25644 - AP3700 Randomly entering into continous boot loop due to Radio Fw image download error

Freerk Terpstra — Tue, 05 Jun 2018 06:42:27 GMT

I found that based on the output of the commands below it can be established if a operational access-point is having communication issues with the internal flashcard. Combine this with your favorite programming language and you can perform proactive "access-point health checks".

The good news is that bug CSCvf16302 (and others related to flash problems) will be fixed in 8.2 MR7, based on the release notes related to the recently posted interim version.

AP#show file systems
File Systems:
       Size(b)       Free(b)      Type  Flags  Prefixes
*            -             -     flash     rw   flash:

AP#show logg
*Feb 18 13:22:42.603: %LWAPP-3-CLIENTERRORLOG: Save LWAPP Config: error saving config file

AP#fsck flash:
Fsck operation may take a while. Continue? [confirm]
%Error fscking flash: (Device or resource busy)

Please rate useful posts... 🙂

Update: Cisco published a document regarding the flash problems covering multiple bugs.