AP'S flapping after OS upgrade

[Philip Brown]

Hi, after upgrading OS 8.3.102.0 to 8.3.112.0 on a SSO pair of 7510's, ca 60 of the AP's landed in some sort of undifinable state and were not managable until pysically power cycled. From the controller i could see that the AP models involved were 2702 I & E and 2602 I.

Has anybody experienced similar ?

Are there any known Bug ID's or statements concerning this behaviour from Cisco ?

Thanks,

Phil

[Leo Laohoo]

Need more details.

Did the APs appear on the WLC with default AP names?

[Philip Brown]

Hi, no the AP names were as previously configured for most of the AP's concerned.
A couple of the AP's did appear on the WLC with their default names though.
After getting most of the AP's disconnected from the appropriate switch and reconnected, those AP's downloaded the new OS and came online successfully.
At present there are 3 AP's with configured hostnames and two with default names flapping or boot looping.
Phil

[Leo Laohoo]

---

@Philip Brown wrote:
two with default names flapping or boot looping.
Phil

---

I'm familiar with this behavour.  Boot looping because the firmware inside the AP is corrupted.  This is a known issue with 8.1.X and 8.2.X.  The only way to stop this is to console into the AP and delete the corrupt folder of the CAPWAP image the AP is trying to boot.  Alternatively, forcing the AP to boot the recovery or "RCV" image will help.  

There is no fix to this.  Regularly, I would get two or three of these behaviours weekly and I have no choice but to pull down the AP and fix them manually.  

I will be trying out 8.5.103.0 in the next two weeks to determine if the bug has been fixed or not.

[Philip Brown]

Hi, thanks for your comments. If this behavior is known then I am a bit concerned that there is no bug ID for it !
I'm being asked to supply such to my customer as the will not accept my statement that this is a known problem with no official statement from Cisco.
Has the problem not been reported via a TAC ? If it has, the TAC # would be a great help.
Phil

[Leo Laohoo]

CSCuv08570

[Philip Brown]

Hi Leo, the bug ID you supplied would only appear to address part of the problem/s being encountered.

I'm now being asked to explain why 20 od AP's are showing the following on the WLC, Backup SW version 0.0.0.0

Most of the AP's concerned have been online for some time and should be showing some backup software version or another.

Could it be that these AP's were ones that were in the boot loop status and when they were power cycled, their Flash memory got purged ?

[Leo Laohoo · Accepted solution]

You're asking questions only TAC can answer.  The only thing I've noticed, when running firmware versions 8.0, 8.1 and 8.2, are the following behaviours: 

1.  APs would boot into default config; 
2. APs will appear on the controller list with an up time of of 00:00:00; 
3. APs does not show any reason or cause of this behaviour in the event.logs

The second one is the boot-crash-loop.  The AP would go into a boot-crash-loop.  The only way to fix this problem is to delete the CAPWAP image and/or force the AP to boot the "RCV" file.    Like the other behaviour, there is no evidence as to why the AP would go into this state.

I am currently running 8.5.103.0 (since 27 September 2017) and I am not seeing this behaviour.

8.5 has a different "version" of this bug but the behaviour is still the same.  The new version of the bug makes it more difficult to detect because the AP will join/leave very often the only way to find out that it's happening is to look at the Statistics > AP Join and notice the large amount of APs failing to join.

[Leo Laohoo]

Hey Philip, 

Just want to give you a "heads up".  

Another "version" of this Bug is currently available in 8.5.  The behaviours are: 

1.  The AP constantly joins/leaves the controller; 

2.  The AP will retain the AP name, Primary/Secondary/Tertiary WLC details preset, AP Groups correct;

3.  AP, however, will show up in the list of APs with an invalid IP address of "0.0.0.0"; 

4.  If you do a "sh cdp neighbor detail" it will show that the AP has loaded the RCV image but unable to load & download the CAPWAP IOS.  

My workaround are: 

1.  Force the AP to load the RCV image; 

2.  MANDATORY:  Change the boot variable string.  Failure to change the boot variable string will send the AP to ROMmon because the AP keeps loading the CAPWAP image; 

3.  Re-format the AP and upload the RCV image. 

[Philip Brown]

Hi Leo, many thanks for the heads up.

In the meantime, I've been investigating some of these AP's. The problem appears to be that they do not have Flash: memory allocated.

Two identical AP models, order, installed, powered on at almost the same time. One shows 40 odd Meg Flash: the other 0, zero, nothing. These are 2702i's but we also have some 2602i's showing the same symptoms.

I have asked that these AP's be powercycled but, in the words of Diana Ross "I'm still waiting".

Phil

[Philip Brown]

Hi Leo or everybody for that matter,   is the boot sequence for the AP's documented anywhere ?

I'd be really interested to know just exactly what the boot loader runs. Does fsck.repair=yes get run when the AP's are powered up ? How much Flash: should get allocated ?

 thanks in advance,

Phil