After WLC reboot only one wireless AP can join, several hours later all WAPs can join again

[Anna Pytka]

I have an issue with my primary WLC AIR-WLC4402-50-K9 running version 7.0.220.

After a reboot of the WLC only one AP can join, all others are rejected. After waiting a few hours (about 3) all APs can join the WLC.

What I have done is a software upgrade (from 6.0.199.4)  on both primary wlc (wlc with the issue) and backup. Following procedure:

1. Reboot primary wlc so that APs failover to backup (succeeded with no issue).

2. Upgarde software on primary wlc

3. To failback all APs from backup to primary I reboot all APs, so that they join again their primary wlc.

A reboot of the first AP was fine, It joined the primary wlc, get new software downloaded, rebootet, joined wlc successful again.

Then rebooting the next AP. But this one then does not join the primary wlc for several hours. But it is also not falling back to the backup wlc as the primary is up and reachable.

I tried a reboot with the next AP, but same behavior.

I was waiting then until both APs were joining the primary wlc again. This happened after around 3 hours.

After that I have rebootet all other APs on the backup wlc also. All was fine they immediately joined the primary without any issue.

There are no log messages during the time when the WLC does not allow APs to join..

I had similar behaviour last year with version 6 when I rebooted the device.

So I dont think this is software related. Also I dont think it's a configuration issue as 1) I have another WLC that uses the same backup controller and this one has the same configuration but works fine also after a reboot. 2) why doesn't it work after a reboot for a few hours, but then all is fine without any config change. After a few hours all is fine for months (until the next reboot).

Does it look like hardware fault?

Please advise.

Thanks in advance.

[Leo Laohoo]

I haven't seen this type of behaviour since the "good ole days" of the 4.X firmware.

By the way, did you also upgrade the ER-files?

[glen.grant]

Do you have AP fallback configured on both the primary and secondary wlc's ? 

[Anna Pytka]

yes, all is configured. As I have other WLC controllers configured exactly the same as this primary one (with the issue), I dont think this is a configuration issue. Any other ideas? thanks.

[tuhpatel]

On the Web GUI :

Monitor-> Statistics-> AP join

What is the failure rror message reason indicated

Also if you have access to the AP console

Run a debug capwap client errors

Let me know if you have any questions

[Anna Pytka]

I have a bit more information:

yesterday we upgraded 2 other WLCs and one of the 2 controllers have shown the same issue. After a reboot only one AP was joining. About 1.5 hours later additional APs were joining and all went back to normal.

The other one that was upgraded hasn't shown this behavior, more than 1 AP were able to join after the reboot.

The difference between the 2 controllers is only, that the one with issues has only APs running in HREAP mode on it. The other one has APs running in local mode.

The controller I first saw the issue (two weeks ago) has also only APs in HREAP mode connected.

So it looks to me now, that there is only an issue with HREAP APs, local mode ones are fine.

The following messages wers shown in the message log:

*spamReceiveTask: Mar 12 15:04:34.039: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1517 Discarding discovery request in LWAPP from AP 00:1a:30:8d:8c:a0 supporting CAPWAP

*mmListen: Mar 12 15:03:26.692: %MM-4-PMKCACHE_DEL_FAILED: mm_listen.c:6724 Failed to delete PMK cache entry for station 0c:60:76:59:9e:83 with request from controller x.x.x.x

*spamReceiveTask: Mar 12 15:03:23.699: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1517 Discarding discovery request in LWAPP from AP 00:1a:30:8d:8c:a0 supporting CAPWAP

Mar 12 15:29:38.670: [WARNING] ews_form.c 3580: ewBuildRepeatForm: repeates exceeds maximum

*apfReceiveTask: Mar 12 15:29:23.811: %RRM-3-RRM_LOGMSG: rrmChanUtils.c:290 RRM LOG: Airewave Director: Could not find valid channel lists for 802.11bg

*spamReceiveTask: Mar 12 15:29:23.730: %DTLS-5-ESTABLISHED_TO_PEER: openssl_dtls.c:573 DTLS connection established to x.x.x.x

*apfReceiveTask: Mar 12 15:29:13.892: %RRM-3-RRM_LOGMSG: rrmChanUtils.c:290 RRM LOG: Airewave Director: Could not find valid channel lists for 802.11bg

*spamReceiveTask: Mar 12 15:29:13.811: %DTLS-5-ESTABLISHED_TO_PEER: openssl_dtls.c:573 DTLS connection established to x.x.x.x

*spamReceiveTask: Mar 12 15:29:12.195: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1517 Discarding discovery request in LWAPP from AP 00:1a:30:8d:8c:a0 supporting CAPWAP

Thanks.