Re: Log spamming - %LWAPP-3-INVALID_SLOT All the AIDs are in use

Davide Fiumi · ‎11-07-2017

Good morning,

on a active-active 2x5508 WLC on 8.0.152.0 we are receiving a constant spam of these messages

*apfMsConnTask_2: Nov 08 09:47:03.898: %LWAPP-3-INVALID_SLOT: spam_api.c:1469 The system detects an invalid slot identifier (1) - All the AIDs are in use. Could not allocate association identifier; AP xx:xx:xx:xx:xx:xx
*apfMsConnTask_2: Nov 08 09:46:58.793: %LWAPP-3-INVALID_SLOT: spam_api.c:1469 The system detects an invalid slot identifier (1) - All the AIDs are in use. Could not allocate association identifier; AP xx:xx:xx:xx:xx:xx
*apfMsConnTask_5: Nov 08 09:46:52.952: %LWAPP-3-INVALID_SLOT: spam_api.c:1469 The system detects an invalid slot identifier (1) - All the AIDs are in use. Could not allocate association identifier; AP xx:xx:xx:xx:xx:xx

The MAC Identifier is related not to the AP MAC, but to the AP Base Radio MAC. Aside from flooding the log, some clients report connectivity issues (unverified). In the log nothing of the kind seems to appear, and debugging connected clients doesn't show any issues with auth and deauth, or reauth - which lead me to believe that only these messages might have something to do with it.

For good measure, I added all of the reported AIDs missing MACs to the MAC-Filter list - nothing changed though.

The WLAN used by clients is a WPA2-AES with Dot1X-CCKM on PEAP and AAA Override (for automatic VLAN assignment). Same exact WLAN is running in a parallel environment without any complaints. What should be noted, this is a mainly Aironet 1532 network (90 APs).

I have enabled on the WLAN Advanced page:

- Allow AAA Override

- Coverage Hole Detection

- Aironet IE

- Client Exclusion

- Client Band Select

- Passive Client

- BSS Max Idle Service

and disabled mDNS Snooping.

Suggestions/ideas? If it's just a graphical bug, would love to get rid of it - could finally syslog the WLC.

Flavio Miranda · ‎11-08-2017

Hi @Davide Fiumi

I'll try to dig on it but at first sight I wouldn't worry about it. The log says LWAPP which is not used anymore in 5500 WLC.

For now I'd recommend to disable - Aironet IE. Some clients does not handle it properly.

-If I helped you somehow, please, rate it as useful.-

Davide Fiumi · ‎11-08-2017

Thank you @Flavio Miranda, I have disabled Aironet IE. Regarding LWAPP not being used any longer in 5500 WLCs that is good, still the message is flooding the system and I'd like to get rid of it if possible.

I've looked for solutions everywhere I could but couldn't come up with one.

Let me know if you have any tips.

Cheers.

Flavio Miranda · ‎11-08-2017

By any chance do you have some old model AP alive on the Network? This could explain the logs.

-If I helped you somehow, please, rate it as useful.-

Vengatesa Prasath · ‎11-08-2017

Hi David,

Not sure, if you are hitting this bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf06182/?reffering_site=dumpcr

above bug is duplicate of this bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve81314/?reffering_site=dumpcr

Workaround: Reload the AP

____________________________

Symptom:
After some time of operation, all association attempts to a given AP radio will be rejected with a status code 17.
The msglog will show an error similar to the following:

*apfMsConnTask_3: Jun 29 09:02:24.301: %LWAPP-3-INVALID_SLOT: spam_api.c:1636 The system detects an invalid slot identifier (1) - All the AIDs are in use. Could not allocate association identifier; AP 00:11:22:33:44:50

"debug client" will show output similar to the following:

apfMsConnTask_3: Jun 29 10:48:43.339: 00:22:44:66:88:00 Sending assoc-resp with status 17 station:00:22:44:66:88:00 AP:00:11:22:33:44:50-01 on apVapId 7
*apfMsConnTask_3: Jun 29 10:48:43.339: 00:22:44:66:88:00 Sending Assoc Response to station on BSSID 00:11:22:33:44:59 (status maximum station reached) ApVapId 7 Slot 1

Conditions:
AP has experienced numerous associations, since last reload.

Running AireOS with the CSCvd27365 fix.

APs in local mode.

L3 roaming may be in play.

Workaround:
reload AP

Further Problem Description:
This bug will address AID problems with local mode APs. For fixes to AID problems with FlexConnect local switching, track CSCve81269.

Special note: with early builds of AireOS containing this fix CSCve81314, you will need to enter the following command each time the WLC boots, for the AID fix to be in effect:

test client aid-audit enable

In later AireOS builds, which have the fix CSCvg24954 integrated, the AID fix will be enabled by default.

********** Please rate Useful post*********************

Davide Fiumi · ‎11-08-2017

@Vengatesa Prasath it seems you are correct, but version 8.0.152.0 shouldn't be affected by this issues (https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn80mr5.html).

Anyway, I restarted all of the APs and the issue persists.

@Flavio Miranda this network is composed only of Aironet 2702I and 1532I/E.

Do you think this should be reported as a live bug? Also, the description of the bug states

"Clients fails to connect to AID with message as All AID are in use when the AP is in Local mode"thus it really could be affecting clients from being able to connect, what do you think?

Davide Fiumi · ‎11-16-2017

I would like to inform you all that we are in the process of testing WLC 5508 v. 8.2.166.0.

Users helped me notice that the issues occur only outside, where the 1532 antennas are installed. These problems are a lot mitigated by the 2702s. I also found in the log of the 1532s this error

CSCva82261 - Cisco 1532 AP uplink drops when sending heavy upstream traffic

As well as the bug that @Vengatesa Prasath reported before.

I have migrated 8 antennas to the new firmware, and the AIDs messages have completely disappeared and users stopped complaining about dropping connection. I noticed that Apple iPhones still disconnect, but that could also be due to power saving features inside the device itself.

We are still testing the fix and I will update the post in the future and mark it solved in case it is fixed for future reference. I also contacted CISCO TAC to report that 8.0.152.0 is affected by

CSCvf06182 - Clients fails to connect to AID with message as All AID are in use when the AP is in Local mode

Thank you very much for your help.

yprasannas · ‎02-26-2018

Ours is 8.0.152 and hitting CSCve81314.

WLC logs are flooded with this message from tons of AP's. AP reload seems to fix the issue...not sure how long will it last.

TAC gave us dev build 8.0.152.2. We will wait for public release.

yprasannas · ‎03-27-2018

Reload does not fix the issue. Apart from log messages we are seeing disconnects to clients so this is a critical issue for us.

We currently have engineering build 8.0.152.2 from TAC.

Confusing thing is 8.0 is EOL starting from April 2018 so not sure if this fix will ever be released to general 8.0x release.

aweittmann · ‎05-03-2018

Same issue !

In version 8.0.152.0 - numerous clients are complaining...need a fix for 8.0.x, please

liors · ‎05-31-2018

Hey,

do you solved the problem? i having the same issue at the same version.

aweittmann · ‎05-31-2018

Hi,

After one week without errors, I think I could say yes, it is solved.

This issue is fixed in bugfix version 8.0.152.7 (provided by the TAC)

After upgrade in that specific version, you need to add specific extra command on WLC :

config ap aid-audit enable

PS: This bugfix version looks pretty stable.

matthew.goli1 · ‎06-06-2018

I'm experiencing this issue as well. We have both 5508 and 2504 WLC's in our environment.

TAC is telling me the same new WLC code will work on both the 5508 and the 2504. the file they gave me was named "AS_5500_8_0_152_8.aes"

does anyone have experience that shows the same image file will work on both the 5508 and the 2504 wireless controllers?

patoberli · ‎06-07-2018

I'm pretty sure it's the same file for the 5508 and 2504.

Davide Fiumi · ‎05-31-2018

8.0.152 sorted it for us as well.