Mobility Express randomly dropping clients

[emanuel32804]

Hello,

I'm facing an issue with our APs dropping clients for some reason. I've examined the logs and I can see it going in a cycle of removing and adding users. We are experiencing this on a range of devices including Android phones and Windows laptops. Currently using ME version 8.10.196.0. The issue only seems to occur with radius users, the other WLAN that uses regular WPA2 doesn't seem to face the same issue. We are currently using 3802 APs in our network.

Here's the log I keep seeing:

2025-10-27 18:24:32	Local0.Info	192.168.100.2	Cisco-7488.bbb7.fe80: *apfReceiveTask: Oct 27 18:24:31.208: %LOG-6-Q_IND: apf_ms_radius_override.c:213 Radius overrides disabled, ignoring source 4 
2025-10-27 18:24:32	Local0.Info	192.168.100.2	Cisco-7488.bbb7.fe80: *apfReceiveTask: Oct 27 18:24:31.208: %APF-6-USER_NAME_DELETED: apf_ms.c:8805 Username entry (xxxxxxxx) is deleted for mobile xx:xx:xx:xx:xx:xx 
2025-10-27 18:24:32	Local0.Warning	192.168.100.2	Cisco-7488.bbb7.fe80: *spamApTask0: Oct 27 18:24:31.232: %APF-4-MOBILESTATION_NOT_FOUND: apf_api.c:57734 Could not find the mobile xx:xx:xx:xx:xx:xx in internal database
2025-10-27 18:24:33	Local0.Info	192.168.100.2	Cisco-7488.bbb7.fe80: *Dot1x_NW_MsgTask_0: Oct 27 18:24:32.948: %APF-6-RADIUS_OVERRIDE_DISABLED: apf_ms_radius_override.c:213 Radius overrides disabled, ignoring source 4 
2025-10-27 18:24:33	Local0.Info	192.168.100.2	Cisco-7488.bbb7.fe80: *Dot1x_NW_MsgTask_0: Oct 27 18:24:32.948: %APF-6-RADIUS_OVERRIDE_DISABLED: apf_ms_radius_override.c:213 Radius overrides disabled, ignoring source 2 
2025-10-27 18:24:33	Local0.Info	192.168.100.2	Cisco-7488.bbb7.fe80: *Dot1x_NW_MsgTask_0: Oct 27 18:24:32.948: %APF-6-USER_NAME_CREATED: apf_ms.c:9003 Username entry (xxxxxxxx) with length (253) created for mobile xx:xx:xx:xx:xx:xx 
2025-10-27 18:24:33	Local0.Info	192.168.100.2	Cisco-7488.bbb7.fe80: *Dot1x_NW_MsgTask_0: Oct 27 18:24:32.948: %APF-6-RADIUS_OVERRIDE_DISABLED: apf_ms_radius_override.c:213 Radius overrides disabled, ignoring source 4 
2025-10-27 18:24:33	Local0.Warning	192.168.100.2	Cisco-7488.bbb7.fe80: *apfReceiveTask: Oct 27 18:24:32.972: %OSAPI-4-MUTEX_LOCK_FAILED: osapi_sem.c:1362 Failed to acquire a mutual exclusion object. waitforever: mutex lock operation failed. errcode = Resource deadlock avoided
2025-10-27 18:24:33	Local0.Notice	192.168.100.2	Cisco-7488.bbb7.fe80: *apfReceiveTask: Oct 27 18:24:32.972: %OSAPI-5-MUTEX_UNLOCK_FAILED: osapi_sem.c:1253 Failed to release a mutual exclusion object. mutex unlock failed, not owned by the calling thread. errcode = Operation not permitted

Does anyone have any idea what could be causing this?

Thanks,

Emanuel

[Kasun Bandara]

Hi @emanuel32804 as a tart, you can use this tool to upload and analyze the debug logs to get clear understanding about the issue.

Wireless Debug Analyzer

[emanuel32804]

I just get error try again later when I upload my tech_support.tgz to that. It also does not work if I just upload the log text file. Am I uploading the wrong things?

[Mark Elsen]

  - @emanuel32804    Uploading the logfile (cleartext) should work, what error do you get ?
                                  As far as the original problem is concerned : reboot the AP that the client is using
                                  and check if that can help.

  M.

[emanuel32804]

The error I get is "Error Try again"

Restarting the AP also did not help the issue.

[Mark Elsen]

  - @emanuel32804     Attach the file which you tried to upload ,

  M.

[emanuel32804]

Here is the 2 files I tried.

[Mark Elsen]

  - @emanuel32804                    Those aren't outputs from client debugs needed for Wireless Debug Analyzer

                                                            Execute client debugging according to 
                                  https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/213258-collect-debugs-from-wireless-lan-control.html

  M.

[emanuel32804]

Ok that was much more successful. Attached the log. In the log analyzer, it gets all the way through EAP authentication and DHCP, and it goes into RUN state. Then luckily (for debugging) about 4 seconds later it dropped it, which shows

Nov 05 02:18:54.519	*apfReceiveTask	Client session has timed out
Nov 05 02:18:54.519	*apfReceiveTask	Client disassociation event has occured. Possible reasons may be due to AP Radio Reset usually due to channel change or wlan was manually disabled or Client unable to get valid DHCP IP for WLAN using DHCP required
Nov 05 02:18:54.519	*apfReceiveTask	Client has been deauthenticated
Nov 05 02:18:54.520	*apfReceiveTask	Client session has timed out

[Mark Elsen]

  - @emanuel32804            Validate the configuration of the mobility express  based controller using :
                                           WirelessAnalyzer input (procedure) for AireOs controllers

                                                      Feed the result into Wireless Config Analyzer

  M.

[emanuel32804]

See attached for the results. The analyzer didn't have anything that really jumped out at me as being related to this issue, but it's very possible I don't know what I'm looking for.

I did also see these in the debug logs, they seem to state the issue more specifically:

Nov 05 02:18:54.519	*osapiBsnTimer	Authentication session timer expired: mark mobile for immediate deletion
Nov 05 02:18:54.519	*osapiBsnTimer	apfMsSessionExpireCallback (apf_ms.c:762) Expiring Mobile!
Nov 05 02:18:54.519	*apfReceiveTask	apfMsExpireMobileStation (apf_ms.c:8148) Changing state for mobile 46:f4:73:a3:72:77 on AP a0:93:51:c8:e4:40 from Associated to Disassociated 

What could cause that timeout?

[Mark Elsen · Accepted solution]

  - @emanuel32804    Disable   802.11v  for the SSID and check if that can help ,

  M.

[emanuel32804]

That worked! Client has been on network for 10 hours with no disconnects! Thank you