topic Re: Client roaming between APs at high traffic load cause 3 second gap in Wireless

Client roaming between APs at high traffic load cause 3 second gap

Anders Hedlund — Fri, 19 Apr 2024 09:37:52 GMT

We are operating our vehicles remotely in the underground mines over Wi-Fi, the data load from the machine is constant around 30Mbit/s (yes, six to eight HD-cameras).
When switching between APs we have noticed that we sometimes get a transmission gap of around 3 seconds, causing the safety mechanism on the trucks to stop the machine.
We have activated Radio Active Traces in the WLC and gets the response included below.
We have more than 150 access points, and this happens between a few of them, most of them never has this problem, but on a few one it happens, we have tried changed the cables and also the access points (we even switched model).
We have tested roaming between them with low load and this does not happen then.
We are on WLC 9800-L version 17.13.1 (but we had the issue on the 17.9.4A also).
The AP version/models are:
17.13.0.107, C9124AXE-E
17.13.0.107, C9124AXE-B
17.13.0.107, C9120AXE-E
This is the reported issue in the debugTrace:
2024/04/04 04:07:20.696836577 {wncd_x_R0-0}{1}: [client-orch-sm] [18090]: (info): MAC: 0090.e89c.47bd Deleting the client, reason: 286, CO_CLIENT_DELETE_REASON_MN_AP_DRIVER_EVENT_CLASS3_RECV, Client state S_CO_RUN
2024/04/04 04:07:20.697072661 {wncd_x_R0-0}{1}: [errmsg] [18090]: (debug): %CLIENT_ORCH_LOG-7-CLIENT_MOVED_TO_DELETE_STATE: R0/0: wncd: Username (null), MAC: 0090.e89c.47bd, IP 10.88.19.154 disconnected from AP (G1N-AP-S-356-1) with SSID (Voysys-AX)
2024/04/04 04:07:20.697082437 {wncd_x_R0-0}{1}: [client-orch-sm] [18090]: (note): MAC: 0090.e89c.47bd Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_MN_AP_DRIVER_EVENT_CLASS3_RECV, details: , fsm-state transition 2c|2d|2e|15|1a|1b|2c|37|46|48|4a|4c|51|60|62|83|86|8e|15|1a|1b|2c|37|46|48|4a|4c|51|60|62|83|ab|

And in the analysis tool, we get:
Controller initiated client deletion with code: CO_CLIENT_DELETE_REASON_MN_AP_DRIVER_EVENT_CLASS3_RECV. Explanation: AP triggered delete, as client sent class 3 frame (data, power save, some management) from non-authenticated client, may happen occasionally on connection recovery. Actions: Nothing required, unless this is happening frequently. This may need RA trace and OTA capture
Has anyone seen this? Any ideas how we can debug this further to better find a solution or work-around?
/Anders

Re: Client roaming between APs at high traffic load cause 3 second gap

Mark Elsen — Fri, 19 Apr 2024 10:53:17 GMT

You can feed Radio Active traces into : Wireless Debug Analyzer for high level analysis
Outputs from https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc5
can also be useful

- You may try to tweak e.g. parameters related to fast roaming (802.11k,...)
- I can assume that the mining environment is very challenging for wireless ; has sufficient site survey been done ?
- Check if wireless drivers of the camera's are up to date , and or can be updated or else , look at new revisions of the
camera models with other firmware if applicable ,

Also have a checkup of the WLC 9800-L configuration with the CLI command show tech wireless and feed the output to : Wireless Config Analyzer

Re: Client roaming between APs at high traffic load cause 3 second gap

Leo Laohoo — Fri, 19 Apr 2024 11:31:36 GMT

CSCwi49862

Re: Client roaming between APs at high traffic load cause 3 second gap

Mark Elsen — Fri, 19 Apr 2024 11:53:19 GMT

- @Leo Laohoo Leo , do you have shared and or allowed content availability for this bug because I get :
This bug is not available for viewing at this time

Re: Client roaming between APs at high traffic load cause 3 second gap

Leo Laohoo — Fri, 19 Apr 2024 12:20:25 GMT

https://imgur.com/pKY4D6t

Re: Client roaming between APs at high traffic load cause 3 second gap

JPavonM — Fri, 19 Apr 2024 14:04:46 GMT

@Anders Hedlund have you tested 17.12 train?

Cisco only recommends 17.13 for deployments that need new features introduced in that code, but TC will tell you to move to recommended 17.9.5 or 17.12.2 (17.12.3 in few weeks)

Re: Client roaming between APs at high traffic load cause 3 second gap

Leo Laohoo — Sun, 21 Apr 2024 00:05:56 GMT

17.12.3 has been out since 21 March 2024 and 17.14.1 was released last week, 13 April 2024.

Release Notes for Cisco Catalyst 9800 Series Wireless Controller, Cisco IOS XE Dublin 17.12.x

Release Notes for Cisco Catalyst 9800 Series Wireless Controller, Cisco IOS XE 17.14.x

Re: Client roaming between APs at high traffic load cause 3 second gap

Leo Laohoo — Sun, 21 Apr 2024 00:08:14 GMT

@Anders Hedlund,

Since the APs are in a vehicle, are the APs powered OFF when the vehicle is not in use?

Re: Client roaming between APs at high traffic load cause 3 second gap

Anders Hedlund — Mon, 22 Apr 2024 21:43:55 GMT

Thanks Marce, yes, we have gone through the usual troubleshooting processes, what makes it tricky is that it's so tight correlated to the traffic load. We have tested adding a UDP packet generator (then skipping the cameras, so only the safety mechanism and the UDP packets) and if we set it to like 30Mbit/s UDP packets can clearly see the problem, but not if we switch it off or have less load).

Re: Client roaming between APs at high traffic load cause 3 second gap

Anders Hedlund — Mon, 22 Apr 2024 21:46:14 GMT

Now this was interesting, thanks, we might have some contact that can dig this one out for us (it has some security classification).

Re: Client roaming between APs at high traffic load cause 3 second gap

Anders Hedlund — Mon, 22 Apr 2024 21:51:34 GMT

Yes, we have to use 17.13, we have many many AP model 9124, and we need 17.13 to be able to run then in indoor-mode enabling non-DFS channel usage (we can not use DFS channels in our setup, it causes too long communucation gaps so the machines will make a safety stop).

Re: Client roaming between APs at high traffic load cause 3 second gap

Anders Hedlund — Mon, 22 Apr 2024 21:55:47 GMT

We don't have the APs on the vehicles, sorry for not beeing clear about this. On the vehicles we have MOXA 4131A clients.

Re: Client roaming between APs at high traffic load cause 3 second gap

Leo Laohoo — Mon, 22 Apr 2024 22:57:25 GMT

@Anders Hedlund wrote:
We don't have the APs on the vehicles, sorry for not beeing clear about this. On the vehicles we have MOXA 4131A clients.

Good.

Reboot the APs and see if the roaming has improved. If it has, consider the possibility of rebooting the APs daily -- Read between the lines.

Re: Client roaming between APs at high traffic load cause 3 second gap

PierreWallin4026 — Thu, 26 Sep 2024 11:10:02 GMT

I dont have access to this one, is there any update? Is there a fix?

Re: Client roaming between APs at high traffic load cause 3 second gap

PierreWallin4026 — Thu, 26 Sep 2024 11:53:10 GMT

I dont have access to this one, is there any update? Is there a fix?

Re: Client roaming between APs at high traffic load cause 3 second gap

Anders Hedlund — Mon, 04 Nov 2024 12:24:14 GMT

After a couple of weeks with extensive testing with hundreds of AP roaming events at high uplink load (30Mbit/s), we found that the problem is gone after upgrading to WLC version 17.15.1.