Users can't connect to WiFi sporadically

[ACleverUsername]

I've been getting reports from several of my users that they can't connect to our WiFI.  I'm working with a specific user and have attached the debug log here.  The issues start at the top and go straight until line 37.  It's basically this...

Logging display requested on 2025/02/11 11:33:48 (EDT) for Hostname: [], Model: [C9800-CL-K9], Version: [17.12.03], SN: [], MD_SN: []
2025/02/11 09:17:27.858718730 {wncd_x_R0-0}{2}: [client-orch-sm] [17192]: (note): MAC: 5af8.2b70.995d Data path not plumbed on standby, Adding client to poision list
2025/02/11 09:17:27.859023063 {wncd_x_R0-0}{2}: [sisf-ha] [17192]: (ERR): Data path plumb create/update failed on standby for mac 5af8.2b70.995d
2025/02/11 09:17:27.860946663 {wncd_x_R0-0}{2}: [dpath_svc] [17192]: (note): MAC: 5af8.2b70.995d client datapath plumb on standby for client IFID: 0xa00002e6
2025/02/11 09:17:38.903305596 {wncd_x_R0-0}{2}: [sisf-ha] [17192]: (ERR): Data path plumb create/update failed on standby for mac 5af8.2b70.995d
2025/02/11 09:17:38.925533380 {wncd_x_R0-0}{2}: [dpath_svc] [17192]: (note): MAC: 5af8.2b70.995d client datapath plumb on standby for client IFID: 0xa00002e6
2025/02/11 09:18:01.900085090 {wncd_x_R0-0}{2}: [sisf-ha] [17192]: (ERR): Data path plumb create/update failed on standby for mac 5af8.2b70.995d

...over and over and over again.  I've tried searching for any of these error messages (especially the misspelled "poision") and found nothing.  This is on a Catalyst 9800-CL running v17.12.3.  Any advice would be greatly appreciated.

[Flavio Miranda]

@ACleverUsername 

 The log you pasted is basically IOS logs. The logs attached have some information's that can help. 

I would recommend you to disable  802.11k.

Cisco Catalyst 9800 Series Wireless Controller Software Configuration Guide, Cisco IOS XE Gibraltar 16.10.x - Assisted Roaming [Cisco Catalyst 9800 Series Wireless Controllers] - Cisco

[ACleverUsername]

Much appreciated.  Can you clarify what you mean by "basically IOS logs"?  I mean, that's what they are since they're what I got when I downloaded the radioactive trace, but are there other logs that would be more helpful?

And what makes you think that disabling 802.11k would be beneficial?  Was there something in the logs that makes you think that?

[Flavio Miranda]

"IEEE802.11v, Process dot11 wnm action frame failed"

  feature like 11v usually cause more problem then benefits.  When clients gets disconnect often like this, the first thing you do is start disabling features and protocols that is not a must on the WLC.  11v is one of them. 

[Scott Fella]

I agree with @Flavio Miranda... unless you have time to test all the features with all the devices you have to support, the more features the more problems.  That is why I always tell folks to create a test SSID, with the basic stuff and disable any other enabled features.  Test and validate and then add one feature at a time and go through the whole test and validate step again. Folks love features and vendors are good at marketing them.  In a controlled environment with a couple devices, they all work great, but in production if you don't manage the end devices, good luck.

[ACleverUsername]

@Scott Fella @Flavio Miranda - REALLY appreciate your help on this.  I'll look into disabling that - just gotta run it past one other person before I do.

And since I'm clearly new to the forums and not sure if this is allowed, but if I were to upload some screenshots of my settings, do you think you could give 'em a once-over and see if they're sane?

[Flavio Miranda]

Sure. 

[ACleverUsername]

Much appreciated.  I've uploaded the Security and Advanced options I currently have set.

[Flavio Miranda]

Security seems to be fine, nothing special.

Advanced have a lot of configuration that can potentially cause problem, inclusing 802.11ax. 

If your problem is critical I would recommend disabling feature by feature  and testing.

Will list some

Just do it in maintenance window and test before and after. 

Make sure your wireless adapter is working with latest driver version. 

[Leo Laohoo]

Start with the most basic:  If the APs are Cheetah OS (CoS), reboot the APs.  

If the problem persist, turn off 802.11k, v and r.  

If the WLC is on 17.12.3, what APSP/SMU are installed?

[ACleverUsername]

I'm showing no APSP nor SMU:

[Leo Laohoo]

This could be a long-shot, but have a look at CSCwm42613.  

[networksupportiap]

We are also facing similar client re-association issue in the 9800-CL WLC. User getting disconnected randomly. Can someone share a solution ?

2025/02/24 17:01:36.834450808 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (note): MAC: 9065.84a9.c973 Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_CLIENT_EAP_TIMEOUT_FAILURE, details: , fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|01|07|15|16|17|15|1a|1b|2c|37|46|48|4a|4c|55|64|66|87|af|
2025/02/24 17:01:36.834503716 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (note): MAC: 9065.84a9.c973 Delete mobile payload sent for BSSID: 889c.ad20.f8ee WTP mac: 889c.ad20.f8e0 slot id: 1
2025/02/24 17:01:36.834510637 {wncd_x_R0-0}{2}: [client-orch-state] [17158]: (note): MAC: 9065.84a9.c973 Client state transition: S_CO_RUN -> S_CO_DELETE_IN_PROGRESS
2025/02/24 17:01:36.834880270 {wncd_x_R0-0}{2}: [dpath_svc] [17158]: (note): MAC: 9065.84a9.c973 Client datapath entry deleted for ifid 0xa0000073
2025/02/24 17:01:36.835006083 {wncd_x_R0-0}{2}: [sanet-shim-translate] [17158]: (note): MAC: 9065.84a9.c973 Session manager disconnect event called, session label: 0x4c00035c
2025/02/24 17:01:36.835274709 {wncd_x_R0-0}{2}: [sisf-tdl] [17158]: (ERR): DAD Pending Rec Del Failed fe80::a9:6533:3f34:787a:-2147483631
2025/02/24 17:01:36.835401011 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (ERR): MAC: 0000.0000.0000 Unable to determine vlan and switching mode. fsm_ctxt not found for client
2025/02/24 17:01:36.835475339 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (ERR): MAC: 0000.0000.0000 Unable to determine vlan and switching mode. fsm_ctxt not found for client
2025/02/24 17:01:36.837418623 {wncd_x_R0-0}{2}: [client-orch-state] [17158]: (note): MAC: 9065.84a9.c973 Client state transition: S_CO_DELETE_IN_PROGRESS -> S_CO_DELETED
2025/02/24 17:01:36.837522010 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (ERR): MAC: 9065.84a9.c973 Failed to receive dot11 disassoc or deauth request. Client FSM not exist, ignore the disassoc/deauth req
2025/02/24 17:01:36.837522350 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (ERR): MAC: 9065.84a9.c973 Failed to get ewlc dot11 packet handler. Disassoc/Deauth request processing failed in CO. Dropping request
2025/02/24 17:01:36.841662725 {wncd_x_R0-0}{1}: [sisf-ha] [17472]: (ERR): Failed to find client 9065.84a9.c973 in db mac table with rc 2
2025/02/24 17:13:27.999162885 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (note): MAC: 9065.84a9.c973 Association received. BSSID 889c.ad20.f8ee, WLAN FCA-WiFi, Slot 1 AP 889c.ad20.f8e0, rflciporaph4, Site tag FCA-CHN, Policy tag FCA-CHN, Policy profile FCA-CHN_WLANID_1, Switching Local, Socket delay 0ms
2025/02/24 17:13:27.999272982 {wncd_x_R0-0}{2}: [client-orch-state] [17158]: (note): MAC: 9065.84a9.c973 Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2025/02/24 17:13:27.999713536 {wncd_x_R0-0}{2}: [dot11] [17158]: (note): MAC: 9065.84a9.c973 Association success. AID 22, Roaming = False, WGB = False, 11r = False, 11w = True Fast roam = False
2025/02/24 17:13:27.999844505 {wncd_x_R0-0}{2}: [client-orch-state] [17158]: (note): MAC: 9065.84a9.c973 Client state transition: S_CO_ASSOCIATING -> S_CO_L2_AUTH_IN_PROGRESS
2025/02/24 17:13:27.999973835 {wncd_x_R0-0}{2}: [client-auth] [17158]: (note): MAC: 9065.84a9.c973 ADD MOBILE sent. Client state flags: 0x1 BSSID: MAC: 889c.ad20.f8ee capwap IFID: 0x90000065, Add mobiles sent: 1
2025/02/24 17:13:28.000167697 {wncd_x_R0-0}{2}: [client-orch-sm] [17158]: (note): MAC: 9065.84a9.c973 Association received. BSSID 889c.ad20.f8ee, WLAN FCA-WiFi, Slot 1 AP 889c.ad20.f8e0, rflciporaph4, Site tag FCA-CHN, Policy tag FCA-CHN, Policy profile FCA-CHN_WLANID_1, Switching Local, old BSSID 889c.ad20.f8ee, Socket delay 2ms
2025/02/24 17:13:28.000198338 {wncd_x_R0-0}{2}: [client-orch-state] [17158]: (note): MAC: 9065.84a9.c973 Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_L2_AUTH_IN_PROGRESS
2025/02/24 17:13:28.000425898 {wncd_x_R0-0}{2}: [dot11] [17158]: (note): MAC: 9065.84a9.c973 Association success. AID 22, Roaming = False, WGB = False, 11r = False, 11w = True Fast roam = False
2025/02/24 17:13:28.000501373 {wncd_x_R0-0}{2}: [client-orch-state] [17158]: (note): MAC: 9065.84a9.c973 Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_L2_AUTH_IN_PROGRESS
2025/02/24 17:13:28.005376629 {wncd_x_R0-0}{2}: [client-auth] [17158]: (note): MAC: 9065.84a9.c973 L2 Authentication initiated. method DOT1X, Policy VLAN 0, AAA override = 0 , NAC = 0
2025/02/24 17:13:28.006344886 {wncd_x_R0-0}{2}: [ewlc-infra-evq] [17158]: (note): Authentication Success. Resolved Policy bitmap:11 for client 9065.84a9.c973
2025/02/24 17:13:31.242092648 {wncd_x_R0-0}{2}: [sanet-shim-miscellaneous] [17158]: (ERR): authc policy update from SANet vlan 17
2025/02/24 17:13:31.242185316 {wncd_x_R0-0}{2}: [client-auth] [17158]: (note): MAC: 9065.84a9.c973 L2 Authentication Key Exchange Start. Resolved VLAN: 17, Audit Session id: C5046A0A000020EE37C4ECD7