We are working on Deploying our new 9800 controllers and have ran into an issues with the N+1 failover.
We are running -CL 17.9.3 and have the Latest APSP applied.
We are using flex connect and all the configs are synced. When we move an AP from one controller to another any Windows dot1x clients stay connected to the SSID but traffic stops passing. The only way to fix is for the client to manually disconnect and re-connect then everything works as normal. The Client shows in a RUN state once the AP joins the second controller. This also happens when we flip back to the first controller.
It seems like the AP is not telling the client to re-authenticate.
We have a TAC case open but they are not exactly the quickest to reply so i figured I would post here to see if anyone else has ran into this.
Here is a Radioactive trace of the Client when moved to the second controller:
From the last 12:39 Entry to the 12:42 entry the client is connected, shows as run but no traffic is passing... 12:42 is when we disconnect the client.
Logging display requested on 2023/08/09 12:43:16 (EST) for Hostname: [WLC006], Model: [C9800-CL-K9], Version: [17.09.03], SN: [x], MD_SN: [x]
2023/08/09 12:39:05.471529145 {wncd_x_R0-1}{1}: [client-orch-sm] [14537]: (note): MAC: f4a4.75a3.87bb HREAP Fault tolerance payload received BSSID 1484.7361.e94e, WLAN GroupNet, slot 1 AP 1484.7361.e940 AP-Test, AID 9WlanId: 1
2023/08/09 12:39:05.471564006 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2023/08/09 12:39:05.471864199 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_ASSOCIATING -> S_CO_CREATE_SM_SESSION_IN_PROGRESS
2023/08/09 12:39:05.473558728 {wncd_x_R0-1}{1}: [ewlc-infra-evq] [14537]: (note): Authentication Success. Resolved Policy bitmap:11 for client f4a4.75a3.87bb
2023/08/09 12:39:05.474223460 {wncd_x_R0-1}{1}: [client-orch-sm] [14537]: (note): MAC: f4a4.75a3.87bb Mobility discovery triggered. Client mode: Flex - Local Switching
2023/08/09 12:39:05.474227999 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_CREATE_SM_SESSION_IN_PROGRESS -> S_CO_MOBILITY_DISCOVERY_IN_PROGRESS
2023/08/09 12:39:05.474557057 {wncd_x_R0-1}{1}: [ewlc-infra-evq] [14537]: (note): Authentication Success. Resolved Policy bitmap:0 for client f4a4.75a3.87bb
2023/08/09 12:39:05.474564045 {wncd_x_R0-1}{1}: [ewlc-infra-evq] [14537]: (note): Authentication Success. Resolved Policy bitmap:11 for client f4a4.75a3.87bb
2023/08/09 12:39:05.474573931 {wncd_x_R0-1}{1}: [ewlc-infra-evq] [14537]: (note): Authentication Success. Resolved Policy bitmap:0 for client f4a4.75a3.87bb
2023/08/09 12:39:05.474770934 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_MOBILITY_DISCOVERY_IN_PROGRESS -> S_CO_MOBILITY_DISCOVERY_IN_PROGRESS
2023/08/09 12:39:05.482284058 {wncd_x_R0-1}{1}: [mm-client] [14537]: (note): MAC: f4a4.75a3.87bb Mobility Successful. Roam Type None, Sub Roam Type MM_SUB_ROAM_TYPE_NONE, Client IFID: 0xa000002d, Client Role: Local PoA: 0x9040000e PoP: 0x0
2023/08/09 12:39:05.482420086 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_MOBILITY_DISCOVERY_IN_PROGRESS -> S_CO_DPATH_PLUMB_IN_PROGRESS
2023/08/09 12:39:05.482428800 {wncd_x_R0-1}{1}: [dot11] [14537]: (note): MAC: f4a4.75a3.87bb Client datapath entry params - ssid:GroupNet,slot_id:1 bssid ifid: 0x0, radio_ifid: 0x90400008, wlan_ifid: 0xf0400001
2023/08/09 12:39:05.482584714 {wncd_x_R0-1}{1}: [dpath_svc] [14537]: (note): MAC: f4a4.75a3.87bb Client datapath entry created for ifid 0xa000002d
2023/08/09 12:39:05.482938659 {wncd_x_R0-1}{1}: [client-iplearn] [14537]: (note): MAC: f4a4.75a3.87bb Client IP learn successful. Method: IP Snooping IP: 10.114.23.111
2023/08/09 12:39:05.483165267 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_DPATH_PLUMB_IN_PROGRESS -> S_CO_IP_LEARN_IN_PROGRESS
2023/08/09 12:39:05.483485868 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_IP_LEARN_IN_PROGRESS -> S_CO_RUN
2023/08/09 12:39:24.997212019 {wncd_x_R0-1}{1}: [client-auth] [14537]: (ERR): MAC: f4a4.75a3.87bb Zero iPSK tag received from AP in extended info payload. Failed to update client common oper data.
2023/08/09 12:39:24.997245790 {wncd_x_R0-1}{1}: [client-iplearn] [14537]: (ERR): MAC: f4a4.75a3.87bb IAPP client IP create binding, sisf create binding failed, sisf_err: 9, vlan: 2023, Zone-id: 0x00000000, ifhdl: 0x9040000e, IP: ::
2023/08/09 12:39:24.997409008 {wncd_x_R0-1}{1}: [ewlc-infra-evq] [14537]: (note): Authentication Success. Resolved Policy bitmap:0 for client f4a4.75a3.87bb
2023/08/09 12:39:24.997787207 {wncd_x_R0-1}{1}: [ewlc-infra-evq] [14537]: (note): Authentication Success. Resolved Policy bitmap:11 for client f4a4.75a3.87bb
2023/08/09 12:39:24.997958502 {wncd_x_R0-1}{1}: [client-auth] [14537]: (note): MAC: f4a4.75a3.87bb ADD MOBILE sent. Client state flags: 0x8 BSSID: MAC: 1484.7361.e94e capwap IFID: 0x9040000e, Add mobiles sent: 1
2023/08/09 12:42:07.617327611 {wncd_x_R0-1}{1}: [client-orch-sm] [14537]: (note): MAC: f4a4.75a3.87bb Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_MN_SA_QUERY_TIMEOUT, details: , fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|b4|b7|ba|bb|96|37|46|48|4a|4c|51|60|62|83|94|aa|
2023/08/09 12:42:07.617408668 {wncd_x_R0-1}{1}: [client-orch-sm] [14537]: (note): MAC: f4a4.75a3.87bb Delete mobile payload sent for BSSID: 1484.7361.e94e WTP mac: 1484.7361.e940 slot id: 1
2023/08/09 12:42:07.617422632 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_RUN -> S_CO_DELETE_IN_PROGRESS
2023/08/09 12:42:07.617763881 {wncd_x_R0-1}{1}: [dpath_svc] [14537]: (note): MAC: f4a4.75a3.87bb Client datapath entry deleted for ifid 0xa000002d
2023/08/09 12:42:07.617903620 {wncd_x_R0-1}{1}: [sanet-shim-translate] [14537]: (note): MAC: f4a4.75a3.87bb Session manager disconnect event called, session label: 0x1700008d
2023/08/09 12:42:07.618171956 {wncd_x_R0-1}{1}: [client-orch-sm] [14537]: (ERR): MAC: 0000.0000.0000 vlan_mode_api: fsm_ctxt not found
2023/08/09 12:42:07.619288878 {wncd_x_R0-1}{1}: [client-orch-state] [14537]: (note): MAC: f4a4.75a3.87bb Client state transition: S_CO_DELETE_IN_PROGRESS -> S_CO_DELETED
