Users are getting disconnected on 9800 WLC - DELETE_REASON_RADIO_DOWN

stole · ‎06-23-2022

Hello all,

I have issues with some users staying connected on 9800 WLC (AP 9115AXI-E). From the logs I can see the below error message:

Delete client from bssid. Sending ack notification to apmgr for bssid disable/delete reason: 174, CO_CLIENT_DELETE_REASON_RADIO_DOWN, BSSID: 2c1a.0520.61a1

I will appreciate any suggestions on what could be the issue. Thank you

---------------------------------

2022/06/23 09:51:27.904553 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): MAC: 8086.f215.f2b6 Client move to idle state, delete reason: CO_CLIENT_DELETE_REASON_RADIO_DOWN, BSSID MAC: 2c1a.0520.61a0, WTP MAC: 2c1a.0520.61a0
2022/06/23 09:51:27.904566 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): CO process BSSID event from apmgr . Received BSSID event 2 from AP manager for bssid 2c1a.0520.61a1 wtp mac 2c1a.0520.61a0 vap id 2, slot id 0
2022/06/23 09:51:27.904571 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): Delete client from bssid. Sending ack notification to apmgr for bssid disable/delete reason: 174, CO_CLIENT_DELETE_REASON_RADIO_DOWN, BSSID: 2c1a.0520.61a1
2022/06/23 09:51:27.904578 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): CO process BSSID event from apmgr . Received BSSID event 2 from AP manager for bssid 2c1a.0520.61a2 wtp mac 2c1a.0520.61a0 vap id 3, slot id 0
2022/06/23 09:51:27.904580 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): Delete client from bssid. Sending ack notification to apmgr for bssid disable/delete reason: 174, CO_CLIENT_DELETE_REASON_RADIO_DOWN, BSSID: 2c1a.0520.61a2
2022/06/23 09:51:27.904734 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): MAC: 8086.f215.f2b6 Process idle request, delete reason: CO_CLIENT_DELETE_REASON_RADIO_DOWN
2022/06/23 09:51:27.904735 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): MAC: 8086.f215.f2b6 Process idle request, moving client from bssid: 2c1a.0520.61a0 to IDLE BSSID handle: 12364632845715509364
2022/06/23 09:51:27.904821 {wncd_x_R0-0}{1}: [client-orch-state] [24682]: (note): MAC: 8086.f215.f2b6 Client state transition: S_CO_RUN -> S_CO_IDLE
2022/06/23 09:51:27.904873 {wncd_x_R0-0}{1}: [apmgr-bssid] [24682]: (note): MAC: 2c1a.0520.61a0 Client delete ACK from co for bssid 2c1a.0520.61a1 slot id 0, state: 1
2022/06/23 09:51:27.904890 {wncd_x_R0-0}{1}: [apmgr-bssid] [24682]: (note): MAC: 2c1a.0520.61a0 Client delete ACK from co for bssid 2c1a.0520.61a2 slot id 0, state: 1
2022/06/23 09:51:27.905222 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): MAC: 8086.f215.f2b6 Client idle subevent, idle BSSID MAC: 2c1a.0520.61a0, WTP MAC: 2c1a.0520.61a0, idle subevent delete reason: CO_CLIENT_DELETE_REASON_RADIO_DOWN
2022/06/23 09:51:27.905232 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): Schedule client delete subevnt. Subevent for client delete scheduled successfully with delete reason 174, CO_CLIENT_DELETE_REASON_RADIO_DOWN, previous client: 8086.f215.f2b6, BSSID MAC: 2c1a.0520.61a0, WTP MAC: 2c1a.0520.61a0
2022/06/23 09:51:27.905250 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): Delete client from bssid. Sending ack notification to apmgr for bssid disable/delete reason: 174, CO_CLIENT_DELETE_REASON_RADIO_DOWN, BSSID: 2c1a.0520.61a0
2022/06/23 09:51:27.905275 {wncd_x_R0-0}{1}: [apmgr-bssid] [24682]: (note): MAC: 2c1a.0520.61a0 Client delete ACK from co for bssid 2c1a.0520.61a0 slot id 0, state: 1
2022/06/23 09:51:37.905761 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [24682]: (ERR): 8086.f215.f2b6CLIENT_STAGE_TIMEOUT State = IDLE, WLAN profile = SWSGP01, Policy profile = SWSGP01-flex-prof, AP name = w-ro-sbzb-parter-2
2022/06/23 09:51:37.905814 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): MAC: 8086.f215.f2b6 Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_RADIO_DOWN, details: , fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|01|07|13|17|18|28|33|42|44|46|48|4d|5c|5e|7f|a8|55|
2022/06/23 09:51:37.905903 {wncd_x_R0-0}{1}: [apmgr-bssid] [24682]: (ERR): Failed to get bssid mac address from bssid handle.
2022/06/23 09:51:37.905904 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (ERR): MAC: 8086.f215.f2b6 Failed to get ap mac from bssid handle required for sending the trap
2022/06/23 09:51:37.905910 {wncd_x_R0-0}{1}: [client-orch-state] [24682]: (note): MAC: 8086.f215.f2b6 Client state transition: S_CO_IDLE -> S_CO_DELETE_IN_PROGRESS
2022/06/23 09:51:37.906638 {wncd_x_R0-0}{1}: [dpath_svc] [24682]: (note): MAC: 8086.f215.f2b6 Client datapath entry deleted for ifid 0xa0000147
2022/06/23 09:51:37.907474 {wncd_x_R0-0}{1}: [sanet-shim-translate] [24682]: (note): MAC: 8086.f215.f2b6 Session manager disconnect event called, session label: 0x9f000b86
2022/06/23 09:51:37.909057 {wncd_x_R0-0}{1}: [client-orch-state] [24682]: (note): MAC: 8086.f215.f2b6 Client state transition: S_CO_DELETE_IN_PROGRESS -> S_CO_DELETED
2022/06/23 10:17:23.903682 {wncd_x_R0-0}{1}: [client-orch-sm] [24682]: (note): MAC: 8086.f215.f2b6 Association received. BSSID 2c1a.0520.61a0, WLAN SWSGP01, Slot 0 AP 2c1a.0520.61a0, w-ro-sbzb-parter-2
2022/06/23 10:17:23.903833 {wncd_x_R0-0}{1}: [client-orch-state] [24682]: (note): MAC: 8086.f215.f2b6 Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2022/06/23 10:17:23.904191 {wncd_x_R0-0}{1}: [dot11] [24682]: (note): MAC: 8086.f215.f2b6 Association success. AID 1, Roaming = False, WGB = False, 11r = False, 11w = False Fast roam = False
2022/06/23 10:17:23.904305 {wncd_x_R0-0}{1}: [client-orch-state] [24682]: (note): MAC: 8086.f215.f2b6 Client state transition: S_CO_ASSOCIATING -> S_CO_L2_AUTH_IN_PROGRESS
2022/06/23 10:17:23.904388 {wncd_x_R0-0}{1}: [client-auth] [24682]: (note): MAC: 8086.f215.f2b6 ADD MOBILE sent. Client state flags: 0x1 BSSID: MAC: 2c1a.0520.61a0 capwap IFID: 0x900000f8
2022/06/23 10:17:23.941322 {wncd_x_R0-0}{1}: [client-auth] [24682]: (note): MAC: 8086.f215.f2b6 L2 Authentication initiated. method DOT1X, Policy VLAN 0,AAA override = 1 , NAC = 0
2022/06/23 10:17:23.942150 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [24682]: (note): Authentication Success. Resolved Policy bitmap:11 for client 8086.f215.f2b6
2022/06/23 10:18:08.243510 {rogued_R0-0}{1}: [rogue-capwap] [25174]: (debug): Client info (4/14), MAC:<8086.f215.f2b6>, reported by 2c1a.0521.0280/0, GW: d884.6680.b6d9, chan:1, rssi:-66, snr: 25, ipv4: 0.0.0.0 , ipv6:0000:0000:0000:0000:0000:0000:0000:0000 , BSSID:d884.6680.b6d9
2022/06/23 10:18:08.243517 {rogued_R0-0}{1}: [rogue-core] [25174]: (info): Reported rogue client <8086.f215.f2b6> by AP <2c1a.0521.0280/0>
2022/06/23 10:18:08.243517 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client ADD 8086.f215.f2b6
2022/06/23 10:18:08.243524 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client exists under ap 586d.8f89.8acb
2022/06/23 10:18:08.243621 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client LRAD report 2c1a.0521.0280, slot_id: 0, last_heard: 1655979488.243621
2022/06/23 10:18:08.244134 {rogued_R0-0}{1}: [rogue-core] [25174]: (debug): Validation for client <8086.f215.f2b6>, [cfg/req] cmx:0/0, aaa:0/0
2022/06/23 10:18:08.244278 {rogued_R0-0}{1}: [rogue-rule] [25174]: (debug): Apply all classification rules to rogue <586d.8f89.8acb>
2022/06/23 10:18:08.244282 {rogued_R0-0}{1}: [rogue-rule] [25174]: (debug): Updating generation-id for rogue AP <586d.8f89.8acb>. 6 -> 6.
2022/06/23 10:18:08.244300 {rogued_R0-0}{1}: [rogue-rule] [25174]: (debug): Apply all classification rules to rogue <d884.6680.b6d9>
2022/06/23 10:18:08.244303 {rogued_R0-0}{1}: [rogue-rule] [25174]: (debug): Updating generation-id for rogue AP <d884.6680.b6d9>. 6 -> 6.
2022/06/23 10:18:08.244433 {rogued_R0-0}{1}: [rogue-core] [25174]: (debug): RLDP not enabled <586d.8f89.8acb>
2022/06/23 10:18:08.244445 {rogued_R0-0}{1}: [rogue-core] [25174]: (debug): RLDP not enabled <d884.6680.b6d9>
2022/06/23 10:18:08.245501 {rogued_R0-0}{1}: [rogue-capwap] [25174]: (debug): Client info (12/15), MAC:<8086.f215.f2b6>, reported by 2c1a.0521.0280/1, GW: f02f.7498.5700, chan:44, rssi:-76, snr: 19, ipv4: 0.0.0.0 , ipv6:0000:0000:0000:0000:0000:0000:0000:0000 , BSSID:586d.8f89.8acb
2022/06/23 10:18:08.245504 {rogued_R0-0}{1}: [rogue-core] [25174]: (info): Reported rogue client <8086.f215.f2b6> by AP <2c1a.0521.0280/1>
2022/06/23 10:18:08.245504 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client ADD 8086.f215.f2b6
2022/06/23 10:18:08.245509 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client exists under ap d884.6680.b6d9
2022/06/23 10:18:08.245562 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client LRAD report 2c1a.0521.0280, slot_id: 1, last_heard: 1655979488.245562
2022/06/23 10:18:08.245931 {rogued_R0-0}{1}: [rogue-core] [25174]: (debug): Validation for client <8086.f215.f2b6>, [cfg/req] cmx:0/0, aaa:0/0
2022/06/23 10:18:08.246091 {rogued_R0-0}{1}: [rogue-rule] [25174]: (debug): Apply all classification rules to rogue <d884.6680.b6d9>
2022/06/23 10:18:08.246094 {rogued_R0-0}{1}: [rogue-rule] [25174]: (debug): Updating generation-id for rogue AP <d884.6680.b6d9>. 6 -> 6.
2022/06/23 10:18:08.246210 {rogued_R0-0}{1}: [rogue-core] [25174]: (debug): RLDP not enabled <d884.6680.b6d9>

Flavio Miranda · ‎06-23-2022

Hi

Does this AP 2c1a.0520.61a0 had radio flap? check if capwap tunnel uptime, if this SSID is central switch.

stole · ‎06-23-2022

The SSID is locally switched. The AP was restarted later, not when the logs were taken. We can see that the user is able to connect on one AP, but when it goes to the next one, the user is not able to connect.

The below log is strange for me, the AP 2c1a.0521.0280 is known to us but is reporting the users as Rogue.

2022/06/23 10:18:08.245504 {rogued_R0-0}{1}: [rogue-core] [25174]: (info): Reported rogue client <8086.f215.f2b6> by AP <2c1a.0521.0280/1>

Also, the below AP is Extreme, is there a chance that there is an Extreme AP that is working because from the logs it looks like that is been reported to him:

2022/06/23 10:18:08.245509 {rogued_R0-0}{1}: [rogue-db] [25174]: (debug): Client exists under ap d884.6680.b6d9 <<<<<

stole · ‎06-24-2022

We found out that the customer has connected two from the old Extreme AP's without telling us and they were causing the issue. After we shut down them, all users were able to connect.