Solved: Re: C9800 exclusion due to wrong PSK - macOS clients only - Page 2

Jaroslav Ksinan · ‎11-30-2021

Hello,

We're migrating APs from old AireOS 2504 WLC to C9800-CL (running on 17.3.3) and when APs are migrated to C9800, macOS clients are unable to connect to WPA2 SSID with PSK authentication (FlexConnect local switching). Windows laptops and phones work like before with the same PSK, so clients definitely use correct password.

I checked bug search tool, didn't find anything relevant. We removed SSID from known networks on client side and joined SSID as new connection but that also didn't help.

Perhaps somebody faced the same and have some ideas what can be causing this. Any idea is greatly appreciated.

WLAN setup:

WLC-P-01#sh wlan name WIFI_PSK
WLAN Profile Name     : WIFI_PSK
================================================
Identifier                                     : 1
Description                                    :
Network Name (SSID)                            : WIFI_PSK
Status                                         : Enabled
Broadcast SSID                                 : Enabled
Advertise-Apname                               : Disabled
Universal AP Admin                             : Disabled
Max Associated Clients per WLAN                : 0
Max Associated Clients per AP per WLAN         : 0
Max Associated Clients per AP Radio per WLAN   : 200
OKC                                            : Enabled
Number of Active Clients                       : 0
CHD per WLAN                                   : Enabled
WMM                                            : Allowed
WiFi Direct Policy                             : Disabled
Channel Scan Defer Priority:
  Priority (default)                           : 5
  Priority (default)                           : 6
Scan Defer Time (msecs)                        : 100
Media Stream Multicast-direct                  : Disabled
CCX - AironetIe Support                        : Disabled
Peer-to-Peer Blocking Action                   : Disabled
Radio Policy                                   : 802.11a and 802.11g only
DTIM period for 802.11a radio                  :
DTIM period for 802.11b radio                  :
Local EAP Authentication                       : Disabled
Mac Filter Authorization list name             : Disabled
Mac Filter Override Authorization list name    : Disabled
Accounting list name                           :
802.1x authentication list name                : Disabled
802.1x authorization list name                 : Disabled
Security
    802.11 Authentication                      : Open System
    Static WEP Keys                            : Disabled
    Wi-Fi Protected Access (WPA/WPA2/WPA3)     : Enabled
        WPA (SSN IE)                           : Disabled
        WPA2 (RSN IE)                          : Enabled
            MPSK                               : Disabled
            AES Cipher                         : Enabled
            CCMP256 Cipher                     : Disabled
            GCMP128 Cipher                     : Disabled
            GCMP256 Cipher                     : Disabled
            Randomized GTK                     : Disabled
        WPA3 (WPA3 IE)                         : Disabled
        Auth Key Management
            802.1x                             : Disabled
            PSK                                : Enabled
            CCKM                               : Disabled
            FT dot1x                           : Disabled
            FT PSK                             : Disabled
            Dot1x-SHA256                       : Disabled
            PSK-SHA256                         : Disabled
            SAE                                : Disabled
            OWE                                : Disabled
            SUITEB-1X                          : Disabled
            SUITEB192-1X                       : Disabled
    CCKM TSF Tolerance (msecs)                 : 1000
    OWE Transition Mode                        : Disabled
    OSEN                                       : Disabled
    FT Support                                 : Disabled
        FT Reassociation Timeout (secs)        : 20
        FT Over-The-DS mode                    : Disabled
    PMF Support                                : Disabled
        PMF Association Comeback Timeout (secs): 1
        PMF SA Query Time (msecs)              : 200
    Web Based Authentication                   : Disabled
    Conditional Web Redirect                   : Disabled
    Splash-Page Web Redirect                   : Disabled
    Webauth On-mac-filter Failure              : Disabled
    Webauth Authentication List Name           : Disabled
    Webauth Authorization List Name            : Disabled
    Webauth Parameter Map                      : Disabled
Band Select                                    : Enabled
Load Balancing                                 : Disabled
Multicast Buffer                               : Disabled
Multicast Buffers (frames)                     : 0
IP Source Guard                                : Disabled
Assisted-Roaming
    Neighbor List                              : Enabled
    Prediction List                            : Disabled
    Dual Band Support                          : Disabled
IEEE 802.11v parameters
    Directed Multicast Service                 : Enabled
    BSS Max Idle                               : Enabled
        Protected Mode                         : Disabled
    Traffic Filtering Service                  : Disabled
    BSS Transition                             : Disabled
        Disassociation Imminent                : Disabled
            Optimised Roaming Timer (TBTTS)    : 40
            Timer (TBTTS)                      : 200
        Dual Neighbor List                     : Disabled
    WNM Sleep Mode                             : Disabled
802.11ac MU-MIMO                               : Enabled
802.11ax parameters
    OFDMA Downlink                             : Enabled
    OFDMA Uplink                               : Enabled
    MU-MIMO Downlink                           : Enabled
    MU-MIMO Uplink                             : Enabled
    BSS Target Wake Up Time                    : Enabled
    BSS Target Wake Up Time Broadcast Support  : Enabled
mDNS Gateway Status                            : Bridge
WIFI Alliance Agile Multiband                  : Disabled
Device Analytics
    Advertise Support                          : Enabled
    Share Data with Client                     : Disabled
Client Scan Report (11k Beacon Radio Measurement)
    Request on Association                     : Disabled
    Request on Roam                            : Disabled
WiFi to Cellular Steering                      : Disabled

WLC logs:

Nov 30 13:55:41.068: %CLIENT_EXCLUSION_SERVER-5-ADD_TO_BLACKLIST_REASON_DYNAMIC: Chassis 1 R0/0: wncmgrd: Client MAC: 6692.f9bc.0f0d was added to exclusion list associated with AP Name:ap-portu-flex-2, BSSID:MAC: b811.4b5a.e60f, reason:Wrong PSK

Nov 30 13:55:27.821: %CLIENT_EXCLUSION_SERVER-5-ADD_TO_BLACKLIST_REASON_DYNAMIC: Chassis 1 R0/0: wncmgrd: Client MAC: 3c06.301a.8e54 was added to exclusion list associated with AP Name:ap-portu-flex-2, BSSID:MAC: b811.4b5a.e60f, reason:Wrong PSK

I also ran a Radioactive Trace, example for one client:

2021/11/30 10:42:37.929244 {wncd_x_R0-0}{1}: [client-orch-sm] [20392]: (note): MAC: 8866.5a45.400b  Association received. BSSID b811.4b5a.e600, WLAN WIFI_PSK, Slot 0 AP b811.4b5a.e600, ap-portu-flex-2
2021/11/30 10:42:37.929350 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2021/11/30 10:42:37.929570 {wncd_x_R0-0}{1}: [dot11] [20392]: (note): MAC: 8866.5a45.400b  Association success. AID 1, Roaming = False, WGB = False, 11r = False, 11w = False 
2021/11/30 10:42:37.929644 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_ASSOCIATING -> S_CO_L2_AUTH_IN_PROGRESS
2021/11/30 10:42:37.929657 {wncd_x_R0-0}{1}: [client-auth] [20392]: (note): MAC: 8866.5a45.400b  L2 Authentication initiated. method PSK, Policy VLAN 132,AAA override = 0, NAC = 0
2021/11/30 10:42:37.929670 {wncd_x_R0-0}{1}: [sanet-shim-translate] [20392]: (ERR): 8866.5a45.400b  wlan_profile Not Found : Device information attributes not populated
2021/11/30 10:42:37.930166 {wncd_x_R0-0}{1}: [epm] [20392]: (ERR): [0000.0000.0000:unknown] HDL = 0x0 Vlan info not found for vlan id 132
2021/11/30 10:42:37.930427 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [20392]: (note): Authentication Success. Resolved Policy bitmap:11 for client 8866.5a45.400b 
2021/11/30 10:42:37.930489 {wncd_x_R0-0}{1}: [client-auth] [20392]: (note): MAC: 8866.5a45.400b  ADD MOBILE sent. Client state flags: 0x1  BSSID: MAC: b811.4b5a.e600  capwap IFID: 0x90000010
2021/11/30 10:42:37.960637 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:37.960638 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed
2021/11/30 10:42:38.960620 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:38.960620 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed
2021/11/30 10:42:39.963961 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:39.963962 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed
2021/11/30 10:42:40.955093 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to eapol key m1 retransmit failure. Max retries for M1 over
2021/11/30 10:42:40.955422 {wncd_x_R0-0}{1}: [client-orch-sm] [20392]: (note): MAC: 8866.5a45.400b  Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_EXCLUDE_WRONG_PSK, fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|01|07|13|1a|23|
2021/11/30 10:42:40.955476 {wncd_x_R0-0}{1}: [client-orch-sm] [20392]: (note): MAC: 8866.5a45.400b  Delete mobile payload sent forbssid: b811.4b5a.e600 WTP mac: b811.4b5a.e600 slot id: 0 
2021/11/30 10:42:40.955482 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_DELETE_IN_PROGRESS
2021/11/30 10:42:40.955559 {wncd_x_R0-0}{1}: [mm-client] [20392]: (ERR): MAC: 8866.5a45.400b  Client not present in DB. Responding to CO with Delete Ack
2021/11/30 10:42:40.955576 {wncd_x_R0-0}{1}: [sanet-shim-translate] [20392]: (note): MAC: 8866.5a45.400b  Session manager disconnect event called, session label: 0xd10003a3
2021/11/30 10:42:40.955710 {wncd_x_R0-0}{1}: [epm-misc] [20392]: (ERR): [0000.0000.0000:unknown] auth mgr get vn called
2021/11/30 10:42:40.955717 {wncd_x_R0-0}{1}: [epm-misc] [20392]: (ERR): [0000.0000.0000:unknown] misc_plugin_get_vn: session_hdl invalid
2021/11/30 10:42:40.955798 {wncd_x_R0-0}{1}: [svm] [20392]: (ERR): SVM-ERR: SVM wlan apply cb: session ctx missing
2021/11/30 10:42:40.955901 {wncd_x_R0-0}{1}: [auth-mgr] [20392]: (ERR): [8866.5a45.400b:capwap_90000010] Failed to search/create timer main rec while timer stop
2021/11/30 10:42:40.955983 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_DELETE_IN_PROGRESS -> S_CO_DELETED

Thanks in advance.

Jaroslav Ksinan · ‎04-25-2023

Hello,

Actually yes, we were able to fix the issue by coincidence when trying something else.

The "wrong PSK" issue stopped occurring when I enabled 802.11r BSS Fast Transition on this WPA2/PSK WLAN.

It stopped even when set to "Adaptive" but then some old Android couldn't connect, so after setting up FT to "Enabled" (and ticking both "PSK" and "FT-PSK" as AKM), both Android was able to associate and macOS devices stopped having an issue with wrong PSK.

I still don't understand why it helped since afaik FT doesn't have anything to do with the actual PSK passphrase (and FT is not really that crucial in WPA2/PSK), and also it's FT "Disabled" that should provide maximum client compatibility. However, in my case, it's the other way around and FT "Enabled" made all clients able to connect

Give it a try and you might be surprised like I was.

billydailey · ‎06-05-2023

if i am reading this correctly, certain phones were having problems connecting to the APs? I am having the same issue now. I am using a 9800L and 9136 and 1852 aps. I think droids were connecting fine along with PCs. But once i set up policys and tags, the phones are staying in an IP learn state and never connect.

Jaroslav Ksinan · ‎06-05-2023

Hi,

all phones were working fine (both iPhones and Androids), it was just MacBooks having issues after migration from AireOS WLC to C9800.

How do you handle DHCP for clients? Are you using Cisco best practices for C9800 (https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#DHCPproxy) i.e. DHCP bridging = DHCP IP helper configured on L3 interface?

What does the client's radioactive trace show regarding IP_LEARN state and disconnection?

billydailey · ‎06-06-2023

Ive attached the radioactive trace. Had to look that up, never used that before. Pretty neat. We are using IPv6 Dhcp on a 4351 with a switching module connecting to the WLC for DHCP.

So the included trace is from a droid device. I just connected 2 iphones and they pull an IPv6 address and connect. Very odd, at one point, droids worked and iphones didnt. Doesnt make sense to me, but i am new to the 9800L.

Any thoughts?

marce1000 · ‎06-07-2023

>.... I have attached the radioactive trace.
Below you will find the output of the radioactive trace when processed with : https://cway.cisco.com/wireless-debug-analyzer/ (Show All flag was checked).

TimeTaskTranslated

Connection attempt #1
Connection attempt #2
2023/06/06 09:56:19.944	client-orch-sm	Client made a new Association to an AP/BSSID: BSSID 687d.b45f.686f, WLAN MCS-Site-1, Slot 1 AP 687d.b45f.6860, 9136I-B
2023/06/06 09:56:19.944	dot11	Association success for client, assigned AID is: 1. Client performed fast roam.
2023/06/06 09:56:19.945	client-orch-sm	Client started layer 2 authentication (either dot1X or PSK)
2023/06/06 09:56:19.950	client-keymgmt	Sent M1 for EAPOL 4-Way Handshake
2023/06/06 09:56:20.024	client-keymgmt	Received and validated M2 for EAPOL 4-Way Handshake
2023/06/06 09:56:20.024	client-keymgmt	Sent M3 for EAPOL 4-Way Handshake
2023/06/06 09:56:20.034	client-keymgmt	Received and validated M4 for EAPOL 4-Way Handshake
2023/06/06 09:56:20.034	client-keymgmt	Negotiated the following encryption mechanism: AKM:FT-PSK Cipher:CCMP WPA Version: WPA2
2023/06/06 09:56:20.034	client-auth	Client successfully completed Pre-shared Key authentication. Assigned VLAN: 301
2023/06/06 09:56:20.034	client-orch-sm	Client passed layer 2 authentication
2023/06/06 09:56:20.034	client-orch-state	Starting Mobility Anchor discovery for client
2023/06/06 09:56:20.036	avc-afc	AVC is enabled for the client session
2023/06/06 09:56:20.036	client-orch-state	Entering IP learn state
2023/06/06 09:56:20.999	client-iplearn

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

billydailey · ‎06-07-2023

Thank you! Any diagnosis from that? Thats were it stays in iplearn before it is then moved to excluded.

billydailey · ‎06-12-2023

Following up, any diagnosis from this?