C9800 exclusion due to wrong PSK - macOS clients only

[Jaroslav Ksinan]

Hello,

We're migrating APs from old AireOS 2504 WLC to C9800-CL (running on 17.3.3) and when APs are migrated to C9800, macOS clients are unable to connect to WPA2 SSID with PSK authentication (FlexConnect local switching). Windows laptops and phones work like before with the same PSK, so clients definitely use correct password.

I checked bug search tool, didn't find anything relevant. We removed SSID from known networks on client side and joined SSID as new connection but that also didn't help.

Perhaps somebody faced the same and have some ideas what can be causing this. Any idea is greatly appreciated.

WLAN setup:

WLC-P-01#sh wlan name WIFI_PSK
WLAN Profile Name     : WIFI_PSK
================================================
Identifier                                     : 1
Description                                    :
Network Name (SSID)                            : WIFI_PSK
Status                                         : Enabled
Broadcast SSID                                 : Enabled
Advertise-Apname                               : Disabled
Universal AP Admin                             : Disabled
Max Associated Clients per WLAN                : 0
Max Associated Clients per AP per WLAN         : 0
Max Associated Clients per AP Radio per WLAN   : 200
OKC                                            : Enabled
Number of Active Clients                       : 0
CHD per WLAN                                   : Enabled
WMM                                            : Allowed
WiFi Direct Policy                             : Disabled
Channel Scan Defer Priority:
  Priority (default)                           : 5
  Priority (default)                           : 6
Scan Defer Time (msecs)                        : 100
Media Stream Multicast-direct                  : Disabled
CCX - AironetIe Support                        : Disabled
Peer-to-Peer Blocking Action                   : Disabled
Radio Policy                                   : 802.11a and 802.11g only
DTIM period for 802.11a radio                  :
DTIM period for 802.11b radio                  :
Local EAP Authentication                       : Disabled
Mac Filter Authorization list name             : Disabled
Mac Filter Override Authorization list name    : Disabled
Accounting list name                           :
802.1x authentication list name                : Disabled
802.1x authorization list name                 : Disabled
Security
    802.11 Authentication                      : Open System
    Static WEP Keys                            : Disabled
    Wi-Fi Protected Access (WPA/WPA2/WPA3)     : Enabled
        WPA (SSN IE)                           : Disabled
        WPA2 (RSN IE)                          : Enabled
            MPSK                               : Disabled
            AES Cipher                         : Enabled
            CCMP256 Cipher                     : Disabled
            GCMP128 Cipher                     : Disabled
            GCMP256 Cipher                     : Disabled
            Randomized GTK                     : Disabled
        WPA3 (WPA3 IE)                         : Disabled
        Auth Key Management
            802.1x                             : Disabled
            PSK                                : Enabled
            CCKM                               : Disabled
            FT dot1x                           : Disabled
            FT PSK                             : Disabled
            Dot1x-SHA256                       : Disabled
            PSK-SHA256                         : Disabled
            SAE                                : Disabled
            OWE                                : Disabled
            SUITEB-1X                          : Disabled
            SUITEB192-1X                       : Disabled
    CCKM TSF Tolerance (msecs)                 : 1000
    OWE Transition Mode                        : Disabled
    OSEN                                       : Disabled
    FT Support                                 : Disabled
        FT Reassociation Timeout (secs)        : 20
        FT Over-The-DS mode                    : Disabled
    PMF Support                                : Disabled
        PMF Association Comeback Timeout (secs): 1
        PMF SA Query Time (msecs)              : 200
    Web Based Authentication                   : Disabled
    Conditional Web Redirect                   : Disabled
    Splash-Page Web Redirect                   : Disabled
    Webauth On-mac-filter Failure              : Disabled
    Webauth Authentication List Name           : Disabled
    Webauth Authorization List Name            : Disabled
    Webauth Parameter Map                      : Disabled
Band Select                                    : Enabled
Load Balancing                                 : Disabled
Multicast Buffer                               : Disabled
Multicast Buffers (frames)                     : 0
IP Source Guard                                : Disabled
Assisted-Roaming
    Neighbor List                              : Enabled
    Prediction List                            : Disabled
    Dual Band Support                          : Disabled
IEEE 802.11v parameters
    Directed Multicast Service                 : Enabled
    BSS Max Idle                               : Enabled
        Protected Mode                         : Disabled
    Traffic Filtering Service                  : Disabled
    BSS Transition                             : Disabled
        Disassociation Imminent                : Disabled
            Optimised Roaming Timer (TBTTS)    : 40
            Timer (TBTTS)                      : 200
        Dual Neighbor List                     : Disabled
    WNM Sleep Mode                             : Disabled
802.11ac MU-MIMO                               : Enabled
802.11ax parameters
    OFDMA Downlink                             : Enabled
    OFDMA Uplink                               : Enabled
    MU-MIMO Downlink                           : Enabled
    MU-MIMO Uplink                             : Enabled
    BSS Target Wake Up Time                    : Enabled
    BSS Target Wake Up Time Broadcast Support  : Enabled
mDNS Gateway Status                            : Bridge
WIFI Alliance Agile Multiband                  : Disabled
Device Analytics
    Advertise Support                          : Enabled
    Share Data with Client                     : Disabled
Client Scan Report (11k Beacon Radio Measurement)
    Request on Association                     : Disabled
    Request on Roam                            : Disabled
WiFi to Cellular Steering                      : Disabled

WLC logs:

Nov 30 13:55:41.068: %CLIENT_EXCLUSION_SERVER-5-ADD_TO_BLACKLIST_REASON_DYNAMIC: Chassis 1 R0/0: wncmgrd: Client MAC: 6692.f9bc.0f0d was added to exclusion list associated with AP Name:ap-portu-flex-2, BSSID:MAC: b811.4b5a.e60f, reason:Wrong PSK

Nov 30 13:55:27.821: %CLIENT_EXCLUSION_SERVER-5-ADD_TO_BLACKLIST_REASON_DYNAMIC: Chassis 1 R0/0: wncmgrd: Client MAC: 3c06.301a.8e54 was added to exclusion list associated with AP Name:ap-portu-flex-2, BSSID:MAC: b811.4b5a.e60f, reason:Wrong PSK

I also ran a Radioactive Trace, example for one client:

2021/11/30 10:42:37.929244 {wncd_x_R0-0}{1}: [client-orch-sm] [20392]: (note): MAC: 8866.5a45.400b  Association received. BSSID b811.4b5a.e600, WLAN WIFI_PSK, Slot 0 AP b811.4b5a.e600, ap-portu-flex-2
2021/11/30 10:42:37.929350 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2021/11/30 10:42:37.929570 {wncd_x_R0-0}{1}: [dot11] [20392]: (note): MAC: 8866.5a45.400b  Association success. AID 1, Roaming = False, WGB = False, 11r = False, 11w = False 
2021/11/30 10:42:37.929644 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_ASSOCIATING -> S_CO_L2_AUTH_IN_PROGRESS
2021/11/30 10:42:37.929657 {wncd_x_R0-0}{1}: [client-auth] [20392]: (note): MAC: 8866.5a45.400b  L2 Authentication initiated. method PSK, Policy VLAN 132,AAA override = 0, NAC = 0
2021/11/30 10:42:37.929670 {wncd_x_R0-0}{1}: [sanet-shim-translate] [20392]: (ERR): 8866.5a45.400b  wlan_profile Not Found : Device information attributes not populated
2021/11/30 10:42:37.930166 {wncd_x_R0-0}{1}: [epm] [20392]: (ERR): [0000.0000.0000:unknown] HDL = 0x0 Vlan info not found for vlan id 132
2021/11/30 10:42:37.930427 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [20392]: (note): Authentication Success. Resolved Policy bitmap:11 for client 8866.5a45.400b 
2021/11/30 10:42:37.930489 {wncd_x_R0-0}{1}: [client-auth] [20392]: (note): MAC: 8866.5a45.400b  ADD MOBILE sent. Client state flags: 0x1  BSSID: MAC: b811.4b5a.e600  capwap IFID: 0x90000010
2021/11/30 10:42:37.960637 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:37.960638 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed
2021/11/30 10:42:38.960620 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:38.960620 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed
2021/11/30 10:42:39.963961 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:39.963962 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed
2021/11/30 10:42:40.955093 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to eapol key m1 retransmit failure. Max retries for M1 over
2021/11/30 10:42:40.955422 {wncd_x_R0-0}{1}: [client-orch-sm] [20392]: (note): MAC: 8866.5a45.400b  Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_EXCLUDE_WRONG_PSK, fsm-state transition 00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|00|01|07|13|1a|23|
2021/11/30 10:42:40.955476 {wncd_x_R0-0}{1}: [client-orch-sm] [20392]: (note): MAC: 8866.5a45.400b  Delete mobile payload sent forbssid: b811.4b5a.e600 WTP mac: b811.4b5a.e600 slot id: 0 
2021/11/30 10:42:40.955482 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_DELETE_IN_PROGRESS
2021/11/30 10:42:40.955559 {wncd_x_R0-0}{1}: [mm-client] [20392]: (ERR): MAC: 8866.5a45.400b  Client not present in DB. Responding to CO with Delete Ack
2021/11/30 10:42:40.955576 {wncd_x_R0-0}{1}: [sanet-shim-translate] [20392]: (note): MAC: 8866.5a45.400b  Session manager disconnect event called, session label: 0xd10003a3
2021/11/30 10:42:40.955710 {wncd_x_R0-0}{1}: [epm-misc] [20392]: (ERR): [0000.0000.0000:unknown] auth mgr get vn called
2021/11/30 10:42:40.955717 {wncd_x_R0-0}{1}: [epm-misc] [20392]: (ERR): [0000.0000.0000:unknown] misc_plugin_get_vn: session_hdl invalid
2021/11/30 10:42:40.955798 {wncd_x_R0-0}{1}: [svm] [20392]: (ERR): SVM-ERR: SVM wlan apply cb: session ctx missing
2021/11/30 10:42:40.955901 {wncd_x_R0-0}{1}: [auth-mgr] [20392]: (ERR): [8866.5a45.400b:capwap_90000010] Failed to search/create timer main rec while timer stop
2021/11/30 10:42:40.955983 {wncd_x_R0-0}{1}: [client-orch-state] [20392]: (note): MAC: 8866.5a45.400b  Client state transition: S_CO_DELETE_IN_PROGRESS -> S_CO_DELETED

Thanks in advance.

[patoberli]

Which APs are that? 

There are a few fixes in 17.3.4c (TAC recommended build) https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html for certain AP models that stop transmitting data, one specific to macbooks.

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-3/release-notes/rn-17-3-9800.html#Cisco_Reference.dita_f6d8b12b-79a3-4f0c-a101-eef2a38f91a1

[Jaroslav Ksinan]

Thanks for your answer and provided links, will check it.

Issue is currently observed on AIR-AP2802I-E-K9 that are already migrated, different APs to be migrated are AIR-CAP2702E-E-K9 and AIR-AP1832I-E-K9

Due to 2702E support we can't go above 17.3 train, so 17.3.4c could be worth a try.

[patoberli]

That reminds me, try to disable the session timeout (set it to 0) on the SSID. I seem to remember some issue with that option.

One more thing. Can you temporarily test with an Open unencrypted SSID to see if the macOS clients can connect there? I assume that will work.  

[Jaroslav Ksinan]

Good idea, I will definitely try also open SSID when macOS clients are on site.

Regarding session timeout, I've come across these settings recommendations - value 0 was recommended value in AireOS, but is definitely not a good idea on C9800 as per C9800 config best practices: https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html

These are the recommended values:

●      Depending on the deployment policies, a good value for the session timeout could be 7200 seconds (120 minutes); this is the minimum time before client reauthentication is enforced. Starting with Release 17.4 the default session timeout is set to 86400 seconds (24 hours) and has to be considered the new recommended value to apply to all releases.

Note:     In AireOS, a session timeout that is set to 0 (zero) means the maximum possible timeout. In the C9800, it actually means “no session timeout,” so if you use the same setting as in AireOS, every roam will require a full reauthentication.

●      Set the per-WLAN user idle timeout to 3600 seconds (60 minutes) to reduce the likelihood of client deletion when moving out of coverage areas or when the client is battery operated and may go to sleep frequently.

●      The exclusion timeout should be enabled, normally with exclusion set to 180 seconds (3 minutes).

So I've configured 86400 seconds for session timeout.

[patoberli]

I believe to remember that there was a bug in the past with macOS clients, if that value wasn't set to 0. It might have been fixed since, but it surely is worth a try, just in the case the bug isn't yet closed.

[Jaroslav Ksinan]

I'll give it a try when we have a chance to troubleshoot properly (it's now hard due to pandemic restrictions).

Btw. I performed upgrade to 17.3.4c but unfortunately that didn't resolve the issue. So if this is a bug, it's not yet fixed.

[Mark Elsen]

 - Below is the result from : https://cway.cisco.com/wireless-debug-analyzer/ for the Radio Active trace the you posted. Looks like the reply from @patoberli  may be advisable. You can also obtain an overall checkup of the 9800-configuration when issuing show tech wireless  on it and have that analyzed with : https://cway.cisco.com/tools/WirelessAnalyzer/

TimeTaskTranslated2021/11/30 10:42:37.929client-orch-smClient made a new Association to an AP/BSSID: BSSID b811.4b5a.e600, WLAN WIFI_PSK, Slot 0 AP b811.4b5a.e600, ap-portu-flex-22021/11/30 10:42:37.929dot11Association success for client, assigned AID is: 12021/11/30 10:42:37.960client-keymgmtCould not validate MIC received in M2 message2021/11/30 10:42:38.960client-keymgmtCould not validate MIC received in M2 message2021/11/30 10:42:39.963client-keymgmtCould not validate MIC received in M2 message2021/11/30 10:42:40.955client-keymgmtReached maximum retries for M12021/11/30 10:42:40.955client-orch-smController initiated client deletion with code: CO_CLIENT_DELETE_REASON_EXCLUDE_WRONG_PSK. Code means: Client provided wrong pre-shared key

[Jaroslav Ksinan]

Many thanks for useful links, I ran WirelessAnalyzer and got nice output from it, however no reported issue was relevant to my issue.

I got error messages regarding MIC validation also from Radioactive Trace (below), and this is the most confusing thing for me, because:

• I doublechecked and re-entered PSK passphrase on WLAN
• macOS clients that were working with same PSK passphrase on AireOS WLC are without any passphrase change unable to authenticate on C9800
• other clients seamlessly authenticated with same passphrase without any change on their side
• macOS clients deleted known SSID and reconnected to SSID as new connection, again making sure they're using correct passphrase

It doesn't make any sense to me why would MIC (which is a hash of passphrase + other stuff) differ only on macOS clients..

2021/11/30 10:42:37.960637 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol mic. MIC mismatch.
2021/11/30 10:42:37.960638 {wncd_x_R0-0}{1}: [client-keymgmt] [20392]: (ERR): MAC: 8866.5a45.400b  Keymgmt: Failed to validate eapol key m2. MIC validation failed

 It indeed seems like some kind of bug, I will probably have to try upgrading to latest bug-fix release.

[Piotr Gorka]

Hi Jaroslav 

Did you fix this problem? 

Regards

[TomDavid]

We just had this issue was an absolute nightmare - Cisco Support unable to sort 

We found that Configuration -> Security -> Wireless Protection Policies -> Client Exclusion Policies (Untick them all)

Then: Configuration -> Tags & Profiles -> Policy -> Click each one including the default one and set Client Exclusion Timeout (Sec) to 1

Do this on every Policy Name including the default one. 

Seems to have sorted ours, bit of a work around. 

[Jaroslav Ksinan]

Hi TomDavid,

thanks for your insights, I tried to change proposed settings but unfortunately with no luck.

[Jaroslav Ksinan]

Hi Piotr,

No, we didn't. Mostly because the affected macOS clients weren't really willing to cooperate in some testing so the customer IT guy told me that we don't care about this issue for now and we'll continue with troubleshooting when someone complains about it

So it's still an issue, I see client exclusions "due to wrong PSK" in WLC logs but they probably use different methods to access network and are okay with it.

[ammahend]

CO_CLIENT_DELETE_REASON_EXCLUDE_WRONG_PSK

If you think the password is correct. Set a different more complex password, try 15 character is possible. 
also share Radioactive trace for this user.

[mgonzalez15]

Hi everyone! Hi Jaroslav!

Do you found the solution with this issue? Could you fix it? Already I have the same issue but envolved all type of clients, the only thing that resolve the issue for now has to disable all the client exclusion polices. but that is not the root solution of the problem.

I would appreciate any support! Regards