cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
515
Views
20
Helpful
6
Replies
Highlighted
Beginner

PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

Hello All,

 

We have been seeing a PSK issue across our enterprise on our guest wireless network [WPA2-AES][Auth(PSK)]. This is seen at our branch offices, which all use AP2802I-B-K9 APs on 2504 WLC 8.3.121.7 locally that failover to our core 5508 controller running the same version. The same PSK for this guest wireless network is configured on each local controller. When a client attempts to connect, they begin the association process to an AP and WLAN, however, when they enter the PSK, they are prompted on their devices that the password for the network is incorrect. The error is seen across multiple clients, but mainly iOS devices of various versions and Windows 10 PCs. Clients become disassociated on the controller, we then have no visibility to their policy manager state. The only temporary solution that we found is just re-entering the PSK on the guest network Layer 2 security at locations that are having this issue, this resolves the problem 75% of the time. This is how clients who fail to connect appear in logs:

 

*Dot1x_NW_MsgTask_5: Jan 04 10:59:21.157: %DOT1X-3-PSK_CONFIG_ERR: 1x_ptsm.c:730 Client aa:bb:cc:dd:ee:ff may be using an incorrect PSK
*Dot1x_NW_MsgTask_5: Jan 04 10:58:58.357: %DOT1X-3-PSK_CONFIG_ERR: 1x_ptsm.c:730 Client aa:bb:cc:dd:ee:ff may be using an incorrect PSK

 

This message also appears a lot in logging:

 

*Dot1x_NW_MsgTask_4: Jan 04 08:28:11.923: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client aa:bb:cc:dd:ee:ff - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03
*Dot1x_NW_MsgTask_1: Jan 04 08:24:09.545: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client aa:bb:cc:dd:ee:ff - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03

 

Similar logs are found across all branches. We have adjusted our EAP timers to this, but the same issue still periodically occurs:

 

(Cisco Controller) show>advanced eap

EAP-Identity-Request Timeout (seconds)........... 30
EAP-Identity-Request Max Retries................. 2
EAP Key-Index for Dynamic WEP.................... 0
EAP Max-Login Ignore Identity Response........... enable
EAP-Request Timeout (seconds).................... 30
EAP-Request Max Retries.......................... 2
EAPOL-Key Timeout (milliseconds)................. 1000
EAPOL-Key Max Retries............................ 2
EAP-Broadcast Key Interval....................... 3600

 

Here is a capture of an affected client:

 

*Dot1x_NW_MsgTask_3: Jan 04 10:16:07.769: aa:bb:cc:dd:ee:ff Starting key exchange to mobile aa:bb:cc:dd:ee:ff, data packets will be dropped

*Dot1x_NW_MsgTask_3: Jan 04 10:16:07.769: aa:bb:cc:dd:ee:ff Sending EAPOL-Key Message to mobile aa:bb:cc:dd:ee:ff

state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00

*Dot1x_NW_MsgTask_3: Jan 04 10:16:07.770: aa:bb:cc:dd:ee:ff Allocating EAP Pkt for retransmission to mobile aa:bb:cc:dd:ee:ff

*osapiBsnTimer: Jan 04 10:16:08.798: aa:bb:cc:dd:ee:ff 802.1x 'timeoutEvt' Timer expired for station aa:bb:cc:dd:ee:ff and for message = M2

*Dot1x_NW_MsgTask_3: Jan 04 10:16:08.799: aa:bb:cc:dd:ee:ff Retransmit 1 of EAPOL-Key M1 (length 121) for mobile aa:bb:cc:dd:ee:ff

*osapiBsnTimer: Jan 04 10:16:09.799: aa:bb:cc:dd:ee:ff 802.1x 'timeoutEvt' Timer expired for station aa:bb:cc:dd:ee:ff and for message = M2

*Dot1x_NW_MsgTask_3: Jan 04 10:16:09.799: aa:bb:cc:dd:ee:ff Retransmit 2 of EAPOL-Key M1 (length 121) for mobile aa:bb:cc:dd:ee:ff

*osapiBsnTimer: Jan 04 10:16:10.798: aa:bb:cc:dd:ee:ff 802.1x 'timeoutEvt' Timer expired for station aa:bb:cc:dd:ee:ff and for message = M2

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Retransmit failure for EAPOL-Key M1 to mobile aa:bb:cc:dd:ee:ff, retransmit count 3, mscb deauth count 0

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Resetting MSCB PMK Cache Entry 0 for station aa:bb:cc:dd:ee:ff

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Removing BSSID 70:79:b3:4c:13:ce from PMKID cache of station aa:bb:cc:dd:ee:ff

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Setting active key cache index 0 ---> 8

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Sent Deauthenticate to mobile on BSSID bb:bb:bb:bb:bb:ce slot 1(caller 1x_ptsm.c:656)

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Setting active key cache index 8 ---> 8

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Deleting the PMK cache when de-authenticating the client.

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Global PMK Cache deletion failed.

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Scheduling deletion of Mobile Station: (callerId: 57) in 10 seconds

*Dot1x_NW_MsgTask_3: Jan 04 10:16:10.799: aa:bb:cc:dd:ee:ff Freeing EAP Retransmit Bufer for mobile aa:bb:cc:dd:ee:ff

*osapiBsnTimer: Jan 04 10:16:20.799: aa:bb:cc:dd:ee:ff apfMsExpireCallback (apf_ms.c:638) Expiring Mobile!

*apfReceiveTask: Jan 04 10:16:20.799: aa:bb:cc:dd:ee:ff apfMsExpireMobileStation (apf_ms.c:7657) Changing state for mobile aa:bb:cc:dd:ee:ff on AP bb:bb:bb:bb:bb:c0 from Associated to Disassociated

 

*apfReceiveTask: Jan 04 10:16:20.799: aa:bb:cc:dd:ee:ff Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds

*osapiBsnTimer: Jan 04 10:16:30.798: aa:bb:cc:dd:ee:ff apfMsExpireCallback (apf_ms.c:638) Expiring Mobile!

*apfReceiveTask: Jan 04 10:16:30.799: aa:bb:cc:dd:ee:ff apfMsAssoStateDec

*apfReceiveTask: Jan 04 10:16:30.799: aa:bb:cc:dd:ee:ff apfMsWepPskStateDec

*apfReceiveTask: Jan 04 10:16:30.799: aa:bb:cc:dd:ee:ff apfMsExpireMobileStation (apf_ms.c:7793) Changing state for mobile aa:bb:cc:dd:ee:ff on AP bb:bb:bb:bb:bb:c0 from Disassociated to Idle

 

*apfReceiveTask: Jan 04 10:16:30.799: aa:bb:cc:dd:ee:ff pemApfDeleteMobileStation2: APF_MS_PEM_WAIT_L2_AUTH_COMPLETE = 0.

*apfReceiveTask: Jan 04 10:16:30.799: aa:bb:cc:dd:ee:ff 0.0.0.0 START (0) Deleted mobile LWAPP rule on AP [bb:bb:bb:bb:bb:c0]

*apfReceiveTask: Jan 04 10:16:30.799: aa:bb:cc:dd:ee:ff Deleting mobile on AP bb:bb:bb:bb:bb:c01)

*spamApTask2: Jan 04 10:16:30.800: aa:bb:cc:dd:ee:ff Delete Mobile request sent to the AP 192.168.***.***:5272

 

What we are thinking is that when branch office APs failover to our core, then associate back to their local WLC, clients then attempt to connect to the network using the same PSK but the key rotation is not synced. We have a script that is configured such that if the Internet connection at the branch office is detected down, we failover the APs primary WLC from the local controller to our core headquarters controller, the 5508. This way client data from the APs is tunneled to the 5508, which has AP groups and interfaces for each individual branch WLC. DHCP for this network is configured locally on each controller, but when failover occurs to our core we pull DHCP from dedicated wireless pools on a core switch. 

 

We have not been able to replicate this just yet in a lab environment. Before we upgrade to an 8.5 WLC version, we want to confirm if there is an error in our config that is causing this. I attached the WLAN Config as well. Please let me know if anybody needs more info or has any solutions, thanks

 

Everyone's tags (8)
6 REPLIES 6
VIP Mentor

Re: PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

Before asking more detail..

 

did you enable fast ssid on wlc ?  if not then do it and try again:

https://rscciew.wordpress.com/2014/06/07/fast-ssid-change/

 

also paste the sh wlan <id> output here.

 

Regards

Dont forget to rate helpful posts

Beginner

Re: PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

Yes, Fast SSID Change is enabled on the controllers. sh wlan is the attached text file. I would also like to specify to everyone that this issue is seen across multiple client devices, usually Win 10 and iOS

VIP Advocate

Re: PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

Try to disable this (under SSID - Advanced):
DHCP Address Assignment Required................. Enabled
That should at least fix some roaming issues you might have once an outage happens/recovers. If you don't have any issues, leave it enabled.

Is the mobility path UP on the affected controllers?
Beginner

Re: PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

Mobility path is up, but the DHCP required setting is something interesting to look into, I have seen clients with 0.0.0.0 IP addresses and figured it was attributed to that setting. I will try this out, any ideas on EAP parameters and timers?

VIP Advocate

Re: PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

I had issues in the past, with several active WLCs and DHCP Required Enabled. For some reason, the client IP Address wasn't always communicated through the Mobility Path (I think when the device was put to sleep while on WLC1 moved around and then waked up on WLC2). In this situation the user needed to disable and re-enable his Wi-Fi to get the connection working again. Once I had DHCP Required disabled, this started to working fine.

 

Leave the EAP timers on default, usually they are fine. It would look different if you used Username/Password for authentication, then some tuning is sometimes needed.

Hall of Fame Master

Re: PSK Issues on 2504 WLC 8.3.121.7 [WPA2-AES][Auth(PSK)]

Well he is my thought. Your design isn’t right for failover. If you are using a local wlc and then the backup is a centralized wlc, that is your issue. That design should probably be FlexConnect local switch, AP and wlan.  This way the dhcp is always local to the client. The controllers don’t communicate with each other except for roaming with the assumption that clients are placed on the same subnet. PSK issue I have not seen as long as the wlan settings and PSK entries are identical. 

-Scott
*** Please rate helpful posts ***
CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards