Wireless barcode scanners dropping off the network momentarily

[dot1x]

Hi All,

We have WLC 7510 running v8.1.x. (We cannot upgrade as other sites have few old APs running on the same WLC)

Recently we encountered an issue with the wireless scanners dropping off the network momentarily.

We ran the logs and could see that there are some 'timeoutEvt' events (see below).

The SSID has WPA2 security.

*Dot1x_NW_MsgTask_5: Jan 07 08:28:28.115: [SA] aa:bb:cc:dd:ee:ff Reusing allocated memory for EAP Pkt for retransmission to mobile aa:bb:cc:dd:ee:ff
*osapiBsnTimer: Jan 07 08:28:49.260: [SA] aa:bb:cc:dd:ee:ff 802.1x 'timeoutEvt' Timer expired for station aa:bb:cc:dd:ee:ff and for message = M3
*Dot1x_NW_MsgTask_5: Jan 07 08:28:29.260: [SA] aa:bb:cc:dd:ee:ff Retransmit 1 of EAPOL-Key M3 (length 155) for mobile aa:bb:cc:dd:ee:ff
*osapiBsnTimer: Jan 07 08:28:30.280: [SA] aa:bb:cc:dd:ee:ff 802.1x 'timeoutEvt' Timer expired for station aa:bb:cc:dd:ee:ff and for message = M3
*Dot1x_NW_MsgTask_5: Jan 07 08:28:31.280: [SA] aa:bb:cc:dd:ee:ff Retransmit 2 of EAPOL-Key M3 (length 155) for mobile aa:bb:cc:dd:ee:ff
*osapiBsnTimer: Jan 07 08:28:31.300: [SA] aa:bb:cc:dd:ee:ff 802.1x 'timeoutEvt' Timer expired for station aa:bb:cc:dd:ee:ff and for message = M3
*Dot1x_NW_MsgTask_5: Jan 07 08:28:31.300: [SA] aa:bb:cc:dd:ee:ff Retransmit failure for EAPOL-Key M3 to mobile aa:bb:cc:dd:ee:ff, retransmit count 3, mscb deauth count 0
*Dot1x_NW_MsgTask_5: Jan 07 08:28:31.300: [SA] aa:bb:cc:dd:ee:ff Resetting MSCB PMK Cache Entry 0 for station aa:bb:cc:dd:ee:ff
*Dot1x_NW_MsgTask_5: Jan 07 08:28:31.300: [SA] aa:bb:cc:dd:ee:ff Removing BSSID 11:22:33:44:55:66 from PMKID cache of station aa:bb:cc:dd:ee:ff
*Dot1x_NW_MsgTask_5: Jan 07 08:28:31.300: [SA] aa:bb:cc:dd:ee:ff Setting active key cache index 0 ---> 8
*Dot1x_NW_MsgTask_5: Jan 07 08:28:31.300: [SA] aa:bb:cc:dd:ee:ff Sent Deauthenticate to mobile on BSSID 11:22:33:44:55:66 slot 0(caller 1x_ptsm.c:598)

[Leo Laohoo]

Based on the logs, these events happened on 07 January. What year was this?

[dot1x]

Hi Leo,

Thanks for the prompt response.
I changed the date/time/MAC Address on purpose.
It is an on-going issue.

[marce1000]

 - Have a sanity check of the controller configuration with https://cway.cisco.com/tools/WirelessAnalyzer/ , debugs can be analyzed with https://cway.cisco.com/tools/WirelessDebugAnalyzer/  - the restriction to not be able to test later controller release must be considered as  serious deadlock, often such issues are then resolved. 

 M.

[dot1x]

Hi Marce,

I've tried the wireless debug analyzer and found the following:

Aug 06 16:26:48.090 *Dot1x_NW_MsgTask_5 4-Way PTK Handshake, Sending M1
Aug 06 16:26:48.115 *Dot1x_NW_MsgTask_5 4-Way PTK Handshake, Received M2
Aug 06 16:26:48.115 *Dot1x_NW_MsgTask_5 4-Way PTK Handshake, Sending M3
Aug 06 16:26:49.260 *osapiBsnTimer 4-Way PTK Handshake, Client did not respond with M3
Aug 06 16:26:49.260 *Dot1x_NW_MsgTask_5 4-Way PTK Handshake, Retransmitting M3 retry #1
Aug 06 16:26:50.280 *osapiBsnTimer 4-Way PTK Handshake, Client did not respond with M3
Aug 06 16:26:50.280 *Dot1x_NW_MsgTask_5 4-Way PTK Handshake, Retransmitting M3 retry #2
Aug 06 16:26:51.300 *osapiBsnTimer 4-Way PTK Handshake, Client did not respond with M3
Aug 06 16:26:51.300 *Dot1x_NW_MsgTask_5 Client disassociation due to Authentication timeout. Auth or Key Exchange max-retransmissions reached. Check/update client driver, security config, certificates etc.
Aug 06 16:26:51.300 *Dot1x_NW_MsgTask_5 Client has been deauthenticated
Aug 06 16:26:51.300 *Dot1x_NW_MsgTask_5 Client expiration timer code set for 10 seconds. The reason: Roaming failed due to WLAN security policy mismatch between controllers (configuration error). It can also be used to report EAPoL retry errors, and GTK rotation failure (in 8.5)

We have already updated the client driver, still no luck.

The WLAN is running only WPA2 PSK.

Any thoughts?

Thanks.

[Leo Laohoo]

And what happens if the scanner is running on OPEN authentication?

[dot1x]

Hi Leo,

The client won't allow creating an open WLAN.

However, we ran some more debugs and found the following:

*spamApTask4: Sep 09 12:57:45.197: [SA] aa:bb:cc:dd:ee:ff Received DELETE mobile, reason MN_AP_AUTH_STOP, from AP aa:11:bb:22:cc:33, slot 0 ...cleaning up mscb
*spamApTask4: Sep 09 12:57:45.197: [SA] aa:bb:cc:dd:ee:ff Warning, ignore the DELETE_MOBILE_PAYLOAD from AP: aa:11:bb:22:cc:33, slot 0. STA connecting AP: dd:44:ee:55:ff:66, slot 0
*apfMsConnTask_2: Sep 09 12:57:56.150: [SA] aa:bb:cc:dd:ee:ff Processing assoc-req station:aa:bb:cc:dd:ee:ff AP:aa:11:bb:22:cc:33-00 thread:9d89cf0
*apfMsConnTask_2: Sep 09 12:57:56.150: [SA] aa:bb:cc:dd:ee:ff Association received from mobile on BSSID ab:cd:ef:12:34:56 AP ACCESSPOINTNAME

Any thoughts?

Thanks.

[Leo Laohoo]

Aug 06 16:26:48.115 *Dot1x_NW_MsgTask_5 4-Way PTK Handshake, Sending M3
Aug 06 16:26:49.260 *osapiBsnTimer 4-Way PTK Handshake, Client did not respond with M3

This is the reason why I'm asking if the barcode scanner can associate using OPEN authentication.  
Debug shows the barcode scanner stopped responding.

[dot1x]

Hi Leo,

We changed the EAP timeout and retries, after which we couldn’t see that error again.

But now we are having the following error: (both on the scanner and a laptop)

*spamApTask4: Sep 09 12:57:45.197: [SA] aa:bb:cc:dd:ee:ff Received DELETE mobile, reason MN_AP_AUTH_STOP, from AP aa:11:bb:22:cc:33, slot 0 ...cleaning up mscb
*spamApTask4: Sep 09 12:57:45.197: [SA] aa:bb:cc:dd:ee:ff Warning, ignore the DELETE_MOBILE_PAYLOAD from AP: aa:11:bb:22:cc:33, slot 0. STA connecting AP: dd:44:ee:55:ff:66, slot 0
*apfMsConnTask_2: Sep 09 12:57:56.150: [SA] aa:bb:cc:dd:ee:ff Processing assoc-req station:aa:bb:cc:dd:ee:ff AP:aa:11:bb:22:cc:33-00 thread:9d89cf0
*apfMsConnTask_2: Sep 09 12:57:56.150: [SA] aa:bb:cc:dd:ee:ff Association received from mobile on BSSID ab:cd:ef:12:34:56 AP ACCESSPOINTNAME

[Leo Laohoo]

What exact 8.1.X.X version?  I smell an expired certificate.

[dot1x]

8.1.102.0

I checked it’s deferred release.

The problem is that multiple sites are running off this controller and only one site reported this issue.

[Scott Fella]

First things first, you need to test either on one ap or in a lab with an open ssid to rule out any issues with the devices or possibly the code. If you start changing things around to hopefully get something to work, that’s the wrong thing to do. Look at it this way, if this was working for a very long time and all of a sudden it’s not, something in the environment changed. Was there a firmware push on the device or was the layout of the environment changed, maybe something was installed that is causing interference. Find out first if anything has changed at that site. I have seen warehouses change the layout of the racks or products and that has caused issues because the aps are now mounted in poor locations. It’s something you need to find out.

[rayyaanfayker0006]

hi

were you able to solve your issue?