Clients disconnect and are unable to re-connect.

vpavlovwa · ‎08-14-2023

Could you please help me figure out this problem? The clients are disconnected and unable to re-connect. Supposedly nothing has been updated, changed or reconfigured.

WLC Log:

---------------Show msglog---------------

Message Log Severity Level ...................... ERROR
*Dot1x_NW_MsgTask_5: Aug 11 06:08:05.457: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:31:15 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_5: Aug 11 06:07:47.445: %DOT1X-3-AAA_AUTH_SEND_FAIL: [PA]1x_aaa.c:848 Unable to send AAA message for client 70:bc:10:5c:31:15
*dot1xMsgTask: Aug 11 05:51:22.404: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client 58:91:cf:e6:01:d6
*spamApTask5: Aug 11 05:11:40.116: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 08:cc:68:90:4b:20
*dot1xMsgTask: Aug 11 05:00:47.517: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 11 04:50:45.627: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client 58:91:cf:e6:01:d6
*dot1xMsgTask: Aug 11 04:00:07.737: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 11 03:50:00.733: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client 58:91:cf:e6:01:d6
*spamApTask1: Aug 11 03:22:19.692: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 70:10:5c:b9:b7:c0
*dot1xMsgTask: Aug 11 02:59:13.848: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 11 02:49:49.959: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client 58:91:cf:e6:01:d6
*dot1xMsgTask: Aug 10 23:57:42.361: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*Dot1x_NW_MsgTask_5: Aug 10 23:55:19.868: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 4c:03:4f:a5:04:dd Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*dot1xMsgTask: Aug 10 22:56:58.476: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 10 21:56:25.698: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 10 20:56:18.802: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 10 19:46:41.026: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client 58:91:cf:e6:01:d6
*spamApTask1: Aug 10 19:41:39.158: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 70:10:5c:b9:b7:c0
*Dot1x_NW_MsgTask_7: Aug 10 19:19:49.022: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 00:28:f8:99:89:5f Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*dot1xMsgTask: Aug 10 18:55:24.138: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8
*dot1xMsgTask: Aug 10 18:46:27.138: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client 58:91:cf:e6:01:d6
*Dot1x_NW_MsgTask_3: Aug 10 18:20:18.841: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:1a:c3 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_3: Aug 10 17:40:36.825: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:1a:c3 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_3: Aug 10 17:40:03.848: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:1a:c3 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_7: Aug 10 17:17:55.583: %DOT1X-3-INVALID_REPLAY_CTR: [PA]1x_eapkey.c:452 Invalid replay counter from client f4:26:79:b4:11:17 - got 00 00 00 00 00 00 00 01, expected 00 00 00 00 00 00 00 02
*spamApTask2: Aug 10 17:14:55.954: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 08:cc:68:62:bf:70
*Dot1x_NW_MsgTask_0: Aug 10 16:59:40.964: %DOT1X-3-INVALID_REPLAY_CTR: [PA]1x_eapkey.c:452 Invalid replay counter from client 36:83:b3:bd:1a:78 - got 00 00 00 00 00 00 00 04, expected 00 00 00 00 00 00 00 05
*spamApTask6: Aug 10 16:47:47.591: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 15, count 1 from AP dc:a5:f4:9c:12:90
*spamApTask2: Aug 10 16:38:56.079: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 08:cc:68:62:bf:70
*Dot1x_NW_MsgTask_5: Aug 10 16:35:56.352: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:20:d5 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_1: Aug 10 16:34:53.554: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client c8:34:8e:63:7e:71 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_5: Aug 10 16:31:33.872: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:20:d5 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*spamApTask6: Aug 10 16:21:43.360: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 08:cc:68:62:c0:a0
*spamApTask2: Aug 10 16:14:02.047: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 70:10:5c:b9:b8:70
*spamApTask6: Aug 10 16:08:54.742: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 50:06:04:ba:00:60
*Dot1x_NW_MsgTask_3: Aug 10 16:05:52.286: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 70:bc:10:5c:20:d3 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*spamApTask0: Aug 10 16:04:35.084: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 20:bb:c0:5b:fb:d0
*spamApTask6: Aug 10 15:59:43.360: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 08:cc:68:62:c0:a0
*Dot1x_NW_MsgTask_7: Aug 10 15:51:16.583: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client a0:4a:5e:d3:36:e7 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*spamApTask6: Aug 10 15:49:43.474: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 1 from AP 08:cc:68:62:c0:a0
*Dot1x_NW_MsgTask_2: Aug 10 15:25:55.936: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client b8:31:b5:98:f1:a2 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_2: Aug 10 15:25:36.215: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client b8:31:b5:98:f1:a2 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_2: Aug 10 15:25:17.365: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client b8:31:b5:98:f1:a2 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_2: Aug 10 15:24:59.323: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client b8:31:b5:98:f1:a2 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_2: Aug 10 15:21:36.434: %DOT1X-3-INVALID_REPLAY_CTR: [PA]1x_eapkey.c:452 Invalid replay counter from client 7e:55:4a:2e:0a:c2 - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03
*spamApTask6: Aug 10 15:19:38.225: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 4, count 3 from AP 08:cc:68:62:c0:a0
*Dot1x_NW_MsgTask_1: Aug 10 15:04:20.421: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client 12:d5:0f:f8:00:f9 Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*Dot1x_NW_MsgTask_5: Aug 10 15:01:56.115: %DOT1X-3-ABORT_AUTH: [PA]1x_bauth_sm.c:487 Authentication Aborted for client c4:03:a8:f5:57:fd Abort Reason:DOT1X RESTARTED DUE TO EAPOL-START/CLIENT ROAM
*spamApTask3: Aug 10 14:52:18.035: %LWAPP-3-REPLAY_ERR: [PA]spam_lrad.c:45310 The system has received replay error on slot 0, WLAN ID 0, count 1 from AP b4:e9:b0:b3:d9:40
*dot1xMsgTask: Aug 10 14:51:30.790: %DOT1X-3-WPA_SEND_STATE_ERR: [PA]1x_kxsm.c:1724 Unable to send EAPOL-key msg - invalid WPA state (0) - client d8:f8:83:c7:cb:f8

Client Debug Log:

(Cisco Controller) >debug mobility handoff enable

(Cisco Controller) >debug aaa all enable

(Cisco Controller) >*apfReceiveTask: Aug 11 13:27:14.895: [PA] Sending Mobile Announce for client AA:BB:CC:DD:EE:FF

*aaaQueueReader: Aug 11 13:56:39.864: [PA] Request Authenticator 2e:53:a6:4a:5f:2a:1b:14:c3:cb:da:86:45:ba:bb:e0
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=45, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=203, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=73, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=28, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=24, vId=14179)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=45, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=203, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=73, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=28, vId=9)
*radiusTransportThread: Aug 11 13:56:39.876: [PA] Vendor Specif Radius Attribute(code=26, avp_len=24, vId=14179)
*apfReceiveTask: Aug 11 13:56:39.878: [PA] Sending Mobile Announce for client ba:36:91:9e:3e:c6

*aaaQueueReader: Aug 11 13:56:41.486: [PA] Request Authenticator 55:ac:24:33:b6:c7:d8:8e:94:2a:40:85:f9:2d:c2:3b
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=45, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=203, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=73, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=28, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=24, vId=14179)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=45, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=203, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=73, vId=9)
*radiusTransportThread: Aug 11 13:56:41.496: [PA] Vendor Specif Radius Attribute(code=26, avp_len=28, vId=9)
*radiusTransportThread: Aug 11 13:56:41.497: [PA] Vendor Specif Radius Attribute(code=26, avp_len=24, vId=14179)
*radiusTransportThread: Aug 11 13:56:54.958: [PA] Vendor Specif Radius Attribute(code=26, avp_len=58, vId=311)
*radiusTransportThread: Aug 11 13:56:54.958: [PA] Vendor Specif Radius Attribute(code=26, avp_len=58, vId=311)
*radiusTransportThread: Aug 11 13:56:54.958: [PA] Vendor Specif Radius Attribute(code=26, avp_len=26, vId=14179)
*radiusTransportThread: Aug 11 13:56:54.958: [PA] Vendor Specif Radius Attribute(code=26, avp_len=58, vId=311)
*radiusTransportThread: Aug 11 13:56:54.958: [PA] Vendor Specif Radius Attribute(code=26, avp_len=58, vId=311)
*radiusTransportThread: Aug 11 13:56:54.959: [PA] Vendor Specif Radius Attribute(code=26, avp_len=26, vId=14179)

The Wireless Debug Analyzer keeps giving me an error and asking to try again.

balaji.bandi · ‎08-14-2023

what Radius Server you using also give more information about your environmet

WLC Version - Model

what AP model ?

what kind of clients ? Windows ?

check is the AAA Servers reachable from WLC ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

vpavlovwa · ‎08-14-2023

Hi Balaji,

it is a 5508 with 8.5.171.0 software.

APs - I am not sure, checking.

The clients having a problem are all Android phones and iPhones.

Radius is pingable from the WLC.

The corporate SSID is not affected. The issue is only happening on the guest SSID with customers who disconnect and try to re-connect, or with new customer. Customers who are already connected to the guest SSID have no issue.

Leo Laohoo · ‎08-14-2023

Certificate issue (expired).

vpavlovwa · ‎08-15-2023

Thank you Leo, I will check the certificates.

marce1000 · ‎08-15-2023

- As per https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#toc-hId-937503882 use this recommended release :
https://software.cisco.com/download/specialrelease/2702eede2b47a5c3bb40795bbe836af6
(aireos controllers are getting older ; use last release available for the particular product)

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

vpavlovwa · ‎08-15-2023

Thank you Marce, I will have the customer download the 8.5.182.11 firmware.

balaji.bandi · ‎08-15-2023

in this case as suggested check the certificate expiry, check the Controller Logs and enable debug and post the logs here.

on the phones check the check is the device getting DHCP IP and check the Cert ?

you can also test using PC connecting to that SSID see what is the behavior and Cert expiry ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

vpavlovwa · ‎08-15-2023

Thank you Balaji,

I will check the certificates, enable debug and will post the logs. I will have a call with the customer tomorrow, so we can test PC connectivity, check certs and collect logs.

issam.niyamathullah@ncs.com.sg · ‎05-19-2024

have you managed to resolve this issue? if so, can explain how?

thosmore · ‎01-17-2024

I'm exploring a similar problem at the tiny non-profit I volunteer for: when a working phone is disconnected, reconnecting it to the same plug gets neither power nor (obviously) connection. When I supply power to the phone using a 48V adapter, it boots and ultimately displays an IP, but fails to connect. We're using Cisco 7960 phones, a Cisco 2800 series server, and a Catalyst 3750. My question: are those of you who see a certificate problem here basing that on behavior, or something you're seeing in the logs?