Cisco Aironet randomly disconnects clients after 10 - 15 minutes

[supportgns]

Hi,

Currently we have a wireless environment with 18 APs and a WLC 2504 for a customer. Today our customer had a connectivity issue and told us several users disconnected randomly after 10 or 15 min being connected to the WiFi.

We did a first check and went to the logs, and this is what we found (I attach the logs between 9:30 and 12:00 because the problem occurred at that time):

*spamApTask2: Nov 15 12:05:04.008: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 1 from AP 00:c8:8b:ff:34:e0
*spamApTask0: Nov 15 11:41:11.489: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11856 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 00:c1:64:b3:80:30
*Dot1x_NW_MsgTask_7: Nov 15 11:23:22.179: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client c0:17:4d:f1:64:07 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_0: Nov 15 11:15:56.012: %DOT1X-3-INVALID_WPA_KEY_STATE: 1x_eapkey.c:2900 Received EAPOL-key message while in invalid state (4) - version 2, type 3, descriptor 2, client d0:c5:f3:89:a0:a0
*Dot1x_NW_MsgTask_0: Nov 15 11:15:56.012: %LOG-3-Q_IND: 1x_eapkey.c:452 Invalid replay counter from client d0:c5:f3:89:a0:a0 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_0: Nov 15 11:15:56.005: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client d0:c5:f3:89:a0:a0 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask3: Nov 15 11:09:56.823: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:99:29:f0
*spamApTask7: Nov 15 11:07:27.796: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11856 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 00:c1:64:99:29:40
*spamApTask3: Nov 15 11:06:39.614: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11856 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 00:c1:64:b3:80:30
*spamApTask0: Nov 15 11:04:48.645: %CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c:7551 Max retransmissions reached on AP(00:c1:64:99:29:40),message (CAPWAP_CONFIGURATION_UPDATE_REQUEST
),number of pending messages(4)
*spamApTask5: Nov 15 11:04:08.845: %CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c:7551 Max retransmissions reached on AP(00:c1:64:b3:80:30),message (CAPWAP_CONFIGURATION_UPDATE_REQUEST
),number of pending messages(1)
*spamApTask3: Nov 15 11:02:45.425: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:90
*spamApTask2: Nov 15 10:51:04.555: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 2 from AP 00:c8:8b:ff:34:e0
*spamApTask2: Nov 15 10:49:04.570: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 1 from AP 00:c8:8b:ff:34:e0
*spamApTask2: Nov 15 10:43:04.616: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 5 from AP 00:c8:8b:ff:34:e0
*spamApTask2: Nov 15 10:41:04.630: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 2 from AP 00:c8:8b:ff:34:e0
*spamApTask7: Nov 15 10:34:22.552: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:50
*spamApTask7: Nov 15 10:32:22.485: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:99:29:40
*Dot1x_NW_MsgTask_0: Nov 15 10:19:33.768: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client cc:af:78:86:13:c8 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_0: Nov 15 10:17:58.705: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client cc:af:78:86:13:c8 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask7: Nov 15 10:17:05.632: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 0, count 1 from AP 00:c8:8b:c5:22:60
*spamApTask7: Nov 15 10:10:22.657: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:99:29:40
*spamApTask3: Nov 15 10:06:45.891: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:90
*Dot1x_NW_MsgTask_0: Nov 15 10:05:39.476: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 3c:2e:f9:df:14:20 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_5: Nov 15 10:04:49.433: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 80:00:0b:48:c2:bd - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask2: Nov 15 10:03:04.911: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 0, count 2 from AP 00:c8:8b:ff:34:e0
*Dot1x_NW_MsgTask_2: Nov 15 10:02:04.729: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 34:02:86:22:cc:ca - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask7: Nov 15 09:56:22.862: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 0, count 1 from AP 00:c1:64:b3:7d:50
*Dot1x_NW_MsgTask_3: Nov 15 09:50:25.239: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client fc:a6:21:65:25:1b - got 00 00 00 00 00 00 00 03, expected 00 00 00 00 00 00 00 04
*Dot1x_NW_MsgTask_7: Nov 15 09:46:41.332: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client a8:5c:2c:09:7e:7f - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03
*Dot1x_NW_MsgTask_3: Nov 15 09:45:58.797: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 08:f4:ab:6d:f2:a3 - got 00 00 00 00 00 00 00 03, expected 00 00 00 00 00 00 00 04
*Dot1x_NW_MsgTask_3: Nov 15 09:45:58.796: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 08:f4:ab:6d:f2:a3 - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 04
*spamApTask3: Nov 15 09:39:57.539: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 3 from AP 00:c1:64:99:29:f0
*Dot1x_NW_MsgTask_0: Nov 15 09:39:13.159: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client a8:51:5b:fe:be:88 - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03
*spamApTask7: Nov 15 09:34:23.030: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:50

I've read that these outputs could be some firmware-related errors and it would be necessary to upgrade the firmware, though the WLC is a 2504 model and it's running 8.5.105.0 version which we upgraded to 6 months ago.

Another thing I'd like to mention is that half of the APs installed have their wired interfaces operating at 100Mbps, when they should be at 1000Mbps because of being connected to a brand-new Catalyst 9200 switch. On-site engineer is gonna check that this weekend. I'd like to know if that speed issue might have something to do with the disconnection problem, because we noticed one of the APs with the random disconnections also has the 100Mbps issue.

We discarded bad signal reception as the office is just 2 floors and the floors' areas are medium-sized, so the 18 APs are more than enough, and in fact the engineer told me the disconnection also happened even if being just 2 meters away the AP.

What could be happening and what should we do in that case? (We hadn't touched anything in the WLC nor the APs because it's still business hours)

[Leo Laohoo]

I'm going to start by saying upgrade the firmware of the WLC. There is a reason why Cisco has "deferred" WLC firmware version 8.5.105.0.

[supportgns]

Hi Leo,

We updated the WLC to 8.5.151.0 last night, as this one is the version marked with the star on Cisco Software Download. However, customer has contacted me again and told me the disconnection issue persists; in addition, connection speed is slow and there are high latencies, up to 900ms.

About the cabling and those AP which were operating at 100Mbps, on-site engineer checked them last week and effectively it was a cabling issue, now it's fixed and all the 19 APs operate at 1Gbps.

We're sure it's not a signal strength issue because even a user who is right below an AP (2-3 meters above him) is the one who reported the 900ms latency. In addition there's no dropped ceiling; the end-users can see directly the APs.

What else can we do? Will it be a co-channel interference? Thanks.

[Leo Laohoo]

Gather some facts.
What kind of wireless clients? Get their MAC addresses.
What radio(s) was/were the client last "seen" to be connected on?

[Scott Fella]

Seems like you might need to look at your txpower levels and like Leo mentioned, capture more data. If you do not have the lower data rates disabled for example, the clients can be sticky to one ap and that is why even though the client is beneath the ap, the have poor connection. It can also be that the client is connected to 2.4ghz and that band has high utilization. Can also be that you enabled features in the SSID that is causing issues with various clients. Is the implementation that was done high density or are the ap’s pretty sparsly deployed? Do you have -65dbi coverage on 5ghz in all locations that you want coverage? Do you have more 5ghz connected client than 2.4ghz? All this information can help us try to understand what can be done or can’t be done.

[YaminSaputra29393]

Hi Leo,

I got same error detail capture log on WLC 2504 using firmware version 8.5.120.0 in attachment.

What can i do for fix this problem ?
Waiting you in advance and thank you.

[Leo Laohoo]

Thanks.
The screen DOES NOT contain anything useful.

[YaminSaputra29393]

On our customers have a problem till today, all AP on Cisco WLC 2500 series suddently down every 11AM and 2PM everyday, disconnect and reconnect again after 10-15 minutes.
What can i do for fix this problem ?
Waiting you in advance and thank you.

[ziqex]

Have you found solution to your issues? If yes please share. Thank you.

[Scott Fella]

It is best to start your own thread and add as much information as possible to help us help you. Every environment is different, with different controllers, firmware access points and configuration. Even device types that connect to the wireless are different.

[supportgns]

Hi,

We recently found that the issue was not related to the AP nor WLC, but due to external sources. As the offices are located in floors 8th and 9th, there are many radio antennas close to this building, that generate some sort of interference with the WiFi provided by the Aironet APs. We reached to this conclusion because those offices were located before in another building (prior to a moving) and there were no connection drops, even when both the Aironet APs and the WLC have been always the same.

We made some workarounds by manually setting the proper channels (1,6,11) and power, and enabling Fast Transition. 

Now I have received no complaints from my customer though there are some random cases of dropping connections, but mainly in Windows 10 endpoints.

[marce1000]

          Ref : https://developer.cisco.com/docs/wireless-troubleshooting-tools/

 You may want to use the Wireless Debug Analyzer for further troubleshooting and  or https://cway.cisco.com/wireless-config-analyzer/  to have a sanity check of your controller configuration.

 M.