11-15-2019 02:33 PM - edited 07-05-2021 11:19 AM
Hi,
Currently we have a wireless environment with 18 APs and a WLC 2504 for a customer. Today our customer had a connectivity issue and told us several users disconnected randomly after 10 or 15 min being connected to the WiFi.
We did a first check and went to the logs, and this is what we found (I attach the logs between 9:30 and 12:00 because the problem occurred at that time):
*spamApTask2: Nov 15 12:05:04.008: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 1 from AP 00:c8:8b:ff:34:e0
*spamApTask0: Nov 15 11:41:11.489: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11856 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 00:c1:64:b3:80:30
*Dot1x_NW_MsgTask_7: Nov 15 11:23:22.179: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client c0:17:4d:f1:64:07 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_0: Nov 15 11:15:56.012: %DOT1X-3-INVALID_WPA_KEY_STATE: 1x_eapkey.c:2900 Received EAPOL-key message while in invalid state (4) - version 2, type 3, descriptor 2, client d0:c5:f3:89:a0:a0
*Dot1x_NW_MsgTask_0: Nov 15 11:15:56.012: %LOG-3-Q_IND: 1x_eapkey.c:452 Invalid replay counter from client d0:c5:f3:89:a0:a0 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_0: Nov 15 11:15:56.005: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client d0:c5:f3:89:a0:a0 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask3: Nov 15 11:09:56.823: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:99:29:f0
*spamApTask7: Nov 15 11:07:27.796: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11856 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 00:c1:64:99:29:40
*spamApTask3: Nov 15 11:06:39.614: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: spam_lrad.c:11856 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 00:c1:64:b3:80:30
*spamApTask0: Nov 15 11:04:48.645: %CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c:7551 Max retransmissions reached on AP(00:c1:64:99:29:40),message (CAPWAP_CONFIGURATION_UPDATE_REQUEST
),number of pending messages(4)
*spamApTask5: Nov 15 11:04:08.845: %CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c:7551 Max retransmissions reached on AP(00:c1:64:b3:80:30),message (CAPWAP_CONFIGURATION_UPDATE_REQUEST
),number of pending messages(1)
*spamApTask3: Nov 15 11:02:45.425: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:90
*spamApTask2: Nov 15 10:51:04.555: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 2 from AP 00:c8:8b:ff:34:e0
*spamApTask2: Nov 15 10:49:04.570: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 1 from AP 00:c8:8b:ff:34:e0
*spamApTask2: Nov 15 10:43:04.616: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 5 from AP 00:c8:8b:ff:34:e0
*spamApTask2: Nov 15 10:41:04.630: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 2 from AP 00:c8:8b:ff:34:e0
*spamApTask7: Nov 15 10:34:22.552: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:50
*spamApTask7: Nov 15 10:32:22.485: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:99:29:40
*Dot1x_NW_MsgTask_0: Nov 15 10:19:33.768: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client cc:af:78:86:13:c8 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_0: Nov 15 10:17:58.705: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client cc:af:78:86:13:c8 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask7: Nov 15 10:17:05.632: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 0, count 1 from AP 00:c8:8b:c5:22:60
*spamApTask7: Nov 15 10:10:22.657: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:99:29:40
*spamApTask3: Nov 15 10:06:45.891: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:90
*Dot1x_NW_MsgTask_0: Nov 15 10:05:39.476: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 3c:2e:f9:df:14:20 - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*Dot1x_NW_MsgTask_5: Nov 15 10:04:49.433: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 80:00:0b:48:c2:bd - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask2: Nov 15 10:03:04.911: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 0, count 2 from AP 00:c8:8b:ff:34:e0
*Dot1x_NW_MsgTask_2: Nov 15 10:02:04.729: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 34:02:86:22:cc:ca - got 00 00 00 00 00 00 00 00, expected 00 00 00 00 00 00 00 01
*spamApTask7: Nov 15 09:56:22.862: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 0, count 1 from AP 00:c1:64:b3:7d:50
*Dot1x_NW_MsgTask_3: Nov 15 09:50:25.239: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client fc:a6:21:65:25:1b - got 00 00 00 00 00 00 00 03, expected 00 00 00 00 00 00 00 04
*Dot1x_NW_MsgTask_7: Nov 15 09:46:41.332: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client a8:5c:2c:09:7e:7f - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03
*Dot1x_NW_MsgTask_3: Nov 15 09:45:58.797: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 08:f4:ab:6d:f2:a3 - got 00 00 00 00 00 00 00 03, expected 00 00 00 00 00 00 00 04
*Dot1x_NW_MsgTask_3: Nov 15 09:45:58.796: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client 08:f4:ab:6d:f2:a3 - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 04
*spamApTask3: Nov 15 09:39:57.539: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 0, WLAN ID 1, count 3 from AP 00:c1:64:99:29:f0
*Dot1x_NW_MsgTask_0: Nov 15 09:39:13.159: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:452 Invalid replay counter from client a8:51:5b:fe:be:88 - got 00 00 00 00 00 00 00 02, expected 00 00 00 00 00 00 00 03
*spamApTask7: Nov 15 09:34:23.030: %LWAPP-3-REPLAY_ERR: spam_lrad.c:44958 The system has received replay error on slot 1, WLAN ID 1, count 1 from AP 00:c1:64:b3:7d:50
I've read that these outputs could be some firmware-related errors and it would be necessary to upgrade the firmware, though the WLC is a 2504 model and it's running 8.5.105.0 version which we upgraded to 6 months ago.
Another thing I'd like to mention is that half of the APs installed have their wired interfaces operating at 100Mbps, when they should be at 1000Mbps because of being connected to a brand-new Catalyst 9200 switch. On-site engineer is gonna check that this weekend. I'd like to know if that speed issue might have something to do with the disconnection problem, because we noticed one of the APs with the random disconnections also has the 100Mbps issue.
We discarded bad signal reception as the office is just 2 floors and the floors' areas are medium-sized, so the 18 APs are more than enough, and in fact the engineer told me the disconnection also happened even if being just 2 meters away the AP.
What could be happening and what should we do in that case? (We hadn't touched anything in the WLC nor the APs because it's still business hours)
11-15-2019 03:20 PM
11-27-2019 11:40 AM
Hi Leo,
We updated the WLC to 8.5.151.0 last night, as this one is the version marked with the star on Cisco Software Download. However, customer has contacted me again and told me the disconnection issue persists; in addition, connection speed is slow and there are high latencies, up to 900ms.
About the cabling and those AP which were operating at 100Mbps, on-site engineer checked them last week and effectively it was a cabling issue, now it's fixed and all the 19 APs operate at 1Gbps.
We're sure it's not a signal strength issue because even a user who is right below an AP (2-3 meters above him) is the one who reported the 900ms latency. In addition there's no dropped ceiling; the end-users can see directly the APs.
What else can we do? Will it be a co-channel interference? Thanks.
11-30-2019 11:43 PM
12-02-2019 08:53 AM
06-11-2020 09:44 PM
06-11-2020 11:48 PM
06-12-2020 12:24 AM
09-29-2020 06:27 AM
Have you found solution to your issues? If yes please share. Thank you.
09-29-2020 08:23 AM
09-29-2020 08:38 AM - edited 09-29-2020 08:40 AM
Hi,
We recently found that the issue was not related to the AP nor WLC, but due to external sources. As the offices are located in floors 8th and 9th, there are many radio antennas close to this building, that generate some sort of interference with the WiFi provided by the Aironet APs. We reached to this conclusion because those offices were located before in another building (prior to a moving) and there were no connection drops, even when both the Aironet APs and the WLC have been always the same.
We made some workarounds by manually setting the proper channels (1,6,11) and power, and enabling Fast Transition.
Now I have received no complaints from my customer though there are some random cases of dropping connections, but mainly in Windows 10 endpoints.
06-12-2020 12:09 AM
Ref : https://developer.cisco.com/docs/wireless-troubleshooting-tools/
You may want to use the Wireless Debug Analyzer for further troubleshooting and or https://cway.cisco.com/wireless-config-analyzer/ to have a sanity check of your controller configuration.
M.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide