03-05-2020 04:15 AM - edited 07-05-2021 11:49 AM
At present, I encountered a problem, some AP1600 showed abnormal symptoms. They can register to the WLC normally, but after a few hours of operation, they will disconnect from the WLC and cannot register again.
I extracted some key information:
*spamApTask5: Mar 01 11:59:55.452: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:6985 0c:68:03:dd:6b:b0: DTLS connection closed forAP 10:191:2:5 (18943), Controller: 10:1:2:253 (5246) Echo Timer Expiry <<<<<<<<
*spamApTask4: Mar 01 12:00:38.393: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:6985 ac:7e:8a:68:77:80: DTLS connection closed forAP 10:191:2:4 (23864), Controller: 10:1:2:253 (5246) Echo Timer Expiry
.....
*osapiBsnTimer: Mar 01 12:03:40.785: %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:3191 Failed to complete DTLS handshake with peer 10.1.2.4
*osapiBsnTimer: Mar 01 12:03:01.689: %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:3191 Failed to complete DTLS handshake with peer 10.1.2.5
.....
A lot of the same information:
*bcastReceiveTask: Mar 01 12:03:01.481: %APF-3-INVALID_INET: apf_net.c:4752 Invalid Inet address source: (fe80::1ede:a7ff:fe06:85e0) destination: (::). Discarding packet.
*apfOrphanSocketTask: Mar 01 12:03:01.136: %LOG-3-Q_IND: apf_net.c:4752 Invalid Inet address source: (fe80::1ede:a7ff:fe06:85e0) destination: (::). Discarding packet.[...It occurred 22 times.!]
(Cisco Controller) >show ap join stats summary all
Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable
Discovery phase statistics
- Discovery requests received.............................. 64
- Successful discovery responses sent...................... 0
- Unsuccessful discovery request processing................ 64
- Reason for last unsuccessful discovery attempt........... Discovery request decoding with subnet broadcast and wrong AP IP address
- Time at last successful discovery attempt................ Not applicable
- Time at last unsuccessful discovery attempt.............. Mar 01 12:15:40.844
Join phase statistics
- Join requests received................................... 0
- Successful join responses sent........................... 0
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Not applicable
- Time at last unsuccessful join attempt................... Not applicable
Configuration phase statistics
--More-- or (q)uit
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable
Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable
Last AP disconnect details
- Reason for last AP connection failure.................... Not applicable
- Last AP disconnect reason................................ Not applicable
Last join error summary
- Type of error that occurred last......................... Lwapp discovery request rejected
- Reason for error that occurred last...................... Discovery request decoding with subnet broadcast and wrong AP IP address
- Time at which the last join error occurred............... Mar 01 12:15:40.844
AP disconnect details
- Reason for last AP connection failure.................... Not applicable
Ethernet Mac : 00:00:00:00:00:00 Ip Address : 10.1.3.32
*Mar 1 12:00:05.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.1.2.253 peer_port: 5246
*Mar 1 12:01:04.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.1.2.253:5246
*Mar 1 12:01:05.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.1.2.253 peer_port: 5246
*Mar 1 12:01:10.999: %DTLS-5-SEND_ALERT: Send FATAL : Unexpected message Alert to 10.1.2.253:5246
*Mar 1 12:01:10.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.1.2.253:5246
*Mar 1 12:02:38.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.1.2.253 peer_port: 5246
*Mar 1 12:02:51.999: %DTLS-5-SEND_ALERT: Send FATAL : Unexpected message Alert to 10.1.2.253:5246
Based on this information, I found some Cisco documentation.
Problem 5: Controller receives AP discovery message on wrong VLAN (you see the discovery message debug, but not response)
You see this message in the debug capwap events enable command output:
Received a Discovery Request with subnet broadcast with wrong AP IP address (A.B.C.D)!
This message means that the controller received a discovery request via a broadcast IP address that has a source IP address which is not in any configured subnets on the controller. This also means the controller is dropping the packet.
The problem is that the AP is not sending the discovery request to the management IP address. The controller is reporting a broadcast discovery request from a VLAN that is not configured on the controller. This typically occurs when the customer trunks allowed VLANs instead of restricting them to wireless VLANs.
Complete these steps in order to resolve this problem:
1.If the controller is on another subnet, the APs must be primed for the controller IP address, or the APs must receive the controllers IP address using any one of the discovery methods.
2.The switch is configured to allow some VLANs that are not on the controller. Restrict the allowed VLANs on the trunks.
The switch interfaces connecting the WLC and AP are in trunk mode.So I suspect this problem caused by the switch interface configuration.I would like to ask everyone to help me analyze whether it is the same problem as "Problem 5" or what other reasons?
Thanks
03-05-2020 04:33 AM
AP is in local mode or flexconnect ?
Are AP and WLC in same subnet or ?
Paste the complete output of these commands:
show sysinfo from WLC
show version from AP
Bootup process from AP console
show run inter <WLC connected>
show run inter <AP connected>
show time form WLC
Regards
Dont forget to rate helpful posts
03-06-2020 03:10 AM
I am sorry for late reply
All APs went offline this morning and cannot register to the WLC. It's weird. Basically, APs will be disconnected after a few hours. Very unstable. We need to use the "shu / no shu" command to restart the switch interfaces connected to APs, and restart these APs to register with the WLC again.
Attached is the information I collected. But there is no information about the AP bootup process, but I attached the show logging information of the AP.
Please help to see if there is any problem.
Thanks
03-06-2020 08:08 AM
Hi
You're hitting a well-known bug (CSCvn08995), upgrade the controller to a stabler version.
Regards,
Cristian Matei.
03-06-2020 04:00 PM
03-06-2020 04:18 PM
03-06-2020 04:51 PM
Doesn't seem to be this bug:
I do not see debugging similar to the following in the debug information:
* spamApTask1: Failed to create DTLS connection for: 54455
* spamApTask1: System reached max concurrent DTLS Handshakes
* spamApTask0: Ignoring peer connection because concurrent handshake threshold reached.
In addition, the msglog on the WLC looks like this:
* osapiBsnTimer: Mar 06 09: 07: 20.977:% DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c: 3191 Failed to complete DTLS handshake with peer 10.1.2.4
* spamApTask7: Mar 06 08: 08: 43.969:% CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c: 7533 Max retransmissions reached on AP (00: 2a: 10: xx: xx: xx), message (CAPWAP_CONFIGURATION_UPDATE_REQUEST
), number of pending messages (37)
* spamApTask0: Mar 06 08: 08: 35.417:% CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c: 6985 f0: 29: 29: 27: 03: 00: DTLS connection closed forAP 10: 1: 2: 4 (24630), Controller: 10: 1: 2: 253 (5246) Echo Timer Expiry
There are also a large number of logs on WLC:
* spamApTask4: Mar 06 08: 05: 05.790:% LOG-3-Q_IND: apf_net.c: 4752 Invalid Inet address source: (fe80 :: c6e7: d5ff: ffe4: 6877) destination: (: :). Discarding packet. [... It occurred 26 times.!]
* bcastReceiveTask: Mar 06 08: 05: 02.172:% APF-3-INVALID_INET: apf_net.c: 4752 Invalid Inet address source: (fe80 :: c6e7: d5ff: ffe4: 6877) destination: (: :). Discarding packet.
* spamApTask3: Mar 06 08: 04: 41.317:% LOG-3-Q_IND: apf_net.c: 4752 Invalid Inet address source: (fe80 :: c6e7: d5ff: ffe4: 6877) destination: (: :). Discarding packet. [... It occurred 26 times.!]
* bcastReceiveTask: Mar 06 08: 04: 40.404:% APF-3-INVALID_INET: apf_net.c: 4752 Invalid Inet address source: (fe80 :: c6e7: d5ff: ffe4: 6877) destination: (: :). Discarding packet.
03-07-2020 05:56 AM
Hi,
The Bug Search Toll is a bit sneaky here, to say the least. go to the bug search tool, look for the ONLY relevant error from your attached file (%DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to) and you'll see the bug; if you click it, it will redirect you to the other bug, ever the errors are different, but outcome and behaviour is the same. See the attached pic for the correct bug output.
03-07-2020 07:12 AM
03-10-2020 11:33 PM
03-11-2020 02:37 AM
Hi,
You've seen the bug, currently opened. Open a TAC case urgently and see how it goes, you may need to do some RMA.
Regards,
Cristian Matei.
03-11-2020 03:15 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide