Re: CAPWAP/DTLS Handshake Error Joining Cisco 2802i AP to 5508 WLC

Rafael Rubik · ‎12-04-2020

Hello friends,

I tried to join a Cisco 2802i AP to a 5508 WLC but found an issue that is not allowing it joining successfully. I believe the problem occurs in capwap / dtls handshake (CAPWAP State "Configure").

Some details:

. We have 170 sites

. We have 4 5508 WLCs
. The problem only occurs on a new site that we are deploying

. This site ( 1x 4431 ISR, 1x 3560CX-12PC-S, 3x 2802i )

. We already changed the router, switch, ap, trunk cable, electrical outlet.

. We already installed a new MPLS circuit (we are the nw operator)

. I took the switch and AP to another site and everything worked fine.

. There is no firewall between this site and the wlc (WLC-1) we are testing.
. There is a firewall for 3 other WLCs (WLC-2, 3, 4) but the log shows "passed" for all traffic.

Some debugs:

(Cisco Controller) debug>capwap errors enable

(Cisco Controller) debug>*spamApTask1: Dec 03 17:44:54.194: 10:b3:d5:3a:81:00 Unable to get Ap mode in Join request

*spamReceiveTask: Dec 04 09:59:42.286: 50:3e:aa:d4:63:ca Unable to get RadId. Sending of PMK cache entry to all APs in flexconnect group failed :: bssid 00:00:00:00:00:00
*spamReceiveTask: Dec 04 09:59:45.881: 70:0b:4f:7b:d3:8d Unable to get RadId. Sending of PMK cache entry to all APs in flexconnect group failed :: bssid 00:00:00:00:00:00
*spamReceiveTask: Dec 04 09:59:50.552: 70:c9:c6:4b:55:1c Unable to get RadId. Sending of PMK cache entry to all APs in flexconnect group failed :: bssid 00:00:00:00:00:00
*spamReceiveTask: Dec 04 10:00:14.064: a4:4e:31:ba:93:08 Unable to get RadId. Sending of PMK cache entry to all APs in flexconnect group failed :: bssid 00:00:00:00:00:00
*spamReceiveTask: Dec 04 10:00:19.434: 5c:cd:5b:20:5a:e9 Unable to get RadId. Sending of PMK cache entry to all APs in flexconnect group failed :: bssid 00:00:00:00:00:00
*spamApTask1: Dec 04 10:00:19.986: 10:b3:d5:3a:81:00 DTLS connection was closed
*spamApTask1: Dec 04 10:00:25.792: 10:b3:d5:3a:81:00 ApModel: AIR-AP2802I-Z-K9

*spamApTask1: Dec 04 10:00:25.810: 10:b3:d5:3a:81:00 ApModel: AIR-AP2802I-Z-K9

*spamApTask1: Dec 04 10:00:35.622: 10:b3:d5:3a:81:00 Unable to get Ap mode in Join request

*spamReceiveTask: Dec 04 10:00:46.339: 80:86:f2:fc:a5:0d Unable to get RadId. Sending of PMK cache entry to all APs in flexconnect group failed :: bssid 00:00:00:00:00:00
*spamApTask1: Dec 04 10:01:18.525: 10:b3:d5:3a:81:00 DTLS connection was closed
*spamApTask1: Dec 04 10:01:24.336: 10:b3:d5:3a:81:00 ApModel: AIR-AP2802I-Z-K9

debug capwap events

*spamApTask4: Dec 04 10:15:41.531: a0:93:51:fc:b2:c0 Echo Request from 10.15.25.12:5272

*spamApTask4: Dec 04 10:15:41.531: a0:93:51:fc:b2:c0 Echo Response sent to 10.15.25.12:5272

*spamApTask4: Dec 04 10:16:00.111: a0:93:51:fc:b2:c0 WTP Event Request from 10.15.25.12:5272 epoch 1607087760

*spamApTask4: Dec 04 10:16:00.111: a0:93:51:fc:b2:c0 WTP Event Response sent to 10.15.25.12:5272

*spamApTask4: Dec 04 10:16:00.125: a0:93:51:fc:b2:c0 WTP Event Request from 10.15.25.12:5272 epoch 1607087760

*spamApTask4: Dec 04 10:16:00.125: a0:93:51:fc:b2:c0 WTP Event Response sent to 10.15.25.12:5272

*spamApTask4: Dec 04 10:16:03.996: a0:93:51:fc:b2:c0 WTP Event Request from 10.15.25.12:5272 epoch 1607087763

*spamApTask4: Dec 04 10:16:03.996: a0:93:51:fc:b2:c0 WTP Event Response sent to 10.15.25.12:5272

*spamApTask1: Dec 04 10:16:16.042: 10:b3:d5:3a:81:00 acDtlsPlumbControlPlaneKeys: lrad:10.15.102.14(5248) mwar:10.15.152.5(5246)

*spamApTask1: Dec 04 10:16:16.042: 10:b3:d5:3a:81:00 DTLS keys for Control Plane deleted successfully for AP 10.15.102.14

*spamApTask1: Dec 04 10:16:16.043: 10:b3:d5:3a:81:00 DTLS connection closed event receivedserver (10.15.152.5/5246) client (10.15.102.14/5248)
*spamApTask1: Dec 04 10:16:16.043: 10:b3:d5:3a:81:00 Entry exists for AP (10.15.102.14/5248)
*spamApTask1: Dec 04 10:16:16.043: 10:b3:d5:3a:81:00 Capwap State Change Event (DeReg) from capwap_ac_sm.c 1787

*spamApTask1: Dec 04 10:16:16.044: 10:b3:d5:3a:81:00 apfSpamProcessStateChangeInSpamContext: Deregister LWAPP event for AP 10:b3:d5:3a:81:00 slot 0
*apfReceiveTask: Dec 04 10:16:16.044: 10:b3:d5:3a:81:00 Deregister LWAPP event for AP 10:b3:d5:3a:81:00 slot 0
*spamApTask1: Dec 04 10:16:16.044: 10:b3:d5:3a:81:00 Capwap State Change Event (DeReg) from capwap_ac_sm.c 1787

*spamApTask1: Dec 04 10:16:16.044: 10:b3:d5:3a:81:00 apfSpamProcessStateChangeInSpamContext: Deregister LWAPP event for AP 10:b3:d5:3a:81:00 slot 1
*spamApTask1: Dec 04 10:16:16.044: 10:b3:d5:3a:81:00 AP Delete request
*apfReceiveTask: Dec 04 10:16:16.044: 10:b3:d5:3a:81:00 Deregister LWAPP event for AP 10:b3:d5:3a:81:00 slot 1
*spamApTask1: Dec 04 10:16:16.045: 10:b3:d5:3a:81:00 No AP entry exist in temporary database for 10.15.102.14:5248
*spamApTask1: Dec 04 10:16:21.854: 10:b3:d5:3a:81:00 Discovery Request from 10.15.102.14:5248

debug dtls all

*capwapSocketTask: Dec 04 10:20:07.737: 10:b3:d5:3a:81:00 DTLS connection found! Acquiring lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:07.737: 10:b3:d5:3a:81:00 Releasing lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:07.737: 10:b3:d5:3a:81:00 DTLS connection find by 0x10e7908c with Local 10.15.152.5:5246 Peer 10.15.102.14:5248

*capwapSocketTask: Dec 04 10:20:07.737: 10:b3:d5:3a:81:00 DTLS connection found! Acquiring lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:07.737: 10:b3:d5:3a:81:00 Releasing lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:12.488: 10:b3:d5:3a:81:00 DTLS connection find by 0x10e7908c with Local 10.15.152.5:5246 Peer 10.15.102.14:5248

*capwapSocketTask: Dec 04 10:20:12.488: 10:b3:d5:3a:81:00 DTLS connection found! Acquiring lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:12.488: 10:b3:d5:3a:81:00 Releasing lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:12.488: 10:b3:d5:3a:81:00 DTLS connection find by 0x10e7908c with Local 10.15.152.5:5246 Peer 10.15.102.14:5248

*capwapSocketTask: Dec 04 10:20:12.488: 10:b3:d5:3a:81:00 DTLS connection found! Acquiring lock for 0x1a528e20
*capwapSocketTask: Dec 04 10:20:12.488: 10:b3:d5:3a:81:00 Releasing lock for 0x1a528e20
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 DTLS connection find by 0x10e7908c with Local 10.15.152.5:5246 Peer 10.15.102.14:5248

*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 DTLS connection found! Acquiring lock for 0x1a528e20
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 Called... for connection 0x1a528e20
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 Buffer length 61, alloc_len 65
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 record=Alert epoch=1 seq=29
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 handshake_done
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 con->rx_seq_valid 255 con->rx_epoch 1 epoch 1
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 seq_num 29 epoch 1con->rx_seq 0 con->rx_epoch 1
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 con rx_seq_valid 255 rx_seq 0 rx_epoch 1
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 Got DTLS_RECORD_ALERT
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 record length 48
*spamApTask1: Dec 04 10:20:17.240: 00000000: 15 fe fd 00 01 00 00 00 00 00 1d 00 30 d9 7d 31 ............0.}1
*spamApTask1: Dec 04 10:20:17.240: 10:b3:d5:3a:81:00 Calling BIO_write! 0x1a528e20, buflen 61

*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: SSL state = 0x3; where = 0x4004; ret = 0x100
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: ret_type_string=warning
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: ret_desc_string=close notify
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: SSL_state_string=SSL negotiation finished successfully
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 SSL_read: peer 10.15.102.14 has closed the connection
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 Requested by openssl_dtls_process_packet
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 in openssl_dtls.c:1032
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 Deleting hash for Local 10.15.152.5:5246 Peer 10.15.102.14:5248

*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 Called...
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: SSL state = 0x3; where = 0x4008; ret = 0x100
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: ret_type_string=warning
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: ret_desc_string=close notify
*spamApTask1: Dec 04 10:20:17.241: openssl_shim_info_callback: SSL_state_string=SSL negotiation finished successfully
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 Shutdown completed
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 Sending 61 bytes
*spamApTask1: Dec 04 10:20:17.241: 00000000: 15 fe fd 00 01 00 00 00 00 00 01 00 30 64 af dd ............0d..
*spamApTask1: Dec 04 10:20:17.241: 00000010: f5 21 d8 cb 6e 9b a1 86 bd 1c 1a 02 a6 72 bf e1 .!..n........r..
*spamApTask1: Dec 04 10:20:17.241: 00000020: 68 b1 60 d0 73 5b f7 af a8 3f 39 01 39 3e b9 07 h.`.s[...?9.9>..
*spamApTask1: Dec 04 10:20:17.241: 00000030: 86 f4 4b 24 0e a4 e6 53 a4 78 79 25 5f ..K$...S.xy%_
*spamApTask1: Dec 04 10:20:17.241: 10:b3:d5:3a:81:00 No data to send
*spamApTask1: Dec 04 10:20:17.242: 10:b3:d5:3a:81:00 DTLS Connection 0x1a528e20 closed by controller
*spamApTask1: Dec 04 10:20:17.243: __dtls_timer_stop: Called...

AP Logging (loop)

kernel: [*12/04/2020 13:56:32.5816] CAPWAP State: Configure
Dec 4 13:56:32 kernel: [*12/04/2020 13:56:32.5838] DOT11_CFG[0] Radio Mode is changed from FlexConnect to FlexConnect
Dec 4 13:56:32 kernel: [*12/04/2020 13:56:32.5873] DOT11_CFG[1] Radio Mode is changed from FlexConnect to FlexConnect
Dec 4 13:56:37 kernel: [*12/04/2020 13:56:37.3211] Re-Tx Count=1, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:56:37 kernel: [*12/04/2020 13:56:37.3211]
Dec 4 13:56:42 kernel: [*12/04/2020 13:56:42.0725] Re-Tx Count=2, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:56:42 kernel: [*12/04/2020 13:56:42.0725]
Dec 4 13:56:46 kernel: [*12/04/2020 13:56:46.8240] Re-Tx Count=3, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:56:46 kernel: [*12/04/2020 13:56:46.8240]
Dec 4 13:56:51 kernel: [*12/04/2020 13:56:51.5754] Re-Tx Count=4, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:56:51 kernel: [*12/04/2020 13:56:51.5754]
Dec 4 13:56:56 kernel: [*12/04/2020 13:56:56.3269] Re-Tx Count=5, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:56:56 kernel: [*12/04/2020 13:56:56.3269]
Dec 4 13:57:01 kernel: [*12/04/2020 13:57:01.0782] Re-Tx Count=6, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:01 kernel: [*12/04/2020 13:57:01.0782]
Dec 4 13:57:05 kernel: [*12/04/2020 13:57:05.8297] Re-Tx Count=7, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:05 kernel: [*12/04/2020 13:57:05.8297]
Dec 4 13:57:10 kernel: [*12/04/2020 13:57:10.5811] Re-Tx Count=8, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:10 kernel: [*12/04/2020 13:57:10.5811]
Dec 4 13:57:15 kernel: [*12/04/2020 13:57:15.3324] Re-Tx Count=9, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:15 kernel: [*12/04/2020 13:57:15.3325]
Dec 4 13:57:15 kernel: [*12/04/2020 13:57:15.3325] Max retransmission count exceeded, going back to DISCOVER mode.
Dec 4 13:57:15 kernel: [*12/04/2020 13:57:15.3331]
Dec 4 13:57:15 kernel: [*12/04/2020 13:57:15.3331] CAPWAP State: DTLS Teardown
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.1692] Discovery Response from 10.15.224.5
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.1768] Discovery Response from 10.15.152.5
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.1841] Discovery Response from 10.4.24.25
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.1912] Discovery Response from 10.4.8.25
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.1984] Discovery Response from 10.15.224.5
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.2056] Discovery Response from 10.15.152.5
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.2127] Discovery Response from 10.4.24.25
Dec 4 13:57:21 kernel: [*12/04/2020 13:57:21.2198] Discovery Response from 10.4.8.25
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.0000]
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.0000] CAPWAP State: DTLS Setup
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.4327]
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.4327] CAPWAP State: Join
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.4342] Sending Join request to 10.15.152.5 through port 5248
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.4467] Join Response from 10.15.152.5
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5422]
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5422] CAPWAP State: Image Data
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5796] do NO_UPGRADE, part1 is active part
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5830]
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5830] CAPWAP State: Configure
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5851] DOT11_CFG[0] Radio Mode is changed from FlexConnect to FlexConnect
Dec 4 13:57:31 kernel: [*12/04/2020 13:57:31.5857] DOT11_CFG[1] Radio Mode is changed from FlexConnect to FlexConnect
Dec 4 13:57:36 kernel: [*12/04/2020 13:57:36.3211] Re-Tx Count=1, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:36 kernel: [*12/04/2020 13:57:36.3211]
Dec 4 13:57:41 kernel: [*12/04/2020 13:57:41.0726] Re-Tx Count=2, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:41 kernel: [*12/04/2020 13:57:41.0726]
Dec 4 13:57:45 kernel: [*12/04/2020 13:57:45.8239] Re-Tx Count=3, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:45 kernel: [*12/04/2020 13:57:45.8239]
Dec 4 13:57:50 kernel: [*12/04/2020 13:57:50.5754] Re-Tx Count=4, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:50 kernel: [*12/04/2020 13:57:50.5754]
Dec 4 13:57:55 kernel: [*12/04/2020 13:57:55.3268] Re-Tx Count=5, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:57:55 kernel: [*12/04/2020 13:57:55.3268]
Dec 4 13:58:00 kernel: [*12/04/2020 13:58:00.0782] Re-Tx Count=6, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:58:00 kernel: [*12/04/2020 13:58:00.0782]
Dec 4 13:58:04 kernel: [*12/04/2020 13:58:04.8297] Re-Tx Count=7, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:58:04 kernel: [*12/04/2020 13:58:04.8297]
Dec 4 13:58:09 kernel: [*12/04/2020 13:58:09.5811] Re-Tx Count=8, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:58:09 kernel: [*12/04/2020 13:58:09.5811]
Dec 4 13:58:14 kernel: [*12/04/2020 13:58:14.3325] Re-Tx Count=9, Max Re-Tx Value=8, SendSeqNum=1, NumofPendingMsgs=1
Dec 4 13:58:14 kernel: [*12/04/2020 13:58:14.3325]
Dec 4 13:58:14 kernel: [*12/04/2020 13:58:14.3325] Max retransmission count exceeded, going back to DISCOVER mode.

I've been working on this for several days, without success. Some help please.

Tks

Rafael Rubik

ajc · ‎12-04-2020

Please clarify something, when you tested successfully the 2800 AP in another site, was that connection against the WLC-2, WLC-3 and WLC-4?.

You are indicating that the 2800 is unable to connect to a separate WLC-1 being tested. In that case, I would check on WLC-1 that you have the license properly installed as step 1, check as well ntp/date/timezone is correct, that the license on your WLC-1 has not expired, that the regulatory domain on your WLC-1 is properly configured based on the AP = AIR-AP2802I-Z-K9 (where Z is the country where this AP is running).

Rafael Rubik · ‎12-04-2020

Hello ajc,

. Yes, when tested successfully the 2800 AP in another site, was tested with WLC-1 and WLC-2. Ok on both.

. The WLC-1 is my backup (N+1), usually without APs. However it is working normally. I can migrate APs from another WLC to WLC-1 and associate normally. The problem seems to be in the new site. But I still don't know where...

. We have only the AIR-AP2802I-Z-K model on the network. I don't believe it's a regulatory domain issue.

. Look at the command below (2x). The AP Laguna-04 (other site) is stable on the WLC-1. The AP AP084Fxxxx (new site) is in loop and disappears from the command.

(Cisco Controller) show>dtls connections (1st)

Total DTLS Connections Count..................... 2
Control DTLS Connections Count................... 2
Data DTLS Connections Count...................... 0
Handshaking DTLS Connections Count............... 0
Licensed AP Support Capacity..................... 500
AP Name Local Port Peer IP Peer Port Ciphersuite
-------------------- ------------- ---------------- ------------- ------------------------------
AP-Laguna_04 Capwap_Ctrl 10.15.25.12 5272 TLS_RSA_WITH_AES_128_CBC_SHA
AP084F.A9E9.0A84 Capwap_Ctrl 10.15.102.14 5248 TLS_RSA_WITH_AES_128_CBC_SHA

(Cisco Controller) show>dtls connections (2nd)

Total DTLS Connections Count..................... 1
Control DTLS Connections Count................... 1
Data DTLS Connections Count...................... 0
Handshaking DTLS Connections Count............... 0
Licensed AP Support Capacity..................... 500
AP Name Local Port Peer IP Peer Port Ciphersuite
-------------------- ------------- ---------------- ------------- ------------------------------
AP-Laguna_04 Capwap_Ctrl 10.15.25.12 5272 TLS_RSA_WITH_AES_128_CBC_SHA

Other outputs (WLC-1):

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.161.9
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... WLC-1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.15.152.5
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 9 days 4 hrs 57 mins 24 secs
System Timezone Location......................... (GMT -3:00) Buenos Aires (Agentina)

--More-- or (q)uit
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries : AR,BR,US
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +38 C
External Temperature............................. +21 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 15
Number of Active Clients......................... 2

OUI Classification Failure Count................. 262

Memory Current Usage............................. 57
Memory Average Usage............................. 57
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card

--More-- or (q)uit
Flash Size....................................... 1073741824

Burned-in MAC Address............................ E8:B7:48:A1:9C:A0
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 500
System Nas-Id.................................... WLC-1
WLC MIC Certificate Types........................ SHA1

(Cisco Controller) >show time

Time............................................. Fri Dec 4 14:36:16 2020

Timezone delta................................... 0:0
Timezone location................................ (GMT -3:00) Buenos Aires (Agen tina)

NTP Servers
NTP Version.................................. 3
NTP Polling Interval......................... 600

Index NTP Key Index NTP Server Status NTP Msg Auth Status
------- ------------------------------------------------------------------- --
1 0 10.4.0.155 In Sync AUTH DISABLED

(Cisco Controller) >show country channels

Configured Country............................. Multiple Countries : AR,BR,US
KEY: * = Channel is legal in this country and may be configured manually.
A = Channel is the Auto-RF default in this country.
. = Channel is not legal in this country.
C = Channel has been configured for use by Auto-RF.
x = Channel is available to be configured for use by Auto-RF.
(-,-) = (indoor, outdoor) regulatory domain allowed by this country.
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-
802.11bg :
Channels : 1 1 1 1 1
: 1 2 3 4 5 6 7 8 9 0 1 2 3 4
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-
AR (-A ,-AR A * * * * A * * * * A . . .
BR (-A ,-AR A * * * * A * * * * A . . .
US (-A ,-AB A * * * * A * * * * A . . .
Auto-RF : C x x x x C x x x x C . . .
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
802.11a : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Channels : 3 3 3 4 4 4 4 4 5 5 6 6 0 0 0 1 1 2 2 2 3 3 4 4 4 5 5 6 6 6 7
: 4 6 8 0 2 4 6 8 2 6 0 4 0 4 8 2 6 0 4 8 2 6 0 4 9 3 7 1 5 9 3

--More-- or (q)uit
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
AR (-A ,-A . A . A . A . A A A A A A A A A A . . . A A A . A A A A * . .
BR (-TZ ,-ANZ . A . A . A . A A A A A A A A A A A A A A A A . A A A A * . .
US (-AB ,-AB . A . A . A . A A A A A A A A A A A A A A A A A A A A A * . .
Auto-RF : . C . C . C . C C C C C x x x x x x x x x x x x x x x x x . .
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
4.9GHz 802.11a :
Channels : 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
: 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
AR (-A ,-A . . . . . . . . . . . . . . . . . . . . . . . . . .
BR (-TZ ,-ANZ . . . . . . . . . . . . . . . . . . . . . . . . . .
US (-AB ,-AB * * * * * * * * * * * * * * * * * * * A * * * * * A
Auto-RF : . C . C . C . C C C C C x x x x x x x x x x x x x x x x x . .
-----------------:+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

(Cisco Controller) >show license capacity

Licensed Feature Max Count Current Count Remaining Count
-----------------------------------------------------------------------
AP Count 500 2 498

(Cisco Controller) >show license all

License Store: Primary License Storage
StoreIndex: 0 Feature: base Version: 1.0
License Type: Permanent
License State: Inactive
License Count: Non-Counted
License Priority: Medium
License Store: Primary License Storage
StoreIndex: 0 Feature: base-ap-count Version: 1.0
License Type: Permanent
License State: Inactive
License Count: 50 / 0 (Active/In-use)
License Priority: Medium
License Store: Primary License Storage
StoreIndex: 0 Feature: base-ap-count Version: 1.0
License Type: Permanent
License State: Inactive
License Count: 150 / 0 (Active/In-use)
License Priority: Medium
License Store: Primary License Storage
StoreIndex: 0 Feature: base-ap-count Version: 1.0
License Type: Permanent

--More-- or (q)uit
License State: Active, Not in Use
License Count: 300 / 0 (Active/In-use)
License Priority: Medium
License Store: Primary License Storage
StoreIndex: 0 Feature: base Version: 1.0
License Type: Permanent
License State: Active, Not in Use
License Count: Non-Counted
License Priority: Medium
License Store: Evaluation License Storage
StoreIndex: 1 Feature: base-ap-count Version: 1.0
License Type: Evaluation
License State: Inactive
Evaluation total period: 8 weeks 4 days
Evaluation period left: 8 weeks 4 days
License Count: 500 / 0 (Active/In-use)
License Priority: None

(Cisco Controller) >show switchconfig

802.3x Flow Control Mode......................... Disable
FIPS prerequisite features....................... Disabled
WLANCC prerequisite features..................... Disabled
UCAPL prerequisite features...................... Disabled
Last login information display................... Disabled
DTLS WLC MIC .................................... SHA2
secret obfuscation............................... Enabled
Strong Password Check Features
case-check.................................... Enabled
consecutive-check............................. Enabled
default-check................................. Enabled
username-check................................ Enabled
position-check................................ Disabled
case-digit-check.............................. Disabled
Min. Password length.......................... 3
Min. Upper case chars......................... 0
Min. Lower case chars......................... 0
Min. Digits chars............................. 0
Min. Special chars............................ 0
Mgmt User
Password Lifetime [days]...................... 0

--More-- or (q)uit
Password Lockout.............................. Disabled
Lockout Attempts.............................. 3
Lockout Timeout [mins]........................ 5
SNMPv3 User
Password Lifetime [days]...................... 0
Password Lockout.............................. Disabled
Lockout Attempts.............................. 3
Lockout Timeout [mins]........................ 5

(Cisco Controller) >show ap dtls-version

DTLS Version..................................... dtls_all

(Cisco Controller) >show ap dtls-cipher-suite

DTLS Cipher Suite................................ RSA-AES128-SHA

(Cisco Controller) >show network summary

RF-Network Name............................. mpsc
DNS Server IP............................... 0.0.0.0
Web Mode.................................... Disable
Secure Web Mode............................. Enable
Secure Web Mode Cipher-Option High.......... Disable
Secure Web Mode SSL Protocol................ Disable
Web CSRF check.............................. Enable
OCSP........................................ Disabled
OCSP responder URL..........................
Secure Shell (ssh).......................... Enable
Secure Shell (ssh) Cipher-Option High....... Disable
Telnet...................................... Disable
Ethernet Multicast Forwarding............... Disable
Ethernet Broadcast Forwarding............... Disable
IPv4 AP Multicast/Broadcast Mode............ Unicast
IPv6 AP Multicast/Broadcast Mode............ Unicast
IGMP snooping............................... Disabled
IGMP timeout................................ 60 seconds
IGMP Query Interval......................... 20 seconds
MLD snooping................................ Disabled
MLD timeout................................. 60 seconds

--More-- or (q)uit
MLD query interval.......................... 20 seconds
User Idle Timeout........................... 300 seconds
ARP Idle Timeout............................ 300 seconds
Cisco AP Default Master..................... Disable
AP Join Priority............................ Enabled
Mgmt Via Wireless Interface................. Enable
Mgmt Via Dynamic Interface.................. Disable
Bridge MAC filter Config.................... Enable
Bridge Security Mode........................ EAP
Mesh Full Sector DFS........................ Enable
Mesh Backhaul RRM........................... Disable
AP Fallback ................................ Enable
AP EasyAdmin ............................... Disable
AP Virtual IP .............................. 0.0.0.0
Web Auth CMCC Support ...................... Disabled
Web Auth Redirect Ports .................... 80
Web Auth Proxy Redirect ................... Disable
Web Auth Captive-Bypass .................. Disable
Web Auth Secure Web ....................... Enable
Web Auth Secure Web Cipher Option ......... Disable
Web Auth Secure Web Sslv3 ................. Disable
Web Auth Secure Redirection ............... Disable
Fast SSID Change ........................... Disabled

--More-- or (q)uit
AP Discovery - NAT IP Only ................. Enabled
IP/MAC Addr Binding Check .................. Enabled
Link Local Bridging Status ................. Disabled
CCX-lite status ............................ Disable
oeap-600 dual-rlan-ports ................... Disable
oeap local-network ......................... Enable
oeap-600 Split Tunneling (Printers)......... Disable
WebPortal Online Client .................... 0
WebPortal NTF_LOGOUT Client ................ 0
mDNS snooping............................... Disabled
mDNS Query Interval......................... 15 minutes
Web Color Theme............................. Default
Capwap Prefer Mode.......................... IPv4
Network Profile............................. Disabled
Client ip conflict detection (DHCP) ........ Disabled
Mesh BH RRM ................................ Disable
Mesh Aggressive DCA......................... Disable
Mesh Auto RF................................ Disable
HTTP Profiling Port......................... 80
HTTP-Proxy Ip Address....................... 0.0.0.0
HTTP-Proxy Port............................. 80
WGB Client Forced L2 Roam................... Disabled

(Cisco Controller) >show nmheartbeat summary

Network Manager Heart Beat is disabled
Network Manager Heart Beat Interval.............. 180 seconds

Leo Laohoo · ‎12-04-2020

First, factory erase the config of the AP using the command "capwap ap erase all".

After the AP reboots, post the complete output to the following command: show ap join stats detailed <AP NAME>

Rafael Rubik · ‎12-05-2020

Hello Leo,

after erase, he tried to associate with my wlc-2. see the commands output and images attached:

(Cisco Controller) >show ap join stats detailed 08:4f:a9:e9:0a:84

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 159
- Successful discovery responses sent...................... 159
- Unsuccessful discovery request processing................ 0
- Reason for last unsuccessful discovery attempt........... Not applicable
- Time at last successful discovery attempt................ Dec 05 10:58:34.538
- Time at last unsuccessful discovery attempt.............. Not applicable

Join phase statistics
- Join requests received................................... 5
- Successful join responses sent........................... 5
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Dec 05 10:58:44.328
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics

--More-- or (q)uit
- Configuration requests received.......................... 5
- Successful configuration responses sent.................. 2
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Dec 05 10:58:55.258
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Timed out while waiting for ECHO repsonse from the AP
- Last AP disconnect reason................................ Unknown failure reason

Last join error summary
- Type of error that occurred last......................... AP got or has been disconnected
- Reason for error that occurred last...................... Timed out while waiting for ECHO repsonse from the AP
- Time at which the last join error occurred............... Dec 05 10:55:52.381

AP disconnect details
- Reason for last AP connection failure.................... Timed out while waiting for ECHO repsonse from the AP
Ethernet Mac : 08:4f: a9:e9:0a:84 Ip Address : 10.15.102.11

AP084F.A9E9.0A84# show ap logg (loop)

Dec 5 14:02:49 kernel: [*12/05/2020 14:02:49.6748] Max retransmission count exceeded, going back to DISCOVER mode.
Dec 5 14:02:49 kernel: [*12/05/2020 14:02:49.6748] GOING BACK TO DISCOVER MODE
Dec 5 14:02:49 kernel: [*12/05/2020 14:02:49.6754]
Dec 5 14:02:49 kernel: [*12/05/2020 14:02:49.6754] CAPWAP State: DTLS Teardown
Dec 5 14:02:50 kernel: [*12/05/2020 14:02:50.6845] Dropping dtls packet since session is not established. Peer 10.15.224.5-5246, Local 10.15.102.11-5272, conn (nil)
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4819]
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4819] CAPWAP State: Discovery
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4827] IP DNS query for CISCO-CAPWAP-CONTROLLER.ad.mpsc.mp.br
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4935] DNS resolved CISCO-CAPWAP-CONTROLLER.ad.mpsc.mp.br
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4935] DNS discover IP addr: 10.15.224.5
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4935] DNS discover IP addr: 10.4.24.25
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4935] DNS discover IP addr: 10.4.8.25
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4948] Discovery Request sent to 10.15.224.5, discovery type STATIC_CONFIG(1)
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.4959] Discovery Request sent to 10.4.24.25, discovery type STATIC_CONFIG(1)
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5000] Discovery Request sent to 10.15.152.5, discovery type STATIC_CONFIG(1)
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5011] Discovery Request sent to 10.4.8.25, discovery type STATIC_CONFIG(1)
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5022] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5049] Discovery Response from 10.15.224.5
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5127] Discovery Response from 10.4.24.25
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5201] Discovery Response from 10.15.152.5
Dec 5 14:02:55 kernel: [*12/05/2020 14:02:55.5274] Discovery Response from 10.4.8.25
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.0000]
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.0000] CAPWAP State: DTLS Setup
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.4123]
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.4123] CAPWAP State: Join
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.4137] Sending Join request to 10.15.224.5 through port 5272
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.4276] Join Response from 10.15.224.5
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5094] HW CAPWAP tunnel is ADDED
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5243]
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5243] CAPWAP State: Image Data
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5618] do NO_UPGRADE, part2 is active part
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5652]
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5652] CAPWAP State: Configure
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5672] DOT11_CFG[0] Radio Mode is changed from Local to Local
Dec 5 14:03:05 kernel: [*12/05/2020 14:03:05.5677] DOT11_CFG[1] Radio Mode is changed from Local to Local
Dec 5 14:03:08 kernel: [*12/05/2020 14:03:08.4206] Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
Dec 5 14:03:08 kernel: [*12/05/2020 14:03:08.4206]
Dec 5 14:03:11 kernel: [*12/05/2020 14:03:11.2714] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
Dec 5 14:03:11 kernel: [*12/05/2020 14:03:11.2714]
Dec 5 14:03:11 kernel: [*12/05/2020 14:03:11.6274] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).
Dec 5 14:03:14 kernel: [*12/05/2020 14:03:14.1223] Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
Dec 5 14:03:14 kernel: [*12/05/2020 14:03:14.1223]
Dec 5 14:03:16 kernel: [*12/05/2020 14:03:16.9731] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
Dec 5 14:03:16 kernel: [*12/05/2020 14:03:16.9731]
Dec 5 14:03:19 kernel: [*12/05/2020 14:03:19.8239] Re-Tx Count=5, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
Dec 5 14:03:19 kernel: [*12/05/2020 14:03:19.8239]
Dec 5 14:03:22 kernel: [*12/05/2020 14:03:22.6749] Re-Tx Count=6, Max Re-Tx Value=5, SendSeqNum=1, NumofPendingMsgs=1
Dec 5 14:03:22 kernel: [*12/05/2020 14:03:22.6749]
Dec 5 14:03:22 kernel: [*12/05/2020 14:03:22.6749] Max retransmission count exceeded, going back to DISCOVER mode.
Dec 5 14:03:22 kernel: [*12/05/2020 14:03:22.6750] GOING BACK TO DISCOVER MODE

MHM Cisco World · ‎12-04-2020

...

MHM Cisco World · ‎12-04-2020

http://www.my80211.com/cisco-wlc-cli-commands/2011/2/20/wlc-predownload-the-image-to-the-access-points-from-the-cont.html

pre download image in AP may cause this issue,

check predownlaod image in AP and image of WLC use for AP.

Rafael Rubik · ‎12-05-2020

Hello MHM,

I took the switch and access points to another site and everything works normally. Associate with all 4 WLCs.

I don´t believe in image issue. It seems WAN connectivity.

Tks

Leo Laohoo · ‎12-05-2020

@Rafael Rubik wrote:

Reason for last AP connection failure.................... Timed out while waiting for ECHO repsonse from the AP

The error message tells me it is a FW issue.

ajc · ‎12-06-2020

Hi Rafael,

Leo is correct, IF after the factory reset your AP is trying to connect to WLC-2 which has a FW in the middle based on your initial explanation, then the NEW LAN Subnet where the AP is connected, is not allowed on that FW so that is why the AP registration is not completed. Are you using Option 43 for in the DHCP Server for the New LAN Subnet?. If it so, change the WLC value from WLC-2 to WLC-1 and try again or manually force in the AP CLI the WLC to be connected (I prefer the DHCP option 43 instead of manipulating individual AP's).

Rafael Rubik · ‎12-07-2020

Hello ajc and Leo

the AP-Switch connectivity is ok, so I can do SSH. I moved the AP to WLC-1 again with the command: capwap ap primary-base. Problem persists. We also did a new test, we changed LAN address (/24) for the site. No sucess.

debug dtls trace enable (output file is attached)

Tks

Scott Fella · ‎12-07-2020

I think this answers your question. It works at another location, so you eliminated the switch, ap and controller. Now you have the path and FW to troubleshoot.

-Scott
*** Please rate helpful posts ***

Rafael Rubik · ‎12-07-2020

Hello Scott,

Yes, there is no FW on this path (site -> WLC-1). I have the WAN / MPLS team in troubleshoot right now, but they haven't found anything since 5 days. Do you suggest any specific test / tshoot to run on the router?

Tks

Scott Fella · ‎12-07-2020

The only other thing you can try is to lower the mtu on the wlc to 1250 and see if that helps. Maybe they are fragmenting the packets, I don't know.

-Scott
*** Please rate helpful posts ***

Rafael Rubik · ‎12-08-2020

Hello Scott

mtu to 1250. Problem persists.

Tks !