DTLS Error

[roycemathewv]

 All,

I am getting error as below;

%DTLS-3-HANDSHAKE_FAILURE: [PA]openssl_dtls.c:3191 Failed to complete DTLS handshake with peer <AP IP>

I have few AP's connected over Sat link and this was all online and all of a sudden AP's went offline and started throwing error for any AP connected from this site. 

Any guidance in identifying root cause? We have left site with no wireless and its important to get this up and running.

Controller - 5508

8.3.143 Version, in HA

[marce1000]

 - Make sure the Sat link has not worsened the latency value for which a max value is allowed of 300ms for APWAP connections.

  M.

[roycemathewv]

The link RTT is always 550 to 600ms and it has worked in the past

[Scott Fella]

I think you got lucky but you might of been on the edge of it working. Maybe there has been a slight increase in latency that causing the aps not to join. Max latency for Flex as you know is 300ms and that is the first thing TAC would verify.

[Rasika Nayanajith]

Pls check your AP cert status. Pls check below for workaround

https://community.cisco.com/t5/wireless-mobility-documents/lightweight-ap-fail-to-create-capwap-lwapp-connection-due-to/ta-p/3155111 

HTH

Rasika

*** Pls rate all useful responses ***

[roycemathewv]

The WAP we have shows Manufacture date as 2019. So will this be an issue still?

Also we have set to ignore check on MIC/SSC.

(Cisco Controller) >show certificate summary
Web Administration Certificate................... 3rd Party
Web Authentication Certificate................... 3rd Party
Certificate compatibility mode:.................. off
Lifetime Check Ignore for MIC ................... Enable
Lifetime Check Ignore for SSC ................... Enable

[Rasika Nayanajith]

What AP models you having this issue ?

Is it possible to get some debug filter by a affected AP ethernet MAC Address

(WC01) >debug client <affected_ap_eth_mac>

(WC01) >debug capwap errors enable

(WC01) >debug capwap events enable

Pls note that using below command you can disable debugging

(WC01) >debug disable-all

HTH

Rasika

[roycemathewv]

AP model is AIR-AP2802I-Z-K9. 

Some logs 

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 300, MaxLicense=200 joined Aps =130
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 apType = 56 apModel: AIR-AP2802I-Z-K9

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 apType: Ox38 bundleApImageVer:
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Could not find image version of bundled AP(apType: 56)!!!
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Unable to get AP Bundled Version. Using Controller Version!!!

*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 Discovery Request from <ip>:5248

*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 ApModel: AIR-AP2802I-Z-K9

*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 300, MaxLicense=200 joined Aps =130
*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 a

*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 apType: Ox38 bundleApImageVer:
*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 Could not find image version of bundled AP(apType: 56)!!!
*spamApTask6: May 28 14:45:56.982: [PA] 50:2f:a8:81:24:80 Unable to get AP Bundled Version. Using Controller Version!!!

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Discovery Request from <ip>:5264

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 ApModel: AIR-AP2802I-Z-K9

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 300, MaxLicense=200 joined Aps =130
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 apType = 56 apModel: AIR-AP2802I-Z-K9

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 apType: Ox38 bundleApImageVer:
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Could not find image version of bundled AP(apType: 56)!!!
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Unable to get AP Bundled Version. Using Controller Version!!!

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Discovery Response sent to <ip> port 5264

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Discovery Response sent to <ip>:5264

[Sandeep Choudhary]

Post the complete output to the WLC command "sh sysinfo" and "sh time".

[roycemathewv]

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.3.143.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... WLC
System Location.................................. 
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. SSO
IP Address....................................... 10.180.x.x
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 7 days 22 hrs 40 mins 40 secs
System Timezone Location......................... (GMT +8:00) HongKong, Bejing, Chongquing
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries : AU,GB,J2,KE,SG
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +18 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 8
Number of Active Clients......................... 144

OUI Classification Failure Count................. 0

Burned-in MAC Address............................ 00:24:97:6A:03:00
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1

--More-- or (q)uit
(Cisco Controller) >
(Cisco Controller) >show time

Time............................................. Thu May 28 17:42:00 2020

Timezone delta................................... 0:0
Timezone location................................ (GMT +8:00) HongKong, Bejing, Chongquing

NTP Servers
NTP Polling Interval......................... 3600

Index NTP Key Index NTP Server Status NTP Msg Auth Status
------- ----------------------------------------------------------------------------------------------
1 0 NTP In Sync AUTH DISABLED

[Scott Fella]

Just out of curiosity, how many of these 2802’s are at that site and how long had they been working? Can you also send the show ap join stats summary all? And point out the hostname of the aps that no longer work?

[roycemathewv]

AIR-AP2802I-Z-K9 - Model

[roycemathewv]

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Discovery Request from <ip>:5264

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 ApModel: AIR-AP2802I-Z-K9

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 300, MaxLicense=200 joined Aps =130
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 apType = 56 apModel: AIR-AP2802I-Z-K9

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 apType: Ox38 bundleApImageVer:
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Could not find image version of bundled AP(apType: 56)!!!
*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Unable to get AP Bundled Version. Using Controller Version!!!

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Discovery Response sent to <ip> port 5264

*spamApTask5: May 28 14:45:58.474: [PA] c4:b2:39:0b:bc:a0 Discovery Response sent to <ip>:5264

[Rasika Nayanajith]

If it is 2802, you may be hitting this bug listed in given Field Notice

CSCvb93909-AP-COS: AP not joining after enabling MIC certificate expiry check

https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html 

If you hitting this bug, upgrade 8.5.161.0 may be the fix

AP-COS APs can be fixed via CSCvb93909 in AireOS 8.5 and later

If upgrade not possible, you can try below workaround

Workaround:
A temporary workaround is to set the WLC's date back (you will need to disable NTP, if enabled.)

If you have upgraded to software with this fix, then:
* configure "config ap cert-expiry-ignore mic enable" on the WLC
* set your date back, so that unfixed APs can join
* wait till all AP-COS APs have downloaded the fixed code, and have rebooted and rejoined
* now you can set the WLC date back to the current date

HTH

Rasika

*** Pls rate all useful responses ***