WLC 2500 AP registration problem

[CSteve]

Hi.

Currently at my job we had a long power loss, and neither one of our ap s cannot register to our wlc 2500 series. I m scratching my head with this problem , and i m out of ideas. Any input would be appreciated.

Some info: 

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 17

Base Mac AP EthernetMac AP Name IP Address Status
0c:27:24:e1:de:40 N A (4/A) 192.168.5.50 Not Joined
0c:27:24:e1:e1:a0 N A (2/A) 192.168.5.41 Not Joined
50:17:ff:56:17:f0 N A (3/B) 192.168.5.44 Not Joined
50:17:ff:f4:7a:50 N A (-1/B) 192.168.5.55 Not Joined
50:17:ff:f4:7b:a0 N A (2/B) 192.168.5.52 Not Joined
50:17:ff:f4:7c:20 N A (4/B) 192.168.5.47 Not Joined
50:17:ff:f4:7d:50 N A (0/A) 192.168.5.48 Not Joined
50:17:ff:f4:7f:a0 N A (1/B) 192.168.5.57 Not Joined
50:17:ff:f4:81:20 N A Konferencia2/B 192.168.5.42 Not Joined
50:17:ff:f4:c9:f0 N A 4A 192.168.5.45 Not Joined
50:17:ff:f4:cd:70 N A (0/B) 192.168.5.40 Not Joined
50:17:ff:f4:cf:10 N A (5/B) 192.168.5.58 Not Joined
50:17:ff:f4:fa:d0 N A (6/B) 192.168.5.49 Not Joined
e8:ed:f3:1a:6e:30 N A (3/B-hatso) 192.168.5.51 Not Joined
e8:ed:f3:1a:c8:20 N A (3/A) 192.168.5.56 Not Joined
e8:ed:f3:1a:c9:70 N A (7/B) 192.168.5.54 Not Joined
e8:ed:f3:1a:cd:20 N A (1/A) 192.168.5.46

(Cisco Controller) >show ap join stats detail e8:ed:f3:1a:cd:20

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 58
- Successful discovery responses sent...................... 58
- Unsuccessful discovery request processing................ 0
- Reason for last unsuccessful discovery attempt........... Not applicable
- Time at last successful discovery attempt................ Sep 04 10:44:02.466
- Time at last unsuccessful discovery attempt.............. Not applicable

Join phase statistics
- Join requests received................................... 0
- Successful join responses sent........................... 0
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Not applicable
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics

--More-- or (q)uit
(Cisco Controller) >show time

Time............................................. Mon Sep 4 10:46:37 2023

Timezone delta................................... 0:0
Timezone location................................

NTP Servers
NTP Polling Interval......................... 3600

Index NTP Key Index NTP Server NTP Msg Auth Status
------- ---------------------------------------------------------------
1 0 192.168.5.1 AUTH DISABLED

(Cisco Controller) >show ap join stats detailed e8:ed:f3:1a:cd:20

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 58
- Successful discovery responses sent...................... 58
- Unsuccessful discovery request processing................ 0
- Reason for last unsuccessful discovery attempt........... Not applicable
- Time at last successful discovery attempt................ Sep 04 10:44:02.466
- Time at last unsuccessful discovery attempt.............. Not applicable

Join phase statistics
- Join requests received................................... 0
- Successful join responses sent........................... 0
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Not applicable
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics

--More-- or (q)uit
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Not applicable
- Last AP disconnect reason................................ Not applicable

Last join error summary
- Type of error that occurred last......................... None
- Reason for error that occurred last...................... Not applicable
- Time at which the last join error occurred............... Not applicable

AP disconnect details
- Reason for last AP connection failure.................... Not applicable

(Cisco Controller) >show running-config

Notice: "show running-config" has been changed to be an alias to "show run-config".
Use "show run-config commands" to display the configuration commands.
Press Enter to continue or <Ctrl-Z> to abort...

System Inventory
NAME: "Chassis" , DESCR: "Cisco 2500 Series Wireless LAN Controller"
PID: AIR-CT2504-K9, VID: V01, SN: PSZ172701KW

Burned-in MAC Address............................ 4C:00:82:59:77:A0
Maximum number of APs supported.................. 21
Press Enter to continue or <ctrl-z> to abort

System Information
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.100.0
Bootloader Version............................... 1.0.18
Field Recovery Image Version..................... 1.0.0
Firmware Version................................. PIC 16.0

Build Type....................................... DATA + WPS

System Name...................................... Cisco_59:77:a4
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1279
IP Address....................................... 192.168.5.5
Last Reset....................................... Power on reset
System Up Time................................... 0 days 1 hrs 4 mins 32 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

--More or (q)uit current module or <ctrl-z> to abort
Configured Country............................... RO - Romania
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +26 C
External Temperature............................. +31 C
Fan Status....................................... 4700 rpm

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 3
Number of Active Clients......................... 0

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 4C:00:82:59:77:A0
Maximum number of APs supported.................. 21
Press Enter to continue or <ctrl-z> to abort

AP Bundle Information

Primary AP Image Size
---------------- ----
ap1g2 9556
ap3g1 11268
ap3g2 11188
ap801 7152
ap802 8548
c1130 5072
c1140 9408
c1250 6944
c1520 8032
c602i 3736

Secondary AP Image Size
------------------ ----
ap1g2 9556
ap3g1 11268
ap3g2 11188
ap801 7152
ap802 8548

--More or (q)uit current module or <ctrl-z> to abort
c1130 5072
c1140 9408
c1250 6944
c1520 8032
c602i 3736
Press Enter to continue or <ctrl-z> to abort

Switch Configuration
802.3x Flow Control Mode......................... Disable
FIPS prerequisite features....................... Disabled
secret obfuscation............................... Enabled
Strong Password Check Features:

case-check ...........Enabled
consecutive-check ....Enabled
default-check .......Enabled
username-check ......Enabled
Press Enter to continue or <ctrl-z> to abort

Network Information
RF-Network Name............................. Sap
Web Mode.................................... Enable
Secure Web Mode............................. Disable
Secure Web Mode Cipher-Option High.......... Disable
Secure Web Mode Cipher-Option SSLv2......... Disable
Secure Web Mode RC4 Cipher Preference....... Disable
OCSP........................................ Disabled
OCSP responder URL..........................
Secure Shell (ssh).......................... Enable
Telnet...................................... Disable
Ethernet Multicast Forwarding............... Disable
Ethernet Broadcast Forwarding............... Disable
AP Multicast/Broadcast Mode................. Multicast Address : 239.0.0.1
IGMP snooping............................... Disabled
IGMP timeout................................ 60 seconds
IGMP Query Interval......................... 20 seconds
MLD snooping................................ Disabled
MLD timeout................................. 60 seconds
MLD query interval.......................... 20 seconds
User Idle Timeout........................... 300 seconds
ARP Idle Timeout............................ 300 seconds

--More or (q)uit current module or <ctrl-z> to abort
Cisco AP Default Master..................... Disable
AP Join Priority............................ Disable
Mgmt Via Wireless Interface................. Disable
Mgmt Via Dynamic Interface.................. Disable
Bridge MAC filter Config.................... Enable
Bridge Security Mode........................ EAP
Mesh Full Sector DFS........................ Enable
AP Fallback ................................ Enable
Web Auth CMCC Support ...................... Disabled
Web Auth Redirect Ports .................... 80
Web Auth Proxy Redirect ................... Disable
Web Auth Captive-Bypass .................. Disable
Web Auth Secure Web ....................... Enable
Fast SSID Change ........................... Enabled
AP Discovery - NAT IP Only ................. Enabled
IP/MAC Addr Binding Check .................. Enabled
CCX-lite status ............................ Disable
oeap-600 dual-rlan-ports ................... Disable
oeap-600 local-network ..................... Enable
mDNS snooping............................... Disabled
mDNS Query Interval......................... 15 minutes
Press Enter to continue or <ctrl-z> to abort

Port Summary
STP Admin Physical Physical Link Link
Pr Type Stat Mode Mode Status Status Trap POE
-- ------- ---- ------- ---------- ---------- ------ ------- -------
1 Normal Forw Enable Auto 1000 Full Up Enable N/A
2 Normal Disa Enable Auto Auto Down Enable N/A
3 Normal Disa Enable Auto Auto Down Enable Enable (Power Off)
4 Normal Disa Enable Auto Auto Down Enable Enable (Power Off)
Press Enter to continue or <ctrl-z> to abort

AP Summary
Number of APs.................................... 0

Global AP User Name.............................. Not Configured
Global AP Dot1x User Name........................ Not Configured

AP Name Slots AP Model Ethernet MAC Location Port Country Priority
------------------ ----- -------------------- ----------------- ---------------- ---- ------- ------

AP Tcp-Mss-Adjust Info
AP Name TCP State MSS Size
------------------ -------- -------
Press Enter to continue or <ctrl-z> to abort

AP Location
Total Number of AP Groups........................ 0

Site Name........................................ default-group
Site Description................................. <none>
NAS-identifier................................... Cisco_59:77:a4
AP Operating Class............................... Not-configured

RF Profile
----------
2.4 GHz band..................................... <none>
5 GHz band....................................... <none>

WLAN ID Interface Network Admission Control Radio Policy
------- ----------- -------------------------- ------------
1 management Disabled None
2 ssid_sapientia Disabled None
3 ssid_sapientia_guest Disabled None

AP Name Slots AP Model Ethernet MAC Location Port Country Priority
------------------ ----- ------------------- ----------------- ---------------- ---- ------- --------
Press Enter to continue or <ctrl-z> to abort

RF Profile

Number of RF Profiles............................ 0

Out Of Box State................................. Disabled

RF Profile Name Band Description 11n-client-only
--------------------------------- ------- ----------------------------------- ---------------

Press Enter to continue or <ctrl-z> to abort

[Leo Laohoo]

What models are the APs?
Can you configure the timezone of the WLC?

[CSteve]

All of the APs are air cap 16021 e k9

(Cisco Controller) >show time

Time............................................. Mon Sep 4 14:54:25 2023

Timezone delta................................... 0:0
Timezone location................................ (GMT +3:00) Baghdad

NTP Servers
NTP Polling Interval......................... 3600

Index NTP Key Index NTP Server NTP Msg Auth Status
------- ---------------------------------------------------------------
1 0 192.168.5.1 AUTH DISABLED

[marce1000]

 - Post an AP boot process (console output) ; or if it still comes on the network (but does not join)  connect to it (with ssh e.g.) and issue the command show logging ; also check the controller logs when this (or  a) particular AP tries to join.

   Check if these commands can help (on the controller) :
                               config ap cert-expiry-ignore mic enable
                                   config ap cert-expiry-ignore ssc enable

 M.
                                                

[CSteve]

 Unfortunately i can t connect to the AP's , not on ssh , nor on telnet. If i try to connect to it on http i get the following message: The AP is not in FlexConnect mode. 

   Check if these commands can help (on the controller) :
                               config ap cert-expiry-ignore mic enable
                                   config ap cert-expiry-ignore ssc enable

These commands are not working on wlc 2504....

[CSteve]

The message logs on the controller are filled with :

*spamApTask3: Sep 04 15:43:09.099: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.56
*spamApTask3: Sep 04 15:43:08.982: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.42
*spamApTask7: Sep 04 15:43:08.937: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.41
*spamApTask2: Sep 04 15:43:08.697: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.54
*spamApTask3: Sep 04 15:43:08.529: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.47
*spamApTask3: Sep 04 15:43:08.386: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.51
*spamApTask0: Sep 04 15:43:08.267: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.40
*spamApTask1: Sep 04 15:43:08.148: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.57
*spamApTask4: Sep 04 15:43:08.144: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.44
*spamApTask5: Sep 04 15:43:07.912: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.52
*spamApTask1: Sep 04 15:43:06.172: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.50
*spamApTask1: Sep 04 15:43:05.126: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.45
*spamApTask0: Sep 04 15:42:38.679: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.46

[marce1000]

  - It would be advisable to get one of the APs and have it available for further testing ; issue these commands on it :
            debug dtls client error
            debug dtls client event  (color change is not important)

   - The mentioned commands earlier are available on 8.3.x and higher releases ;  so you could be hit by : 
                       https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html

              You could also disable NTP and set the controller time way back ; see what happens , 

 M.

[CSteve]

I set the year to an earlier date, and the aps could join. Looks like the certificates are not valid, because they are older than 10 years, as i ve read in a previous post. I should upgrade to FUS version 1.9 or higher, so i could disable the lifetime validity check

[marce1000 · Accepted solution]

                   >...I should upgrade to FUS version 1.9 or higher, so i could disable the lifetime validity check
  - This is not exactly correct , you do need indeed 8.3.x (aireos software version) or above to  disable the lifetime validity check as per  
               https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html
                                            recommended is 
             https://software.cisco.com/download/specialrelease/2702eede2b47a5c3bb40795bbe836af6 ; however that does indeed needs the FUS to be upgraded to 1.9 first!

  - Appendix  : because of the aireos based platforms getting older , it is advisable to run the last/latest available for it (per model)

  M.

[Leo Laohoo]

---

@CSteve wrote:
*spamApTask0: Sep 04 15:42:38.679: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.5.46

---

What country are you in?  That time is not correct.  

[CSteve]

Romania. Gmt +3. Already set Baghdad as timezone, which is in the same timezone as Romania.

[Leo Laohoo]

Turn off NTP. 

Roll back the date of the controller back to 2006.  Reboot the AP.

[Rich R]

It's covered in detail in field notice 63942 in my signature below.  You might also be affected by FN72524.

1. Roll back time as you've already done to allow APs to join.
2. Upgrade to latest available software which supports your APs and WLC - that's currently 8.5.182.11 (link below)>
3. Make sure you have the "config ap cert-expiry-ignore" commands configured.
4. Allow the APs to download new software and config change from WLC.
5. After that you can re-enable NTP and they should keep working with correct time.