wlc 2504 APs not joining time is correct

[Elesh]

My APs are not joining the controller all of a sudden. The APs used to work fine but after a reboot ALL of them are not joining. 

I set the date and time manually and set a NTP server and rebooted, but still have the same issue. I can ping the APs.

Under Statistics -> ap join, I select the AP. Under discovery phase stats, it shows last successful attempt with a recent time, and last unsuccessful attemp is blank. There are no errors, all blank

One funny thing I notice when i run  Show Sysinfo is this line:

OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

Does anyone have any ideas?

[Leo Laohoo]

Post the complete output to the following commands: 

1. WLC:  sh sysinfo
2. AP:  sh version

[Elesh]

Here is the output form the commands:

wlc: sh sysinfo
(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.151.0
Bootloader Version............................... 1.0.16
Field Recovery Image Version..................... 1.0.0
Firmware Version................................. PIC 16.0

OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... Cisco_db:cc:e5
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1279
IP Address....................................... 192.168.1.200
IPv6 Address..................................... ::
Last Reset....................................... Power on reset
System Up Time................................... 0 days 0 hrs 2 mins 21 secs
System Timezone Location......................... (GMT -8:00) Pacific Time (US a nd Canada)

--More-- or (q)uit
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... US - United States
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +27 C
External Temperature............................. +30 C
Fan Status....................................... 4800 rpm

State of 802.11b Network......................... Disabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 1
Number of Active Clients......................... 0

OUI Classification Failure Count................. 0

Memory Current Usage............................. 34
Memory Average Usage............................. 36
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card

--More-- or (q)uit
Flash Size....................................... 1073741824

Burned-in MAC Address............................ 64:D8:14:DB:CC:E0
Maximum number of APs supported.................. 75
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1

AP: sh version

ROM: Bootstrap program is U-Boot boot loader
BOOTLDR: U-Boot boot loader Version 30

APC064.E444.0628 uptime is 0 days, 0 hours, 8 minutes
Last reload time : Sun Oct 2 14:57:11 UTC 2022
Last reload reason : Capwap Discovery Failed

memory. -AP1832I-B-K9 ARMv7 Processor rev 0 (v7l) with 997268/801636K bytes of --More--
Processor board ID KWC232506ZF
AP Running Image : 8.5.151.0
Primary Boot Image : 8.5.151.0
Backup Boot Image : 8.3.143.4
1 Gigabit Ethernet interfaces
2 802.11 Radios
Radio FW version : f6d58b5e05c762162e07fe70e0671647
NSS FW version : NSS.AK.C.CS-3-fix3

Base ethernet MAC Address : C0:64:E4:44:06:28
Part Number : 0-0000-00
PCA Assembly Number : 074-104694-02
PCA Revision Number : 01
PCB Serial Number : KWC232506ZF
Top Assembly Part Number : 074-104694-02
Top Assembly Serial Number : KWC232506ZF
Top Revision Number : A0
Product/Model Number : AIR-AP1832I-B-K9

[Rich R]

Read this field notice very carefully, twice if necessary: https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html

I've posted a summary of this numerous times before - if you search you'll find it - which is what you should be doing before posting the same thing many others have ...

[Elesh]

I've tried to search but was unable to find your post.. Could you provide a link?

[Rich R]

Admittedly the search is not very good: https://community.cisco.com/t5/wireless/problem-getting-lap-s-to-re-regiser-with-wlc/m-p/4691443/highlight/true#M246270

[Elesh]

Another thing that I noticed is the the NTP is not syncng. Here is my output for sh time:

Time............................................. Sun Oct 2 13:56:58 2022

Timezone delta................................... 0:0
Timezone location................................ (GMT -8:00) Pacific Time (US and Canada)

NTP Servers
NTP Version.................................. 3
NTP Polling Interval......................... 3500

Index NTP Key Index NTP Server Status NTP Msg Auth Status
------- ---------------------------------------------------------------------
1 0 0.pool.ntp.org Not Synched AUTH DISABLED
2 0 1.pool.ntp.org Not Synched AUTH DISABLED
3 0 time.windows.com In Progress AUTH DISABLED

[CarlosdeAssuncao]

Try only set date and time without NTP.

Do you have Mac ACL for the APS?

[Rich R]

Cisco devices (certainly IOS anyway) have a habit of looking up the DNS for those names at boot time only and never again.  So if the IP address changes they will never know - they still keep trying the same old IP forever.  I suspect you'll either need to add IP addresses and check them regularly or delete and re-add the names to force new IP lookup.  You need to disable NTP to fix your cert problem anyway.

[Rich R]

ps: the OUI will be the one which came with the WLC code unless you update it manually but beware: https://bst.cisco.com/bugsearch/bug/CSCvg66702 OUI Update failure and WLC System reloads unexpectedly while updating the OUI file
Also updating the OUI doesn't actually add new OUI's it just updates the ones it already has if they've changed so pretty pointless most of the time.
The only way to really get updates properly is with a new version of WLC code but Cisco's support for that is quite limited now.
What they really want you to do is use ISE for device identification - that gets regular updates.

[Elesh]

I have removed the NTP servers, and it still is having issues. So I connect to a AP using the console port, here is what I see. It seems the certificate has expired? But why would it expire.. Is this an easy fix?

[*10/04/2022 22:00:14.0484] CAPWAP State: Discovery
[*10/04/2022 22:00:14.0484] IP DNS query for CISCO-CAPWAP-CONTROLLER.lan
[*10/04/2022 22:00:14.0584] Discovery Request sent to 172.16.32.8, discovery typ e STATIC_CONFIG(1)
[*10/04/2022 22:00:14.0584] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*10/04/2022 22:00:14.0584] Discovery Response from 172.16.32.8
[*10/04/2022 22:00:14.0784] Discovery Response from 172.16.32.8
[*10/04/2022 22:00:23.0000]
[*10/04/2022 22:00:23.0000] CAPWAP State: DTLS Setup
[*10/04/2022 22:00:23.0000] dtls_connectionDB_add_connection: Number of DTLS con nections exceeded two
[*10/04/2022 22:00:23.2199] Certificate is expired
[*10/04/2022 22:00:23.2199] Certificate Start Date: Aug 15 12:24:27 2012 GMT
[*10/04/2022 22:00:23.2199] Certificate End Date: Aug 15 12:34:27 2022 GMT
[*10/04/2022 22:00:23.2199] display_verify_cert_status: Verify Cert: FAILED at 0 depth: certificate has expired
[*10/04/2022 22:00:23.2199] X509 OpenSSL Errors...
[*10/04/2022 22:00:23.2199]
[*10/04/2022 22:00:23.2199] NONE
[*10/04/2022 22:00:23.2199]
[*10/04/2022 22:00:23.2199]
[*10/04/2022 22:00:23.2199] dtls_verify_con_cert: Controller certificate verific ation error
[*10/04/2022 22:00:23.2199] dtls_process_packet: Controller certificate verifica tion failed
[*10/04/2022 22:00:23.2199] sendPacketToDtls: DTLS: Closing connection 0x18ea000 .
[*10/04/2022 22:00:23.2199]
[*10/04/2022 22:00:23.2199] Lost connection to the controller, going to restart CAPWAP (reason : dtls_rc_connection_closed)...
[*10/04/2022 22:00:23.2199]
[*10/04/2022 22:00:23.2199] Restarting CAPWAP State Machine.
[*10/04/2022 22:00:23.3299] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C APWAP state: DTLS Setup(3).
[*10/04/2022 22:00:23.3299] Failed to disconnect DTLS-CTRL session.
[*10/04/2022 22:00:23.3299]
[*10/04/2022 22:00:23.3299] CAPWAP State: DTLS Teardown
[*10/04/2022 22:00:23.3499] DTLS: Error while processing DTLS packet 0x1934000.
[*10/04/2022 22:00:27.9285] No more AP manager addresses remain..
[*10/04/2022 22:00:27.9285] No valid AP manager found for controller 'Cisco_db:c c:e5' (ip: 172.16.32.8)
[*10/04/2022 22:00:27.9285] Failed to join controller Cisco_db:cc:e5.
[*10/04/2022 22:00:27.9285] Failed to join controller.

[Rich R]

Did you read the field notice or my summary like I suggested? 
Or you just disabled NTP and hoped that would somehow magically fix everything - it will not.  You need to follow ALL the steps in the correct order to resolve the issue.

> But why would it expire..
Because Cisco issued 10 year Manufacturing Installed Certificates (MIC) which expire - yes you guessed it - 10 years after manufacture.  If you'd read the field notice you'd know this already.

[Elesh]

I apologize rrudling, I sure feel like an a**. 

I went through the field notice briefly just now and will go into detail when I'm on site again.. Thanks.

[Rich R]

LOL fair enough glad we got there in the end!

[Elesh]

Just to be clear. Do I need to reset the time to the past and  run config ap cert-expiry-ignore {mic|ssc} enable on the WLC, or on each AP?