3502i can't join WLC5508

[AyoubC]

Hello folks, 

reaching out here after reading lot of article and sorting many issues, but still APs can't join the controller, 

WLC and APs logs are attached, below some quick outputs 

** AP ** 

AP7c69.f694.0065#show version
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JBB6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Sun 01-Nov-15 20:52 by prod_rel_team

ROM: Bootstrap program is C3500 boot loader
BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 15.3 [vtoky-imagetype 106]

AP7c69.f694.0065 uptime is 5 hours, 36 minutes
System returned to ROM by reload
System image file is "flash:/ap3g1-k9w8-mx.153-3.JBB6/ap3g1-k9w8-xx.153-3.JBB6"
Last reload reason:

cisco AIR-CAP3502I-E-K9 (PowerPC460exr) processor (revision B0) with 98294K/32768K bytes of memory.
Processor board ID FCZ1732W007
PowerPC460exr CPU at 666Mhz, revision number 0x18A8
Last reset from reload
LWAPP image version 8.1.131.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 7C:69:F6:94:00:65
Part Number : 73-14857-01
PCA Assembly Number : 800-38797-01
PCA Revision Number : A0
PCB Serial Number : FOC1727AV2B
Top Assembly Part Number : 800-32891-02
Top Assembly Serial Number : FCZ1732W007
Top Revision Number : A0
Product/Model Number : AIR-CAP3502I-E-K9

AP7c69.f694.0065#show clock
*01:04:55.360 UTC Thu Dec 6 2018

** WLC ** 

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.6.120.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 1.27
Build Type....................................... DATA + WPS

System Name...................................... WLC01
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.9.8.3
Last Reset....................................... Software reset
System Up Time................................... 0 days 0 hrs 32 mins 5 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... FR - France

--More-- or (q)uit
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +35 C
External Temperature............................. +18 C
Fan Status....................................... OK

State of 802.11b Network......................... Disabled
State of 802.11a Network......................... Disabled
Number of WLANs.................................. 2
Number of Active Clients......................... 0

Burned-in MAC Address............................ 64:00:F1:F1:26:E0
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 25

(Cisco Controller) >show time

Time............................................. Thu Dec 6 01:04:24 2018

Your assistance please, 

[marce1000]

 - The controller software version is very old : 7.6.x , which seems incompatible with the ap running LWAPP image version 8.1.131.0 (from your outputs) , upgrade the controller to  https://software.cisco.com/download/home/282600534/type/280926587/release/8.5.171.0

 M.

[AyoubC]

got it, my APs have a mix of codes, 8.1 and 8.2, 

Can I upgrade from 7.6 to 8.2 ? is this upgrade will work with the AP w/ 8.1 as well ? 

[AyoubC]

my controller now is running 8.2 

(Cisco Controller) >show system

Incorrect usage. Use the '?' or <TAB> key to list commands.

(Cisco Controller) >show sysin

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Build Name....................................... Engg Special Image

Product Version.................................. 8.2.100.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 1.27
Build Type....................................... DATA + WPS

I'm still getting the same exact errors, any idea ? 

[marce1000]

 - Upgrade to https://software.cisco.com/download/home/282600534/type/280926587/release/8.5.171.0 , also check if the regulatory domain of the AP and the controller match, further  go to "Management > Licenses " add AP count  &  hit "Set count" button (if applicable or needed) , 

 M.

[AyoubC]

Hello @marce1000 

regulatory domains matches. APs are -E- and controller uses France - FR domaine, 

Licenses of 25 are applied already, 

Can I go directly from 8.2 to 8.5 ? my APs 8.1 and 8.2, we will be able to communicate with 8.5 ?  

[marce1000]

 - You can go directly to 8.5 , AP's sync capwap client software with controller and will be able to communicate with 8.5 , 

 M.

[AyoubC]

@marce1000 - Thank you very much for the guidance, now I'm on 8.5 version, 

now i m getting new errrors, 

*** WLC console ****

*spamApTask6: Dec 07 17:58:45.773: 7c:69:f6:94:00:65 Failed to create DTLS connection for 10.9.8.10.16391

*spamApTask4: Dec 07 17:58:47.064: Error retrieving LSC ID cert from WLC cert store
*spamApTask3: Dec 07 17:58:47.683: Error retrieving LSC ID cert from WLC cert store
*spamApTask3: Dec 07 17:58:49.053: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:58:53.723: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:58:53.724: 7c:69:f6:94:00:65 Failed to create DTLS connection for 10.9.8.10:16391

*spamApTask6: Dec 07 17:58:53.724: 7c:69:f6:94:00:65 Failed to create DTLS connection for 10.9.8.10.16391

*spamApTask3: Dec 07 17:58:57.004: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:59:20.443: Error retrieving LSC ID cert from WLC cert store
*spamApTask0: Dec 07 17:59:21.148: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:59:22.442: Error retrieving LSC ID cert from WLC cert store
*spamApTask0: Dec 07 17:59:23.147: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:59:26.443: Error retrieving LSC ID cert from WLC cert store
*spamApTask0: Dec 07 17:59:27.148: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:59:34.443: Error retrieving LSC ID cert from WLC cert store
*spamApTask0: Dec 07 17:59:35.147: Error retrieving LSC ID cert from WLC cert store
*spamApTask6: Dec 07 17:59:39.438: 7c:69:f6:94:00:65 Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.9.8.10:16391)since DTLS session is not established

FYI 

(Cisco Controller) >
(Cisco Controller) >show certificate summary
Web Administration Certificate................... Locally Generated
Web Authentication Certificate................... Locally Generated
Certificate compatibility mode:.................. on
Lifetime Check Ignore for MIC ................... Enable
Lifetime Check Ignore for SSC ................... Enable

*** AP Console *** 
*Dec 7 20:57:46.001: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.9.8.3:5246
*Dec 7 20:57:55.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.9.8.3 peer_port: 5246
*Dec 7 20:58:25.003: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x51D8368!

idea about? 

[marce1000]

 - Provide output of : show certificate lsc summary and show certificate lsc ap-provision on the WLC ,

 M.

[AyoubC]

I tried to disable anything that has certificate verification related feature, I don't care about sec for now, I want just the APs to be online, 

@marce1000 Please let me know if you think I'm doing something wrong, 

[AyoubC]

Adding more Expert folks to this conversation 

@Rasika Nayanajith 
@Leo Laohoo 
@Scott Fella 

[AyoubC]

@marce1000  - sounds I missed your previous response, sorry, here the output you asked me for, 

(Cisco Controller) >show certificate lsc summary

LSC Enabled...................................... No
LSC CA-Server.................................... None

LSC AP-Provisioning.............................. No

LSC Params:
Country......................................
State........................................
City.........................................
Orgn.........................................
Dept.........................................
Email........................................
CN...........................................
KeySize...................................... 2048

LSC Certs:
CA Cert...................................... Not Configured
RA Cert...................................... Not Configured
DEV Cert..................................... Not Configured

(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >show certificate lsc ap-provision

LSC AP-Provisioning.............................. No

(Cisco Controller) >

[marce1000]

 - Have a checkup of the controller configuration with WirelessAnalyzer , for that you need to prepare a config file as follows :  Type "config paging disable" in privileged mode to set your terminal to display without any breaks then type "show run-config" to display the config. This output  can then be used by https://cway.cisco.com/tools/WirelessAnalyzer/

 Also go to  Security > AP Policies : make sure  that any references to LSC (settings) are not used and or unchecked , 

 M.

[AyoubC]

Hello @marce1000 

Thank you so much for the help, I ended up by doing a factory default and reinstall licenses, 

and enable MIC/SSC cert-ignore, and the APs joined the controller, 

to be honest I don't know what was the problem, and hopefully this won't appear again, 

now I'm dealing with some connectivities issues

Thank you @marce1000 again!