AP3802i wont join 5508 controller

[Paul.Ostaszewski]

I have a 5508 controller running 8.3.150 that I have four AIR-CAP3702I-A-K9 running on currently.  I purchased four AIR-AP3802I-B-K9 to replace the four 3702's that I'm currently running. 

My SHA1 cert is expired.

Certificate Name: Cisco SHA1 device cert

Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT5508-K9-503de5aec9a0, emailAddress=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Validity :
Start : Apr 8 13:39:08 2011 GMT
End : Apr 8 13:49:08 2021 GMT

I have disabled NTP and set the clock on the 5508 back to before the certificate expires. 

I have also ran this command: config ap cert-expiry-ignore mic enable

I have a 25 license count for which 4 are being used by the 3702's.

Here is the debug from the controller:

*sshpmLscTask: Apr 08 04:03:02.699: sshpmLscTask: LSC Task received a message 4
*spamApTask5: Apr 08 04:03:34.746: 70:df:2f:05:0e:38 Failed to parse CAPWAP packet from 10.10.10.211:5248

*spamApTask5: Apr 08 04:03:44.230: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask5: Apr 08 04:03:44.230: sshpmGetCID: failed to find matching cert name cscoSha2IdCert

*spamApTask5: Apr 08 04:03:44.230: GetIDCert: Using SHA2 Id cert on WLC

*spamApTask5: Apr 08 04:03:44.230: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask5: Apr 08 04:03:44.230: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask5: Apr 08 04:03:44.230: Get Cert from CID: For CID 1db5a6a2 certType 1
*spamApTask5: Apr 08 04:03:44.230: Get Cert from CID: Found match of ID Cert in row 2
*spamApTask5: Apr 08 04:03:44.230: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask5: Apr 08 04:03:44.230: sshpmGetCID: failed to find matching cert name cscoSha2IdCert

*spamApTask5: Apr 08 04:03:44.231: GetDERIDKey: Using SHA2 Id cert Private Keys on WLC

*spamApTask5: Apr 08 04:03:44.231: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask5: Apr 08 04:03:44.231: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask5: Apr 08 04:03:44.231: GetPrivateKey: called to get key for CID 1db5a6a2

*spamApTask5: Apr 08 04:03:44.231: Private Key found row 2 KeyBufLen 2048 Keylen 1191 PrivateKeyPtr 0x2c4945b0

*spamApTask5: Apr 08 04:03:44.271: OpenSSL Get Issuer Handles: locking ca cert table

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: x509 subject_name /serialNumber=PID:AP3800 SN:FOC212448KU/O=Cisco/OU=ACT-2 Lite SUDI/CN=AP3800

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: issuer_name /O=Cisco/CN=ACT2 SUDI CA

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: CN AP3800

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: issuerCertCN ACT2 SUDI CA

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: Cert Name in subject is AP3800

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: Extracted cert issuer from subject name.

*spamApTask5: Apr 08 04:03:44.272: NMSP:: Algo name matched SHA256

*spamApTask5: Apr 08 04:03:44.272: ACT2 RSA SHA1 certificate

*spamApTask5: Apr 08 04:03:44.272: ACT2 dummy mac: MAC: 1122.3344.5566

*spamApTask5: Apr 08 04:03:44.272: OpenSSL Get Issuer Handles: Cert is issued by Cisco Systems.

*spamApTask5: Apr 08 04:03:44.272: Retrieving x509 cert for CertName cscoAct2RsaCaCert

*spamApTask5: Apr 08 04:03:44.272: sshpmGetCID: called to evaluate <cscoAct2RsaCaCert>

*spamApTask5: Apr 08 04:03:44.272: sshpmGetCID: Found matching CA cert cscoAct2RsaCaCert in row 8
*spamApTask5: Apr 08 04:03:44.272: Found CID 2b40476b for certname cscoAct2RsaCaCert

*spamApTask5: Apr 08 04:03:44.272: CACertTable: Found matching CID cscoAct2RsaCaCert in row 8 x509 0x2cc7be3c
*spamApTask5: Apr 08 04:03:44.273: Retrieving x509 cert for CertName cscoDefaultNewRootCaCert

*spamApTask5: Apr 08 04:03:44.273: sshpmGetCID: called to evaluate <cscoDefaultNewRootCaCert>

*spamApTask5: Apr 08 04:03:44.273: sshpmGetCID: Found matching CA cert cscoDefaultNewRootCaCert in row 4
*spamApTask5: Apr 08 04:03:44.273: Found CID 29307290 for certname cscoDefaultNewRootCaCert

*spamApTask5: Apr 08 04:03:44.273: CACertTable: Found matching CID cscoDefaultNewRootCaCert in row 4 x509 0x2cc7cd00
*spamApTask5: Apr 08 04:03:44.273: cscoAct2RsaCaCert: successfully added ACT2 RSA to store cert Verify User Certificate(?!)

*spamApTask5: Apr 08 04:03:44.279: Verify User Certificate: X509 Cert Verification return code: 1
*spamApTask5: Apr 08 04:03:44.279: Verify User Certificate: X509 Cert Verification result text: ok
*spamApTask5: Apr 08 04:03:44.279: sshpmGetCID: called to evaluate <cscoAct2RsaCaCert>

*spamApTask5: Apr 08 04:03:44.279: sshpmGetCID: Found matching CA cert cscoAct2RsaCaCert in row 8
*spamApTask5: Apr 08 04:03:44.279: Verify User Certificate: OPENSSL X509_Verify: AP Cert Verfied Using >cscoAct2RsaCaCert<

*spamApTask5: Apr 08 04:03:44.279: OpenSSL Get Issuer Handles: Check cert validity times (allow expired YES)
*spamApTask5: Apr 08 04:03:44.279: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask5: Apr 08 04:03:44.279: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask5: Apr 08 04:03:44.279: sshpmFreePublicKeyHandle: called with 0x1b0b5cc8

*spamApTask5: Apr 08 04:03:44.279: sshpmFreePublicKeyHandle: freeing public key

*spamApTask5: Apr 08 04:04:56.104: 70:df:2f:05:0e:38 Failed to parse CAPWAP packet from 10.10.10.211:5248

Here is the debug from the AP3802i:

[*04/08/2021 09:03:34.7848] CAPWAP State: Discovery
[*04/08/2021 09:03:34.7856] Got WLC address 192.168.1.10 from DHCP.
[*04/08/2021 09:03:34.7856] IP DNS query for CISCO-CAPWAP-CONTROLLER.zewsworld.com
[*04/08/2021 09:03:34.8559] Discovery Request sent to 192.168.1.10, discovery type DHCP(2)
[*04/08/2021 09:03:34.8600] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*04/08/2021 09:03:34.8602] Discovery Response from 192.168.1.10
[*04/08/2021 09:03:44.0001] Started wait dtls timer (60 sec)
[*04/08/2021 09:03:44.0005]
[*04/08/2021 09:03:44.0005] CAPWAP State: DTLS Setup
[*04/08/2021 09:03:44.0409] dtls_verify_server_cert: Controller certificate verification successful
[*04/08/2021 09:03:44.7221]
[*04/08/2021 09:03:44.7221] CAPWAP State: Join
[*04/08/2021 09:03:44.7389] Sending Join request to 192.168.1.10 through port 5248
[*04/08/2021 09:04:41.0377]
[*04/08/2021 09:04:41.0377] CAPWAP State: DTLS Teardown
[*04/08/2021 09:04:41.1404] status 'upgrade.sh: Script called with args:[CANCEL]'
[*04/08/2021 09:04:41.1975] do CANCEL, part2 is active part
[*04/08/2021 09:04:41.2120] status 'upgrade.sh: Cleanup tmp files ...'
[*04/08/2021 09:04:41.2453] Dropping dtls packet since session is not established. Peer 192.168.1.10-5246, Local 10.10.10.211-5248, conn (nil)
[*04/08/2021 09:04:41.2454] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/08/2021 09:04:41.2455] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/08/2021 09:04:55.7907]
[*04/08/2021 09:04:55.7907] CAPWAP State: Discovery
[*04/08/2021 09:04:55.7915] Got WLC address 192.168.1.10 from DHCP.
[*04/08/2021 09:04:55.7915] IP DNS query for CISCO-CAPWAP-CONTROLLER.zewsworld.com
[*04/08/2021 09:04:55.8696] Discovery Request sent to 192.168.1.10, discovery type DHCP(2)
[*04/08/2021 09:04:55.8721] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*04/08/2021 09:04:55.8743] Discovery Response from 192.168.1.10
[*04/08/2021 09:05:06.0001] Started wait dtls timer (60 sec)
[*04/08/2021 09:05:06.0005]
[*04/08/2021 09:05:06.0005] CAPWAP State: DTLS Setup
[*04/08/2021 09:05:06.0446] dtls_verify_server_cert: Controller certificate verification successful
[*04/08/2021 09:05:06.7043]
[*04/08/2021 09:05:06.7043] CAPWAP State: Join
[*04/08/2021 09:05:06.7116] Sending Join request to 192.168.1.10 through port 5248
[*04/08/2021 09:06:03.0353]
[*04/08/2021 09:06:03.0353] CAPWAP State: DTLS Teardown
[*04/08/2021 09:06:03.1216] status 'upgrade.sh: Script called with args:[CANCEL]'
[*04/08/2021 09:06:03.1782] do CANCEL, part2 is active part
[*04/08/2021 09:06:03.1927] status 'upgrade.sh: Cleanup tmp files ...'
[*04/08/2021 09:06:03.2260] Dropping dtls packet since session is not established. Peer 192.168.1.10-5246, Local 10.10.10.211-5248, conn (nil)
[*04/08/2021 09:06:03.2261] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*04/08/2021 09:06:03.2261] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).

I think I've tried everything that's available to get these AP's to join.  Anything I missed?  I don't have support on this controller so upgrading to 8.5.182 or higher isn't really an option.

Here's more information:

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.3.150.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 1.27
OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... 
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 192.168.1.10
IPv6 Address..................................... ::
Last Reset....................................... Power on reset
System Up Time................................... 98 days 19 hrs 4 mins 29 secs
System Timezone Location......................... (GMT -6:00) Central Time (US and Canada)
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Configured Country............................... US - United States
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +39 C
External Temperature............................. +21 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 2
Number of Active Clients......................... 35

OUI Classification Failure Count................. 8443

Burned-in MAC Address............................ 50:3D:E5:AE:C9:A0
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 500
System Nas-Id.................................... 
WLC MIC Certificate Types........................ SHA1

Any help would be greatly appreciated!

Thank you in advance

[Leo Laohoo]

---

@Paul.Ostaszewski wrote:
No join information found for AP: 70:df:2f:05:0e:38

---

This means the AP's join request has never reached the controller.  Is there a FW somewhere? Can the AP ping the controller's IP address?  Can the controller ping the AP's IP address?

[Paul.Ostaszewski]

No firewall. I even put the AP on the same vlan as the WLC management and
it won’t join.

The AP can ping the controller and the controller can ping the AP. They
definitely see each other.

[Leo Laohoo]

This is a 3800, right?  Let's try something out: 

Download the AP firmware for 17.11.1 and 17.10.1 and put both files into a TFTP server. 

Next, console or remote into the AP and run the following commands: 

archive download-sw /no-reboot tftp://<IP_ADDRESS>/ap3g3-k9w8-tar.153-3.JPO.tar

Wait for 2 minutes for the entire process to finish and continue with the next. 

archive download-sw tftp://<IP_ADDRESS>/ap3g3-k9w8-tar.153-3.JPP.tar

After 2 minutes the AP will reboot automatically and then make attempts to join the controller.  Make sure to plug a console cable in before it reboots.  

Let's see what this will do.

[Paul.Ostaszewski]

Leo,

Unfortunately I don't have access to those files.

[Leo Laohoo]

---

@Paul.Ostaszewski wrote:
Unfortunately I don't have access to those files.

---

I have a solution for this:  

1.  Read Cisco Wireless LAN Controller HTTP Parsing Engine Denial of Service Vulnerability.
2.  Scroll down to the "Customers Without Service Contracts" section, where it specifically states: 

Customers who purchase directly from Cisco but do not hold a Cisco service contract and customers who make purchases through third-party vendors but are unsuccessful in obtaining fixed software through their point of sale should obtain upgrades by contacting the Cisco TAC.

Customers should have the product serial number available and be prepared to provide the URL of this advisory as evidence of entitlement to a free upgrade.

---

3.  Email TAC.  Provide the filenames, the HTML locations and the Security Bulletin (Step 1).  

NOTE:  Do not call Cisco TAC.  Send them an email. 

[Paul.Ostaszewski]

Thank you. I’ll give it a try.

[Paul.Ostaszewski]

Leo,

Cisco TAC granted me access to the files and I performed the steps you asked.  Here is a sh version and the output of the console.  The 3802i AP will not join...

AP70DF.2F05.0E38#sh version
Restricted Rights Legend

Use, duplication, or disclosure by the Government is subject to
restrictions as set forth in subparagraph (c) of the Commercial
Computer Software - Restricted Rights clause at FAR sec. 52.227-19 and
subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

This product contains some software licensed under the
"GNU General Public License, version 2" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

This product contains some software licensed under the
"GNU Library General Public License, version 2" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Library
General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.0.html

This product contains some software licensed under the
"GNU Lesser General Public License, version 2.1" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Lesser
General Public License, version 2.1", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html

This product contains some software licensed under the
--More-- [*06/12/2023 02:27:34.0160] Set PnP NTP Server pnpntpserver.zewsworld.com.
"GNU General Public License, version 3" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, Version 3", available here:
http://www.gnu.org/licenses/gpl.html.

This product contains some software licensed under the
"GNU Affero General Public License, version 3" provided
with ABSOLUTELY NO WARRANTY under the terms of
"GNU Affero General Public License, version 3", available here:
http://www.gnu.org/licenses/agpl-3.0.html.

Cisco AP Software, (ap3g3), C3802, RELEASE SOFTWARE
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Wed Aug 10 23:07:52 GMT 2022

ROM: Bootstrap program is U-Boot boot loader
BOOTLDR: U-Boot boot loader Version 2013.01-g9ac8b85f2 (Jan 23 2023 - 10:29:05)

AP70DF.2F05.0E38 uptime is 0 days, 0 hours, 3 minutes
Last reload time : Mon Jun 12 02:24:23 UTC 2023
Last reload reason : reload command

cisco AIR-AP3802I-B-K9 ARMv7 Processor rev 1 (v7l) with 1028224/590168K bytes of memory.
Processor board ID FCW2125JNSF
AP Running Image : 17.6.4.56
Primary Boot Image : 17.6.4.56
Backup Boot Image : 17.11.0.155
Primary Boot Image Hash:
Backup Boot Image Hash: 09fa34ba37a1ab41b05f0580ff79bd62c79bc61113dc72d4be662b37e9ce6ba6191aae19c46fda04f35c6d43c3676ca2b18bc3934ecdfb015e34e59413fedb2c
1 Multigigabit Ethernet interfaces
1 Gigabit Ethernet interfaces
2 802.11 Radios
Radio Driver version : 9.0.5.5-W8964
Radio FW version : 9.1.8.1
NSS FW version : 2.4.28

Base ethernet MAC Address : 70:DF:2F:05:0E:38
Part Number : 73-017278-06
PCA Assembly Number : 000-00000-00
PCA Revision Number :
PCB Serial Number : FOC212448KU
Top Assembly Part Number : 068-100730-01
Top Assembly Serial Number : FCW2125JNSF
Top Revision Number : C0
Product/Model Number : AIR-AP3802I-B-K9

AP70DF.2F05.0E38#[*06/12/2023 02:28:04.3292] PNP:Server not reachable, Start CAPWAP Discovery
[*06/12/2023 02:28:04.3298]
[*06/12/2023 02:28:04.3298] CAPWAP State: Discovery
[*06/12/2023 02:28:04.3308] Got WLC address 192.168.1.10 from DHCP.
[*06/12/2023 02:28:04.3308] IP DNS query for CISCO-CAPWAP-CONTROLLER.zewsworld.com
[*06/12/2023 02:28:04.4152] Discovery Request sent to 192.168.1.10, discovery type DHCP(2)
[*06/12/2023 02:28:04.4162] Discovery Response from 192.168.1.10
[*06/12/2023 02:28:04.4200] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*06/12/2023 02:28:04.4205]
[*06/12/2023 02:28:04.4205] CAPWAP State: Discovery
[*06/12/2023 02:29:18.0000] Started wait dtls timer (60 sec)
[*06/12/2023 02:29:18.0005]
[*06/12/2023 02:29:18.0005] CAPWAP State: DTLS Setup
[*06/12/2023 02:29:18.0378] Certificate is expired
[*06/12/2023 02:29:18.0378] Certificate Start Date: Apr 8 13:39:08 2011 GMT
[*06/12/2023 02:29:18.0379] Certificate End Date: Apr 8 13:49:08 2021 GMT
[*06/12/2023 02:29:18.0379] display_verify_cert_status: Verify Cert: FAILED at 0 depth: certificate has expired
[*06/12/2023 02:29:18.0380] X509 OpenSSL Errors...
[*06/12/2023 02:29:18.0380]
[*06/12/2023 02:29:18.0381] 1956715504:error:0909006C:lib(9):func(144):reason(108):NA:0:Expecting: CERTIFICATE
[*06/12/2023 02:29:18.0381] 1956715504:error:0909006C:lib(9):func(144):reason(108):NA:0:Expecting: CERTIFICATE
[*06/12/2023 02:29:18.0381] 1956715504:error:0909006C:lib(9):func(144):reason(108):NA:0:Expecting: CERTIFICATE
[*06/12/2023 02:29:18.0381] 1956715504:error:0909006C:lib(9):func(144):reason(108):NA:0:Expecting: CERTIFICATE
[*06/12/2023 02:29:18.0381]
[*06/12/2023 02:29:18.0381]
[*06/12/2023 02:29:18.0381]
[*06/12/2023 02:29:18.0381] dtls_verify_server_cert: Controller certificate verification error
[*06/12/2023 02:29:18.0386] 1956715504:error:1416F086:lib(20):func(367):reason(134):NA:0:
[*06/12/2023 02:29:18.0386] dtls_process_packet: Error connecting TLS context ERR: 5
[*06/12/2023 02:29:18.0391] DTLS: Error while processing DTLS packet 0x561ae000.
[*06/12/2023 02:30:15.0288]
[*06/12/2023 02:30:15.0288] CAPWAP State: DTLS Teardown
[*06/12/2023 02:30:15.1562] status 'upgrade.sh: Script called with args:[CANCEL]'
[*06/12/2023 02:30:15.2177] do CANCEL, part2 is active part
[*06/12/2023 02:30:15.2353] status 'upgrade.sh: Cleanup tmp files ...'
[*06/12/2023 02:30:15.2694] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*06/12/2023 02:30:15.2695] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*06/12/2023 02:30:19.7807] No more AP manager addresses remain..
[*06/12/2023 02:30:19.7807] No valid AP manager found for controller 'ZEW-WLC-01' (ip: 192.168.1.10)
[*06/12/2023 02:30:19.7807] Failed to join controller ZEW-WLC-01.
[*06/12/2023 02:30:19.7808] Failed to join controller.

[Leo Laohoo]

---

@Paul.Ostaszewski wrote:
[*06/12/2023 02:29:18.0379] Certificate End Date: Apr 8 13:49:08 2021 GMT

---

Disable NTP and roll back the date of the controller to, say, 06 April 2021.  

[Paul.Ostaszewski]

Leo,
SUCCESS!!!!!

Very odd though… 2 of the 4 3802i’s needed to have the two OS upgrades
installed and the second two booted and connected fine without. All 4 are
connected to my controller. I set the time back to actual time and rebooted
the APs and all 4 reconnected.

What did the install of the two OS’ do on the first two APs?

Thank you for all your help!

[Leo Laohoo]

---

@Paul.Ostaszewski wrote:
What did the install of the two OS’ do on the first two APs?

---

Potential bug in the APs operating OS (17.6.4.56).  

The objective of loading two different OS-es is to completely overwrite the two older OS-es in the AP. 

[Paul.Ostaszewski]

Makes sense. Thank you for the explanation and all your help!
Much appreciated!

[karol.krzyzyk]

Did you also already activated this options ?

WLC> config ap cert-expiry-ignore mic enable
WLC> config ap cert-expiry-ignore ssc enable

[karol.krzyzyk]

Hello All

I have WLC 5508 with 8.5.182.0 software and start from last week I have crazy situation . In one location we have Primary WLC and in another Backup wlc. from last week half of our AP jump between backup and primary. this dive me crazy. .

on web log found 

---

*spamApTask6: Jun 09 22:42:41.078: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50757), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask6: Jun 09 22:42:41.077: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask5: Jun 09 22:42:29.078: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50675), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask5: Jun 09 22:42:29.077: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*emWeb: Jun 09 22:41:32.388: %EMWEB-3-FORM_SUBMIT_CSRF_DETECTED: [PA]ews_form.c:1239 Form submit action failed. Cross Site Attack detected form_idx=256 url=/screens/banner.html formCsrfTbl[256]=1.
*spamApTask5: Jun 09 22:41:02.315: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask3: Jun 09 22:40:50.980: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask7: Jun 09 22:39:10.112: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask7: Jun 09 22:39:10.111: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask4: Jun 09 22:39:06.512: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:af:9c:c0: DTLS connection closed forAP 10:4:17:22 (57758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:39:06.511: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:af:9c:c0
*spamApTask4: Jun 09 22:39:00.112: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50674), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:39:00.111: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask6: Jun 09 22:37:33.883: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:af:9c:c0
*spamApTask6: Jun 09 22:37:32.788: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask2: Jun 09 22:37:21.889: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask6: Jun 09 22:35:27.912: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50757), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask6: Jun 09 22:35:27.911: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask5: Jun 09 22:35:16.912: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50675), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask5: Jun 09 22:35:16.911: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask3: Jun 09 22:34:37.112: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:af:9c:c0: DTLS connection closed forAP 10:4:17:22 (57757), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask3: Jun 09 22:34:37.111: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:af:9c:c0
*spamApTask5: Jun 09 22:33:50.769: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask3: Jun 09 22:33:39.797: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask5: Jun 09 22:33:05.824: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:af:9c:c0
*spamApTask7: Jun 09 22:31:46.708: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask7: Jun 09 22:31:46.707: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask4: Jun 09 22:31:34.908: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50674), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:31:34.907: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask4: Jun 09 22:30:13.508: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:af:9c:c0: DTLS connection closed forAP 10:4:17:22 (57758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:30:13.507: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:af:9c:c0
*spamApTask6: Jun 09 22:30:08.631: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask2: Jun 09 22:29:57.574: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask6: Jun 09 22:28:36.743: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:af:9c:c0
*spamApTask6: Jun 09 22:28:04.708: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50757), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask6: Jun 09 22:28:04.707: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask5: Jun 09 22:27:53.708: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50675), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask5: Jun 09 22:27:53.707: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask5: Jun 09 22:26:26.560: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask3: Jun 09 22:26:15.548: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask3: Jun 09 22:25:17.908: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:af:9c:c0: DTLS connection closed forAP 10:4:17:22 (57757), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask3: Jun 09 22:25:17.907: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:af:9c:c0
*spamApTask7: Jun 09 22:24:22.708: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask7: Jun 09 22:24:22.707: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask5: Jun 09 22:24:08.162: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:af:9c:c0
*spamApTask4: Jun 09 22:24:06.708: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50674), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:24:06.707: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask6: Jun 09 22:22:44.459: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask2: Jun 09 22:22:32.714: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask4: Jun 09 22:21:15.308: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:af:9c:c0: DTLS connection closed forAP 10:4:17:22 (57758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:21:15.307: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:af:9c:c0
*spamApTask6: Jun 09 22:20:39.508: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50757), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask6: Jun 09 22:20:39.507: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask5: Jun 09 22:20:28.908: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50675), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask5: Jun 09 22:20:28.907: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask6: Jun 09 22:19:40.102: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:af:9c:c0
*spamApTask5: Jun 09 22:19:02.283: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP d0:c7:89:c6:c8:60
*spamApTask3: Jun 09 22:18:50.592: %LWAPP-3-VENDOR_PLD_VALIDATE_ERR: [PA]spam_lrad.c:12016 Validation of SPAM_VENDOR_SPECIFIC_PAYLOAD(185) with length=9 failed - AP 0c:68:03:38:78:30
*spamApTask7: Jun 09 22:16:58.504: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 d0:c7:89:c6:c8:60: DTLS connection closed forAP 10:4:129:33 (50758), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask7: Jun 09 22:16:58.503: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: d0:c7:89:c6:c8:60
*spamApTask4: Jun 09 22:16:46.704: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:38:78:30: DTLS connection closed forAP 10:4:129:34 (50674), Controller: 10:4:65:248 (5246) Echo Timer Expiry
*spamApTask4: Jun 09 22:16:46.703: %CAPWAP-3-ECHO_ERR: [PA]capwap_ac_sm.c:7875 Did not receive heartbeat reply; AP: 0c:68:03:38:78:30
*spamApTask3: Jun 09 22:16:46.304: %CAPWAP-3-DTLS_CLOSED_ERR: [PA]capwap_ac_sm.c:7130 0c:68:03:af:9c:c0: DTLS connection closed forAP 10:4:17:22 (57757), Controller: 10:4:65:248 (5246) Echo Timer Expiry

I don't know what is going on. tried reboot both WLCs , WAN line , FG, all, and not helped

Any sugestion ?

[Leo Laohoo]

@karol.krzyzyk,

This is a different situation.  Please create a new thread so we do not get confused with troubleshooting.