APs unable to Join 5508 - intermittent

[Najib Akbari]

I have C5508 controller with code 8.3.150.0 hosting ~280 APs with different models ( LAP1142N, AP1832I, CAP1702I etc) and yesterday for unknown reason several APs were missing ( de-associated ) from WLC. no TAC support and no time, reloaded the WLC and made them back up!. now I open this thread so possibly find the technical logic behind it. today tried to simulate the issue so reloaded some APs and here is one of them logs ( debug capwap client event ) and it is unable to join the WLC, im sure if I reload it and or reload AP it will be able to join. Please assist analyzing this situation and find the logic. Thanks!!!!!!!

let me know if more details needed

----------------------------------------

*Apr 29 00:17:45.999: %CAPWAP-3-EVENTLOG: Could not discover any MWAR.
*Apr 29 00:17:45.999: %CAPWAP-3-EVENTLOG: Starting Discovery. Initializing discovery latency in discovery responses.
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: CAPWAP State: Discovery.
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: Not waiting for DHCP options as resolve method on interface BVI1 is 4
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 172.25.4.71
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 172.25.4.71
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 172.25.4.71
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 172.25.4.71
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: Discovery Request sent to 172.25.254.1 with discovery type set to 1
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 172.25.4.71
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 172.25.4.71
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: Discovery Request sent to 172.25.254.1 with discovery type set to 1
*Apr 29 00:17:46.003: %CAPWAP-3-EVENTLOG: Discovery Request sent to 255.255.255.255 with discovery type set to 0
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Send broadcast discovery request
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Failed to send packet to destination FF01::18C
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Discovery Request sent to FF01::18C with discovery type set to 0
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 0
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 1
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 00:17:46.007: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 2
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 3
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 00:17:46.019: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 00:17:46.023: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 00:17:46.023: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 00:17:55.191: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: Ap mgr count=1
*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: Controller: fpwpwcmdf. ApMgr count is 1 ipTransportTried 0 prefer-mode 1

*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 172.25.254.1 to least load
*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: IPv4 Pref mode. Choosing AP Mgr with index 0, IP = 172.25.254.1, load = 272 ap ip: (172.25.254.132)
*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: capwapSetTransportAddr returning: index=0, apMgrCount = 0

*Apr 29 00:17:56.007: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Apr 29 00:17:56.007: %CAPWAP-3-EVENTLOG: Synchronizing time with AC time.
*Apr 29 00:17:56.000: %CAPWAP-3-EVENTLOG: Setting time to 00:17:56 UTC Apr 29 2022

*Apr 29 00:17:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.25.254.1 peer_port: 5246
*Apr 29 00:17:56.000: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Setup.
*Apr 29 00:17:56.000: %CAPWAP-3-EVENTLOG: Setting default MTU: MTU discovery can start with 576
fpwpap013-CT-068#debug capwap client event
*Apr 29 00:18:25.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0xC9C6738!

*Apr 29 00:18:35.183: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Wait DTLS timer has expired
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Dtls session establishment failed
*Apr 29 00:18:55.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.25.254.1:5246
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Teardown.
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: DTLS session cleanup completed. Restarting capwap state machine.
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Previous CAPWAP state was DTLS Setup,numOfCapwapDiscoveryResp = 4.
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Attempting to join next controller
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Go Join the next controller

*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Ap mgr count=0
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Go Join the next controller

*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Remove discovery response at index 0

*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 00:18:55.999: %CAPWAP-3-EVENTLOG: Ap mgr count=1
*Apr 29 00:18:56.003: %CAPWAP-3-EVENTLOG: Controller: fpwpwcmdf. ApMgr count is 1 ipTransportTried 0 prefer-mode 1

*Apr 29 00:18:56.003: %CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 172.25.254.1 to least load
*Apr 29 00:18:56.003: %CAPWAP-3-EVENTLOG: IPv4 Pref mode. Choosing AP Mgr with index 0, IP = 172.25.254.1, load = 272 ap ip: (172.25.254.132)
*Apr 29 00:18:56.003: %CAPWAP-3-EVENTLOG: capwapSetTransportAddr returning: index=0, apMgrCount = 0

*Apr 29 00:18:56.003: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Apr 29 00:18:56.003: %CAPWAP-3-EVENTLOG: Synchronizing time with AC time.
*Apr 29 00:18:56.000: %CAPWAP-3-EVENTLOG: Setting time to 00:18:56 UTC Apr 29 2022

*Apr 29 00:18:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.25.254.1 peer_port: 5246
*Apr 29 00:18:56.000: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Setup.
*Apr 29 00:18:56.000: %CAPWAP-3-EVENTLOG: Setting default MTU: MTU discovery can start with 576
*Apr 29 00:19:15.179: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Apr 29 00:19:25.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0xC1DE60C!

-----------------------------------------------------------------------------

[Leo Laohoo]

FN63942 - Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration

[Najib Akbari]

Thanks for the response, but thats not the issue here. I am already aware of the cert expiry issue and used the work around "ap cert-expiry-ignore {mic|ssc} enable" and changed the date to be withing the AP cert. its been working fine for a while the whole setup and im looking to figure out what happened yesterday and 40% of APs lost the tunnel and I had to reload the WLC.

[Najib Akbari]

in case needed:

WLC:

(Cisco Controller) >show time

Time............................................. Thu Apr 28 18:23:17 2022

Timezone delta................................... 0:0
Timezone location................................ (GMT -8:00) Pacific Time (US and Canada)

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.3.150.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... WLC
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 172.25.254.1
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 1 days 4 hrs 44 mins 38 secs
System Timezone Location......................... (GMT -8:00) Pacific Time (US and Canada)
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Configured Country............................... US - United States
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +39 C
External Temperature............................. +22 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 662

OUI Classification Failure Count................. 0

Burned-in MAC Address............................ F8:C2:88:8C:AD:60
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500
System Nas-Id.................................... WLC
WLC MIC Certificate Types........................ SHA1/SHA2

AP:

AP#sh ver
Cisco IOS Software, C1700 Software (AP3G2-K9W8-M), Version 15.3(3)JD17, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Fri 12-Apr-19 03:21 by prod_rel_team

ROM: Bootstrap program is C1700 boot loader
BOOTLDR: C1700 Boot Loader (AP3G2-BOOT-M) LoaderVersion 15.3() [ TRUE]

fpwpap013-CT-068 uptime is 6 days, 19 hours, 19 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g2-k9w8-mx.153-3.JD17/ap3g2-k9w8-xx.153-3.JD17"
Last reload reason: Reset Request from Controller.

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP1702I-B-K9 (PowerPC) processor (revision A0) with 376814K/134656K bytes of memory.
Processor board ID FCW2111N9PJ
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
LWAPP image version 8.3.150.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: F8:0B:CB:7A:36:D0
Part Number : 73-16776-02
PCB Serial Number : FOC21101X03
Top Assembly Part Number : 068-100893-02
Top Assembly Serial Number : FCW2111N9PJ
Top Revision Number : A0
Product/Model Number : AIR-CAP1702I-B-K9

in case needed:

WLC:

(Cisco Controller) >show time

Time............................................. Thu Apr 28 18:23:17 2022

Timezone delta................................... 0:0
Timezone location................................ (GMT -8:00) Pacific Time (US and Canada)

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.3.150.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... WLC
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 172.25.254.1
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 1 days 4 hrs 44 mins 38 secs
System Timezone Location......................... (GMT -8:00) Pacific Time (US and Canada)
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Configured Country............................... US - United States
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +39 C
External Temperature............................. +22 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 662

OUI Classification Failure Count................. 0

Burned-in MAC Address............................ F8:C2:88:8C:AD:60
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500
System Nas-Id.................................... WLC
WLC MIC Certificate Types........................ SHA1/SHA2

AP:

AP#sh ver
Cisco IOS Software, C1700 Software (AP3G2-K9W8-M), Version 15.3(3)JD17, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Fri 12-Apr-19 03:21 by prod_rel_team

ROM: Bootstrap program is C1700 boot loader
BOOTLDR: C1700 Boot Loader (AP3G2-BOOT-M) LoaderVersion 15.3() [ TRUE]

fpwpap013-CT-068 uptime is 6 days, 19 hours, 19 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g2-k9w8-mx.153-3.JD17/ap3g2-k9w8-xx.153-3.JD17"
Last reload reason: Reset Request from Controller.

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP1702I-B-K9 (PowerPC) processor (revision A0) with 376814K/134656K bytes of memory.
Processor board ID FCW2111N9PJ
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
LWAPP image version 8.3.150.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: F8:0B:CB:7A:36:D0
Part Number : 73-16776-02
PCB Serial Number : FOC21101X03
Top Assembly Part Number : 068-100893-02
Top Assembly Serial Number : FCW2111N9PJ
Top Revision Number : A0
Product/Model Number : AIR-CAP1702I-B-K9

[marce1000]

   - The controller software version is very old , these any 5508 should run  8.5.182.12 (8.5.182.13 for 3504s)
      as pointed out in
https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html

 + Also checkout  https://www.cisco.com/c/en/us/support/docs/wireless/5500-series-wireless-controllers/119286-lap-notjoin-wlc-tshoot.html
Start for instance on the  AP : https://www.cisco.com/c/en/us/support/docs/wireless/5500-series-wireless-controllers/119286-lap-notjoin-wlc-tshoot.html#toc-hId--607814488

               + Look for basic network connectivity issues for APs , such as port counters

  M,

[Najib Akbari]

Thank you! I will go through this documents. by the way 8.3.150.0 is the highest version I can go which supports 1142N AP

[marce1000]

     >.... by the way 8.3.150.0 is the highest version I can go which supports 1142N AP
  - That is a very fundamental problem here ; these days aireos controllers should use the last version available because the product line is end of support and there will be no further bug fixes, if it is due to a bug , which is likely then you can not fix things because of the 1142N APs. But those are also very old , and so you should modernize the wireless infrastructure ,

 M.

[Najib Akbari]

I ran those debug commands on both WLC and one AP, based on the doc it seems WLC does not receive join request from AP and AP logs shows a complain about " DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached"

here are some parts of the logs:

WLC:

(Cisco Controller) >
(Cisco Controller) >*spamApTask7: Apr 29 14:33:11.082: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask7: Apr 29 14:33:11.082: sshpmGetCID: Found matching ID cert cscoSha2IdCert in row 3
*spamApTask7: Apr 29 14:33:11.082: GetIDCert: Using SHA2 Id cert on WLC

*spamApTask7: Apr 29 14:33:11.082: Get Cert from CID: For CID 1a8079b1 certType 1
*spamApTask7: Apr 29 14:33:11.082: Get Cert from CID: Found match of ID Cert in row 3
*spamApTask7: Apr 29 14:33:11.082: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask7: Apr 29 14:33:11.082: sshpmGetCID: Found matching ID cert cscoSha2IdCert in row 3
*spamApTask7: Apr 29 14:33:11.082: GetDERIDKey: Using SHA2 Id cert Private Keys on WLC

*spamApTask7: Apr 29 14:33:11.082: GetPrivateKey: called to get key for CID 1a8079b1

*spamApTask7: Apr 29 14:33:11.082: Private Key found row 3 KeyBufLen 2048 Keylen 1191 PrivateKeyPtr 0x2c490c44

*spamApTask7: Apr 29 14:33:11.311: OpenSSL Get Issuer Handles: locking ca cert table

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: x509 subject_name /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AP1G4-D4ADBD125260/emailAddress=support@cisco.com

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: issuer_name /O=Cisco/CN=Cisco Manufacturing CA SHA2

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: CN AP1G4-D4ADBD125260

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: issuerCertCN Cisco Manufacturing CA SHA2

*spamApTask7: Apr 29 14:33:11.312: GetMac: MAC: d4ad.bd12.5260

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: openssl Mac Address in subject is d4:ad:bd:12:52:60

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: Cert Name in subject is AP1G4-D4ADBD125260

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: Extracted cert issuer from subject name.

*spamApTask7: Apr 29 14:33:11.312: NMSP:: Algo name matched SHA256

*spamApTask7: Apr 29 14:33:11.312: OpenSSL Get Issuer Handles: Cert is issued by Cisco Systems.

*spamApTask7: Apr 29 14:33:11.312: Retrieving x509 cert for CertName cscoMfgSha2CaCert

*spamApTask7: Apr 29 14:33:11.312: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>

*spamApTask7: Apr 29 14:33:11.312: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask7: Apr 29 14:33:11.312: Found CID 233f6707 for certname cscoMfgSha2CaCert

*spamApTask7: Apr 29 14:33:11.312: CACertTable: Found matching CID cscoMfgSha2CaCert in row 7 x509 0x2cc7b950
*spamApTask7: Apr 29 14:33:11.312: Retrieving x509 cert for CertName cscoRootSha2CaCert

*spamApTask7: Apr 29 14:33:11.312: sshpmGetCID: called to evaluate <cscoRootSha2CaCert>

*spamApTask7: Apr 29 14:33:11.312: sshpmGetCID: Found matching CA cert cscoRootSha2CaCert in row 6
*spamApTask7: Apr 29 14:33:11.312: Found CID 2e464774 for certname cscoRootSha2CaCert

*spamApTask7: Apr 29 14:33:11.312: CACertTable: Found matching CID cscoRootSha2CaCert in row 6 x509 0x2cc7ba04
*spamApTask7: Apr 29 14:33:11.318: Verify User Certificate: X509 Cert Verification return code: 1
*spamApTask7: Apr 29 14:33:11.318: Verify User Certificate: X509 Cert Verification result text: ok
*spamApTask7: Apr 29 14:33:11.318: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>

*spamApTask7: Apr 29 14:33:11.318: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask7: Apr 29 14:33:11.318: Verify User Certificate: OPENSSL X509_Verify: AP Cert Verfied Using >cscoMfgSha2CaCert<

*spamApTask7: Apr 29 14:33:11.319: OpenSSL Get Issuer Handles: Check cert validity times (allow expired YES)
*spamApTask7: Apr 29 14:33:11.319: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: Apr 29 14:33:11.319: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: Apr 29 14:33:11.319: sshpmFreePublicKeyHandle: called with 0x1ea1ec00

*spamApTask7: Apr 29 14:33:11.319: sshpmFreePublicKeyHandle: freeing public key

*spamApTask7: Apr 29 14:33:16.556: 74:86:0b:cb:4b:d0 Discovery Request from 172.25.253.51:45248

*spamApTask7: Apr 29 14:33:16.556: 74:86:0b:cb:4b:d0 ApModel: AIR-CAP1532I-B-K9

*spamApTask7: Apr 29 14:33:16.556: 74:86:0b:cb:4b:d0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 400, MaxLicense=300 joined Aps =276
*spamApTask7: Apr 29 14:33:16.556: 74:86:0b:cb:4b:d0 apType = 34 apModel: AIR-CAP1532I-B-K9

*spamApTask7: Apr 29 14:33:16.556: 74:86:0b:cb:4b:d0 apType: Ox22 bundleApImageVer: 8.3.150.0
*spamApTask7: Apr 29 14:33:16.556: 74:86:0b:cb:4b:d0 version:8 release:3 maint:150 build:0
*spamApTask7: Apr 29 14:33:16.557: 74:86:0b:cb:4b:d0 Discovery Response sent to 172.25.253.51 port 45248

*spamApTask7: Apr 29 14:33:16.557: 74:86:0b:cb:4b:d0 Discovery Response sent to 172.25.253.51:45248

*spamApTask7: Apr 29 14:33:16.559: 74:86:0b:cb:4b:d0 Discovery Request from 172.25.253.51:45248

*spamApTask7: Apr 29 14:33:16.559: 74:86:0b:cb:4b:d0 ApModel: AIR-CAP1532I-B-K9

*spamApTask7: Apr 29 14:33:16.559: 74:86:0b:cb:4b:d0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 400, MaxLicense=300 joined Aps =276

AP:

*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Ap mgr count=0
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Go Join the next controller

*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Remove discovery response at index 0

*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Could not discover any MWAR.
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Starting Discovery. Initializing discovery latency in discovery responses.
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: CAPWAP State: Discovery.
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Not waiting for DHCP options as resolve method on interface BVI1 is 4
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 172.25.4.71
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 172.25.4.71
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 172.25.4.71
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 172.25.4.71
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: Discovery Request sent to 172.25.254.1 with discovery type set to 1
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 172.25.4.71
*Apr 29 21:35:25.999: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 172.25.4.71
*Apr 29 21:35:26.003: %CAPWAP-3-EVENTLOG: Discovery Request sent to 172.25.254.1 with discovery type set to 1
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Discovery Request sent to 255.255.255.255 with discovery type set to 0
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Send broadcast discovery request
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Failed to send packet to destination FF01::18C
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Discovery Request sent to FF01::18C with discovery type set to 0
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 0
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 21:35:26.007: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 1
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 2
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 21:35:26.011: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: Discovery Response from 172.25.254.1
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: Entered wtpDecodeDiscoveryResponse: numOfCapwapDiscoveryResp 3
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: Copied ipAddr 172.25.254.1(147E) into discoveryResp
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: CapwapDiscoveryResponse: added mwar 172.25.254.1 to discoveryResponseList
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: AC_DESCRIPTOR
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: AC_NAME fpwpwcmdf
*Apr 29 21:35:26.015: %CAPWAP-3-EVENTLOG: IPV4 MGR 1
*Apr 29 21:35:26.019: %CAPWAP-3-EVENTLOG: return from wtpDecodeDiscoveryResponse
*Apr 29 21:35:26.019: %CAPWAP-3-EVENTLOG: returning from wtpProcessDiscoveryResponse
*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: Ap mgr count=1
*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: Controller: fpwpwcmdf. ApMgr count is 1 ipTransportTried 0 prefer-mode 1

*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 172.25.254.1 to least load
*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: IPv4 Pref mode. Choosing AP Mgr with index 0, IP = 172.25.254.1, load = 276 ap ip: (172.25.253.51)
*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: capwapSetTransportAddr returning: index=0, apMgrCount = 0

*Apr 29 21:35:36.007: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Apr 29 21:35:36.007: %CAPWAP-3-EVENTLOG: Synchronizing time with AC time.
*Apr 29 21:35:32.000: %CAPWAP-3-EVENTLOG: Setting time to 21:35:32 UTC Apr 29 2022

*Apr 29 21:35:32.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.25.254.1 peer_port: 5246
*Apr 29 21:35:32.000: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Setup.
*Apr 29 21:35:32.000: %CAPWAP-3-EVENTLOG: Setting default MTU: MTU discovery can start with 576
*Apr 29 21:35:32.000: DTLS_CLIENT_EVENT: dtls_connectionDB_add_connection: Added Connection 0x67C883E0 Server 172.25.254.1:147E, Client 172.25.253.51:B0C0

*Apr 29 21:35:33.999: DTLS_CLIENT_EVENT: dtls_connection_retransmit: Retransmit Handshake record to 172.25.254.1:5246 for Connection 0x67C883E0

*Apr 29 21:35:37.419: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Apr 29 21:35:37.999: DTLS_CLIENT_EVENT: dtls_connection_retransmit: Retransmit Handshake record to 172.25.254.1:5246 for Connection 0x67C883E0

*Apr 29 21:35:45.999: DTLS_CLIENT_EVENT: dtls_connection_retransmit: Retransmit Handshake record to 172.25.254.1:5246 for Connection 0x67C883E0

*Apr 29 21:36:01.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x67C883E0!

*Apr 29 21:36:17.419: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Wait DTLS timer has expired
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Dtls session establishment failed
*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: wtpCloseAllDtlsConnections:
DISCONNECTING DTLS Session: 0x67C883E0 - CTRL connection

*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: dtls_disconnect: Disconnecting DTLS connection 0x67C883E0
*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: dtls_free_connection: Free Called... for Connection 0x67C883E0
*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: dtls_send_Alert: Sending FATAL : Close notify Alert
*Apr 29 21:36:31.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.25.254.1:5246
*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: wtpDtlsCallback: DTLS-Ctrl Connection 0x67C883E0 closed
*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: dtls_free_connection: Free Done... for Connection 0x67C883E0
*Apr 29 21:36:31.999: DTLS_CLIENT_EVENT: dtls_connectionDB_del_connection: Deleted Connection 0x67C883E0, Server 172.25.254.1:5246, Client 172.25.253.51:45248, Count 0

*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Teardown.
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: DTLS session cleanup completed. Restarting capwap state machine.
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Previous CAPWAP state was DTLS Setup,numOfCapwapDiscoveryResp = 4.
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Attempting to join next controller
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Go Join the next controller

*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Ap mgr count=0
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Go Join the next controller

*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Remove discovery response at index 0

*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Selected MWAR 'fpwpwcmdf' (index 0).
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Ap mgr count=1
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Controller: fpwpwcmdf. ApMgr count is 1 ipTransportTried 0 prefer-mode 1

*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 172.25.254.1 to least load
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: IPv4 Pref mode. Choosing AP Mgr with index 0, IP = 172.25.254.1, load = 276 ap ip: (172.25.253.51)
*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: capwapSetTransportAddr returning: index=0, apMgrCount = 0

*Apr 29 21:36:31.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Apr 29 21:36:31.999: %CAPWAP-3-EVENTLOG: Synchronizing time with AC time.
*Apr 29 21:36:32.000: %CAPWAP-3-EVENTLOG: Setting time to 21:36:32 UTC Apr 29 2022

*Apr 29 21:36:32.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.25.254.1 peer_port: 5246
*Apr 29 21:36:32.000: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Setup.
*Apr 29 21:36:32.000: %CAPWAP-3-EVENTLOG: Setting default MTU: MTU discovery can start with 576
*Apr 29 21:36:32.000: DTLS_CLIENT_EVENT: dtls_connectionDB_add_connection: Added Connection 0x67BEECC0 Server 172.25.254.1:147E, Client 172.25.253.51:B0C0

*Apr 29 21:36:33.999: DTLS_CLIENT_EVENT: dtls_connection_retransmit: Retransmit Handshake record to 172.25.254.1:5246 for Connection 0x67BEECC0

*Apr 29 21:36:37.999: DTLS_CLIENT_EVENT: dtls_connection_retransmit: Retransmit Handshake record to 172.25.254.1:5246 for Connection 0x67BEECC0

*Apr 29 21:36:45.999: DTLS_CLIENT_EVENT: dtls_connection_retransmit: Retransmit Handshake record to 172.25.254.1:5246 for Connection 0x67BEECC0

*Apr 29 21:36:57.419: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Apr 29 21:37:01.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x67BEECC0!

[Leo Laohoo]

Move the WLC time and date back to 2021 and reboot the APs.

[Najib Akbari]

why? I know if I reboot the AP it will join the WLC, but i want to know why it does not sometimes, specially when the switch in between goes down and comes back live the AP will not rejoin until reboot.

if you are referring to the cert validity period it is withing that range:

Certificate Name: Cisco SHA1 device cert

Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT5508-K9-f8c2888cad60, emailAddress=support@cisco.com
Issuer Name :

--More-- or (q)uit
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number (Hex):

Validity :
Start : Dec 30 07:57:17 2020 GMT
End : May 14 20:25:42 2029 GMT

[ahollifield]

https://www.cisco.com/c/en/us/products/collateral/wireless/5500-series-wireless-controllers/eos-eol-notice-c51-740221.html

[Najib Akbari]

@Rasika Nayanajithany advise on the issue please?

[Rich R]

Trying to solve problems on the 1142 is just a pointless exercise.

We had some trouble with them a few years ago and although TAC agreed it looked like a bug they said (as expected) that there was nothing they could do about it because they're past end of support.  If you choose to use end of support products on end of support software you really are on your own.  There will always be hundreds of unresolved bugs that you might encounter now and then, so you'll just have to get used to seeing them occasionally.  If you're not happy with that then upgrade to supported hardware and software.  You'll have new bugs to deal with but at least you can get TAC support for those and (eventually) get them fixed.  Meanwhile if you know that reloads resolve the problem then you just have to do reloads.

[Najib Akbari]

Thank you! it seems we have to deal with it until we upgrade