3702 certificate unknown 17.9.4a

[eglinsky2012]

We have a few 3702 APs that are unable to join our 9800 WLCs that are running 17.9.4a with APSP8. The error shown in the AP's log is:

*Mar 4 15:02:06.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: x.x.x.x peer_port: 5246
*Mar 4 15:02:06.215: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from x.x.x.x
*Mar 4 15:02:06.215: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to x.x.x.x:5246

The 9800 join statistics show:

Access Point Statistics Summary

Is the AP currently connected to controller

NOT JOINED

Time at which the AP joined this controller last time

NA

Type of error that occurred last

DTLS-Handshake

Time at which the last join error occurred

03/04/2024 10:02:06

Last AP Disconnect Details

Reason for last AP connection failure

Certificate verify failed

Last Reboot Reason (Reported by AP)

No reboot reason

...

The APs have both 9800 WLCs in the HA configuration as primary and secondary with no tertiary specified, however, they instead are joining our 8540 WLCs (which are in the same mobility group as the 9800s) and are currently running version 8.10.190.6.

I wouldn't be surprised if this were an "expired" certificate due to the age of these APs, but an "unknown" certificate is something I haven't heard of. Is this a known issue with a workaround?

[jagan.chowdam]

Are there any other AP models joined c9800? 

Have you checked certificate validity on the 3702 APs?

debug capwap console cli
show crypto pki certificates

Try factory reset the AP.

Jagan Chowdam

/**Pls rate useful responses**/ 

[eglinsky2012]

Yes, we have a variety of other APs joined including other x700 models, x800, 9130, 9166.

Here's the output of the certificates command, which shows some certs that are still valid and some that are expired.

Again, the error isn't for an expired cert (as is shown on the 9800 AP join statistics for some old 1142 APs that we have yet to locate), it's for an unknown cert.

HBL-2125.2-STRS#show crypto pki certificates
CA Certificate
  Status: Available
  Certificate Serial Number (hex): 01
  Certificate Usage: Signature
  Issuer:
    cn=Cisco Root CA M2
    o=Cisco
  Subject:
    cn=Cisco Root CA M2
    o=Cisco
  Validity Date:
    start date: 13:00:18 UTC Nov 12 2012
    end   date: 13:00:18 UTC Nov 12 2037
  Associated Trustpoints: Trustpool cisco-m2-root-cert
  Storage:

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 02
  Certificate Usage: Signature
  Issuer:
    cn=Cisco Root CA M2
    o=Cisco
  Subject:
    cn=Cisco Manufacturing CA SHA2
    o=Cisco
  CRL Distribution Points:
    http://www.cisco.com/security/pki/crl/crcam2.crl
  Validity Date:
    start date: 13:50:58 UTC Nov 12 2012
    end   date: 13:00:17 UTC Nov 12 2037
  Associated Trustpoints: Trustpool Cisco_IOS_M2_MIC_cert
  Storage:

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 00
  Certificate Usage: General Purpose
  Issuer:
    e=support@airespace.com
    cn=ca
    ou=none
    o=airespace Inc
    l=San Jose
    st=California
    c=US
  Subject:
    e=support@airespace.com
    cn=ca
    ou=none
    o=airespace Inc
    l=San Jose
    st=California
    c=US
  Validity Date:
    start date: 23:38:55 UTC Feb 12 2003
    end   date: 23:38:55 UTC Nov 11 2012
  Associated Trustpoints: airespace-old-root-cert
  Storage:

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 00
  Certificate Usage: Signature
  Issuer:
    e=support@airespace.com
    cn=Airespace Root CA
    ou=Engineering
    o=Airespace Inc.
    l=San Jose
    st=California
    c=US
  Subject:
    e=support@airespace.com
    cn=Airespace Root CA
    ou=Engineering
    o=Airespace Inc.
    l=San Jose
    st=California
    c=US
  Validity Date:
    start date: 13:41:22 UTC Jul 31 2003
    end   date: 13:41:22 UTC Apr 29 2013
  Associated Trustpoints: airespace-new-root-cert
  Storage:

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 03
  Certificate Usage: General Purpose
  Issuer:
    e=support@airespace.com
    cn=Airespace Root CA
    ou=Engineering
    o=Airespace Inc.
    l=San Jose
    st=California
    c=US
  Subject:
    e=support@airespace.com
    cn=Airespace Device CA
    ou=Engineering
    o=Airespace Inc.
    l=San Jose
    st=California
    c=US
  Validity Date:
    start date: 22:37:13 UTC Apr 28 2005
    end   date: 22:37:13 UTC Jan 26 2015
  Associated Trustpoints: airespace-device-root-cert
  Storage:

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 5FF87B282B54DC8D42A315B568C9ADFF
  Certificate Usage: Signature
  Issuer:
    cn=Cisco Root CA 2048
    o=Cisco Systems
  Subject:
    cn=Cisco Root CA 2048
    o=Cisco Systems
  Validity Date:
    start date: 20:17:12 UTC May 14 2004
    end   date: 20:25:42 UTC May 14 2029
  Associated Trustpoints: Trustpool cisco-root-cert
  Storage:

Certificate
  Status: Available
  Certificate Serial Number (hex): 1A5F8D2C00000007C6E1
  Certificate Usage: General Purpose
  Issuer:
    cn=Cisco Manufacturing CA
    o=Cisco Systems
  Subject:
    Name: AP3G2-7426ac944c2c
    e=support@cisco.com
    cn=AP3G2-7426ac944c2c
    o=Cisco Systems
    l=San Jose
    st=California
    c=US
  CRL Distribution Points:
    http://www.cisco.com/security/pki/crl/cmca.crl
  Validity Date:
    start date: 07:47:28 UTC Feb 15 2014
    end   date: 07:57:28 UTC Feb 15 2024
  Associated Trustpoints: Cisco_IOS_MIC_cert
  Storage:

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 6A6967B3000000000003
  Certificate Usage: Signature
  Issuer:
    cn=Cisco Root CA 2048
    o=Cisco Systems
  Subject:
    cn=Cisco Manufacturing CA
    o=Cisco Systems
  CRL Distribution Points:
    http://www.cisco.com/security/pki/crl/crca2048.crl
  Validity Date:
    start date: 22:16:01 UTC Jun 10 2005
    end   date: 20:25:42 UTC May 14 2029
  Associated Trustpoints: Trustpool Cisco_IOS_MIC_cert
  Storage:

[marce1000]

             - (Also) checkout :  https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin

  M

[jagan.chowdam]

Looks like AP MIC certificate Expired on Feb 15, 2024. AP-to-WLC Control and Provisioning of Wireless Access Points (CAPWAP) connection uses MIC. 

Certificate
  Status: Available
  Certificate Serial Number (hex): 1A5F8D2C00000007C6E1
  Certificate Usage: General Purpose
  Issuer:
    cn=Cisco Manufacturing CA
    o=Cisco Systems
  Subject:
    Name: AP3G2-7426ac944c2c
    e=support@cisco.com
    cn=AP3G2-7426ac944c2c
    o=Cisco Systems
    l=San Jose
    st=California
    c=US
  CRL Distribution Points:
    http://www.cisco.com/security/pki/crl/cmca.crl
  Validity Date:
    start date: 07:47:28 UTC Feb 15 2014
    end   date: 07:57:28 UTC Feb 15 2024
  Associated Trustpoints: Cisco_IOS_MIC_cert

The APs joined the WLC before expiration date will function until they reboot. Any new associations will have issues establishing CAPWAP tunnel.

Refer Cisco Field Notice: https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html

https://community.cisco.com/t5/wireless-mobility-knowledge-base/lightweight-ap-fail-to-create-capwap-lwapp-connection-due-to/ta-p/3155111

https://ipnet.xyz/2022/10/cisco-wlap-and-wlc-failed-to-create-capwap-connection/

Jagan Chowdam

/**Pls rate useful responses**/

[eglinsky2012]

Oh wow, I missed that one. Good catch! And thank you for that information on the field notice. I remember that from when it last struck December 2022. I'm trying to use the workaround of the config ap cert-expiry-ignore {mic|ssc} enable command, but that doesn't seem to be available on IOS, only AireOS.

However, I set an affected AP to go to an AireOS WLC runing 8.10.190.6, and it has associated. It just won't associate to the 9800. Maybe because we have ap cert-expiry-ignore mic enable set on those controllers?

[jagan.chowdam]

Either ignore MIC or NTP rollback.

[Leo Laohoo]

Post the complete output to the AP command of "dir".

[marce1000]

 - Have a checkup of the 9800 WLCs configuration using the CLI command show tech wireless and feed the output into :
                                                                                                                   Wireless Config Analyzer

 M.

[eglinsky2012]

@jagan.chowdam, the ignore MIC command doesn't seem to be available on 9800. All the bugs refer to AireOS syntax, but I can't find any syntax to work with IOS. Am I missing something? In the meantime, I have set the WLC block back and the APs have rejoined.

@Leo Laohoo:

Directory of flash:/

2 -rwx 280 Jan 1 1970 00:03:13 +00:00 info
3 -rwx 64 Feb 27 2024 16:32:27 +00:00 sensord_CSPRNG0
21 drwx 2368 Feb 27 2024 16:32:07 +00:00 ap3g2-k9w8-mx.ap_zmr10_esc.202312090727
4 -rwx 59908 Feb 27 2024 16:32:34 +00:00 event.log
5 -rwx 63 Mar 1 1993 00:00:33 +00:00 mesh_cfg.txt
6 -rwx 79 Jan 6 2016 19:28:05 +00:00 mesh_port_cfg.txt
7 -rwx 367 Mar 4 2024 20:14:59 +00:00 capwap-saved-config-bak
14 -rwx 965 Dec 27 2023 10:48:12 +00:00 lwapp_mm_mwar_hash.cfg
9 drwx 0 Mar 1 1993 00:01:04 +00:00 configs
10 -rwx 64 Feb 27 2024 16:32:27 +00:00 sensord_CSPRNG1
11 -rwx 137676 Feb 18 2024 01:10:52 +00:00 event.r1
15 -rwx 367 Mar 4 2024 20:09:59 +00:00 capwap-saved-config
8 -rwx 52 Dec 27 2023 10:48:12 +00:00 lwapp_ssc_token.cfg
13 -rwx 0 Mar 1 1993 00:00:57 +00:00 config.txt
12 -rwx 142109 Feb 21 2019 18:45:25 +00:00 event.r0
17 -rwx 438 Mar 4 2024 21:49:37 +00:00 env_vars
19 -rwx 12312 Mar 4 2024 20:14:59 +00:00 private-multiple-fs

[Leo Laohoo]

Ok, I've got a workaround or a hack around this.  Here's how it works:  

1.  Download ap3g2-rcvk9w8-tar.153-3.JPQ1.tar and put the file into a TFTP server. 
2.  Remote into the AP and enter the following commands: 

debug capwap console cli
delete /f /r flash:ap3g2-k9w8-mx.ap_zmr10_esc.202312090727
archive tar /x tftp://<IP ADDRESS>/ap3g2-rcvk9w8-tar.153-3.JPQ1.tar flash:

3.  Reboot the AP when the transfer completes.

[jagan.chowdam]

Solution for Expired AP Certificates and/or for Scenario of Encrypted Mobility Tunnels That Fail to Form

C9800 Command to Accept Expired Certificates

configure terminal
crypto pki certificate map map1 1
 issuer-name co cisco manufacturing ca
crypto pki certificate map map1 2
 issuer-name co act2 sudi ca

crypto pki trustpool policy
 match certificate map1 allow expired-certificate
    
exit

Create a Certificate Map and Add the Rules

configure terminal
crypto pki certificate map map1 1
issuer-name co Cisco Manufacturing CA

Use the Certificate Map Under the Trustpool Policy

configure terminal
crypto pki trustpool policy
match certificate map1 allow expired-certificate

 OR, 

Workaround for APs That Fail to Join the WLC Due to an Expired Certificate

Note: This workaround should only be used in order to allow the APs to join the WLC just long enough to upgrade the software and implement the solution provided in this section.

If the AP's and/or WLC's certificates have expired:

1. Disable Network Time Protocol (NTP).
2. Change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs might not be able to join.

Note: When you temporarily disable NTP and change the WLC time settings, it can adversely affect other time-dependent WLC features such as MFP, Simple Network Management Protocol Version 3 (SNMPv3), and Location.

Jagan Chowdam

[Rich R]

@jagan.chowdam has provided the main answer for you.  The commands for 9800 are in the field notice in the "C9800 Command to Accept Expired Certificates" section but easy to miss.

Maybe because we have ap cert-expiry-ignore mic enable set on those controllers? Yes

I remember that from when it last struck December 2022. Different problem, different field notice.  The AP MIC cert expiry issue has been around since at least 2014.

The link to this FN is in my signature below <wink>