cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Community Helping Community

Lightweight AP - Fail to create CAPWAP/LWAPP connection due to certificate expiration

87508
Views
40
Helpful
33
Comments

Problem Description:

Due to the certificate expiration, any new Control and Provisioning of Wireless Access Points (CAPWAP) or Light Weight Access Point Protocol (LWAPP) connection will fail to establish. The main feature that is affected will be the Access Point (AP)-to-controller joining. The secondary feature that is affected will be new mobility connections between the controllers.

When an AP attempts to establish a new connection, the AP fails to join. When you configure mobility between controllers, they will fail to establish a connection.

The likelihood that this issue will be encountered is 100% for wireless products in use (to include APs and controllers) that have a Manufacturer Installed Certificate (MIC) that is older than ten years. Self-Signed Certificates (SSCs) that were generated by the Autonomous-to-lightweight Upgrade Tool will expire on January 1, 2020.

The affected products (listed in the Products Affected section) were released prior to the end of CY2005; beginning in March 2015, the products might begin to experience these symptoms.

Some Cisco CAPWAP based wireless solutions are reaching an age of 10 years from the date of manufacture. When this occurs CAPWAP DTLS tunnels will fail to be established because the certificates on CAPWAP based hardware has expired. The certificate installed in the wireless hardware is used to authenticate the devices when joining the network.

This issue is being tracked via Cisco defect ID: CSCuq19142 and via Field Notice 63942.

*Note: The MIC Lifetime has been documented in past via the Wireless LAN Controller (WLC) Design and Features FAQ at http://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/118833-wlc-design-ftrs-faq.html.
 

Problem Symptom:

Wireless Access Points fail to connect to the Wireless LAN Controller. At the time of the join failure, the WLC's msglog may show messages similar to the following:

Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55

CAPWAP utilizes Datagram Transport Layer Security (DTLS) in order to encrypt communication between the Lightweight AP and the WLC. The MIC or SSC is used in order to authenticate the Lightweight AP to the WLC, and vice versa, during the DTLS session establishment. The CAPWAP/DTLS connection cannot be established after the MIC or SSC validity end date.

 

Affected Products:

Cisco Wireless LAN Controllers - FCS in 2011 or earlier:


Family / SW Type


Last Software Release


FCS

Date


End of Sale Date


Last Date of Support (HW)


End of Sale Notice

2006 Series Wireless LAN Controller

4.2.x

24/Mar/05

02/Apr/07

21/Apr/12

http://www.cisco.com/c/en/us/products/collateral/wireless/2000-series-wireless-lan-controllers/prod_end-of-life_notice0900aecd805d22b0.html


2100 Series Wireless LAN Controller


7.0.x


09/Jan/07


02/May/12


31/May/17


http://www.cisco.com/c/en/us/products/collateral/wireless/2100-series-wireless-lan-controllers/end_of_life_notice_c51-691053.html


4400 Series Wireless LAN Controller


7.0.x


23/Jun/05


13/Jun/11


30/Jun/16


http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/end_of_life_notice_c51-634665.html


Cisco Catalyst 3750G Integrated Wireless LAN Controller


7.0.x


14/Mar/07


13/Jun/11


30/Jun/16


http://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-3750-series-integrated-wireless-lan-controllers/end_of_life_notice_c51-634675.html


Cisco Wireless Services Module 1 (WiSM1)


7.0.x


14/Nov/05


23/Apr/12


30/Apr/17


http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/catalyst-6500-series-7600-series-wireless-services-module-wism/end_of_life_notice_c51-691055.html

NM-AIR-WLC6
(Cisco 6-Access-Point Wireless LAN Controller Network Module)

4.2.x

27/Feb/06

18/Feb/08

16/Feb/13

http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/network-modules/prod_end-of-life_notice0900aecd806aeb34.html

NME-AIR-WLCx (Cisco Wireless LAN Controller Module (WLCM))

7.0.x

15/Feb/07

23/Apr/12

30/Apr/17

http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/wireless-lan-controller-module/end_of_life_notice_c51-691054.html

AIR-CT2504

-

8/Jul/11

-

-

-

AIR-CT5508

-

6/May/09

-

-

-

AIR-CT7510

-

25/Mar/11

10/Apr/17

30/Apr/22

http://www.cisco.com/c/en/us/products/collateral/wireless/flex-7500-series-wireless-controllers/eos-eol-notice-c51-738009.html

WS-SVC-WISM2

-

2/Apr/11

-

-

-

 

 

Cisco Aironet Branded Lightweight Access Points - FCS in 2010 or earlier:

Family / SW Type

Last Software Release

FCS Date

End Of Sale Date

Last date of Support (HW)

End of Sale Notice

Cisco AP801 Integrated Access Point

8.0.x

26-Jun-08 (CISCO888W-GN-A-K9)

31/Mar/16 (C887VA-V-W-E-K9)

31/Mar/21

http://www.cisco.com/c/en/us/products/collateral/routers/800-series-routers/eos-eol-notice-c51-735923.html

Cisco Aironet 1000 Series

4.2.x

24/Mar/05

11/Mar/08

10/Mar/13

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1000-series/prod_end-of-life_notice0900aecd806c0c29.html

Cisco Aironet 1040 Series

8.3.x

24/Aug/10

1/Oct/13

30/Sep/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1140-series/end_of_life_notice_c51-727650.html

Cisco Aironet 1120 Series

7.0.x

02/Oct/02*

19/Jun/09

18/Jun/14

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1100-series/eol_c51-506612.html

Cisco Aironet 1130 Series

8.0.x

24/Nov/04*

26/Jul/13

31/Jul/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1130-ag-series/end_of_life_notice_c51-726426.html

Cisco Aironet 1140 Series

8.3.x

30/Sep/09

1/Oct/13

30/Sep/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1130-ag-series/end_of_life_notice_c51-727649.html

Cisco Aironet 1200/1230 Series

7.0.x

23/Aug/02*

19/Jun/09

30/Jun/14

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1230-ag-series/eol_c51-506614.html

Cisco Aironet 1240 Series

8.0.x

12/Dec/05

26/Jul/13

31/Jul/18

http://www.cisco.com/c/en/us/products/collateral/collaboration-endpoints/unified-ip-phone-7900-series/end_of_life_notice_c51-726425.html

Cisco Aironet 1250 Series

8.0.x

02/Nov/07

20/Jan/12

31/Jan/17

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1130-ag-series/end_of_life_notice_c51-681596.html

Cisco Aironet 1260 Series

8.3.x

27/Apr/10

7/Oct/13

2/Jan/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1260-series/end_of_life_notice_c51-727739.html

Cisco Aironet 1300 Series

7.0.x

04/May/04*

11/Jan/13

31/Jan/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1300-series/end_of_life_notice_c51-711894.html

Cisco Aironet 3500 series

-

26/May/10

1/Apr/16

31/Mar/21

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-3500-series/eos-eol-notice-c51-734304.html


*Note: For AP series whose FCS date is before 2005: APs started being manufactured with MICs on July 18, 2005.  Any Lightweight AP's that were manufactured prior to that date have SSCs.

 

Workaround prior to the fix being available:

If you believe you will be affected by this issue and need a fix before the official code with the associated correction is posted at www.cisco.com, then please contact TAC, who will work to provide an escalation release of code accordingly.

 

Recovery for APs in a failed scenario:

*Note: This workaround should only be used in order to allow APs with expired certificates to join the WLC for long enough to upgrade the software.

If the certificates have expired, disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs may not be able to join. Once the software has been upgraded, and the affected APs have joined, the WLC clock should be reset to the valid time.

*Note: Temporarily disabling NTP and changing the WLC's time settings can adversely affect other time dependent WLC features such as MFP, SNMPv3, and location.

 

Solution:

To allow additional usage of hardware, beyond the 10 year certificate date, Cisco is providing a software maintenance release with a feature to ignore the validity period of the certificates in the CAPWAP authentication process.

Maintenance releases with the feature to ignore the validity period of the certificates are being created for AireOS 7.0, 7.4 and 8.0.

Cisco has released the fix to Cisco.com in AireOS 7.0.252.0 and 7.4.140.0

Cisco will release to Cisco.com a rebuild of AireOS 8.0 (as version 8.0.120.0) before July 2015.

*Note: Cisco has a beta version of AirOS 8.0 MR2 that does contain the needed commands to work around this issue and can be used until the official AireOS 8.0 MR2 (8.0.120.0) is released on Cisco.com, see the following URL for details:

https://supportforums.cisco.com/document/12492986/80mr2-beta-availability

 

These maintenance releases should be updated before the certificate expires on the APs and WLCs.
 

By default, if an AP and/or WLC certificate has expired, then the DTLS connection will fail. To allow AP's to join a WLC after certificate expiration, upgrade to the fixed software version, then use the following commands:

For 7.0.252.0:
(WLC)>config ap lifetime-check {mic|ssc} enable

For 7.4.140.0 and later:
(WLC)>config ap cert-expiry-ignore {mic|ssc} enable

With "config ap lifetime-check {mic|ssc} enable" or "config ap cert-expiry-ignore {mic|ssc} enable" in effect, the WLC and AP will ignore the expiration date on the devices' MICs and SSCs. The above-noted commands must remain in effect as long as devices with expired MIC or SSC certificates are used.

Because 4400 series WLCs that were among the first manufactured had both Airespace and Cisco MICs installed, with the Airespace MIC being given precedence by the WLC, and the fix for CSCuq19142 is only applicable for Cisco MICs,the currently available fix for CSCuq19142 may not work.  This is potentially applicable to most 4400s manufactured in 2005, and other variants, depending on RMA and refurbishment history of the affected unit.  Please see section "How to Identify Hardware Levels" for how to determine the date of manufacture. If the affected unit was refurbished, the SN may have changed with the MIC remaining the same. At present the only remedy is to disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. Contact TAC to get an escalation image with the fix, as per bug ID CSCuu02970.

 

How to Identify Certificate Expiration date:

(via CLI or Serial number or Python Script or WLCCA)

This section describes how to determine when your AP and WLC MICs and/or SSCs expire using show commands when available or via the device serial number.

1) Manufacturing Installed Certificates (MICs):

The serial number can be used to determine the approximate date when the MIC will expire.

The AP's MIC will expire, at the earliest, ten years past the date of manufacture. Please note, some APs may have more recently created MICs under some conditions. For example, if the AP's motherboard was manufactured and stored, but not assembled until some time later or if the AP was subject to RMA and a refurbishing process, etc.

To determine when the AP was manufactured, run this command on the WLC to find the AP SN:

(Cisco Controller) >show ap inventory all
Inventory for lap1130-sw3-9
NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
NAME: "Dot11Radio0" , DESCR: "802.11G Radio"
PID: UNKNOWN, VID: , SN: GAM112706LC
NAME: "Dot11Radio1" , DESCR: "802.11A Radio"
PID: UNKNOWN, VID: , SN: ALP112706LC
The AP chassis SN is in the first section of the output, for example: PID:
AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE

See "Deriving manufactured date from serial number" section below.

Alternatively, the exact date the MIC expires can be found by running this command and looking for the "Certificate" entry; ignore "CA Certificate" entries. The "end date" associated with the "Validity Date" section is the expiration date for the MIC certificate:

AP_CLI#sh crypto pki certificates
CA Certificate
Status: Available...
...
Certificate
Status: Available
Certificate Serial Number: 728AF4350000001E4C89
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: C1130-001c58b5b3a4
ea=support@cisco.com
cn=C1130-001c58b5b3a4
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/crl/cmca.crl
Validity Date:
start date: 04:22:10 UTC Jul 11 2007
end date: 04:32:10 UTC Jul 11 2017
Associated Trustpoints: Cisco_IOS_MIC_cert


2) Self-Signed Certificates (SSCs):

In order to determine if you have an SSC, run this WLC command:

AP_CLI >show auth-list
...
AP with Self-Signed Certificate................ yes
...

All AP SSC's have an expiration date of January 1st, 2020.


3) Wireless LAN Controllers (WLCs):

You can determine the WLC's serial number by running this command:
WLC_CLI>show inventory

Burned-in MAC Address............................ 24:E9:B3:43:C4:E0
Maximum number of APs supported.................. 75
NAME: "Chassis" , DESCR: "Cisco 2500 Series Wireless LAN Controller"
PID: AIR-CT2504-K9, VID: V04, SN: PSZ17441ANT

To determine the WLC serial number via the GUI, navigate: Controller > Inventory

If you have AireOS 8.0 or later, to determine when the WLC certificate expires, run this command and look for the "Cisco SHA1 device cert":

WLC_CLI: show certificate all

Certificate Name: Cisco SHA1 device cert
Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT2504-K9-d0c282d65a20, MAILTO=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number :
454384735992863371807890
Validity :
Start : 2011 Jul 26th, 20:17:17 GMT
End : 2021 Jul 26th, 20:27:17 GMT
Signature Algorithm :
rsa-pkcs1-sha1
Hash key :
SHA1 Fingerprint : 98:89:eb:12:2a:98:bc:fe:ad:5b:8f:23:63:0f:47:d1:36:ce:f5:be
MD5 Fingerprint : ba:f3:98:9a:cd:f8:01:08:84:b8:66:3c:6a:6c:d3:05

This command is not available in AireOS releases prior to 8.0. There is no similarly straightforward command to derive this date in earlier AireOS releases. As an alternate method, use the WLC serial numbers to determine the earliest possible MIC expiration date.

Deriving manufactured date from product serial numbers:
The serial number format is: "LLLYYWWSSSS"; where "YY" is the year of manufacture and "WW" is the week of manufacture. The date code can be found in the 4 middle digits of the serial number.

Manufacturing Year Codes:
01 = 1997 06 = 2002 11 = 2007 16 = 2012
02 = 1998 07 = 2003 12 = 2008 17 = 2013
03 = 1999 08 = 2004 13 = 2009 18 = 2014
04 = 2000 09 = 2005 14 = 2010
05 = 2001 10 = 2006 15 = 2011


Manufacturing Week Codes:
01-05 = January,   15-18 = April,      28-31 = July,              41-44 = October
06-09 = February,  19-22 = May,      32-35 = August,         45-48 = November
10-14 = March,      23-27 = June,     36-40 = September,   49-52 = December

Example: SN FCZ1128Q0PE has year code 11, meaning it was manufactured in the year 2007. The week code is 28, meaning it was manufactured in July of that year.

 

4) Access Point Certificate Check Tool:

A Python script has been written that runs on Windows, Mac and Linux systems that allows a user to check on the certificate expiration date for all AP's on their network. 

The following Cisco Support Forum's article explains how to access and run this tool:

            Access Point Certificate Check Tool - apCertCheck

 

5) Wireless LAN Controller (WLC) Config Analyzer:

WLCCA version 3.6.5 and above has support to check the AP Certificate expiration date.  This check is done based on the AP Serial number and will flag any AP needing checked based on if the AP serial number is within 60 days of expiration. 

The following Cisco Support Forum's article explains how to access and run this tool:

          WLC Config Analyzer

NOTE: Using the AP Serial number is only an approximation of the MIC expiration date.  Any AP's flagged by this method should always be check for the real MIC expiration date via the Access Point commands listed.

 

Comments
Beginner

Hi Cisco Team,

 

Hope you're doing well.

 

We have the same issue but this is only happens when our connection failover from ISP 1 to ISP 2. Is it ISP issue? Please help us to solve this issue, were trying to trouble shoot since last year :(

 

 

 

Beginner

@timsmith 

Hi Timsmith, thanks for sharing

is it possible this is only happens during fail over? Example we have ISP 1 and ISP 2, when using the ISP 1 we did not encountered this certificate expiration but if we do the fail over the ISP 2 we see this issue. All AP's are disconnected to WLC but the WLC can ping the access points. Thank you very much

Cisco Employee
Sorry, I've been out on vacation. To answer your question, anytime an AP would try to join a WLC, the certificate is checked, if expired, then the join would fail. So this can happen on a failover.

Thanks.. Tim
CreatePlease to create content
Content for Community-Ad

August's Community Spotlight Awards