Problem Description:
Due to the certificate expiration, any new Control and Provisioning of Wireless Access Points (CAPWAP) or Light Weight Access Point Protocol (LWAPP) connection will fail to establish. The main feature that is affected will be the Access Point (AP)-to-controller joining. The secondary feature that is affected will be new mobility connections between the controllers.
When an AP attempts to establish a new connection, the AP fails to join. When you configure mobility between controllers, they will fail to establish a connection.
The likelihood that this issue will be encountered is 100% for wireless products in use (to include APs and controllers) that have a Manufacturer Installed Certificate (MIC) that is older than ten years. Self-Signed Certificates (SSCs) that were generated by the Autonomous-to-lightweight Upgrade Tool will expire on January 1, 2020.
The affected products (listed in the Products Affected section) were released prior to the end of CY2005; beginning in March 2015, the products might begin to experience these symptoms.
Some Cisco CAPWAP based wireless solutions are reaching an age of 10 years from the date of manufacture. When this occurs CAPWAP DTLS tunnels will fail to be established because the certificates on CAPWAP based hardware has expired. The certificate installed in the wireless hardware is used to authenticate the devices when joining the network.
This issue is being tracked via Cisco defect ID: CSCuq19142 and via Field Notice 63942.
*Note: The MIC Lifetime has been documented in past via the Wireless LAN Controller (WLC) Design and Features FAQ at http://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/118833-wlc-design-ftrs-faq.html.
Problem Symptom:
Wireless Access Points fail to connect to the Wireless LAN Controller. At the time of the join failure, the WLC's msglog may show messages similar to the following:
Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55
CAPWAP utilizes Datagram Transport Layer Security (DTLS) in order to encrypt communication between the Lightweight AP and the WLC. The MIC or SSC is used in order to authenticate the Lightweight AP to the WLC, and vice versa, during the DTLS session establishment. The CAPWAP/DTLS connection cannot be established after the MIC or SSC validity end date.
Affected Products:
Cisco Wireless LAN Controllers - FCS in 2012 or earlier:
|
|
|
Date |
|
|
|
|
2006 Series Wireless LAN Controller |
4.2.x |
24/Mar/05 |
02/Apr/07 |
21/Apr/12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
NM-AIR-WLC6 |
4.2.x |
27/Feb/06 |
18/Feb/08 |
16/Feb/13 |
|
|
NME-AIR-WLCx (Cisco Wireless LAN Controller Module (WLCM)) |
7.0.x |
15/Feb/07 |
23/Apr/12 |
30/Apr/17 |
|
|
AIR-CT2504 |
8.5.x |
8/Jul/11 |
18/Apr/18 |
30/Apr/23 |
|
|
AIR-CT5508 |
8.5.x |
6/May/09 |
4/May/18 |
31/Jul/23 |
|
|
AIR-CT7510 |
8.5.x |
25/Mar/11 |
10/Apr/17 |
30/Apr/22 |
|
|
AIR-CT8510 |
8.5.x |
30/Aug/12 |
3/Sep/18 |
30/Sep/23 |
|
|
WS-SVC-WISM2 |
8.5.x |
2/Apr/11 |
10/Apr/17 |
30/Apr/22 |
Cisco Aironet Branded Lightweight Access Points - FCS in 2010 or earlier:
|
Family / SW Type |
Last Software Release |
FCS Date |
End Of Sale Date |
Last date of Support (HW) |
End of Sale Notice |
|---|---|---|---|---|---|
|
Cisco AP801 Integrated Access Point |
8.0.x |
26-Jun-08 (CISCO888W-GN-A-K9) |
31/Mar/16 (C887VA-V-W-E-K9) |
31/Mar/21 |
|
|
Cisco Aironet 1000 Series |
4.2.x |
24/Mar/05 |
11/Mar/08 |
10/Mar/13 |
|
|
Cisco Aironet 1040 Series |
8.3.x |
24/Aug/10 |
1/Oct/13 |
30/Sep/18 |
|
|
Cisco Aironet 1120 Series |
7.0.x |
02/Oct/02* |
19/Jun/09 |
18/Jun/14 |
http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1100-series/eol_c51-506612.html |
|
Cisco Aironet 1130 Series |
8.0.x |
24/Nov/04* |
26/Jul/13 |
31/Jul/18 |
|
|
Cisco Aironet 1140 Series |
8.3.x |
30/Sep/09 |
1/Oct/13 |
30/Sep/18 |
|
|
Cisco Aironet 1200/1230 Series |
7.0.x |
23/Aug/02* |
19/Jun/09 |
30/Jun/14 |
http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1230-ag-series/eol_c51-506614.html |
|
Cisco Aironet 1240 Series |
8.0.x |
12/Dec/05 |
26/Jul/13 |
31/Jul/18 |
|
|
Cisco Aironet 1250 Series |
8.0.x |
02/Nov/07 |
20/Jan/12 |
31/Jan/17 |
|
|
Cisco Aironet 1260 Series |
8.3.x |
27/Apr/10 |
7/Oct/13 |
2/Jan/18 |
|
|
Cisco Aironet 1300 Series |
7.0.x |
04/May/04* |
11/Jan/13 |
31/Jan/18 |
|
|
Cisco Aironet 1600 Series |
8.5.x |
16/Nov/12 |
29/Dec/16 |
31/Dec/21 |
|
|
Cisco Aironet 1700 Series |
8.10.x |
01/Jun/14 |
30/Apr/19 |
30/Apr/24 |
https://www.cisco.com/c/en/us/products/collateral/wireless/eos-eol-notice-c51-740712.html |
|
Cisco Aironet 2600 Series |
8.5.x |
18/Jun/12 |
29/Dec/16 |
31/Dec/21 |
|
|
Cisco Aironet 2700 Series |
8.10.x |
21/Mar/14 |
30/Apr/19 |
30/Apr/24 | https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-2700-series-access-point/eos-eol-notice-c51-740711.html |
|
Cisco Aironet 3500 series |
8.5.x |
26/May/10 |
1/Apr/16 |
31/Mar/21 |
|
| Cisco Aironet 3600 series |
8.5.x |
20/Oct/11 |
29/Dec/16 |
31/Dec/21 |
|
| Cisco Aironet 3700 Series |
8.10.x |
02/Sep/13 |
30/Apr/19 |
30/Apr/24 |
|
| Cisco Aironet AIR-CAP1552I series |
8.5.x |
2/May/11 |
30/Mar/16 |
31/Mar/21 |
|
|
Cisco Aironet 1570 series: AIR-AP1572EAC and AIR-AP1572EC |
- |
1/Sep/14 |
13/Nov/20 |
30/Nov/25 |
|
|
Cisco Aironet 1570 series: AIR-AP1572EC3, AIR-AP1572EC4, AIR-AP1572IC3 and AIR-AP1572IC4 |
- |
1/Sep/14 |
30/Apr/19 |
30/Apr/24 |
*Note: For AP series whose FCS date is before 2005: APs started being manufactured with MICs on July 18, 2005. Any Lightweight AP's that were manufactured prior to that date have SSCs.
Workaround prior to the fix being available:
If you believe you will be affected by this issue and need a fix before the official code with the associated correction is posted at www.cisco.com, then please contact TAC, who will work to provide an escalation release of code accordingly.
Recovery for APs in a failed scenario:
*Note: This workaround should only be used in order to allow APs with expired certificates to join the WLC for long enough to upgrade the software.
If the certificates have expired, disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs may not be able to join. Once the software has been upgraded, and the affected APs have joined, the WLC clock should be reset to the valid time.
*Note: Temporarily disabling NTP and changing the WLC's time settings can adversely affect other time dependent WLC features such as MFP, SNMPv3, and location.
Solution:
To allow additional usage of hardware, beyond the 10 year certificate date, Cisco is providing a software maintenance release with a feature to ignore the validity period of the certificates in the CAPWAP authentication process.
Maintenance releases with the feature to ignore the validity period of the certificates are being created for AireOS 7.0, 7.4 and 8.0.
Cisco has released the fix to Cisco.com in AireOS 7.0.252.0 and 7.4.140.0
Cisco will release to Cisco.com a rebuild of AireOS 8.0 (as version 8.0.120.0) before July 2015.
*Note: Cisco has a beta version of AirOS 8.0 MR2 that does contain the needed commands to work around this issue and can be used until the official AireOS 8.0 MR2 (8.0.120.0) is released on Cisco.com, see the following URL for details:
https://supportforums.cisco.com/document/12492986/80mr2-beta-availability
These maintenance releases should be updated before the certificate expires on the APs and WLCs.
By default, if an AP and/or WLC certificate has expired, then the DTLS connection will fail. To allow AP's to join a WLC after certificate expiration, upgrade to the fixed software version, then use the following commands:
For 7.0.252.0:
(WLC)>config ap lifetime-check {mic|ssc} enable
For 7.4.140.0 and later:
(WLC)>config ap cert-expiry-ignore {mic|ssc} enable
With "config ap lifetime-check {mic|ssc} enable" or "config ap cert-expiry-ignore {mic|ssc} enable" in effect, the WLC and AP will ignore the expiration date on the devices' MICs and SSCs. The above-noted commands must remain in effect as long as devices with expired MIC or SSC certificates are used.
Because 4400 series WLCs that were among the first manufactured had both Airespace and Cisco MICs installed, with the Airespace MIC being given precedence by the WLC, and the fix for CSCuq19142 is only applicable for Cisco MICs,the currently available fix for CSCuq19142 may not work. This is potentially applicable to most 4400s manufactured in 2005, and other variants, depending on RMA and refurbishment history of the affected unit. Please see section "How to Identify Hardware Levels" for how to determine the date of manufacture. If the affected unit was refurbished, the SN may have changed with the MIC remaining the same. At present the only remedy is to disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. Contact TAC to get an escalation image with the fix, as per bug ID CSCuu02970.
How to Identify Certificate Expiration date:
(via CLI or Serial number or Python Script or WLCCA)
This section describes how to determine when your AP and WLC MICs and/or SSCs expire using show commands when available or via the device serial number.
1) Manufacturing Installed Certificates (MICs):
The serial number can be used to determine the approximate date when the MIC will expire.
The AP's MIC will expire, at the earliest, ten years past the date of manufacture. Please note, some APs may have more recently created MICs under some conditions. For example, if the AP's motherboard was manufactured and stored, but not assembled until some time later or if the AP was subject to RMA and a refurbishing process, etc.
To determine when the AP was manufactured, run this command on the WLC to find the AP SN:
(Cisco Controller) >show ap inventory all
Inventory for lap1130-sw3-9
NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
NAME: "Dot11Radio0" , DESCR: "802.11G Radio"
PID: UNKNOWN, VID: , SN: GAM112706LC
NAME: "Dot11Radio1" , DESCR: "802.11A Radio"
PID: UNKNOWN, VID: , SN: ALP112706LC
The AP chassis SN is in the first section of the output, for example: PID:
AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
See "Deriving manufactured date from serial number" section below.
Alternatively, the exact date the MIC expires can be found by running this command and looking for the "Certificate" entry; ignore "CA Certificate" entries. The "end date" associated with the "Validity Date" section is the expiration date for the MIC certificate:
AP_CLI#sh crypto pki certificates
CA Certificate
Status: Available...
...
Certificate
Status: Available
Certificate Serial Number: 728AF4350000001E4C89
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: C1130-001c58b5b3a4
ea=support@cisco.com
cn=C1130-001c58b5b3a4
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/crl/cmca.crl
Validity Date:
start date: 04:22:10 UTC Jul 11 2007
end date: 04:32:10 UTC Jul 11 2017
Associated Trustpoints: Cisco_IOS_MIC_cert
2) Self-Signed Certificates (SSCs):
In order to determine if you have an SSC, run this WLC command:
AP_CLI >show auth-list
...
AP with Self-Signed Certificate................ yes
...
All AP SSC's have an expiration date of January 1st, 2020.
3) Wireless LAN Controllers (WLCs):
You can determine the WLC's serial number by running this command:
WLC_CLI>show inventory
Burned-in MAC Address............................ 24:E9:B3:43:C4:E0
Maximum number of APs supported.................. 75
NAME: "Chassis" , DESCR: "Cisco 2500 Series Wireless LAN Controller"
PID: AIR-CT2504-K9, VID: V04, SN: PSZ17441ANT
To determine the WLC serial number via the GUI, navigate: Controller > Inventory
If you have AireOS 8.0 or later, to determine when the WLC certificate expires, run this command and look for the "Cisco SHA1 device cert":
WLC_CLI: show certificate all
Certificate Name: Cisco SHA1 device cert
Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT2504-K9-d0c282d65a20, MAILTO=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number :
454384735992863371807890
Validity :
Start : 2011 Jul 26th, 20:17:17 GMT
End : 2021 Jul 26th, 20:27:17 GMT
Signature Algorithm :
rsa-pkcs1-sha1
Hash key :
SHA1 Fingerprint : 98:89:eb:12:2a:98:bc:fe:ad:5b:8f:23:63:0f:47:d1:36:ce:f5:be
MD5 Fingerprint : ba:f3:98:9a:cd:f8:01:08:84:b8:66:3c:6a:6c:d3:05
This command is not available in AireOS releases prior to 8.0. There is no similarly straightforward command to derive this date in earlier AireOS releases. As an alternate method, use the WLC serial numbers to determine the earliest possible MIC expiration date.
Deriving manufactured date from product serial numbers:
The serial number format is: "LLLYYWWSSSS"; where "YY" is the year of manufacture and "WW" is the week of manufacture. The date code can be found in the 4 middle digits of the serial number.
Manufacturing Year Codes:
01 = 1997 06 = 2002 11 = 2007 16 = 2012
02 = 1998 07 = 2003 12 = 2008 17 = 2013
03 = 1999 08 = 2004 13 = 2009 18 = 2014
04 = 2000 09 = 2005 14 = 2010
05 = 2001 10 = 2006 15 = 2011
Manufacturing Week Codes:
01-05 = January, 15-18 = April, 28-31 = July, 41-44 = October
06-09 = February, 19-22 = May, 32-35 = August, 45-48 = November
10-14 = March, 23-27 = June, 36-40 = September, 49-52 = December
Example: SN FCZ1128Q0PE has year code 11, meaning it was manufactured in the year 2007. The week code is 28, meaning it was manufactured in July of that year.
4) Access Point Certificate Check Tool:
A Python script has been written that runs on Windows, Mac and Linux systems that allows a user to check on the certificate expiration date for all AP's on their network.
The following Cisco Support Forum's article explains how to access and run this tool:
Access Point Certificate Check Tool - apCertCheck
5) Wireless LAN Controller (WLC) Config Analyzer:
WLCCA version 3.6.5 and above has support to check the AP Certificate expiration date. This check is done based on the AP Serial number and will flag any AP needing checked based on if the AP serial number is within 60 days of expiration.
The following Cisco Support Forum's article explains how to access and run this tool:
NOTE: Using the AP Serial number is only an approximation of the MIC expiration date. Any AP's flagged by this method should always be check for the real MIC expiration date via the Access Point commands listed.
