08-06-2018 12:57 AM - edited 07-05-2021 08:56 AM
Updated 5508 WLC from 8.0.152.0 to 8.3.143.0
WLC updated fine but our APs running 8.0.152.0 would not join the WLC (couldn’t even join it to download its updated software)
WLC logs below
*spamApTask3: Aug 05 13:41:25.730: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask3: Aug 05 13:40:39.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP 10.100.0.17 (13504).
*spamApTask3: Aug 05 13:40:39.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask3: Aug 05 13:40:31.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP 10.100.0.17 (13504).
*spamApTask3: Aug 05 13:40:31.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask3: Aug 05 13:40:27.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP 10.100.0.17 (13504).
*spamApTask3: Aug 05 13:40:27.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask3: Aug 05 13:40:25.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP 10.100.0.17 (13504).
*spamApTask3: Aug 05 13:40:25.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
--------
AP logs below
Feb 4 13:20:14.795: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Feb 4 13:20:14.795: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Feb 4 13:20:15.427: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to down
*Feb 4 13:20:15.643: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Feb 4 13:20:15.843: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Feb 4 13:20:16.843: %SOAP_FIPS-2-SELF_TEST_HW_SUCCESS: HW crypto FIPS self test passed (2-16)
*Feb 4 13:20:16.843: DPAA Initialization Complete
*Feb 4 13:20:16.843: %SYS-3-HARIKARI: Process DPAA INIT top-level routine exited
*Feb 4 13:20:17.847: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Feb 4 13:20:18.863: %LINK-6-UPDOWN: Interface BVI1, changed state to up
*Feb 4 13:20:19.863: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up
*Feb 4 13:20:20.315: Currently running a Release Image
validate_sha2_block: Failed to get certificate chain
*Feb 4 13:20:20.731: Using SHA-1 signed certificate for image signing validation.
*Feb 4 13:20:27.075: APAVC: Succeeded to activate all the STILE protocols.
*Feb 4 13:20:27.075: APAVC: Registering with CFT
*Feb 4 13:20:27.075: APAVC: CFT registration of delete callback succeeded
*Feb 4 13:20:27.075: APAVC: Reattaching Original Buffer pool for system use
*Feb 4 13:20:27.075: Pool-ReAtach: paks 18174 radio17566
*Feb 4 13:20:34.275: AP image integrity check PASSED
*Feb 4 13:20:34.387: validate_sha2_block:No SHA2 Block present on this AP.
*Feb 4 13:20:34.403: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Feb 4 13:20:34.403: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Feb 4 13:20:44.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 port 0 CLI Request Triggered
*Feb 4 13:20:55.691: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source
*Feb 4 13:20:56.791: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Feb 4 13:20:57.791: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
Translating "CISCO-CAPWAP-CONTROLLER"...domain server (255.255.255.255)
*Feb 4 13:20:57.887: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Feb 4 13:20:58.887: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Aug 5 13:55:10.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.0.9 peer_port: 5246
*Aug 5 13:55:39.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x6B7919C!
*Aug 5 13:56:09.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.100.0.9:5246
----------
I'm guessing the Cert issue relates to the lack of SHA2 block message on our APs?
---------
The result of show crypto ca certificates as as below
CA Certificate
Status: Available
Certificate Serial Number (hex): 01
Certificate Usage: Signature
Issuer:
cn=Cisco Root CA M2
o=Cisco
Subject:
cn=Cisco Root CA M2
o=Cisco
Validity Date:
start date: 13:00:18 UTC Nov 12 2012
end date: 13:00:18 UTC Nov 12 2037
Associated Trustpoints: Trustpool cisco-m2-root-cert
Storage:
CA Certificate
Status: Available
Certificate Serial Number (hex): 02
Certificate Usage: Signature
Issuer:
cn=Cisco Root CA M2
o=Cisco
Subject:
cn=Cisco Manufacturing CA SHA2
o=Cisco
CRL Distribution Points:
http://www.cisco.com/security/pki/crl/crcam2.crl
Validity Date:
start date: 13:50:58 UTC Nov 12 2012
end date: 13:00:17 UTC Nov 12 2037
Associated Trustpoints: Trustpool Cisco_IOS_M2_MIC_cert
Storage:
CA Certificate
Status: Available
Certificate Serial Number (hex): 00
Certificate Usage: General Purpose
Issuer:
cn=ca
ou=none
o=airespace Inc
l=San Jose
st=California
c=US
Subject:
cn=ca
ou=none
o=airespace Inc
l=San Jose
st=California
c=US
Validity Date:
start date: 23:38:55 UTC Feb 12 2003
end date: 23:38:55 UTC Nov 11 2012
Associated Trustpoints: airespace-old-root-cert
Storage:
CA Certificate
Status: Available
Certificate Serial Number (hex): 00
Certificate Usage: Signature
Issuer:
cn=Airespace Root CA
ou=Engineering
o=Airespace Inc.
l=San Jose
st=California
c=US
Subject:
cn=Airespace Root CA
ou=Engineering
o=Airespace Inc.
l=San Jose
st=California
c=US
Validity Date:
start date: 13:41:22 UTC Jul 31 2003
end date: 13:41:22 UTC Apr 29 2013
Associated Trustpoints: airespace-new-root-cert
Storage:
CA Certificate
Status: Available
Certificate Serial Number (hex): 03
Certificate Usage: General Purpose
Issuer:
cn=Airespace Root CA
ou=Engineering
o=Airespace Inc.
l=San Jose
st=California
c=US
Subject:
cn=Airespace Device CA
ou=Engineering
o=Airespace Inc.
l=San Jose
st=California
c=US
Validity Date:
start date: 22:37:13 UTC Apr 28 2005
end date: 22:37:13 UTC Jan 26 2015
Associated Trustpoints: airespace-device-root-cert
Storage:
CA Certificate
Status: Available
Certificate Serial Number (hex): 5FF87B282B54DC8D42A315B568C9ADFF
Certificate Usage: Signature
Issuer:
cn=Cisco Root CA 2048
o=Cisco Systems
Subject:
cn=Cisco Root CA 2048
o=Cisco Systems
Validity Date:
start date: 20:17:12 UTC May 14 2004
end date: 20:25:42 UTC May 14 2029
Associated Trustpoints: Trustpool cisco-root-cert
Storage:
Certificate
Status: Available
Certificate Serial Number (hex): 7EAD12810000002375BE
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: AP3G2-4c4e35034bfb
cn=AP3G2-4c4e35034bfb
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/pki/crl/cmca.crl
Validity Date:
start date: 20:53:56 UTC Feb 4 2013
end date: 21:03:56 UTC Feb 4 2023
Associated Trustpoints: Cisco_IOS_MIC_cert
Storage:
CA Certificate
Status: Available
Certificate Serial Number (hex): 6A6967B3000000000003
Certificate Usage: Signature
Issuer:
cn=Cisco Root CA 2048
o=Cisco Systems
Subject:
cn=Cisco Manufacturing CA
o=Cisco Systems
CRL Distribution Points:
http://www.cisco.com/security/pki/crl/crca2048.crl
Validity Date:
start date: 22:16:01 UTC Jun 10 2005
end date: 20:25:42 UTC May 14 2029
Associated Trustpoints: Trustpool Cisco_IOS_MIC_cert
Storage:
------------
I have found multiple Cisco Bugs that refer to similar symptoms but our situation doesn’t quite meet all the criteria
This one looks most similar - https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63916.html but our AP serial number doesn’t show as being affected?
These others show similar symptoms but not exact error matches
https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuq19142/?rfs=iqvred
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCur43050/?referring_site=bugqvinvisibleredir
Anyone able to assist?
Thanks
Paul
Solved! Go to Solution.
08-17-2018 06:22 AM
This root cause for this issue has now been established.
It would appear the controller in question had previously been setup to use LSC certs and configured to talk to a (now offline) CA.
Previous OS updates of the WLC didnt cause an issue but the jump from 8.0 to 8.3 caused this issue.
It wasnt possible to disable LSC from the GUI (it errored) but once disabled via the CLI, APs were able to connect.
Thanks to all those who provided input into this thread.
08-06-2018 01:51 AM
08-06-2018 02:07 AM
>show sysinfo
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.152.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS
System Name...................................... XXXXXXXXXXXX
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.100.0.9
IPv6 Address..................................... ::
Last Reset....................................... Power on reset
System Up Time................................... 0 days 17 hrs 5 mins 23 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180
Configured Country............................... GB - United Kingdom
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +35 C
External Temperature............................. +23 C
Fan Status....................................... OK
State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 4
Number of Active Clients......................... 225
Burned-in MAC Address............................ 00:06:F6:62:0B:40
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1
>show time
Time............................................. Mon Aug 6 09:01:37 2018
Timezone delta................................... 0:0
Timezone location................................
NTP Servers
NTP Polling Interval......................... 3600
Index NTP Key Index NTP Server NTP Msg Auth Status
------- ----------------------------------------------------------------------------------
1 0 10.100.0.XX AUTH DISABLED
---------
#sh version
Cisco IOS Software, C2600 Software (AP3G2-K9W8-M), Version 15.3(3)JA12, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Fri 20-Oct-17 20:51 by prod_rel_team
ROM: Bootstrap program is C2600 boot loader
BOOTLDR: C2600 Boot Loader (AP3G2-BOOT-M) LoaderVersion 12.4(25e)JAY, RELEASE SOFTWARE (fc1)
GC01-F00-AP06 uptime is 17 hours, 9 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-xx.153-3.JA12"
Last reload reason:
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco AIR-CAP2602I-E-K9 (PowerPC) processor (revision A0) with 188398K/60928K bytes of memory.
Processor board ID FGL1708Z7XA
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
LWAPP image version 8.0.152.0
1 Gigabit Ethernet interface
2 802.11 Radios
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 4C:4E:35:03:4B:FB
Part Number : 73-14588-02
PCA Assembly Number : 800-37899-01
PCA Revision Number : A0
PCB Serial Number : FOC17055MWR
Top Assembly Part Number : 800-38356-01
Top Assembly Serial Number : FGL1708Z7XA
Top Revision Number : A0
Product/Model Number : AIR-CAP2602I-E-K9
Configuration register is 0xF
#sh ip interface brief
Interface IP-Address OK? Method Status Protocol
BVI1 10.100.0.X YES TFTP up up
Dot11Radio0 unassigned NO unset up up
Dot11Radio1 unassigned NO unset up up
GigabitEthernet0 unassigned NO unset up up
Virtual-WLAN0 unassigned NO unset up up
Virtual-WLAN0.1 unassigned NO unset up up
Virtual-WLAN0.2 unassigned NO unset up up
Virtual-WLAN0.3 unassigned NO unset up up
Virtual-WLAN0.4 unassigned NO unset up up
Virtual-WLAN0.5 unassigned NO unset up up
Virtual-WLAN0.6 unassigned NO unset up up
Virtual-WLAN0.7 unassigned NO unset up up
Virtual-WLAN0.8 unassigned NO unset up up
Virtual-WLAN0.9 unassigned NO unset up up
Virtual-WLAN0.10 unassigned NO unset up up
Virtual-WLAN0.11 unassigned NO unset up up
Virtual-WLAN0.12 unassigned NO unset up up
Virtual-WLAN0.13 unassigned NO unset up up
Virtual-WLAN0.14 unassigned NO unset up up
Virtual-WLAN0.15 unassigned NO unset up up
Virtual-WLAN0.16 unassigned NO unset up up
-------
WLC commands show details of 8.0.152.0 as we had to roll back I'm afraid
08-06-2018 02:17 AM
08-06-2018 02:21 AM
08-06-2018 02:52 AM
@PJR_CDF wrote:
Doesn't entering this command just set the CAPWAP controller IP address for the AP?
The command manually points the AP to the WLC.
Your previous output does not show if the AP knows where the WLC from DHCP Option 43.
If, by entering this command and, the AP joins the WLC then I am very certain I can say that the issue is that DHCP Option 43 is either misconfigured or not configured at all.
08-06-2018 03:01 AM
option 43 isnt used
I believe the AP knows where the WLC is from it's current configuration and is able to locate the WLC and attempt connection which the logs I posted earlier show (unless I am misinterpreting something?)
I appreciate my original post showed a load of info but in terms of connectivity the WLC logs show attempted connection from the AP
WLC Log
*spamApTask2: Aug 05 15:14:43.021: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP 10.100.0.17 (13505).
*spamApTask2: Aug 05 15:14:43.020: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
AP logs showing attempted connection to WLC
*Aug 5 13:55:10.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.0.9 peer_port: 5246
*Aug 5 13:55:39.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x6B7919C!
I dont think the AP is unable to locate the WLC - It appears to me that the 2 devices are unable to negotiate a secure channel on which to communicate?
Thanks for your continued help by the way
08-06-2018 03:44 AM
08-17-2018 06:22 AM
This root cause for this issue has now been established.
It would appear the controller in question had previously been setup to use LSC certs and configured to talk to a (now offline) CA.
Previous OS updates of the WLC didnt cause an issue but the jump from 8.0 to 8.3 caused this issue.
It wasnt possible to disable LSC from the GUI (it errored) but once disabled via the CLI, APs were able to connect.
Thanks to all those who provided input into this thread.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide