2602 AP's wont connect to 5508WLC after update to 8.3.143 - PKI initialization error

[PJR_CDF]

Updated 5508 WLC from 8.0.152.0 to 8.3.143.0

WLC updated fine but our APs running 8.0.152.0 would not join the WLC (couldn’t even join it to download its updated software)

WLC logs below

*spamApTask3: Aug 05 13:41:25.730: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

*spamApTask3: Aug 05 13:40:39.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP  10.100.0.17 (13504).

*spamApTask3: Aug 05 13:40:39.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

*spamApTask3: Aug 05 13:40:31.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP  10.100.0.17 (13504).

*spamApTask3: Aug 05 13:40:31.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

*spamApTask3: Aug 05 13:40:27.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP  10.100.0.17 (13504).

*spamApTask3: Aug 05 13:40:27.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

*spamApTask3: Aug 05 13:40:25.729: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP  10.100.0.17 (13504).

*spamApTask3: Aug 05 13:40:25.728: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

--------

AP logs below

Feb  4 13:20:14.795: %SSH-5-ENABLED: SSH 2.0 has been enabled

*Feb  4 13:20:14.795: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up

*Feb  4 13:20:15.427: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to down

*Feb  4 13:20:15.643: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down

*Feb  4 13:20:15.843: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down

*Feb  4 13:20:16.843: %SOAP_FIPS-2-SELF_TEST_HW_SUCCESS: HW crypto FIPS self test passed (2-16)

*Feb  4 13:20:16.843: DPAA Initialization Complete

*Feb  4 13:20:16.843: %SYS-3-HARIKARI: Process DPAA INIT top-level routine exited

*Feb  4 13:20:17.847: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up

*Feb  4 13:20:18.863: %LINK-6-UPDOWN: Interface BVI1, changed state to up

*Feb  4 13:20:19.863: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up

*Feb  4 13:20:20.315: Currently running a Release Image

validate_sha2_block: Failed to get certificate chain

*Feb  4 13:20:20.731: Using SHA-1 signed certificate for image signing validation.

*Feb  4 13:20:27.075: APAVC: Succeeded to activate all the STILE protocols.

*Feb  4 13:20:27.075: APAVC: Registering with CFT

*Feb  4 13:20:27.075: APAVC: CFT registration of delete callback succeeded

*Feb  4 13:20:27.075: APAVC: Reattaching  Original Buffer pool for system use

*Feb  4 13:20:27.075: Pool-ReAtach: paks 18174 radio17566

*Feb  4 13:20:34.275: AP image integrity check PASSED

*Feb  4 13:20:34.387:  validate_sha2_block:No SHA2 Block present on this AP.

*Feb  4 13:20:34.403: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Feb  4 13:20:34.403: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset

*Feb  4 13:20:44.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 port 0 CLI Request Triggered

*Feb  4 13:20:55.691: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source

*Feb  4 13:20:56.791: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up

*Feb  4 13:20:57.791: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up

Translating "CISCO-CAPWAP-CONTROLLER"...domain server (255.255.255.255)

*Feb  4 13:20:57.887: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up

*Feb  4 13:20:58.887: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up

*Aug  5 13:55:10.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.0.9 peer_port: 5246

*Aug  5 13:55:39.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x6B7919C! 

*Aug  5 13:56:09.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.100.0.9:5246

----------

I'm guessing the Cert issue relates to the lack of SHA2 block message on our APs?

---------

The result of show crypto ca certificates as as below

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 01

  Certificate Usage: Signature

  Issuer:

    cn=Cisco Root CA M2

    o=Cisco

  Subject:

    cn=Cisco Root CA M2

    o=Cisco

  Validity Date:

    start date: 13:00:18 UTC Nov 12 2012

    end   date: 13:00:18 UTC Nov 12 2037

  Associated Trustpoints: Trustpool cisco-m2-root-cert

  Storage:

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 02

  Certificate Usage: Signature

  Issuer:

    cn=Cisco Root CA M2

    o=Cisco

  Subject:

    cn=Cisco Manufacturing CA SHA2

    o=Cisco

  CRL Distribution Points:

    http://www.cisco.com/security/pki/crl/crcam2.crl

  Validity Date:

    start date: 13:50:58 UTC Nov 12 2012

    end   date: 13:00:17 UTC Nov 12 2037

  Associated Trustpoints: Trustpool Cisco_IOS_M2_MIC_cert

  Storage:

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 00

  Certificate Usage: General Purpose

  Issuer:

    e=support@airespace.com

    cn=ca

    ou=none

    o=airespace Inc

    l=San Jose

    st=California

    c=US

  Subject:

    e=support@airespace.com

    cn=ca

    ou=none

    o=airespace Inc

    l=San Jose

    st=California

    c=US

  Validity Date:

    start date: 23:38:55 UTC Feb 12 2003

   end   date: 23:38:55 UTC Nov 11 2012

  Associated Trustpoints: airespace-old-root-cert

  Storage:

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 00

  Certificate Usage: Signature

  Issuer:

    e=support@airespace.com

    cn=Airespace Root CA

    ou=Engineering

    o=Airespace Inc.

    l=San Jose

    st=California

    c=US

  Subject:

    e=support@airespace.com

    cn=Airespace Root CA

    ou=Engineering

    o=Airespace Inc.

    l=San Jose

    st=California

    c=US

  Validity Date:

    start date: 13:41:22 UTC Jul 31 2003

    end   date: 13:41:22 UTC Apr 29 2013

  Associated Trustpoints: airespace-new-root-cert

  Storage:

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 03

  Certificate Usage: General Purpose

  Issuer:

    e=support@airespace.com

    cn=Airespace Root CA

    ou=Engineering

    o=Airespace Inc.

    l=San Jose

    st=California

    c=US

  Subject:

    e=support@airespace.com

    cn=Airespace Device CA

    ou=Engineering

    o=Airespace Inc.

    l=San Jose

    st=California

    c=US

  Validity Date:

    start date: 22:37:13 UTC Apr 28 2005

    end   date: 22:37:13 UTC Jan 26 2015

  Associated Trustpoints: airespace-device-root-cert

  Storage:

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 5FF87B282B54DC8D42A315B568C9ADFF

  Certificate Usage: Signature

  Issuer:

    cn=Cisco Root CA 2048

    o=Cisco Systems

  Subject:

    cn=Cisco Root CA 2048

    o=Cisco Systems

  Validity Date:

    start date: 20:17:12 UTC May 14 2004

    end   date: 20:25:42 UTC May 14 2029

  Associated Trustpoints: Trustpool cisco-root-cert

  Storage:

Certificate

  Status: Available

  Certificate Serial Number (hex): 7EAD12810000002375BE

  Certificate Usage: General Purpose

  Issuer:

    cn=Cisco Manufacturing CA

    o=Cisco Systems

  Subject:

    Name: AP3G2-4c4e35034bfb

    e=support@cisco.com

    cn=AP3G2-4c4e35034bfb

    o=Cisco Systems

    l=San Jose

    st=California

    c=US

  CRL Distribution Points:

    http://www.cisco.com/security/pki/crl/cmca.crl

  Validity Date:

    start date: 20:53:56 UTC Feb 4 2013

    end   date: 21:03:56 UTC Feb 4 2023

  Associated Trustpoints: Cisco_IOS_MIC_cert

  Storage:

CA Certificate

  Status: Available

  Certificate Serial Number (hex): 6A6967B3000000000003

  Certificate Usage: Signature

  Issuer:

    cn=Cisco Root CA 2048

    o=Cisco Systems

  Subject:

    cn=Cisco Manufacturing CA

    o=Cisco Systems

  CRL Distribution Points:

    http://www.cisco.com/security/pki/crl/crca2048.crl

  Validity Date:

    start date: 22:16:01 UTC Jun 10 2005

    end   date: 20:25:42 UTC May 14 2029

  Associated Trustpoints: Trustpool Cisco_IOS_MIC_cert

  Storage:

------------

I have found multiple Cisco Bugs that refer to similar symptoms but our situation doesn’t quite meet all the criteria

This one looks most similar - https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63916.html but our AP serial number doesn’t show as being affected?

These others show similar symptoms but not exact error matches

https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuq19142/?rfs=iqvred

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCur43050/?referring_site=bugqvinvisibleredir

Anyone able to assist?

Thanks

Paul

[Leo Laohoo]

Post the complete output to the following commands:
1. WLC: sh sysinfo;
2. WLC: sh time;
3. AP: sh version; and
4. AP: sh ip interface brief

[PJR_CDF]

>show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.152.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... XXXXXXXXXXXX
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.100.0.9
IPv6 Address..................................... ::
Last Reset....................................... Power on reset
System Up Time................................... 0 days 17 hrs 5 mins 23 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... GB - United Kingdom
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +35 C
External Temperature............................. +23 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 4
Number of Active Clients......................... 225

Burned-in MAC Address............................ 00:06:F6:62:0B:40
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1

>show time

Time............................................. Mon Aug 6 09:01:37 2018

Timezone delta................................... 0:0
Timezone location................................

NTP Servers
NTP Polling Interval......................... 3600

Index NTP Key Index NTP Server NTP Msg Auth Status
------- ----------------------------------------------------------------------------------
1 0 10.100.0.XX AUTH DISABLED

 ---------

#sh version
Cisco IOS Software, C2600 Software (AP3G2-K9W8-M), Version 15.3(3)JA12, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Fri 20-Oct-17 20:51 by prod_rel_team

ROM: Bootstrap program is C2600 boot loader
BOOTLDR: C2600 Boot Loader (AP3G2-BOOT-M) LoaderVersion 12.4(25e)JAY, RELEASE SOFTWARE (fc1)

GC01-F00-AP06 uptime is 17 hours, 9 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g2-k9w8-mx.153-3.JA12/ap3g2-k9w8-xx.153-3.JA12"
Last reload reason:

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP2602I-E-K9 (PowerPC) processor (revision A0) with 188398K/60928K bytes of memory.
Processor board ID FGL1708Z7XA
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
LWAPP image version 8.0.152.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 4C:4E:35:03:4B:FB
Part Number : 73-14588-02
PCA Assembly Number : 800-37899-01
PCA Revision Number : A0
PCB Serial Number : FOC17055MWR
Top Assembly Part Number : 800-38356-01
Top Assembly Serial Number : FGL1708Z7XA
Top Revision Number : A0
Product/Model Number : AIR-CAP2602I-E-K9

Configuration register is 0xF
#sh ip interface brief
Interface IP-Address OK? Method Status Protocol
BVI1 10.100.0.X YES TFTP up up
Dot11Radio0 unassigned NO unset up up
Dot11Radio1 unassigned NO unset up up
GigabitEthernet0 unassigned NO unset up up
Virtual-WLAN0 unassigned NO unset up up
Virtual-WLAN0.1 unassigned NO unset up up
Virtual-WLAN0.2 unassigned NO unset up up
Virtual-WLAN0.3 unassigned NO unset up up
Virtual-WLAN0.4 unassigned NO unset up up
Virtual-WLAN0.5 unassigned NO unset up up
Virtual-WLAN0.6 unassigned NO unset up up
Virtual-WLAN0.7 unassigned NO unset up up
Virtual-WLAN0.8 unassigned NO unset up up
Virtual-WLAN0.9 unassigned NO unset up up
Virtual-WLAN0.10 unassigned NO unset up up
Virtual-WLAN0.11 unassigned NO unset up up
Virtual-WLAN0.12 unassigned NO unset up up
Virtual-WLAN0.13 unassigned NO unset up up
Virtual-WLAN0.14 unassigned NO unset up up
Virtual-WLAN0.15 unassigned NO unset up up
Virtual-WLAN0.16 unassigned NO unset up up

-------

WLC commands show details of 8.0.152.0 as we had to roll back I'm afraid

[Leo Laohoo]

Enter this command from the AP: capwap ap primary-base <WLC name> <WLC Management IP address>

[PJR_CDF]

I'm not allowed to make changes outside of maintenance windows I'm afraid.

Doesn't entering this command just set the CAPWAP controller IP address for the AP?

Just trying to understand what adding this command will show us?

Thanks for your assistance as always though :)

[Leo Laohoo]

---

@PJR_CDF wrote:
Doesn't entering this command just set the CAPWAP controller IP address for the AP?

---

The command manually points the AP to the WLC. 

Your previous output does not show if the AP knows where the WLC from DHCP Option 43.

If, by entering this command and, the AP joins the WLC then I am very certain I can say that the issue is that DHCP Option 43 is either misconfigured or not configured at all.

[PJR_CDF]

option 43 isnt used

I believe the AP knows where the WLC is from it's current configuration and is able to locate the WLC and attempt connection which the logs I posted earlier show (unless I am misinterpreting something?)

I appreciate my original post showed a load of info but in terms of connectivity the WLC logs show attempted connection from the AP

WLC Log

*spamApTask2: Aug 05 15:14:43.021: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP 10.100.0.17 (13505).
*spamApTask2: Aug 05 15:14:43.020: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

AP logs showing attempted connection to WLC

*Aug  5 13:55:10.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.0.9 peer_port: 5246

*Aug  5 13:55:39.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x6B7919C!

I dont think the AP is unable to locate the WLC - It appears to me that the 2 devices are unable to negotiate a secure channel on which to communicate?

Thanks for your continued help by the way

[patoberli]

Correct, this is what this command does. It will probably also cause the AP to directly switch to that WLC.

[PJR_CDF · Accepted solution]

This root cause for this issue has now been established.

It would appear the controller in question had previously been setup to use LSC certs and configured to talk to a (now offline) CA.

Previous OS updates of the WLC didnt cause an issue but the jump from 8.0 to 8.3 caused this issue.

It wasnt possible to disable LSC from the GUI (it errored) but once disabled via the CLI, APs were able to connect.

Thanks to all those who provided input into this thread.