topic Hi Sandeep, in Wireless

New 5520 won't accept APs (DTLS issues?)

Jim Blake — Mon, 05 Jul 2021 11:43:34 GMT

I have a pair of 5520 WLCs running as an HA Pair, fresh out of the box (8.1). The network they are on is simple, and devices including APs on the network can ping the WLCs. However, when a primed AP attempts to associate with the WLC, there looks to be a problem with DTLS. The WLCs were purchased with DTLS licences, but on examination, they are just right-to-use certificates, no xml file or anything else.I have attempted to find out about licencing and DTLS, but have not got far.

I've tried APs with the WLC's Evaluation licences enabled, the Permanent licences enabled and with both enabled, and its made no difference.

I have tried new, out-of-the box 3701 APs, and also an elderly 1131 from my own Lab, which I know works OK; it associates fine with a vWLC on evaluation licences, but that made no difference.

The APs were primed, as shown below, but just don't get past the attempt to initiate DTLS:

AP0024.1444.492c>ping 10.0.110.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.110.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
AP0024.1444.492c>

AP0024.1444.492c>show capwap ip config

LWAPP Static IP Configuration
IP Address 10.0.10.60
IP netmask 255.255.255.0
Default Gateway 10.0.10.254
Primary Controller 10.0.110.10

AP0024.1444.492c>
*Mar 4 14:12:36.116: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Mar 4 14:13:36.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.110.10 peer_port: 5246
*Mar 4 14:14:06.001: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2051 Max retransmission count reached!
*Mar 4 14:14:35.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.0.110.10:5246
*Mar 4 14:14:36.000: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Mar 4 14:13:36.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.110.10 peer_port: 5246

The WLC Log implicates DTLS as well...

*spamApTask0: Mar 04 14:16:28.338: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:8629 8c:60:4f:ab:ec:c1: Failed to create DTLS connection for AP 10.0.10.60 (17556).
*spamApTask0: Mar 04 14:16:28.338: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed
*spamApTask0: Mar 04 14:16:28.338: %LOG-3-Q_IND: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:28.338: %SSHPM-3-CERT_TABLE_INVALID: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:20.340: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:8629 8c:60:4f:ab:ec:c1: Failed to create DTLS connection for AP 10.0.10.60 (17556).
*spamApTask0: Mar 04 14:16:20.340: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed
*spamApTask0: Mar 04 14:16:20.340: %LOG-3-Q_IND: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:20.340: %SSHPM-3-CERT_TABLE_INVALID: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:16.342: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:8629 8c:60:4f:ab:ec:c1: Failed to create DTLS connection for AP 10.0.10.60 (17556).
*spamApTask0: Mar 04 14:16:16.342: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed

but Google hasn't helped with "[PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed"

and the DTLS debug off the WLC continies the theme of DTLS, but I can't get any further than that.

There is just one last hint that something is not right with the controllers: The boot log (attached below) contains the lines:

1) Web Server:    CLI:    Secure Web: Web Admin Certificate not found (error).

2) Initializing Licensing Storage: failed (22)

3) Starting VPN Services: Unable to load system certificate!!! Contact your Cisco Systems Inc. technical support representativeok

4) Web Server:    CLI:    Secure Web: Web Admin Certificate not found (error).

I'm not sure if these represent an issue or not....especially when it ALSO says "Starting DTLS server: enabled in CAPWAP"

Any help or suggestions would be great!

Thanks

Jim

                       .o88b. d888888b .d8888. .o88b. .d88b.
                     d8P Y8   `88'   88' YP d8P Y8 .8P Y8.
                     8P         88    `8bo.   8P      88    88
                     8b         88      `Y8b. 8b      88    88
                     Y8b d8   .88.   db   8D Y8b d8 `8b d8'
                      `Y88P' Y888888P `8888Y' `Y88P' `Y88P'

Booting Primary Image...

<Stuff Deleted>

Detecting hardware . . . . 3

INIT: version 2.88 booting

Configuring network interfaces... done.
Setting up the kernel dump handler..

INIT: Entering runlevel: 3

sh: 0: unknown operand
Detecting Hardware ...
Loading host drivers..
Loading host NIC drivers..
Starting Hardware Acceleration...

Cryptographic library self-test....Testing SHA1 Short Message 1
Testing SHA256 Short Message 1
Testing SHA1 Short Message 1
SHA1 POST PASSED
passed!

XML config selected
Validating XML configuration
octeon_device_init: found 1 DPs
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 8.1.102.0
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Storage: failed (22)
Initializing Licensing Services: ok
Starting Statistics Service: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services:
License daemon start initialization.....

License daemon running.....
ok
Starting System Services: == simStartTasks
ok
Starting FIPS Features: ok : Not enabled
Starting Fastpath Hardware Acceleration: ok
ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-3.1.0, build 549. Flags-[DUTY CYCLE] : ok

Fastpath CPU0.00: Initializing last packet received queue. Num of cores(24)

Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2040

Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok

Fastpath CPU0.00: 24 Cores are being initialized

Fastpath CPU0.00: Initializing Timer...

Fastpath CPU0.00: Initializing Timer...done.

Fastpath CPU0.00: Initializing Timer...

Fastpath CPU0.00: Initializing NBAR AGING Timer...done.

Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok

Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok

Fastpath CPU0.02: ERROR reading temp sensor
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok

Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok

Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok

Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok

Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok

Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok

Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok

Fastpath CPU0.10: Core 10 Initialization and FIPS self-test: ok

Fastpath CPU0.11: Core 11 Initialization and FIPS self-test: ok

Fastpath CPU0.12: Core 12 Initialization and FIPS self-test: ok

Fastpath CPU0.13: Core 13 Initialization and FIPS self-test: ok

Fastpath CPU0.14: Core 14 Initialization and FIPS self-test: ok

Fastpath CPU0.15: Core 15 Initialization and FIPS self-test: ok

Fastpath CPU0.16: Core 16 Initialization and FIPS self-test: ok

Fastpath CPU0.17: Core 17 Initialization and FIPS self-test: ok

Fastpath CPU0.18: Core 18 Initialization and FIPS self-test: ok

Fastpath CPU0.19: Core 19 Initialization and FIPS self-test: ok

Fastpath CPU0.20: Core 20 Initialization and FIPS self-test: ok

Fastpath CPU0.21: Core 21 Initialization and FIPS self-test: ok

Fastpath CPU0.22: Core 22 Initialization and FIPS self-test: ok

Fastpath CPU0.23: Core 23 Initialization and FIPS self-test: ok

Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Certificate Database: ok
Starting VPN Services: Unable to load system certificate!!! Contact your Cisco Systems Inc. technical support representativeok
Starting Management Frame Protection: ok
Starting DNS Services: ok
HBL initialization is successful
Starting Licensing Services: ok
Starting Redundancy: Starting Peer Search Timer of 120 seconds

Initiate Role Negotiation Message to peer

Found the Peer. Starting Role Determination...
ok

Start rmgrPingTaskStarting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting AVC Flex Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Environment Status Monitoring Service: ok
Starting RAID Volume Status Monitoring Service: ok
Starting Mesh Services: ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server: enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Tunnel Services New: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
   Web Server:    CLI:    Secure Web: Web Admin Certificate not found (error).

(Cisco Controller)

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User: Admin

Password:**********

(Cisco Controller) >

Can you paste the output of

Sandeep Choudhary — Fri, 04 Mar 2016 16:09:27 GMT

Can you paste the output of this command from WLC: show auth-list

Regards

Hi Jim

Sandeep Choudhary — Fri, 04 Mar 2016 16:30:11 GMT

Hi Jim

I think this may be a hardware problem and would suggest to open a TAC case.

check this:

https://supportforums.cisco.com/document/102661/troubleshooting-network-access-telnetsshhttphttps-wireless-lan-controller

Workaround: This issue requires hardware replacement for resolution.

Regards

Dont forget to rate helpful posts

Hi Sandeep,

Jim Blake — Sat, 05 Mar 2016 00:02:51 GMT

Hi Sandeep,

I'll do a "show auth list", but I'm not able to till Monday...bear with me till then and thanks for the interest

Jim

Here it is

Jim Blake — Mon, 07 Mar 2016 09:03:09 GMT

Here it is

(Cisco Controller) >show auth-list

Authorize MIC APs against Auth-list or AAA ...... disabled
Authorize LSC APs against Auth-List ............. disabled
APs Allowed to Join
AP with Manufacturing Installed Certificate.... yes
AP with Self-Signed Certificate................ yes
AP with Locally Significant Certificate........ yes

Hi,

Sandeep Choudhary — Mon, 07 Mar 2016 09:25:16 GMT

Hi,

I will try to disable LSC and SSC on WLC and give a try again.

AP with Self-Signed Certificate................ Disable

AP with Locally Significant Certificate........ Disable

How to do it:

Go to Security > AAA > AP Policies , uncheck box for LSC and SSC then save the config on WLC.

*** Don't disable MIC check box.

After save , just reboot WLC.

Regards

Don't forget to rate helpful posts

Hi Sandeep,

Jim Blake — Mon, 07 Mar 2016 10:09:53 GMT

Hi Sandeep,

That was how it was originally. I changed the settings from "Disable" to "Enable" to check that it wasn't something like that which was causing the problem. The issue remains, with either "enable" or "disable" set fo both those configurations

Thanks

Jim

Then you must open a TAC case

Sandeep Choudhary — Mon, 07 Mar 2016 13:09:17 GMT

Then you must open a TAC case, there is no other-way around.

Please keep updating....It may help others.

Regards

Don't forget to rate helpful posts

Well that was fun....NOT! I

Jim Blake — Fri, 18 Mar 2016 12:25:04 GMT

Well that was fun....NOT! I raised a TAC on this, and got given some info:

bug ID: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuv97685/?reffering_site=dumpcr

and a link to a fix

http://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/troubleshooting/trb-guide-wlc-5520-8540.html#pgfId-1310108

Basically, access the CIMC port on the 5520 and enable the “Hypervisor”. Then reboot the controller.

I did all that but it made no difference, and the WLCs are now being RMA'ed as not field fixable

Jim

Re: Hi,

S Jhon — Wed, 07 Feb 2018 05:43:28 GMT

This issue is generally with the timings mismatch between wlc and AP's. change the time setting in WLC and try.