I have a pair of 5520 WLCs running as an HA Pair, fresh out of the box (8.1). The network they are on is simple, and devices including APs on the network can ping the WLCs. However, when a primed AP attempts to associate with the WLC, there looks to be a problem with DTLS. The WLCs were purchased with DTLS licences, but on examination, they are just right-to-use certificates, no xml file or anything else.I have attempted to find out about licencing and DTLS, but have not got far.
I've tried APs with the WLC's Evaluation licences enabled, the Permanent licences enabled and with both enabled, and its made no difference.
I have tried new, out-of-the box 3701 APs, and also an elderly 1131 from my own Lab, which I know works OK; it associates fine with a vWLC on evaluation licences, but that made no difference.
The APs were primed, as shown below, but just don't get past the attempt to initiate DTLS:
AP0024.1444.492c>ping 10.0.110.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.110.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
AP0024.1444.492c>
AP0024.1444.492c>show capwap ip config
LWAPP Static IP Configuration
IP Address 10.0.10.60
IP netmask 255.255.255.0
Default Gateway 10.0.10.254
Primary Controller 10.0.110.10
AP0024.1444.492c>
*Mar 4 14:12:36.116: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Mar 4 14:13:36.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.110.10 peer_port: 5246
*Mar 4 14:14:06.001: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2051 Max retransmission count reached!
*Mar 4 14:14:35.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.0.110.10:5246
*Mar 4 14:14:36.000: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Mar 4 14:13:36.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.110.10 peer_port: 5246
The WLC Log implicates DTLS as well...
*spamApTask0: Mar 04 14:16:28.338: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:8629 8c:60:4f:ab:ec:c1: Failed to create DTLS connection for AP 10.0.10.60 (17556).
*spamApTask0: Mar 04 14:16:28.338: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed
*spamApTask0: Mar 04 14:16:28.338: %LOG-3-Q_IND: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:28.338: %SSHPM-3-CERT_TABLE_INVALID: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:20.340: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:8629 8c:60:4f:ab:ec:c1: Failed to create DTLS connection for AP 10.0.10.60 (17556).
*spamApTask0: Mar 04 14:16:20.340: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed
*spamApTask0: Mar 04 14:16:20.340: %LOG-3-Q_IND: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:20.340: %SSHPM-3-CERT_TABLE_INVALID: [PA]sshpmcert.c:884 Accessing certificate table before initialization
*spamApTask0: Mar 04 14:16:16.342: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:8629 8c:60:4f:ab:ec:c1: Failed to create DTLS connection for AP 10.0.10.60 (17556).
*spamApTask0: Mar 04 14:16:16.342: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed
but Google hasn't helped with "[PA]openssl_dtls.c:456 PKI initialization error : Certificate initialization failed"
and the DTLS debug off the WLC continies the theme of DTLS, but I can't get any further than that.
There is just one last hint that something is not right with the controllers: The boot log (attached below) contains the lines:
1) Web Server: CLI: Secure Web: Web Admin Certificate not found (error).
2) Initializing Licensing Storage: failed (22)
3) Starting VPN Services: Unable to load system certificate!!! Contact your Cisco Systems Inc. technical support representativeok
4) Web Server: CLI: Secure Web: Web Admin Certificate not found (error).
I'm not sure if these represent an issue or not....especially when it ALSO says "Starting DTLS server: enabled in CAPWAP"
Any help or suggestions would be great!
Thanks
Jim
.o88b. d888888b .d8888. .o88b. .d88b.
d8P Y8 `88' 88' YP d8P Y8 .8P Y8.
8P 88 `8bo. 8P 88 88
8b 88 `Y8b. 8b 88 88
Y8b d8 .88. db 8D Y8b d8 `8b d8'
`Y88P' Y888888P `8888Y' `Y88P' `Y88P'
Booting Primary Image...
<Stuff Deleted>
Detecting hardware . . . . 3
INIT: version 2.88 booting
Configuring network interfaces... done.
Setting up the kernel dump handler..
INIT: Entering runlevel: 3
sh: 0: unknown operand
Detecting Hardware ...
Loading host drivers..
Loading host NIC drivers..
Starting Hardware Acceleration...
Cryptographic library self-test....Testing SHA1 Short Message 1
Testing SHA256 Short Message 1
Testing SHA1 Short Message 1
SHA1 POST PASSED
passed!
XML config selected
Validating XML configuration
octeon_device_init: found 1 DPs
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.
Cisco AireOS Version 8.1.102.0
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Storage: failed (22)
Initializing Licensing Services: ok
Starting Statistics Service: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services:
License daemon start initialization.....
License daemon running.....
ok
Starting System Services: == simStartTasks
ok
Starting FIPS Features: ok : Not enabled
Starting Fastpath Hardware Acceleration: ok
ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-3.1.0, build 549. Flags-[DUTY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(24)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2040
Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok
Fastpath CPU0.00: 24 Cores are being initialized
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok
Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok
Fastpath CPU0.02: ERROR reading temp sensor
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok
Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok
Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok
Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok
Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok
Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok
Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok
Fastpath CPU0.10: Core 10 Initialization and FIPS self-test: ok
Fastpath CPU0.11: Core 11 Initialization and FIPS self-test: ok
Fastpath CPU0.12: Core 12 Initialization and FIPS self-test: ok
Fastpath CPU0.13: Core 13 Initialization and FIPS self-test: ok
Fastpath CPU0.14: Core 14 Initialization and FIPS self-test: ok
Fastpath CPU0.15: Core 15 Initialization and FIPS self-test: ok
Fastpath CPU0.16: Core 16 Initialization and FIPS self-test: ok
Fastpath CPU0.17: Core 17 Initialization and FIPS self-test: ok
Fastpath CPU0.18: Core 18 Initialization and FIPS self-test: ok
Fastpath CPU0.19: Core 19 Initialization and FIPS self-test: ok
Fastpath CPU0.20: Core 20 Initialization and FIPS self-test: ok
Fastpath CPU0.21: Core 21 Initialization and FIPS self-test: ok
Fastpath CPU0.22: Core 22 Initialization and FIPS self-test: ok
Fastpath CPU0.23: Core 23 Initialization and FIPS self-test: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Certificate Database: ok
Starting VPN Services: Unable to load system certificate!!! Contact your Cisco Systems Inc. technical support representativeok
Starting Management Frame Protection: ok
Starting DNS Services: ok
HBL initialization is successful
Starting Licensing Services: ok
Starting Redundancy: Starting Peer Search Timer of 120 seconds
Initiate Role Negotiation Message to peer
Found the Peer. Starting Role Determination...
ok
Start rmgrPingTaskStarting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting AVC Flex Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Environment Status Monitoring Service: ok
Starting RAID Volume Status Monitoring Service: ok
Starting Mesh Services: ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server: enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Tunnel Services New: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
Web Server: CLI: Secure Web: Web Admin Certificate not found (error).
(Cisco Controller)
Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)
User: Admin
Password:**********
(Cisco Controller) >
