3650 - AP's cannot join 3650 controllers. - Certificate initialization failed

martijn.ffwd · ‎11-23-2016

Hi,

I have a problem with 3 new Cisco CS 3650 24PS –S switches. Whenever I connect (several) Aironet 1142n access points. The access point cannot join the controller.

The 3650 controllers keeps displaying the error message:

*%DTLS-3-PKI_ERROR: 1 wcm: PKI initialization error : Certificate initialization failed

I’ve tried different IOS images, working AP’s from different (working)3560 switches. But I cannot get it working.

IOS XE versions:

3.6.4e
3.65.bE

The only thing i have done differently comparing with the other 3650switches is in first instance i installed the LDPE (W/O DTLS) version. But this image is already replaced with one of the versions above.

Does one of you have a suggestion?

I’ve added some additional trace output below:

A command: “show trace messages group-ap” shows these messages:

Error retrieving ID cert from SSHPM shim cache

0000.0000.0000 Failed to create DTLS connection for 192.168.100.125.24310

sshpmShimGetDevCert: sshpm shim device cert not valid

Thank you

Paul Chapman · ‎11-23-2016

Hi -

That's a known issue. You have 2 choices. 1) Replace the legacy APs, or 2) roll back the clock on the switch to 2014 and disable NTP(not recommended due to security implications).

See bug CSCuq19142.

PSC

martijn.ffwd · ‎11-24-2016

Hi Paul,

Thank you for your quick response. I saw the bug report and already tried to change the time/date to 2011 and 2014: No difference. I don't get the same error messages as described in the bugreport.

i checked the manufacturing date from the access points and it is 2011 ( so not near the 10 years certificate experation date)

I have multiple 3650 with the same IOSxe installation and the same access points have no problem joining these controllers. I even have a old WLC that also has no problemens with the access points.

The only messages the 3650 controller gives are the following:

[01/01/14 12:42:27.624 CET 70b 8963] 0000.0000.0000 Discarding non-ClientHello H andshake OR DTLS encrypted packet from 192.168.100.107:24311)since DTLS session is not established

[01/01/14 12:42:28.241 CET 70c 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:29.241 CET 70d 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:30.241 CET 70e 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:30.740 CET 70f 8963] 1caa.076e.b1b0 Unable to find the First RCB index. Return Value: 2
[01/01/14 12:42:31.241 CET 710 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:32.241 CET 711 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:33.242 CET 712 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:34.242 CET 713 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:35.242 CET 714 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:36.242 CET 715 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:37.242 CET 716 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:38.242 CET 717 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:39.242 CET 718 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:40.242 CET 719 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:40.741 CET 71a 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

[01/01/14 12:42:40.741 CET 71b 8963] Error retrieving ID cert from SSHPM shim cache
[01/01/14 12:42:40.741 CET 71c 8963] 0000.0000.0000 Failed to create DTLS connection for 192.168.100.107.24310

[01/01/14 12:42:41.242 CET 71d 8963] sshpmShimGetDevCert: sshpm shim device cert not valid

The 1142n access point itself gives these messages:

*Jan 1 11:34:57.036: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.100.160:5246
*Jan 1 11:34:57.075: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Jan 1 11:34:57.075: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Jan 1 11:34:57.123: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*Jan 1 11:34:57.123: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*Jan 1 11:34:57.132: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Jan 1 11:34:57.150: %LINK-3-UPDOWN: Interface Dot11Radio1, changed state to up
*Jan 1 11:34:57.163: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up
*Jan 1 11:34:57.175: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Jan 1 11:35:00.150: status of voice_diag_test from WLC is false
*Jan 1 11:35:10.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.100.160 peer_port: 5246
*Jan 1 11:35:10.000: %CAPWAP-5-CHANGED: CAPWAP changed state to
*Jan 1 11:35:39.999: DTLS_CLIENT_ERROR: ../dtls/dtls_connection_db.c:2017 Max retransmission count reached!
*Jan 1 11:35:39.999: %DTLS-3-HANDSHAKE_RETRANSMIT: Max retransmit count for 192.168.100.160 is reached.

Do you have any more ideas?

Paul Chapman · ‎11-24-2016

Hi -

There are some manufacturer installed certificates embedded in the switch. The only other thing I can think of is that the onboard certificate store on the 3650 is corrupt in some way. Try reloading the OS to see if that resolves the issue. Otherwise, I think you're going to be calling TAC.

PSC

martijn.ffwd · ‎11-28-2016

Hi Paul,

Thanks again. Ok, it is strange that i have 3x 3650 switches with the same problem. I've tested a Cisco 2602i AP but i get the same results. I will contact our supplier and create a TAC case. I will post the results if i got any.

martijn.ffwd · ‎01-11-2017

Hi Paul,

We send the switches to our supplier but they are not able to get the switches repaired. The switches are being replaced.

Thank you for your help.