APs won't join vWLC 8.3

[Austin Godbey]

Have a customer with all APs already associated to their main vWLC running 8.3. Created a new vWLC to act as their secondary controller. Statically assigned WLCs in HA tab for all APs. None of the APs will join the controller. APs are 3602i, 3702i, and 3802i. Same code on both vWLCs.

I see the following logs on the WLC:

*spamApTask7: May 16 18:11:37.611: %CAPWAP-3-DECODE_ERR: capwap_ac_sm.c:2732 Error decoding discovery request from AP 00:00:00:00:00:00
*spamApTask7: May 16 18:11:37.611: %CAPWAP-3-INVALID_PAYLOAD3: capwap_ac_decode.c:629 The system detects an invalid vendor type 12846 in WTP descriptor message element

On the AP Join status page the APs keep rising the discovery counter, but never transition to the join phase.

Capwap debugs don't show any errors, just keep repeating the discovery phase. Packet captures show the same.

[Austin Godbey]

(Cisco Controller) >debug capwap packet enable

Warning! Enabling this debug with large number of AP's connected, May flood console with message logs !!!.
Are you sure you want to continue? (y/N)y

(Cisco Controller) >
(Cisco Controller) >*spamApTask7: May 21 14:37:07.957: <<<< Start of CAPWAP Packet >>>>
*spamApTask7: May 21 14:37:07.957: CAPWAP Control mesg Recd from 172.19.38.26, Port 5264
*spamApTask7: May 21 14:37:07.957: HLEN 4, Radio ID 0, WBID 1
*spamApTask7: May 21 14:37:07.957: Msg Type : CAPWAP_PRIMARY_DISCOVERY_REQUEST
*spamApTask7: May 21 14:37:07.957: Msg Length : 187
*spamApTask7: May 21 14:37:07.957: Msg SeqNum : 0
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_DISCOVERY_TYPE, Length 1
*spamApTask7: May 21 14:37:07.957: Discovery Type : CAPWAP_DISCOVERY_TYPE_STATIC_CONFIG
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_WTP_BOARD_DATA, Length 73
*spamApTask7: May 21 14:37:07.957: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:07.957: WTP_SERIAL_NUMBER : AIR-AP3802I-B-K9
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_WTP_DESCRIPTOR, Length 40
*spamApTask7: May 21 14:37:07.957: Maximum Radios Supported : 2
*spamApTask7: May 21 14:37:07.957: Radios in Use : 2
*spamApTask7: May 21 14:37:07.957: Encryption Capabilities : 0x00 0x01
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_WTP_FRAME_TUNNEL, Length 1
*spamApTask7: May 21 14:37:07.957: WTP Frame Tunnel Mode : NATIVE_FRAME_TUNNEL_MODE
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_WTP_MAC_TYPE, Length 1
*spamApTask7: May 21 14:37:07.957: WTP Mac Type : SPLIT_MAC
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:07.957: Radio ID : 0
*spamApTask7: May 21 14:37:07.957: Radio Type : RADIO_80211A
*spamApTask7: May 21 14:37:07.957: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:07.957: Radio ID : 1
*spamApTask7: May 21 14:37:07.957: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*spamApTask7: May 21 14:37:07.957: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:07.957:
IE : UNKNOWN IE 207

*spamApTask7: May 21 14:37:07.957: IE Length : 4

*spamApTask7: May 21 14:37:07.957: Decode routine not available, Printing Hex Dump

*spamApTask7: May 21 14:37:07.957: 00000000: 01 00 00 01 ....

*spamApTask7: May 21 14:37:07.957:
*spamApTask7: May 21 14:37:07.957: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 15
*spamApTask7: May 21 14:37:07.957: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:07.957:
IE : RAD_NAME_PAYLOAD

*spamApTask7: May 21 14:37:07.957: IE Length : 9

*spamApTask7: May 21 14:37:07.957: Rad Name :
*spamApTask7: May 21 14:37:07.957: AP-BFP-31

*spamApTask7: May 21 14:37:07.957: <<<< End of CAPWAP Packet >>>>

*spamApTask7: May 21 14:37:13.659: <<<< Start of CAPWAP Packet >>>>
*spamApTask7: May 21 14:37:13.659: CAPWAP Control mesg Recd from 172.19.38.26, Port 5264
*spamApTask7: May 21 14:37:13.659: HLEN 4, Radio ID 0, WBID 1
*spamApTask7: May 21 14:37:13.659: Msg Type : CAPWAP_DISCOVERY_REQUEST
*spamApTask7: May 21 14:37:13.659: Msg Length : 187
*spamApTask7: May 21 14:37:13.659: Msg SeqNum : 0
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_DISCOVERY_TYPE, Length 1
*spamApTask7: May 21 14:37:13.659: Discovery Type : CAPWAP_DISCOVERY_TYPE_STATIC_CONFIG
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_WTP_BOARD_DATA, Length 73
*spamApTask7: May 21 14:37:13.659: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.659: WTP_SERIAL_NUMBER : AIR-AP3802I-B-K9
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_WTP_DESCRIPTOR, Length 40
*spamApTask7: May 21 14:37:13.659: Maximum Radios Supported : 2
*spamApTask7: May 21 14:37:13.659: Radios in Use : 2
*spamApTask7: May 21 14:37:13.659: Encryption Capabilities : 0x00 0x01
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_WTP_FRAME_TUNNEL, Length 1
*spamApTask7: May 21 14:37:13.659: WTP Frame Tunnel Mode : NATIVE_FRAME_TUNNEL_MODE
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_WTP_MAC_TYPE, Length 1
*spamApTask7: May 21 14:37:13.659: WTP Mac Type : SPLIT_MAC
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:13.659: Radio ID : 0
*spamApTask7: May 21 14:37:13.659: Radio Type : RADIO_80211A
*spamApTask7: May 21 14:37:13.659: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:13.659: Radio ID : 1
*spamApTask7: May 21 14:37:13.659: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*spamApTask7: May 21 14:37:13.659: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.659:
IE : UNKNOWN IE 207

*spamApTask7: May 21 14:37:13.659: IE Length : 4

*spamApTask7: May 21 14:37:13.659: Decode routine not available, Printing Hex Dump

*spamApTask7: May 21 14:37:13.659: 00000000: 01 00 00 01 ....

*spamApTask7: May 21 14:37:13.659:
*spamApTask7: May 21 14:37:13.659: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 15
*spamApTask7: May 21 14:37:13.659: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.659:
IE : RAD_NAME_PAYLOAD

*spamApTask7: May 21 14:37:13.659: IE Length : 9

*spamApTask7: May 21 14:37:13.659: Rad Name :
*spamApTask7: May 21 14:37:13.659: AP-BFP-31

*spamApTask7: May 21 14:37:13.659: <<<< End of CAPWAP Packet >>>>

*spamApTask7: May 21 14:37:13.660: <<<< Start of CAPWAP Packet >>>>
*spamApTask7: May 21 14:37:13.660: CAPWAP Control mesg Recd from 172.19.38.26, Port 5264
*spamApTask7: May 21 14:37:13.660: HLEN 4, Radio ID 0, WBID 1
*spamApTask7: May 21 14:37:13.660: Msg Type : CAPWAP_DISCOVERY_REQUEST
*spamApTask7: May 21 14:37:13.660: Msg Length : 187
*spamApTask7: May 21 14:37:13.660: Msg SeqNum : 0
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_DISCOVERY_TYPE, Length 1
*spamApTask7: May 21 14:37:13.660: Discovery Type : CAPWAP_DISCOVERY_TYPE_STATIC_CONFIG
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_WTP_BOARD_DATA, Length 73
*spamApTask7: May 21 14:37:13.660: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.660: WTP_SERIAL_NUMBER : AIR-AP3802I-B-K9
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_WTP_DESCRIPTOR, Length 40
*spamApTask7: May 21 14:37:13.660: Maximum Radios Supported : 2
*spamApTask7: May 21 14:37:13.660: Radios in Use : 2
*spamApTask7: May 21 14:37:13.660: Encryption Capabilities : 0x00 0x01
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_WTP_FRAME_TUNNEL, Length 1
*spamApTask7: May 21 14:37:13.660: WTP Frame Tunnel Mode : NATIVE_FRAME_TUNNEL_MODE
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_WTP_MAC_TYPE, Length 1
*spamApTask7: May 21 14:37:13.660: WTP Mac Type : SPLIT_MAC
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:13.660: Radio ID : 0
*spamApTask7: May 21 14:37:13.660: Radio Type : RADIO_80211A
*spamApTask7: May 21 14:37:13.660: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:13.660: Radio ID : 1
*spamApTask7: May 21 14:37:13.660: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*spamApTask7: May 21 14:37:13.660: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.660:
IE : UNKNOWN IE 207

*spamApTask7: May 21 14:37:13.660: IE Length : 4

*spamApTask7: May 21 14:37:13.660: Decode routine not available, Printing Hex Dump

*spamApTask7: May 21 14:37:13.660: 00000000: 01 00 00 01 ....

*spamApTask7: May 21 14:37:13.660:
*spamApTask7: May 21 14:37:13.660: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 15
*spamApTask7: May 21 14:37:13.660: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.660:
IE : RAD_NAME_PAYLOAD

*spamApTask7: May 21 14:37:13.660: IE Length : 9

*spamApTask7: May 21 14:37:13.660: Rad Name :
*spamApTask7: May 21 14:37:13.660: AP-BFP-31

*spamApTask7: May 21 14:37:13.660: <<<< End of CAPWAP Packet >>>>

*spamApTask7: May 21 14:37:13.666: <<<< Start of CAPWAP Packet >>>>
*spamApTask7: May 21 14:37:13.666: CAPWAP Control mesg Recd from 172.19.38.26, Port 5264
*spamApTask7: May 21 14:37:13.666: HLEN 4, Radio ID 0, WBID 1
*spamApTask7: May 21 14:37:13.666: Msg Type : CAPWAP_DISCOVERY_REQUEST
*spamApTask7: May 21 14:37:13.666: Msg Length : 187
*spamApTask7: May 21 14:37:13.666: Msg SeqNum : 0
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_DISCOVERY_TYPE, Length 1
*spamApTask7: May 21 14:37:13.666: Discovery Type : CAPWAP_DISCOVERY_TYPE_STATIC_CONFIG
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_WTP_BOARD_DATA, Length 73
*spamApTask7: May 21 14:37:13.666: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.666: WTP_SERIAL_NUMBER : AIR-AP3802I-B-K9
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_WTP_DESCRIPTOR, Length 40
*spamApTask7: May 21 14:37:13.666: Maximum Radios Supported : 2
*spamApTask7: May 21 14:37:13.666: Radios in Use : 2
*spamApTask7: May 21 14:37:13.666: Encryption Capabilities : 0x00 0x01
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_WTP_FRAME_TUNNEL, Length 1
*spamApTask7: May 21 14:37:13.666: WTP Frame Tunnel Mode : NATIVE_FRAME_TUNNEL_MODE
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_WTP_MAC_TYPE, Length 1
*spamApTask7: May 21 14:37:13.666: WTP Mac Type : SPLIT_MAC
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:13.666: Radio ID : 0
*spamApTask7: May 21 14:37:13.666: Radio Type : RADIO_80211A
*spamApTask7: May 21 14:37:13.666: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION, Length 5
*spamApTask7: May 21 14:37:13.666: Radio ID : 1
*spamApTask7: May 21 14:37:13.666: Radio Type : RADIO_80211B
*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*spamApTask7: May 21 14:37:13.666: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.666:
IE : UNKNOWN IE 207

*spamApTask7: May 21 14:37:13.666: IE Length : 4

*spamApTask7: May 21 14:37:13.666: Decode routine not available, Printing Hex Dump

*spamApTask7: May 21 14:37:13.666: 00000000: 01 00 00 01 ....

*spamApTask7: May 21 14:37:13.666:
*spamApTask7: May 21 14:37:13.666: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 15
*spamApTask7: May 21 14:37:13.666: Vendor Identifier : 0x00409600
*spamApTask7: May 21 14:37:13.666:
IE : RAD_NAME_PAYLOAD

*spamApTask7: May 21 14:37:13.666: IE Length : 9

*spamApTask7: May 21 14:37:13.666: Rad Name :
*spamApTask7: May 21 14:37:13.666: AP-BFP-31

*spamApTask7: May 21 14:37:13.666: <<<< End of CAPWAP Packet >>>>

[Austin Godbey]

(Cisco Controller) >debug capwap events enable

(Cisco Controller) >*spamApTask7: May 21 14:30:17.438: sshpmFreePublicKeyHandle: freeing public key

*spamApTask7: May 21 14:34:44.466: 6c:8b:d3:ee:a0:60 Primary Discovery Request from 172.19.38.26:5264

*spamApTask7: May 21 14:34:44.466: 6c:8b:d3:ee:a0:60 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 200, MaxLicense=93 joined Aps =0
*spamApTask7: May 21 14:34:44.466: 6c:8b:d3:ee:a0:60 apType = 52 apModel: AIR-AP3802I-B-K9

*spamApTask7: May 21 14:34:44.466: 6c:8b:d3:ee:a0:60 apType: Ox34 bundleApImageVer: 8.3.150.0
*spamApTask7: May 21 14:34:44.466: 6c:8b:d3:ee:a0:60 version:8 release:3 maint:150 build:0
*spamApTask7: May 21 14:34:44.466: 6c:8b:d3:ee:a0:60 Primary Discovery Response sent to 172.19.38.26:5264

*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 Discovery Request from 172.19.38.26:5264

*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 200, MaxLicense=93 joined Aps =0
*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 apType = 52 apModel: AIR-AP3802I-B-K9

*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 apType: Ox34 bundleApImageVer: 8.3.150.0
*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 version:8 release:3 maint:150 build:0
*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 Discovery Response sent to 172.19.38.26 port 5264

*spamApTask7: May 21 14:34:50.165: 6c:8b:d3:ee:a0:60 Discovery Response sent to 172.19.38.26:5264

*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 Discovery Request from 172.19.38.26:5264

*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 200, MaxLicense=93 joined Aps =0
*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 apType = 52 apModel: AIR-AP3802I-B-K9

*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 apType: Ox34 bundleApImageVer: 8.3.150.0
*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 version:8 release:3 maint:150 build:0
*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 Discovery Response sent to 172.19.38.26 port 5264

*spamApTask7: May 21 14:34:50.166: 6c:8b:d3:ee:a0:60 Discovery Response sent to 172.19.38.26:5264

*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 Discovery Request from 172.19.38.26:5264

*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 200, MaxLicense=93 joined Aps =0
*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 apType = 52 apModel: AIR-AP3802I-B-K9

*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 apType: Ox34 bundleApImageVer: 8.3.150.0
*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 version:8 release:3 maint:150 build:0
*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 Discovery Response sent to 172.19.38.26 port 5264

*spamApTask7: May 21 14:34:50.170: 6c:8b:d3:ee:a0:60 Discovery Response sent to 172.19.38.26:5264

*spamApTask7: May 21 14:34:59.666: 00:08:e3:ff:fd:90 DTLS connection not found, creating new connection for 172.19.38.26 (5264) 10.2.0.98 (5246)

*spamApTask7: May 21 14:34:59.698: 00:08:e3:ff:fd:90 DTLS Session established server (10.2.0.98:5246), client (172.19.38.26:5264)
*spamApTask7: May 21 14:34:59.699: 00:08:e3:ff:fd:90 Starting wait join timer for AP: 172.19.38.26:5264

*spamApTask7: May 21 14:34:59.709: 00:08:e3:ff:fd:90 DTLS connection closed event receivedserver (10.2.0.98/5246) client (172.19.38.26/5264)
*spamApTask7: May 21 14:34:59.710: 00:08:e3:ff:fd:90 No entry exists for AP (172.19.38.26/5264)
*spamApTask7: May 21 14:34:59.710: 00:08:e3:ff:fd:90 Deleting AP entry 172.19.38.26:5264 from temporary database.
*spamApTask7: May 21 14:35:04.421: 00:08:e3:ff:fd:90 DTLS connection not found, creating new connection for 172.19.38.26 (5264) 10.2.0.98 (5246)

*spamApTask7: May 21 14:35:04.453: 00:08:e3:ff:fd:90 DTLS Session established server (10.2.0.98:5246), client (172.19.38.26:5264)
*spamApTask7: May 21 14:35:04.453: 00:08:e3:ff:fd:90 Starting wait join timer for AP: 172.19.38.26:5264

*spamApTask7: May 21 14:35:04.464: 00:08:e3:ff:fd:90 DTLS connection closed event receivedserver (10.2.0.98/5246) client (172.19.38.26/5264)
*spamApTask7: May 21 14:35:04.464: 00:08:e3:ff:fd:90 No entry exists for AP (172.19.38.26/5264)
*spamApTask7: May 21 14:35:04.464: 00:08:e3:ff:fd:90 Deleting AP entry 172.19.38.26:5264 from temporary database.
*spamApTask7: May 21 14:35:09.169: 00:08:e3:ff:fd:90 DTLS connection not found, creating new connection for 172.19.38.26 (5264) 10.2.0.98 (5246)

*spamApTask7: May 21 14:35:09.204: 00:08:e3:ff:fd:90 DTLS Session established server (10.2.0.98:5246), client (172.19.38.26:5264)
*spamApTask7: May 21 14:35:09.204: 00:08:e3:ff:fd:90 Starting wait join timer for AP: 172.19.38.26:5264

*spamApTask7: May 21 14:35:09.215: 00:08:e3:ff:fd:90 DTLS connection closed event receivedserver (10.2.0.98/5246) client (172.19.38.26/5264)
*spamApTask7: May 21 14:35:09.215: 00:08:e3:ff:fd:90 No entry exists for AP (172.19.38.26/5264)
*spamApTask7: May 21 14:35:09.215: 00:08:e3:ff:fd:90 Deleting AP entry 172.19.38.26:5264 from temporary database.

(Cisco Controller) >

[Austin Godbey]

(Cisco Controller) >debug pm pki enable

(Cisco Controller) >*spamApTask7: May 21 14:42:10.147: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask7: May 21 14:42:10.147: sshpmGetCID: failed to find matching cert name cscoSha2IdCert

*spamApTask7: May 21 14:42:10.147: GetIDCert: Using SHA2 Id cert on WLC

*spamApTask7: May 21 14:42:10.147: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: May 21 14:42:10.147: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: May 21 14:42:10.147: Get Cert from CID: For CID 148fdb8c certType 1
*spamApTask7: May 21 14:42:10.147: Get Cert from CID: Found match of ID Cert in row 2
*spamApTask7: May 21 14:42:10.147: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask7: May 21 14:42:10.147: sshpmGetCID: failed to find matching cert name cscoSha2IdCert

*spamApTask7: May 21 14:42:10.147: GetDERIDKey: Using SHA2 Id cert Private Keys on WLC

*spamApTask7: May 21 14:42:10.147: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: May 21 14:42:10.147: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: May 21 14:42:10.147: GetPrivateKey: called to get key for CID 148fdb8c

*spamApTask7: May 21 14:42:10.147: Private Key found row 2 KeyBufLen 2048 Keylen 1190 PrivateKeyPtr 0x7f31ad87a9b0

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: locking ca cert table

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: x509 subject_name /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AP3G3-6C8BD313E052/emailAddress=support@cisco.com

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: issuer_name /O=Cisco/CN=Cisco Manufacturing CA SHA2

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: CN AP3G3-6C8BD313E052

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: issuerCertCN Cisco Manufacturing CA SHA2

*spamApTask7: May 21 14:42:10.157: GetMac: MAC: 6c8b.d313.e052

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: openssl Mac Address in subject is 6c:8b:d3:13:e0:52

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: Cert Name in subject is AP3G3-6C8BD313E052

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: Extracted cert issuer from subject name.

*spamApTask7: May 21 14:42:10.157: NMSP:: Algo name matched SHA256

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: Cert is issued by Cisco Systems.

*spamApTask7: May 21 14:42:10.157: Retrieving x509 cert for CertName cscoMfgSha2CaCert

*spamApTask7: May 21 14:42:10.157: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>

*spamApTask7: May 21 14:42:10.157: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask7: May 21 14:42:10.157: Found CID 227056ab for certname cscoMfgSha2CaCert

*spamApTask7: May 21 14:42:10.157: CACertTable: Found matching CID cscoMfgSha2CaCert in row 7 x509 0x7f31b05cb218
*spamApTask7: May 21 14:42:10.157: Retrieving x509 cert for CertName cscoRootSha2CaCert

*spamApTask7: May 21 14:42:10.157: sshpmGetCID: called to evaluate <cscoRootSha2CaCert>

*spamApTask7: May 21 14:42:10.157: sshpmGetCID: Found matching CA cert cscoRootSha2CaCert in row 6
*spamApTask7: May 21 14:42:10.157: Found CID 2883937e for certname cscoRootSha2CaCert

*spamApTask7: May 21 14:42:10.157: CACertTable: Found matching CID cscoRootSha2CaCert in row 6 x509 0x7f31b05cb4e8
*spamApTask7: May 21 14:42:10.157: Verify User Certificate: X509 Cert Verification return code: 1
*spamApTask7: May 21 14:42:10.157: Verify User Certificate: X509 Cert Verification result text: ok
*spamApTask7: May 21 14:42:10.157: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>

*spamApTask7: May 21 14:42:10.157: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask7: May 21 14:42:10.157: Verify User Certificate: OPENSSL X509_Verify: AP Cert Verfied Using >cscoMfgSha2CaCert<

*spamApTask7: May 21 14:42:10.157: OpenSSL Get Issuer Handles: Check cert validity times (allow expired YES)
*spamApTask7: May 21 14:42:10.157: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: May 21 14:42:10.157: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: May 21 14:42:10.157: sshpmFreePublicKeyHandle: called with 0x7f31ad64d368

*spamApTask7: May 21 14:42:10.157: sshpmFreePublicKeyHandle: freeing public key

*spamApTask7: May 21 14:42:14.898: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask7: May 21 14:42:14.898: sshpmGetCID: failed to find matching cert name cscoSha2IdCert

*spamApTask7: May 21 14:42:14.898: GetIDCert: Using SHA2 Id cert on WLC

*spamApTask7: May 21 14:42:14.899: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: May 21 14:42:14.899: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: May 21 14:42:14.899: Get Cert from CID: For CID 148fdb8c certType 1
*spamApTask7: May 21 14:42:14.899: Get Cert from CID: Found match of ID Cert in row 2
*spamApTask7: May 21 14:42:14.899: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask7: May 21 14:42:14.899: sshpmGetCID: failed to find matching cert name cscoSha2IdCert

*spamApTask7: May 21 14:42:14.899: GetDERIDKey: Using SHA2 Id cert Private Keys on WLC

*spamApTask7: May 21 14:42:14.899: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: May 21 14:42:14.899: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: May 21 14:42:14.899: GetPrivateKey: called to get key for CID 148fdb8c

*spamApTask7: May 21 14:42:14.899: Private Key found row 2 KeyBufLen 2048 Keylen 1190 PrivateKeyPtr 0x7f31ad87a9b0

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: locking ca cert table

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: x509 subject_name /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AP3G3-6C8BD313E052/emailAddress=support@cisco.com

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: issuer_name /O=Cisco/CN=Cisco Manufacturing CA SHA2

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: CN AP3G3-6C8BD313E052

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: issuerCertCN Cisco Manufacturing CA SHA2

*spamApTask7: May 21 14:42:14.920: GetMac: MAC: 6c8b.d313.e052

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: openssl Mac Address in subject is 6c:8b:d3:13:e0:52

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: Cert Name in subject is AP3G3-6C8BD313E052

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: Extracted cert issuer from subject name.

*spamApTask7: May 21 14:42:14.920: NMSP:: Algo name matched SHA256

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: Cert is issued by Cisco Systems.

*spamApTask7: May 21 14:42:14.920: Retrieving x509 cert for CertName cscoMfgSha2CaCert

*spamApTask7: May 21 14:42:14.920: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>

*spamApTask7: May 21 14:42:14.920: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask7: May 21 14:42:14.920: Found CID 227056ab for certname cscoMfgSha2CaCert

*spamApTask7: May 21 14:42:14.920: CACertTable: Found matching CID cscoMfgSha2CaCert in row 7 x509 0x7f31b05cb218
*spamApTask7: May 21 14:42:14.920: Retrieving x509 cert for CertName cscoRootSha2CaCert

*spamApTask7: May 21 14:42:14.920: sshpmGetCID: called to evaluate <cscoRootSha2CaCert>

*spamApTask7: May 21 14:42:14.920: sshpmGetCID: Found matching CA cert cscoRootSha2CaCert in row 6
*spamApTask7: May 21 14:42:14.920: Found CID 2883937e for certname cscoRootSha2CaCert

*spamApTask7: May 21 14:42:14.920: CACertTable: Found matching CID cscoRootSha2CaCert in row 6 x509 0x7f31b05cb4e8
*spamApTask7: May 21 14:42:14.920: Verify User Certificate: X509 Cert Verification return code: 1
*spamApTask7: May 21 14:42:14.920: Verify User Certificate: X509 Cert Verification result text: ok
*spamApTask7: May 21 14:42:14.920: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>

*spamApTask7: May 21 14:42:14.920: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask7: May 21 14:42:14.920: Verify User Certificate: OPENSSL X509_Verify: AP Cert Verfied Using >cscoMfgSha2CaCert<

*spamApTask7: May 21 14:42:14.920: OpenSSL Get Issuer Handles: Check cert validity times (allow expired YES)
*spamApTask7: May 21 14:42:14.920: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask7: May 21 14:42:14.920: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask7: May 21 14:42:14.921: sshpmFreePublicKeyHandle: called with 0x7f31ad64d800

*spamApTask7: May 21 14:42:14.921: sshpmFreePublicKeyHandle: freeing public key

(Cisco Controller) >debug pm pki disable

[kai schoene]

Hi

I have the same issue. Did you figure it out?

[Austin Godbey]

Yes. We rebuilt the new secondary controller and got the AP to join, but with other errors. Found the bug 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva69352 that was specific to two virtual WLCs used this way.

Under "Controller > Mobility Management > Mobility Groups" we added each of the controllers to each other and used their Hash Key. Once they were added the APs would be able to join either and fail back to either.

Good luck!

[miguelmagr]

After restarting one of the controllers. I have some issues when an AP gets restarted and then tries to synchronize with the WLC, but it cannot be done.

[Rdph8214]

Hello 

We have a similar issue

We have factory reset the AP with the command. capwap ap erase all

And we configured it to connect to only one WLCs and it works, and when we do the HA and force it to go to the other WLCs it works. But when we want to return the AP to the initial WLCs it does not manage to join.

Open a ticket here but it is not approved, I don't understand why.