topic Re: APs not joining a new WLC in Wireless

APs not joining a new WLC

davidrodriguez — Wed, 07 Jun 2023 00:00:39 GMT

Hi guys!
We need help with two WLCs, one with a free license and another with 27 APs licenses.
Currently, we are trying to change these WLCs one for another so we can use the WLC Licensed.
When we disconnect the current WLC and connect the new one, the APs are not joining. On the logs, the only thing we can see is this:

CAPWAP LOG ERROR:

*spamApTask3: Jun 06 17:45:36.919: DTLS connection was closed

CAPWAP LOGS EVENTS:

*spamApTask2: Jun 06 17:44:58.092: DTLS Session established server (1.1.1.1:5246), client (1.1.1.2:5264)
*spamApTask2: Jun 06 17:44:58.092: Starting wait join timer for AP: 1.1.1.2:5264

*spamApTask2: Jun 06 17:44:58.103: acDtlsPlumbControlPlaneKeys: lrad:1.1.1.2(5264) mwar:1.1.1.1(5246)

*spamApTask2: Jun 06 17:44:58.103: DTLS keys for Control Plane deleted successfully for AP 1.1.1.2

*spamApTask2: Jun 06 17:44:58.104: DTLS connection closed event receivedserver (1.1.1.1/5246) client (1.1.1.2/5264)
*spamApTask2: Jun 06 17:44:58.104: No entry exists for AP (1.1.1.2/5264)
*spamApTask2: Jun 06 17:44:58.105: Deleting AP entry 1.1.1.2:5264 from temporary database.

Both WLCs are Cisco 5508 and are on version: 8.3.150.0. The APs are 3802, on version 8.3.150.0.
What we tried:
We tried to configure manually the new WLC IPs on the APs under Wireless/All APs/[AP]/High Availability.
We checked the Country, and it's set to US-Canada.
We checked the date/time and Time zone.

Any help is much appreciated!

Re: APs not joining a new WLC

davidrodriguez — Wed, 07 Jun 2023 00:10:22 GMT

These are the logs from the AP:

[*06/06/2023 22:48:08.0171] CAPWAP State: DTLS Teardown
[*06/06/2023 22:48:12.7751] Discovery Response from 1.1.1.1
[*06/06/2023 22:48:28.9232] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C
[*06/06/2023 22:48:22.0000]
[*06/06/2023 22:48:22.0000] CAPWAP State: DTLS Setup
[*06/06/2023 22:48:22.0004] dtls_connectionDB_add_connection: Number of DTLS con
[*06/06/2023 22:49:05.5706] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C
[*06/06/2023 22:49:05.5706] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C
[*06/06/2023 22:49:17.0682] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C
[*06/06/2023 22:49:18.3720] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C
[*06/06/2023 22:49:18.3721] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in C
[*06/06/2023 22:49:19.0166] dtls_disconnect: ERROR shutting down dtls connection

Re: APs not joining a new WLC

Flavio Miranda — Wed, 07 Jun 2023 00:21:01 GMT

Take one AP, factory reset it and try. Looking to what you said, all seems to be correct. It should join.

Re: APs not joining a new WLC

Leo Laohoo — Wed, 07 Jun 2023 04:00:05 GMT

On the AP, post the complete output to the command "sh version".

Re: APs not joining a new WLC

davidrodriguez — Wed, 07 Jun 2023 20:39:12 GMT

Restricted Rights Legend

Use, duplication, or disclosure by the Government is subject to
restrictions as set forth in subparagraph (c) of the Commercial
Computer Software - Restricted Rights clause at FAR sec. 52.227-19 and
subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

This product contains some software licensed under the
"GNU General Public License, version 2" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

This product contains some software licensed under the
"GNU Library General Public License, version 2" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Library
General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.0.html

This product contains some software licensed under the
"GNU Lesser General Public License, version 2.1" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Lesser
General Public License, version 2.1", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html

Cisco AP Software, (ap3g3), [cheetah-build3:/san/BUILD/workspace/83MR5_Cheetah_CCO/label/barbados]
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu Apr 11 13:36:09 PDT 2019

ROM: Bootstrap program is U-Boot boot loader
BOOTLDR: U-Boot boot loader Version 2013.01-gf899470 (Aug 16 2018 - 15:50:45)

[AP_NAME] uptime is 0 days, 22 hours, 12 minutes
Last reload time : Tue Jun 6 22:24:34 UTC 2023
Last reload reason : Admin Reload

cisco AIR-AP3802I-A-K9 ARMv7 Processor rev 1 (v7l) with 1030640/713764K bytes of memory.
Processor board ID FJC26061FTL
AP Running Image : 8.3.150.0
Primary Boot Image : 8.3.150.0
Backup Boot Image : 8.7.106.0
1 Multigigabit Ethernet interfaces
1 Gigabit Ethernet interfaces
2 802.11 Radios
Radio Driver version : 9.0.5.5-W8964
Radio FW version : 9.1.8.1
NSS FW version :

Base ethernet MAC Address : B8:11:4B:7B:C5:98
Part Number : 0-000000-00
PCA Assembly Number : 800-105809-01
PCA Revision Number : B0
PCB Serial Number : FOC255025H7
Top Assembly Part Number : 800-105809-01
Top Assembly Serial Number : FJC26061FTL
Top Revision Number : C0
Product/Model Number : AIR-AP3802I-A-K9

@Leo Laohoohere you can see the full sh version from one of the APs.

Re: APs not joining a new WLC

Dustin Anderson — Wed, 07 Jun 2023 21:17:24 GMT

I know you said you checked the date/time/timezone, but is is correct on the old WLC since the APs would have that date/time?

Re: APs not joining a new WLC

davidrodriguez — Wed, 07 Jun 2023 21:28:58 GMT

I tried your suggestion but no luck. I factory reset one AP, and it didn't join the new WLC

Re: APs not joining a new WLC

davidrodriguez — Wed, 07 Jun 2023 21:42:09 GMT

Yes, both have the same time. Seconds of difference, but the same date/time/timezone.

Re: APs not joining a new WLC

Leo Laohoo — Wed, 07 Jun 2023 23:51:20 GMT

Factory-reset the AP using the command "capwap ap erase all".

Re: APs not joining a new WLC

davidrodriguez — Thu, 08 Jun 2023 00:43:56 GMT

I did that on one of the APs, but it didn't join the new WLC.
On the new WLC logs, I didn't see the AP's Mac address in the errors anymore, but it was not joining the WLC.
I was planning on clearing the config on the new WLC, disconnecting the old WLC while configuring the new one, and trying to see if that solves the issue.

Re: APs not joining a new WLC

Leo Laohoo — Thu, 08 Jun 2023 01:43:11 GMT

Please share the output to the command "sh capwap client rcb".

Re: APs not joining a new WLC

Rich R — Thu, 08 Jun 2023 14:02:46 GMT

> "one with a free license"
Is that the new one which the APs can't join?
What exactly do you mean by "free license"?
What do "show sysyinfo" and "show license summary", "show license all" show?

Re: APs not joining a new WLC

davidrodriguez — Thu, 08 Jun 2023 15:01:11 GMT

The one with free license is the WLC we want to remove. It has the evaluation license that longs 3 weeks.

The other WLC we are trying to set up has license for 27 APs.

Re: APs not joining a new WLC

davidrodriguez — Thu, 08 Jun 2023 15:18:09 GMT

See the output here:

AP#sh capwap client rcb
AdminState : ADMIN_ENABLED
OperationState : UP
Name : AP5
SwVer : 8.3.150.0
HwVer : 1.0.0.0
MwarApMgrIp : 1.1.2.1 (This IP is the OLD WLC we want to remove)
MwarName : Cisco_c0:6f:c4
MwarHwVer : 0.0.0.0
Location : default location
ApMode : FlexConnect
ApSubMode : Not Configured
CAPWAP Path MTU : 1485
CAPWAP UDP-Lite : Enabled
IP Prefer-mode : IPv4
AP Link DTLS Encryption : OFF
AP Tcp MSS Adjust : Disabled
LinkAuditing : disabled
Efficient Upgrade State : Disabled
Flex Group Name : default-flex-group

Re: APs not joining a new WLC

Rich R — Thu, 08 Jun 2023 20:03:47 GMT

Ok can you run those commands on the new one then?

Re: APs not joining a new WLC

davidrodriguez — Fri, 16 Jun 2023 15:30:02 GMT

Sorry for the late response, i was out. Here are the outputs on the other WLC (the one with the license)

(Cisco Controller) >show sysinfo Manufacturer's Name.............................. Cisco Systems Inc. Product Name..................................... Cisco Controller Product Version.................................. 8.3.150.0 Bootloader Version............................... 1.0.1 Field Recovery Image Version..................... 6.0.182.0 Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27 OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014 Build Type....................................... DATA + WPS System Name...................................... WLC01 System Location.................................. System Contact................................... System ObjectID.................................. 1.3.6.1.4.1.9.1.1069 Redundancy Mode.................................. Disabled IP Address....................................... 1.1.1.2 IPv6 Address..................................... :: Last Reset....................................... Power on reset System Up Time................................... 3 days 1 hrs 33 mins 13 secs System Timezone Location......................... (GMT -5:00) Eastern Time (US and Canada) System Stats Realtime Interval................... 5 System Stats Normal Interval..................... 180 Configured Country............................... Multiple Countries : CA,US Operating Environment............................ Commercial (0 to 40 C) Internal Temp Alarm Limits....................... 0 to 65 C Internal Temperature............................. +32 C External Temperature............................. +22 C Fan Status....................................... OK State of 802.11b Network......................... Enabled State of 802.11a Network......................... Enabled Number of WLANs.................................. 2 Number of Active Clients......................... 0 OUI Classification Failure Count................. 0 Burned-in MAC Address............................ 44:2B:03:B4:C0:20 Power Supply 1................................... Present, OK Power Supply 2................................... Absent Maximum number of APs supported.................. 500 System Nas-Id.................................... WLC MIC Certificate Types........................ SHA1

(Cisco Controller) >show license all License Store: Primary License Storage StoreIndex: 0 Feature: base Version: 1.0 License Type: Permanent License State: Active, Not in Use License Count: Non-Counted License Priority: Medium License Store: Primary License Storage StoreIndex: 0 Feature: base-ap-count Version: 1.0 License Type: Permanent License State: Inactive License Count: 12 / 0 (Active/In-use) License Priority: Medium License Store: Primary License Storage StoreIndex: 0 Feature: base-ap-count Version: 1.0 License Type: Permanent License State: Inactive License Count: 17 / 0 (Active/In-use) License Priority: Medium License Store: Primary License Storage StoreIndex: 0 Feature: base-ap-count Version: 1.0 License Type: Permanent License State: Inactive License Count: 22 / 0 (Active/In-use) License Priority: Medium License Store: Primary License Storage StoreIndex: 0 Feature: base-ap-count Version: 1.0 License Type: Permanent License State: Active, In Use License Count: 27 / 27 (Active/In-use) License Priority: Medium License Store: Evaluation License Storage StoreIndex: 1 Feature: base-ap-count Version: 1.0 License Type: Evaluation License State: Inactive Evaluation total period: 8 weeks 4 days Evaluation period left: 8 weeks 4 days License Count: 500 / 0 (Active/In-use) License Priority: None

Thanks for all the help!

Re: APs not joining a new WLC

Rich R — Fri, 16 Jun 2023 15:56:42 GMT

Hmmmm! One last idea (although I don't think that's it): Do you have "config ap cert-expiry-ignore mic enable" configured on both of the WLCs? APs must pick up the config from the WLC they've joined before trying to join the other one. (FN-63942)

Otherwise you need to attach a complete console log from an AP trying to join (attach as text file), from power-on, and debugs from the WLC at the same time. Packet capture of the CAPWAP control packets (UDP 5246) at the same time would also be good to see.

Re: APs not joining a new WLC

davidrodriguez — Fri, 16 Jun 2023 16:18:28 GMT

Do you know which cert I should be looking for?

I see from this post: https://www.wiresandwi.fi/blog/cisco-wlc-or-ap-device-certificate-expired-what-you-can-do

That I should check the Certificate Name: Cisco SHA1 device cert; if that is so, then I could see that the certificate is expired, and you could be in the right direction:

Certificate Name: Cisco SHA1 device cert Subject Name : C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT5508-K9-442b03b4c020, emailAddress=support@cisco.com Issuer Name : O=Cisco Systems, CN=Cisco Manufacturing CA Serial Number (Hex): 6BC13D7800000011CF23 Validity : Start : May 4 02:13:03 2012 GMT End : May 4 02:23:03 2022 GMT

Then I'll try to run that command you mentioned on the new WLC to force the APs to join it.

Re: APs not joining a new WLC

Rich R — Fri, 16 Jun 2023 16:50:50 GMT

Aha then it is that!
Configure the command on the OLD WLC - then the APs will pick up the config and then they will be able to join the NEW WLC (with the expired cert). But you must also configure it on the NEW WLC otherwise the APs will remove the config and then not be able to join again the next they reset.

In summary: configure that on new and old WLCs.

The command is harmless - it won't impact the APs at all - but if for some reason you don't want to apply it to the old WLC then on the new WLC you'll need to:
- apply config ap cert-expiry-ignore mic enable
- disable NTP, configure date to earlier than May 4 2022
- allow all APs to join and get the new config
- after all APs have joined and picked up the config fix then re-enable NTP

Warning: if you don't configure it on the old WLC and one of the APs joins the old WLC again then the AP will lose the config then not be able to join the new WLC again - so your easy path is just configure both.

Re: APs not joining a new WLC

davidrodriguez — Fri, 16 Jun 2023 16:53:21 GMT

Thank you! I'll run the command on both WLC, and then next week, we are planning a maintenance window to switch to the other WLC.

I already apply the command config ap cert-expiry-ignore mic enable on the new WLC.

And disabled NTP, when we are ready to switch them, i'll change the date on the new WLC.

I'll run the command on the old WLC too today.

I'll let you know how it goes next week.