topic Re: C9115AXi leave EWC to try associated on C9800-CL with no success in Wireless

C9115AXi leave EWC to try associated on C9800-CL with no success

ANTONIO DEUS — Fri, 06 Oct 2023 14:24:27 GMT

Hello,
About 1 year ago, to expand WiFi coverage, we purchased AP Catalyst C9115AXI-E. As the version of the WLC 5520, in SSO and CAPWAP, did not support the new APs, because of the AIR-CAP1602I-E-K9 series AP, it was decided to make use of the EWC that the C9115Axi series APs provide.
So far there is no problem, just to contextualize the scenario.
However, WiFi coverage needs increased and more Catalyst C9115AXI-E and 2 C9800-CL Series APs were purchased.
We started the process of installing the C9800-CL in HyverV, on the same network segment (/21), where de other WLC e EWC are and without any problems until the moment SSO was established. At this point, the APs left the EWC and started trying to register with the new Wilress Controller without success, with the message: DTLS cert-chain not available

When consulting the messages, here is the result:
c9800-wc01#show wireless stats ap history
AP Name Radio MAC Event Time Recent Disconnect Time Disconnect Reason Disconnect Count
-------------------------------------------------- --------------------------------------------------
edif1-ap11 QQQQ.a8bf.7280 Disjointed 10/03/23 23:32:05 NA DTLS cert-chain not available 1
edif2-ap108 QQQQ.a8cc.3b40 Disjoined 10/03/23 23:32:15 NA DTLS cert-chain not available 1
edif1-ap24 TTTT.d528.6f40 Disjoined 03/10/23 23:32:16 NA DTLS cert-chain not available 1
edif1-ap20 TTTT.d52a.e0a0 Disjoined 10/03/23 23:32:13 NA DTLS cert-chain not available 1

c9800-wc01#show wireless stats ap join summary
Number of APs: 4

Base MAC Ethernet MAC AP Name IP Address Status Last Failure Phase Last Disconnect Reason
-------------------------------------------------- --------------------------------------------------
QQQQ.a8bf.7280 xxxx.7758.ee74 edif1-ap11 10.3.6.11 Not Joined Dtls-Handshake DTLS cert-chain not available
QQQQ.a8cc.3b40 xxxx.7759.078c edif2-ap108 10.3.6.72 Not Joined Dtls-Handshake DTLS cert-chain not available
TTTT.d528.6f40 yyyy.3718.8e08 edif1-ap24 10.3.6.24 Not Joined Dtls-Handshake DTLS cert-chain not available
TTTT.d52a.e0a0 yyyy.3718.dc34 edif1-ap20 10.3.6.20 Not Joined Dtls-Handshake DTLS cert-chain not available

With this attempt to register the APs on the C9800-CL, it caused a WiFi outage in buildings that were previously registered with EWC, something that is serious for us!

With this scenario of loss of connectivity, the new C9800-CL were turned off and they could only be worked on after office hours (11:30pm - 7:00am).

Questions:
A) Why are the APs abandoning the EWC and trying to register with the C9800-CL, when the high availability parameters (EWC name and IP) are configured?
B) Why is this abandonment happening when the EWC didn't have any problems, that is, it never lost connectivity and the AP itself didn't try to register with the new C9800-CL in SSO?
C) Regarding the error presented (DTLS cert-chain not available), the solution described here => https://community.cisco.com/t5/wireless/ap-does-not-connect-to-the-controller/td-p/4529464 has already been tried, but unsuccessfully. What is the next step?

EWC version: 17.6.5
C9800-Cl version: 17.9.4

With these conditions, how can this problem be overcome?
Helps are appreciated.
Best regards,
Antonio

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Mark Elsen — Fri, 06 Oct 2023 14:52:49 GMT

         - In general you should work to a single centralized controller setup (C9800-CL)

                >....DTLS cert-chain not available
For c9800-wc01 , check it's configuration with the CLI command : show tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/ , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.     Checkout all advisories!

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Leo Laohoo — Sat, 07 Oct 2023 00:20:55 GMT

@ANTONIO DEUS wrote:
A) Why are the APs abandoning the EWC and trying to register with the C9800-CL, when the high availability parameters (EWC name and IP) are configured?

It could be a bug or it could be the strain on the EWC.

Even with a 9800-80 and only 5500 APs, I sometimes catch the APs join the secondary unit even though the secondary unit is not in the configuration of the AP.

The primary controller is running 25% memory utilization which is already very high. (Normal rate should be <12%.)

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

ANTONIO DEUS — Sat, 07 Oct 2023 00:44:59 GMT

Hi @Mark Elsen ,

Thank you for the tips, but after run the command (show tech wireless) and analyzed the results (in attachment) there is nothing can related with the problem.

My original questions remains 😞

Thanks anyway!

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

ANTONIO DEUS — Sat, 07 Oct 2023 00:48:24 GMT

Hi @Leo Laohoo ,

My concern is the AP do not register on C9800-Cl. If teh AP leave the EWC I don't care, but the APs leave the EWC and do not associated to the C9800-CL. That's the big issue.

Thanks anyway.

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Leo Laohoo — Sat, 07 Oct 2023 02:10:34 GMT

@Mark Elsen

Please read THIS first before encouraging people to use the WLC Debug Analyzer.

If the WLC Debug Analyzer could not analyze a simple debug, what are the chances of WLC Debug Analyzer providing accurate information to more complicated debugs?

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Leo Laohoo — Sat, 07 Oct 2023 02:12:01 GMT

What are these AP (models) that are not going to the 9800?

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Mark Elsen — Sat, 07 Oct 2023 05:26:17 GMT

>...My concern is the AP do not register on C9800-Cl. If the AP leave the EWC I don't care, but the APs leave the EWC and do not associated to the C9800-CL. That's the big issue.
- You should have an environment with all EWC's turned off and test again because the 2 controller being in the same capwap broadcast domain confuses the AP ; then if problems persist use these tools for troubleshooting (further) :
https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Mark Elsen — Sat, 07 Oct 2023 05:31:37 GMT

>...Please read THIS first before encouraging people to use the WLC Debug Analyzer.
@Leo Laohoo Hey Leo , my reply to the customer mentioned the steps for the WirelessAnalyzer procedure not WLC Debug Analyzer , concerning it's usefulness read this too ! : This is so good

Re: C9115AXi leave EWC to try associated on C9800-CL with no success

Rich R — Sat, 07 Oct 2023 13:04:26 GMT

1. EWC and WLC is NOT SUPPORTED to be installed on the same subnet/VLAN - this is clearly stated in the documentation. It WILL cause problems as you have discovered and you should NEVER do that!

2. Make sure you have configured your 9800-CL management interface and certificate correctly as per the best practice guide:
https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#Wirelessmanagementinterface
https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#Configurationfilemanagement

You either have to do a big bang cutover from EWC or phased migration of your APs. I suggest the phased approach so that you can test 1 or 2 before starting to move the rest.
- setup a new VLAN for the WLC
- setup a new VLAN for the migrated APs
- use DHCP with option 43 to point the APs to the new WLC IP. Use ACLs on the switch/router to block any traffic to the EWC IP from those APs.
- Then when you are ready you move the AP from the EWC VLAN to the new AP VLAN and power cycle the AP. It should discover the WLC from DHCP option 43 and proceed with join and download of new software. Better still do factory default reset of the AP.

If the AP has EWC software installed then I recommend doing a complete re-flash of the AP using the procedure at https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9120axi-access-point/217537-repairing-c9120-c9115-access-points-from.html (which installs 8.10.130.0 CAPWAP software) and then upgrade to the latest CAPWAP image. This avoids the possibility of the EWC software trying to re-activate if/when the WLC is momentarily unreachable which will cause you the same problem again.