topic Re: Cisco access points C9115AXI are getting disconnected ramdomly in Wireless

Cisco access points C9115AXI are getting disconnected ramdomly

gustavo.garcia@iosoffices.com — Wed, 06 Apr 2022 21:39:37 GMT

Hello Community, I do have a Cisco C9115AXI-A wireless network, one of them is getting the controller functionality and another 6 APs are providing devices connectivity service. all of them are getting disconnected randomly.

I'm using software release IOS XE Software, Version 17.03.04c

Does someone have the same issue?, does someone have a solution for this?

Thanks in advance.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Leo Laohoo — Wed, 06 Apr 2022 23:22:29 GMT

gustavo.garcia@iosoffices.com wrote:

all of them are getting disconnected randomly.

Start with the basics: Examine the uptime of each APs and determine if the APs crashed or rebooted.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Ambuj M — Thu, 07 Apr 2022 05:16:56 GMT

This version has a known issue with 9115, there is a good chance you might be hitting this bug.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvy51818

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Rich R — Thu, 07 Apr 2022 13:08:11 GMT

@Ambuj M CSCvy51818 is terminated and doesn't even mention IOS-XE. Maybe you're thinking of some other bug?

Like @Leo Laohoo says you need to go back to basics and see whether the APs are crashing or just losing connectivity then work from there.

Crashes = software bug -> update IOS-XE.

Connectivity problems -> troubleshoot your network connectivity.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

gustavo.garcia@iosoffices.com — Thu, 07 Apr 2022 13:14:06 GMT

Thank you Leo, actually, a continous ping never stops, it is only the contact with controller supervision which falls:

%CAPWAPAC_SMGR_TRACE_MESSAGE-3-WLC_GEN_ERR: Chassis 2 R0/0: wncd: Error in Session-IP: 10.137.8.19[5259] Mac: e44e.2d40.c5c0 Heartbeat timer expiry for AP. Close CAPWAP DTLS session

%CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: AP3-S125, MAC: a49b.cd29.d8b8 Disjoined

%PEER_SELECTION-5-EWC_PEER_SELECTION_EVENT: Chassis 2 R0/0: wncd: HA Event: Non-Candidate AP 'AP3-S125' is no longer a peer

%CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: Session-IP:10.137.8.19[5259] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

%CAPWAP_IMGDWNLD_TRACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 2 R0/0: wncd: List Event: AP with wtp_mac e44e.2d40.c5c0 for Image Type ap1g7 is added to Master AP list

Re: Cisco access points C9115AXI are getting disconnected ramdomly

gustavo.garcia@iosoffices.com — Thu, 07 Apr 2022 13:23:12 GMT

Thank you @Rich R it is not a connectivity issue, because a continous ping never falls, the AP is just loosing communication to cntroller and disconnects all the devices connected at that point to it.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

gustavo.garcia@iosoffices.com — Thu, 07 Apr 2022 13:24:33 GMT

Thank you @Ambuj M, I'll try updating software

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Ambuj M — Thu, 07 Apr 2022 13:53:20 GMT

look at the open caveat in IOS XE Amsterdam 17.3.4c

sorry should have pasted this link before, I am not recommending by any means to jump to conclusion and we should follow procedure, but based on code he is running there is a good change he might be hitting the bug.

https://www.cisco.com/c/en/us/td/docs/wireless/controller/ewc/17-3/rel-notes/ewc-rn-17-3-x.html#Cisco_Reference.dita_dc77575e-1ef4-4687-a033-3c380dd7b8d3

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Rich R — Thu, 07 Apr 2022 14:10:28 GMT

So you have to understand how to read those @Ambuj M - those were open (still under investigation) at the time that version was released and like that one can be closed subsequently.

More useful to look at the resolved caveats and open caveats for the latest release - 17.3.5a in this case.

For example https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa12278 fixed.

Looking in bug tool there are actually numerous AP fixes (also relevant to 9115) in 17.3.5 so I would certainly consider using that anyway.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

patoberli — Thu, 07 Apr 2022 14:26:38 GMT

I also suggest upgrading (please note, you also need to install an SMU for 17.3.5 on the controller).

Regarding the error message "Heartbeat timer expiry for AP." This typically means that the AP lost the connection to the controller. This could happen if the latency is to high (more than 300 ms) for example. This can be temporary caused if there is an overloaded link between the AP and the controller.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

gustavo.garcia@iosoffices.com — Thu, 07 Apr 2022 15:52:38 GMT

Thank you very much @patoberli, @Ambuj M and @Rich R. I'll try again to upgrade to 17.3.5, I tried but still back to 03.04c, don't know why, I didn't find the reasson at logs.

Let you know.

Cheers!

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Leo Laohoo — Thu, 07 Apr 2022 22:42:10 GMT

Is the 9800 in an HA?

Re: Cisco access points C9115AXI are getting disconnected ramdomly

stuart.pannell — Fri, 10 Nov 2023 13:30:22 GMT

Hi Gustavo, did you get a fix for this issue? was it AP/Controller related? I am having the same issues at a larger scale and have been banging my head trying to fix it. We have swapped out cabling, switches, AP's no fails on ping tests but random drops during the day. I think its down to congestion on either the 9200L switches or the AP's themselves.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Rich R — Fri, 10 Nov 2023 13:41:03 GMT

@stuart.pannell 17.3 code is almost end of life now and things have moved on a lot since then.

Current recommended code (see TAC recommended link below) is 17.9.4 + SMUs + APSPs.
Also check your config with the config analyzer - link below.
If you still see the issue then open a TAC case so they can capture all the relevant data and escalate with dev team.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

stuart.pannell — Fri, 10 Nov 2023 14:22:28 GMT

I have checked the config via the analyser and also we are running code 17.9.3 with the latest APSP for that version. I have a taC case open so hopefully will get to the bottom of it soon.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

stuart.pannell — Tue, 14 Nov 2023 14:21:31 GMT

Hi All, I have been doing some further investigation and it looks like we are hitting congestion. I believe that CAPWAP control packets are marked at CS6. I have enabled QOS on all of the uplinks, checked the QOS with a "Show policy-map" and cannot see any packets coming to the switchport with CS6. I have converted an old 1832i and joined that to the WLC, I have the same QOS policy on the switchport that is connected to and can see packets in the Output_control_mgmt_queue marked as CS6.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Rich R — Sun, 19 Nov 2023 11:49:04 GMT

Yes looking at pcap CAPWAP control are CS6.
But you would still see some packets even if a few are getting dropped so I'd say either they're getting re-marked or you're not checking the markings correctly.
Where did you think you have congestion? Do you see drops on those interfaces?

Re: Cisco access points C9115AXI are getting disconnected ramdomly

stuart.pannell — Tue, 16 Jul 2024 08:56:01 GMT

Just for information, we have fixed our issue with AP's randomly dropping during the day. It turned out to be a physical electrical issue where the AP is shorting out on the ceiling grid, we have insulated the AP's from the ceiling grids and they are now stable and been up for weeks.

Re: Cisco access points C9115AXI are getting disconnected ramdomly

Rich R — Tue, 16 Jul 2024 09:12:18 GMT

How does a plastic AP case "short out"?

Re: Cisco access points C9115AXI are getting disconnected ramdomly

stuart.pannell — Tue, 16 Jul 2024 09:30:04 GMT

It's metal on the back