Small Junior College | APs Randomly Disconnect From Cisco WLC

[CoachChan]

Hello Everyone,

Intro:

I recently took a job at a small junior college as a support specialist to be trained as a wireless networking manager as my primary task. I am new to the wireless network side of things but have caught on quickly. I will give the details below and thank you in advance for taking the time to help with my issue!

Products Used:

Cisco Catalyst C9800-40-K9 | Wireless Controller | IOS-XE 17.9.4

Cisco Catalyst 2960-X Series | Switches | SW Version 15.2(2)E5

Cisco Catalyst C9105AXW-B | Access Points | SW Version 17.9.4.202

Issue: APs Randomly Disconnect From Cisco WLC

We have a building on campus where approximately 20-50 APs will randomly lose connection to the WLC with errors such as DTLS close alert from peer, DTLS handshake expired, and Heart beat timer expiry. End-users lose WLAN and LAN connections for 10-15 minutes before connections are restored automatically. I have attempted to google each error but have come up short on answers to solve them.

Attempted Solutions That Failed:

Updating WLC to current IOS-XE 17.9.4 | Help a few APs join WLC but no changes to randomly disconnecting APs

Installing all current SMU, APSP, and APDP packages available for WLC | Help a few APs join WLC but no changes to randomly disconnecting APs

Configuring switchports' to failing APs from access to trunk | All ports confirmed to be on correct access vlan before switching to trunk native vlan. | No changes, did not help anything.

Latest Solution Ideas:

PoE Issue? Are they losing power then reconnecting? How do I test/monitor this?

Switch software is not latest version. Would updating it to most current recommended version help?

Configure a VM for 9800-CL WLC and see if the same issues persist on different controller entirely.

[Leo Laohoo]

At the very least, install AP Service Pack 2.  

[CoachChan]

I installed AP service pack 2 last week. Helped a few APs join back but later disconnected. I Installed AP service pack 4 yesterday, did not help anything to my knowledge.

I did find this bug web page that looks to be very similar if not exactly the same thing. Can you look over it and see if you agree that this is the same issue I am experiencing?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwh61011

Thank you for your swift reply!

[Rich R]

Marce has given plenty of good info there but once you've covered that I think you're at the point where you need to open a TAC case.  I have a few cases open for 17.9.4 - no real progress on any of them at the moment.
TAC will also be able to confirm if they think it's caused by CSCwh61011

[marce1000]

 - Here are   a few useful commands concerning troubleshooting DTLS (and or AP-connectivity issues ) :
show wireless stats ap join summary
show wireless dtls connections
show platform hardware chassis active qfp feature wireless capwap datapath statistics drop all
show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> details
show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> statistics
show platform hardware chassis active qfp feature wireless dtls datapath statistics all
show platform hardware chassis active qfp statistics drop all | inc Global | Wls 

 Also look into : https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc4

 Advising to enable syslog  server for APs per https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_wl_16_10_cg/enabling-syslog-messages-in-access-points-and-controller-for-syslog-server.html#task_56DB7D3369B846FA98D2D071AAC9D946

  - Last but not least (perhaps should have been first) : have a checkup of the 9800-40-K9 controller configuration with the CLI command show tech wireless ; feed the output into Wireless Config Analyzer    consider this procedure mandatory and or benefits can for instance be seen from This is so good

 M.

            

[eglinsky2012]

---

@CoachChan wrote:

PoE Issue? Are they losing power then reconnecting? How do I test/monitor this?

Switch software is not latest version. Would updating it to most current recommended version help?

---

From the switch perspective, can you provide a show run for the interface that an affected AP is connected to?

Also do a show log from the switch. That will show you any PoE events as well as link up/down events, assuming logging for those isn't disabled.

Furthermore, you could do a "show int" for an affected interface and pay attention to the lines with input/CRC errors and collisions (these could indicate a physical issue such as bad patch cable or data jack).

It may or may not help, but I'd recommend updating the switch software in general. I think 15.2(4)E10 is the final software for the X series (it is on our 3560X) and it is very reliable for us.

[Gaurav Kansal]

Dear CoachChan,

What is source of powering your access points.? If you are using POE+ switch then please refer your network switch power budge and keep investigate your switch logs after this activity happen.

GoodLuck
Regards

[Spreadlove]

@CoachChan this reply may be coming in a bit late or may not be the solution to this but I also got the same DTLS error, turned out to be an issue with VLAN, the uplink switch from which the switch the AP is connected did not have the vlan needed for the AP to get an IP address.
Explanation: there are two switches and an AP, Switch A is an uplink for Switch B, AP is connected on Switch B, AP is supposed to be on vlan X to operate, switch B has vlan X but Switch A does not have vlan x.

Hope it helps someone