C9130-AXI-E randomly disjoining and joining

[Oblivion_Tech]

Hi all,

I've searched if this question has been asked already and couldn't find anything, hence excuse me if something similar exists and point me to the solution.
If not, please have a look and let me know what you think about this one.

We have a setup of 2 WLC units (9800-L) with 9130-AXI-E and AIR-AP1562I-E-K9 WAPs.
System is working fine, no issues really except for this random one we are facing. Sometimes our WAPs (happens on only 4 WAPs out of 120), will go from blue indication light to red/green and within half a minute back to blue light. During red light status, devices those were connected to the WAP will lose WiFi hence the connection and once blue light is back, everything works fine.

I thought WAP restarted for some reason, but checking the uptime I've noticed it didn't and it's been up for 290+ days.
Everything is showing healthy, functional, registered and I have no idea what is causing this issue.

I've checked WLC logs and found following:

Aug 15 18:51:41.321: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: ***, MAC: *** Disjoined
Aug 15 18:51:41.312: %CAPWAPAC_SMGR_TRACE_MESSAGE-3-EWLC_GEN_ERR: Chassis 1 R0/0: wncd: Error in Session-IP: *** Mac: 34ed.1b6a.9800 Maximum retries for sending CAPWAP message reached. Close CAPWAP DTLS session
Aug 15 18:51:41.312: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: ***, MAC: ac7a.5695.3602 Disjoined
Aug 15 18:53:41.564: %LINEPROTO-5-UPDOWN: Line protocol on Interface Capwap105, changed state to up
Aug 15 18:53:41.558: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: ***, MAC: *** Joined

So WAP disjoins and joins randomly, at least from my perspective.

Before I remove and reinstall those WAPs which I guess should help, I'd like to hear a possible reason for this to happen.
Any advice would be highly appreciated.

Thanks guys
Petar

[marce1000]

 - Check if these AP's are having basic network connectivity problems (or not). For that you can for instance look at the port counters on the switches that they are connected too. Make sure all error counters are idle. You could for instance also have an ssh-connected session on the troublesome access points on standby and see if that does get disconnected or not when these problems happen. If the ssh remains then use the term mon command also (at the beginning of the session already) , to have console messages displayed which may provide further insights as to what kind of problems the access points are experiencing. For the rest have a checkup of the configuration of the 9800-L controllers , with the CLI command show  tech wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer

 M.

[Leo Laohoo]

1. What firmware is the controller on? 
2. What is the uptime of the controller? 
3. Flex or Local mode?

[Arshad Safrulla]

You are currently running an obsolete IOS-XE code, consider upgrading it to 17.3.5a or 17.6.3 if you dont have any Wave1 AP's in your network.Recommended Cisco IOS XE Releases for Catalyst 9800 Wireless LAN Controllers - Cisco

That being said,

1. Did you check the CPU/memory utilization in the WLC and the AP when the issue was reported?
2. did you see any abnormal behavior in the switch where the AP or WLC is connected, such as uplink failure or something like that?
3. Also do you have the AP management in the same VLAN as Wireless Management (WMI) interface of WLC? If yes it is not recommended design when you have more than 100AP's.
4. It is very important that you offload AP syslog messages to a syslog server or check the AP logs when the issue was reported. If you have any logs from AP at the time of the reported issue please sanitize and share here.

[Oblivion_Tech]

Hi,

CPU and memory are fine. I think we'll go for an update and then monitor the behavior. 
Why did you recommend 17.3.5 and 17.6.3 and not Cupertino-17.9.1?

Only APs we have are ones I've mentioned in my original post.

[marce1000 · Accepted solution]

 - Ref (and or use 17.9.x only related when features are required) : https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html

                  >....Amsterdam 17.3

Cisco IOS XE 17.3.x is a long-lived train with several MRs planned. 17.3 is the last Cisco IOS-XE release for C9800 WLC to support IOS APs (with the exception of IW3700 which is still supported on later releases). Cisco recommends 17.3.5b CCO image  for all deployments with >>>>>>>>>>IOS APs.

 M.

[Oblivion_Tech]

Hi,

After an upgrade, above mentioned issues disappeared, but now our log is filled with:

Aug 19 06:22:56.607: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:***.80[5264] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Aug 19 06:22:07.193: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:***.79[5264] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Aug 19 06:20:35.990: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:***.79[5264] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Aug 19 06:19:04.731: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:***.79[5264] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Aug 19 06:17:33.514: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:***.79[5264] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

IP address message part goes from .1 to .250 and then starts again. Everything is working fine, no issues on WiFi network.

[Arshad Safrulla]

I would be checking the NTP of the WLC, also I would re-create the certificate and reload the WLC.

wireless config vwlc-ssc key-size 2048 signature-algo sha256 password <12345Password>

[Oblivion_Tech]

Hi Arshad,

I think I found what is causing this issue. Before an upgrade we had 116 APs, while after an upgrade WLC is showing 115. I've checked license logs and found an AP list before and after an upgrade. I've found a missing AP, which I think is causing this error, but I have only model and serial number, no MAC address and there is no way I can find out where this AP is installed in our environment. I would like to find it, disconnect it and check if error is still generated.

Is there a way to find a MAC address from model and serial number only?
Do you think command you've sent me could fix this issue? What about the password part, where do I find that?

Thank you

[Arshad Safrulla]

Which code did you upgrade to? You had any Wave 1 (1700, 2700 and 3700) AP in your network and upgrade to 17.4+ code or may be any other unsupported AP trying to register to your WLC. Check the release notes for the code you are running.

You can check your DHCP server log and match the IP's against the AP IP's registered in the WLC to trace the MAC. You can set any password you wish. 

[Oblivion_Tech]

I've upgraded to recommended Amsterdam 17.3 and there are no above mentioned APs in our setup. Only 9130AXI model and AP1562 model. 

WLC detects 115 and the one that is missing is not getting an IP address due to handshake error. I found only a serial number and if I can find MAC, I'll find it in the network, but I can't find MAC address, so I'm a bit stuck.

[marce1000]

                        >....but I can't find MAC address, so I'm a bit stuck.

          - If you provide ap-addresses through DHCP , then check the dhcp server's logs when this AP gets on the network  (or around that time you get the join error which will probably not 'far away')

 M.

[Oblivion_Tech]

Hi,

Alright, found it.

Thank you very much for your support, time and effort. Highly appreciate it!

[marce1000]

         >...offload AP syslog messages to a syslog server or check the AP logs

 For that check : https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_wl_16_10_cg/enabling-syslog-messages-in-access-points-and-controller-for-syslog-server.html

 M.