CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown

[JustTakeTheFirstStep]

Several APs are being disconnected.

CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown

Does anyone know the cause?

[marce1000]

                    - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa35350

 M.

[Rich R · Accepted solution]

Many possible reasons including the bug @marce1000 suggested.
You're running 17.5.1 which was a limited support release.
https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-xe-17/ios-xe-17-5-x-eol.html

You should move to an extended support release which has the fix for that bug (and many others).  Suggest you upgrade to 17.6.3.

Refer to https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#anc7
Note "Cisco recommends 17.6.3 CCO image for all deployments without IOS APs."

[russell.splain]

Glad to see it's not just my deployment.  I have a similar issue as that shown by Snika:

Random AP's dropping sporadically with AP DISJOIN / JOIN messages.

usually accompanied with log string: "CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown"

Happens between 20 and 50 times daily, across my 300 AP's.

Installation:   9800-L-F in High-Availability pair, running 17.3.5b, due to need to support Wave-1 AP's  (2700/3700).

Interesting item:  All DISJOIN messages in the 9800 logs are associated with "Chassis 2", and all JOIN messages are associated with "Chassis 1".  This is curious because the Active controller in the SSO Redundancy setup is Chassis 1.  Chassis 2 is a hot-standby mirror of Chassis 1.  Examples:

Aug 22 13:27:57 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll2ap205, MAC: bc26.c7a2.bdee Joined
Aug 22 13:30:33 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.6.132.71[5256] Mac: 780c.f04e.b180 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 13:30:33 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll2ap106, MAC: 700f.6a26.337a Disjoined
Aug 22 13:31:02 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll2ap106, MAC: 700f.6a26.337a Joined
Aug 22 13:31:41 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll2ap107, MAC: 7872.5d2c.eaee Disjoined
Aug 22 13:31:41 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.6.132.73[5256] Mac: 7872.5d9f.0080 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 13:31:53 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll2ap213, MAC: f4db.e61a.7df2 Disjoined
Aug 22 13:31:53 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.6.132.183[5272] Mac: f4db.e61d.f020 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 13:32:11 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll2ap107, MAC: 7872.5d2c.eaee Joined
Aug 22 13:32:24 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll2ap213, MAC: f4db.e61a.7df2 Joined
Aug 22 13:35:17 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll2ap208, MAC: 502f.a8ec.aa86 Disjoined
Aug 22 13:35:17 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.6.132.188[5248] Mac: 00ea.bd69.bc40 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 13:35:48 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll2ap208, MAC: 502f.a8ec.aa86 Joined
Aug 22 14:11:11 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.78[5248] Mac: bc26.c7ba.5ce0 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:11:11 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap612, MAC: 502f.a8ec.6f90 Disjoined
Aug 22 14:11:42 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap612, MAC: 502f.a8ec.6f90 Joined
Aug 22 14:13:05 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap812, MAC: 7872.5d2c.eaf0 Disjoined
Aug 22 14:13:05 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.99[5248] Mac: 7872.5d9f.00a0 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:13:11 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap204, MAC: f4db.e61a.7d98 Disjoined
Aug 22 14:13:11 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.49[5264] Mac: f4db.e61d.ea80 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:13:17 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap601, MAC: 700f.6a26.3114 Disjoined
Aug 22 14:13:17 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.175[5256] Mac: 780c.f04e.8b20 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:13:35 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap812, MAC: 7872.5d2c.eaf0 Joined
Aug 22 14:13:47 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap601, MAC: 700f.6a26.3114 Joined
Aug 22 14:14:46 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap104, MAC: 7872.5d2c.eaca Disjoined
Aug 22 14:14:46 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.155[5248] Mac: 7872.5d9e.fe40 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:14:57 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap702, MAC: a093.514b.07f8 Disjoined
Aug 22 14:14:57 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.83[5264] Mac: 502f.a84d.4d00 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:15:16 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap104, MAC: 7872.5d2c.eaca Joined
Aug 22 14:15:27 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap702, MAC: a093.514b.07f8 Joined
Aug 22 14:15:52 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap711, MAC: a093.514b.0860 Disjoined
Aug 22 14:15:52 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.87[5248] Mac: 502f.a84d.5380 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:16:13 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.15.178.85[5248] Mac: 7872.5d48.e980 CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Aug 22 14:16:13 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap717, MAC: 7872.5d46.6c0e Disjoined
Aug 22 14:16:22 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap711, MAC: a093.514b.0860 Joined
Aug 22 14:16:43 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: fll3ap717, MAC: 7872.5d46.6c0e Joined
Aug 22 14:17:06 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: fll3ap202, MAC: f4db.e61a.76b2 Disjoined

[Filip Spisak]

Hello, have you found a solution to this issue? We have the same issue when using 17.6.4 and the C9800-CL. A few AP are  disconnecting from the controller and switching from capwap to ewc-ap.

[Scott Fella]

@Filip Spisak you should open a new thread since you have a different model controller and design. Then describe your whole setup. The only thing I can tell you is to delete the EWC image so that AP’s don’t convert.  Then you still have to troubleshoot why the AP’s loose connection to the 9800-CL, that I think would be your main issue. 

[stobar1101]

Hello, what was the solution to your problem?

Right now I'm going to update to version 17.6.4 I hope this solves this problem

[russell.splain]

In our case, it turned out that NAC policies applied to AP switch ports
interfered with AP CAPWAP messages. Changing NAC switch port policy solved
the problem. 17.6.4 has been running fine ever since.

[hara12386]

Hello @russell.splain

I am facing the same issue, could you please provide more details about the solution? Sorry I didn't understand well what you meant !

Many thanks,

Hamid