CAPWAP state: DTLS Teardown

[a.saad@growup-ts.com]

I have a problem with AP 1815i as it continuously disjoin the controller 9800 and restart and then loop in same action as shown in attached logs.

These are the port configuration

==============
Access switch
==============

interface gi1/0/21
description To-WLC
switchport mode trunk
switchport trunk allowed vlan 5,6,50
switchport trunk native vlan 50

interface gi1/0/15
description To-AP
switchport mode trunk
switchport trunk allowed vlan 5,6,50
switchport trunk native vlan 50

==================
Core Switch
==================
ip dhcp pool DATA-pool
network 192.168.6.0 255.255.255.0
default-router 192.168.6.254
dns-server 192.168.100.1 192.168.100.2 8.8.8.8
!
ip dhcp pool WLC-Mgm-pool
network 192.168.50.0 255.255.255.0
default-router 192.168.50.254
option 43 hex f205.c0a8.3264.01

[Rich R]

If the AP is in local mode or the WLANs are all centrally switched (on a flex AP) then those VLANs are only significant on the WLC trunk port not the AP port because all the client traffic is tunnelled to the WLC over CAPWAP and breaks out on the WLC.  Therefore the AP only needs the management connection to the WLC which can be over trunk native vlan or access port - same result in that the frames are untagged.

[a.saad@growup-ts.com]

Sorry guys, I couldn't access the site remotely till now due to connectivity issue at customer side, once I can access I will try all what you mentioned and feedback you all.

Thanks for support

[a.saad@growup-ts.com]

Find attached AP log

[a.saad@growup-ts.com]

I followed you recommendation, I removed native vlan command and configured AP port in access mode.

Same problem as attached file AP log

[dbandulas]

I have the same problem. is this resolve  

[Scott Fella]

You have 1815's and on at least 17.6.3?  What does the ap console log look like?  Please add more details.

[dbandulas]

AP Model: 9115A
WLC:c9800

Ap down 15:10:32
Ap Up- 15:11:48

this happens regularly on many APs


===============================================================
Apr 20 14:45:25 ntp_update: NTP: Thu Apr 20 14:45:25 2023 :Can not create
ntp process log file.
Apr 20 14:45:25 kernel: [*04/20/2023 14:45:25.9574] systemd[1]: Started
Cisco syslogd watcher.
Apr 20 14:45:25 kernel: [*04/20/2023 14:45:25.9584] Update NTP source to WLC
Apr 20 14:45:25 kernel: [*04/20/2023 14:45:25.9724] systemd[1]: Started ntp
file watcher.
Apr 20 14:45:26 kernel: [*04/20/2023 14:45:26.4664] Got WSA Server config
TLVs
Apr 20 14:45:27 kernel: [*04/20/2023 14:45:27.6274] systemd[1]: Starting
Lighttpd Watcher...
Apr 20 14:45:27 kernel: [*04/20/2023 14:45:27.6584] systemd[1]: Started
Lighttpd Watcher.
Apr 20 14:45:30 kernel: [*04/20/2023 14:45:30.7774] AP tag change to
TAG-POL-TDL_EXTERNAL
Apr 20 14:45:30 kernel: [*04/20/2023 14:45:30.8394] flags value is 1
process iot_radio
Apr 20 14:45:30 root: BLE reset lock acquired
Apr 20 14:45:31 kernel: [*04/20/2023 14:45:31.0834] Powering down BLE radio
Apr 20 14:45:32 root: released BLE reset lock
Apr 20 14:45:53 kernel: [*04/20/2023 14:45:53.8644] set cleanair
[slot0][band0] disable
Apr 20 14:45:53 NCI: CLEANAIR: Slot 0 admin disabled
Apr 20 14:45:53 kernel: [*04/20/2023 14:45:53.8664] set cleanair
[slot1][band1] disable
Apr 20 14:45:55 NCI: CLEANAIR: Slot 1 admin disabled
Apr 20 15:04:40 kernel: [*04/20/2023 15:04:40.4273] Re-Tx Count=1, Max
Re-Tx Value=5, SendSeqNum=2, NumofPendingMsgs=1
Apr 20 15:04:40 kernel: [*04/20/2023 15:04:40.4273]
Apr 20 15:04:43 kernel: [*04/20/2023 15:04:43.2783] Re-Tx Count=2, Max
Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=2
Apr 20 15:04:43 kernel: [*04/20/2023 15:04:43.2783]
Apr 20 15:04:46 kernel: [*04/20/2023 15:04:46.1293] Re-Tx Count=3, Max
Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=2
Apr 20 15:04:46 kernel: [*04/20/2023 15:04:46.1293]
Apr 20 15:04:48 kernel: [*04/20/2023 15:04:48.9803] Re-Tx Count=4, Max
Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=2
Apr 20 15:04:48 kernel: [*04/20/2023 15:04:48.9803]
Apr 20 15:04:51 kernel: [*04/20/2023 15:04:51.8313] Re-Tx Count=5, Max
Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=2
Apr 20 15:04:51 kernel: [*04/20/2023 15:04:51.8313]
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] Max retransmission
count exceeded, going back to DISCOVER mode.
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 54, eleLen = 62, sendSeqNum = 5
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] ....TLV:
TLV_INACTIVE_CLIENT_DATA_PAYLOAD(2213), level: 0, seq: 0, nested: true
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 20, eleLen = 28, sendSeqNum = 5
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 464, eleLen = 472, sendSeqNum = 5
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] ...Vendor SubType:
AP_CDP_CACHE_PAYLOAD(24) len: 460 vendId 409600
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6823] Dropping msg
CAPWAP_ECHO_REQUEST, type = 1, len = 0, eleLen = 8, sendSeqNum = 5
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.6833] Flexconnect Switching
to Standalone Mode!
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.8853]
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.8853] CAPWAP State: DTLS
Teardown
Apr 20 15:04:54 upgrade: Script called with args:[CANCEL]
Apr 20 15:04:54 kernel: [*04/20/2023 15:04:54.9633] status 'upgrade.sh:
Script called with args:[CANCEL]'
Apr 20 15:04:55 kernel: [*04/20/2023 15:04:55.0083] do CANCEL, part2 is
active part
Apr 20 15:04:55 upgrade: Cleanup tmp files ...
Apr 20 15:04:55 kernel: [*04/20/2023 15:04:55.0273] status 'upgrade.sh:
Cleanup tmp files ...'
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.6513] systemd[1]: Starting
dhcpv6 client watcher...
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.6513] Discovery Response from
10.16.4.250
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.6523] Discovery Response from
10.16.4.250
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.6783] systemd[1]: Stopping
DHCPv6 client...
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.6883] systemd[1]: Starting
DHCPv6 client...
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.7203] systemd[1]: Started
DHCPv6 client.
Apr 20 15:05:09 kernel: [*04/20/2023 15:05:09.7433] systemd[1]: Started
dhcpv6 client watcher.
Apr 20 15:05:19 kernel: [*04/20/2023 15:05:19.0353]
Apr 20 15:05:19 kernel: [*04/20/2023 15:05:19.0353] CAPWAP State: DTLS Setup
Apr 20 15:05:19 kernel: [*04/20/2023 15:05:19.1313]
dtls_verify_server_cert: Controller certificate verification successful
Apr 20 15:05:19 kernel: [*04/20/2023 15:05:19.7903]
Apr 20 15:05:19 kernel: [*04/20/2023 15:05:19.7903] CAPWAP State: Join
Apr 20 15:05:19 kernel: [*04/20/2023 15:05:19.8313] Sending Join request to
10.16.4.250 through port 5259
Apr 20 15:05:21 kernel: [*04/20/2023 15:05:21.8853] systemd[1]: Starting
Lighttpd Watcher...
Apr 20 15:05:21 kernel: [*04/20/2023 15:05:21.9143] systemd[1]: Started
Lighttpd Watcher.
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5233] Sending Join request to
10.16.4.250 through port 5259
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5273] Join Response from
10.16.4.250
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5273] AC accepted join
request with result code: 0
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5573] Received wlcType 0,
timer 30
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5753]
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5753] CAPWAP State: Image Data
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5753] AP image version
17.6.4.56 backup 8.10.130.0, Controller 17.6.4.56
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.5753] Version is the same, do
not need update.
Apr 20 15:05:24 upgrade: Script called with args:[NO_UPGRADE]
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.6163] status 'upgrade.sh:
Script called with args:[NO_UPGRADE]'
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.6613] do NO_UPGRADE, part2 is
active part
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.6673]
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.6673] CAPWAP State: Configure
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.6893] capwapd: reading file
/click/nss_lag_control/lacp_state: No such file or directory
Apr 20 15:05:24 kernel: [*04/20/2023 15:05:24.6893] capwapd: reading file
/click/nss_lag_control/capwap_state: No such file or directory
Apr 20 15:05:24 capwapd[5437]: Check lagloadbalance setting flex_mode 1 cfg
0 linkstate 1 ap_type 83
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.3193]
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.3193] CAPWAP State: Run
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.3713] AP has joined
controller TDL-WIFICTRL-01
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.4893] Flexconnect Switching
to Connected Mode!
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.9703] Previous AP mode is 2,
change to 2
Apr 20 15:05:25 capwapd[5437]: Check lagloadbalance setting flex_mode 1 cfg
0 linkstate 1 ap_type 83
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.9723] capwapd: reading file
/click/nss_lag_control/lacp_state: No such file or directory
Apr 20 15:05:25 kernel: [*04/20/2023 15:05:25.9723] capwapd: reading file
/click/nss_lag_control/capwap_state: No such file or directory
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.1233] Current session mode:
ssh, Configured: Telnet-No, SSH-Yes, Console-Yes
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.1233]
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.1233] Current session mode:
telnet, Configured: Telnet-No, SSH-Yes, Console-Yes
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.1233]
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.1723] Current session mode:
console, Configured: Telnet-No, SSH-Yes, Console-Yes
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.1723]
Apr 20 15:05:26 chpasswd: password for user changed
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.2023] chpasswd: password for
user changed
Apr 20 15:05:26 chpasswd: password for user changed
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.2183] chpasswd: password for
user changed
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.3603] systemd[1]: Starting
Cisco syslogd watcher...
Apr 20 15:05:26 syslogd exiting
Apr 20 15:05:26 syslogd started: BusyBox v1.32.1
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.4533] systemd[1]: Started ntp
file watcher.
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.4603] systemd[1]: Started
Cisco syslogd watcher.
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.9413] systemd[1]: Starting
Lighttpd Watcher...
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.9693] systemd[1]: Started
Lighttpd Watcher.
Apr 20 15:05:26 kernel: [*04/20/2023 15:05:26.9803] Got WSA Server config
TLVs
Apr 20 15:05:31 kernel: [*04/20/2023 15:05:31.2663] AP tag change to
TAG-POL-TDL_EXTERNAL
Apr 20 15:05:31 kernel: [*04/20/2023 15:05:31.3293] flags value is 1
process iot_radio
Apr 20 15:05:31 root: BLE reset lock acquired
Apr 20 15:05:31 kernel: [*04/20/2023 15:05:31.5723] Powering down BLE radio
Apr 20 15:05:33 root: released BLE reset lock
Apr 20 15:05:54 NCI: CLEANAIR: Slot 0 admin disabled
Apr 20 15:05:54 kernel: [*04/20/2023 15:05:54.3374] set cleanair
[slot0][band0] disable
Apr 20 15:05:54 kernel: [*04/20/2023 15:05:54.3584] set cleanair
[slot1][band1] disable
Apr 20 15:05:56 NCI: CLEANAIR: Slot 1 admin disabled
Apr 20 15:11:03 kernel: [*04/20/2023 15:11:03.9807] Re-Tx Count=1, Max
Re-Tx Value=5, SendSeqNum=82, NumofPendingMsgs=1
Apr 20 15:11:03 kernel: [*04/20/2023 15:11:03.9807]
Apr 20 15:11:06 kernel: [*04/20/2023 15:11:06.8317] Re-Tx Count=2, Max
Re-Tx Value=5, SendSeqNum=82, NumofPendingMsgs=1
Apr 20 15:11:06 kernel: [*04/20/2023 15:11:06.8317]
Apr 20 15:11:09 kernel: [*04/20/2023 15:11:09.6827] Re-Tx Count=3, Max
Re-Tx Value=5, SendSeqNum=87, NumofPendingMsgs=6
Apr 20 15:11:09 kernel: [*04/20/2023 15:11:09.6827]
Apr 20 15:11:12 kernel: [*04/20/2023 15:11:12.5337] Re-Tx Count=4, Max
Re-Tx Value=5, SendSeqNum=87, NumofPendingMsgs=6
Apr 20 15:11:12 kernel: [*04/20/2023 15:11:12.5337]
Apr 20 15:11:15 kernel: [*04/20/2023 15:11:15.3837] Re-Tx Count=5, Max
Re-Tx Value=5, SendSeqNum=87, NumofPendingMsgs=6
Apr 20 15:11:15 kernel: [*04/20/2023 15:11:15.3837]
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] Max retransmission
count exceeded, going back to DISCOVER mode.
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 20, eleLen = 28, sendSeqNum = 92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 34, eleLen = 42, sendSeqNum = 92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] ....TLV:
TLV_INACTIVE_CLIENT_DATA_PAYLOAD(2213), level: 0, seq: 0, nested: true
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 34, eleLen = 42, sendSeqNum = 92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] ....TLV:
TLV_INACTIVE_CLIENT_DATA_PAYLOAD(2213), level: 0, seq: 0, nested: true
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 863, eleLen = 871, sendSeqNum =
92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 38, eleLen = 46, sendSeqNum = 92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] ....TLV:
TLV_ECHOPAYLOAD(42), level: 0, seq: 0, nested: true
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 64, eleLen = 72, sendSeqNum = 92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] ....TLV:
TLV_APETHERINTFPAYLOAD(46), level: 0, seq: 0, nested: true
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 274, eleLen = 282, sendSeqNum =
92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 274, eleLen = 282, sendSeqNum =
92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 131, eleLen = 139, sendSeqNum =
92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 334, eleLen = 342, sendSeqNum =
92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Dropping msg
CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 334, eleLen = 342, sendSeqNum =
92
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2357] Flexconnect Switching
to Standalone Mode!
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.4117]
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.4117] CAPWAP State: DTLS
Teardown
Apr 20 15:11:18 upgrade: Script called with args:[CANCEL]
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.4897] status 'upgrade.sh:
Script called with args:[CANCEL]'
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.5347] do CANCEL, part2 is
active part
Apr 20 15:11:18 upgrade: Cleanup tmp files ...
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.5537] status 'upgrade.sh:
Cleanup tmp files ...'
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2047] systemd[1]: Starting
dhcpv6 client watcher...
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2137] Discovery Response from
10.16.4.250
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2157] Discovery Response from
10.16.4.250
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2347] systemd[1]: Stopping
DHCPv6 client...
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2437] systemd[1]: Starting
DHCPv6 client...
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2767] systemd[1]: Started
DHCPv6 client.
Apr 20 15:11:33 kernel: [*04/20/2023 15:11:33.2987] systemd[1]: Started
dhcpv6 client watcher.
Apr 20 15:11:42 kernel: [*04/20/2023 15:11:42.5867]
Apr 20 15:11:42 kernel: [*04/20/2023 15:11:42.5867] CAPWAP State: DTLS Setup
Apr 20 15:11:42 kernel: [*04/20/2023 15:11:42.6817]
dtls_verify_server_cert: Controller certificate verification successful
Apr 20 15:11:43 kernel: [*04/20/2023 15:11:43.3387] systemd[1]: Starting
Lighttpd Watcher...
Apr 20 15:11:43 kernel: [*04/20/2023 15:11:43.3397]
Apr 20 15:11:43 kernel: [*04/20/2023 15:11:43.3397] CAPWAP State: Join
Apr 20 15:11:43 kernel: [*04/20/2023 15:11:43.3647] Sending Join request to
10.16.4.250 through port 5259
Apr 20 15:11:43 kernel: [*04/20/2023 15:11:43.3677] systemd[1]: Started
Lighttpd Watcher.
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.0767] Sending Join request to
10.16.4.250 through port 5259
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.0817] Join Response from
10.16.4.250
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.0817] AC accepted join
request with result code: 0
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.1117] Received wlcType 0,
timer 30
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.1297]
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.1297] CAPWAP State: Image Data
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.1307] AP image version
17.6.4.56 backup 8.10.130.0, Controller 17.6.4.56
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.1307] Version is the same, do
not need update.
Apr 20 15:11:48 upgrade: Script called with args:[NO_UPGRADE]
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.1707] status 'upgrade.sh:
Script called with args:[NO_UPGRADE]'
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.2167] do NO_UPGRADE, part2 is
active part
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.2227]
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.2227] CAPWAP State: Configure
Apr 20 15:11:48 capwapd[5437]: Check lagloadbalance setting flex_mode 1 cfg
0 linkstate 1 ap_type 83
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.2437] capwapd: reading file
/click/nss_lag_control/lacp_state: No such file or directory
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.2437] capwapd: reading file
/click/nss_lag_control/capwap_state: No such file or directory
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.8667]
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.8667] CAPWAP State: Run
Apr 20 15:11:48 kernel: [*04/20/2023 15:11:48.9157] AP has joined
controller TDL-WIFICTRL-01
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.0337] Flexconnect Switching
to Connected Mode!
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.5127] Previous AP mode is 2,
change to 2
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.5147] capwapd: reading file
/click/nss_lag_control/lacp_state: No such file or directory
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.5147] capwapd: reading file
/click/nss_lag_control/capwap_state: No such file or directory
Apr 20 15:11:49 capwapd[5437]: Check lagloadbalance setting flex_mode 1 cfg
0 linkstate 1 ap_type 83
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.6627] Current session mode:
ssh, Configured: Telnet-No, SSH-Yes, Console-Yes
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.6627]
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.6627] Current session mode:
telnet, Configured: Telnet-No, SSH-Yes, Console-Yes
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.6627]
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.7107] Current session mode:
console, Configured: Telnet-No, SSH-Yes, Console-Yes
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.7107]
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.7417] chpasswd: password for
user changed
Apr 20 15:11:49 chpasswd: password for user changed
Apr 20 15:11:49 chpasswd: password for user changed
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.7777] chpasswd: password for
user changed
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.8977] systemd[1]: Starting
Cisco syslogd watcher...
Apr 20 15:11:49 syslogd exiting
Apr 20 15:11:49 syslogd started: BusyBox v1.32.1
Apr 20 15:11:49 ntp_update: NTP: Thu Apr 20 15:11:49 2023 :Can not create
ntp process log file.
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.9687] Update NTP source to WLC
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.9747] systemd[1]: Started
Cisco syslogd watcher.
Apr 20 15:11:49 kernel: [*04/20/2023 15:11:49.9867] systemd[1]: Started ntp
file watcher.
Apr 20 15:11:50 kernel: [*04/20/2023 15:11:50.5247] Got WSA Server config
TLVs
Apr 20 15:11:53 kernel: [*04/20/2023 15:11:53.5977] systemd[1]: Starting
Lighttpd Watcher...
Apr 20 15:11:53 kernel: [*04/20/2023 15:11:53.6417] systemd[1]: Started
Lighttpd Watcher.
Apr 20 15:11:54 kernel: [*04/20/2023 15:11:54.7327] AP tag change to
TAG-POL-TDL_EXTERNAL
Apr 20 15:11:54 kernel: [*04/20/2023 15:11:54.7937] flags value is 1
process iot_radio
Apr 20 15:11:54 root: BLE reset lock acquired
Apr 20 15:11:55 kernel: [*04/20/2023 15:11:55.0377] Powering down BLE radio
Apr 20 15:11:56 root: released BLE reset lock
Apr 20 15:12:18 NCI: CLEANAIR: Slot 0 admin disabled
Apr 20 15:12:18 kernel: [*04/20/2023 15:12:18.9007] set cleanair
[slot0][band0] disable
Apr 20 15:12:18 kernel: [*04/20/2023 15:12:18.9027] set cleanair
[slot1][band1] disable
Apr 20 15:12:20 NCI: CLEANAIR: Slot 1 admin disabled
Apr 20 15:14:04 sshd[5754]: Invalid user tdh from 10.16.33.141 port 50170
Apr 20 15:14:43 sshd[5754]: error: Could not get shadow information for
NOUSER
Apr 20 15:14:43 sshd[5754]: Failed password for invalid user tdh from
10.16.33.141 port 50170 ssh2
Apr 20 15:15:03 sshd[5754]: Connection closed by invalid user tdh
10.16.33.141 port 50170 [preauth]
Apr 20 15:15:08 sshd[6051]: error: Could not get shadow information for
admin
Apr 20 15:15:08 sshd[6051]: Accepted password for admin from 10.16.33.141
port 50180 ssh2
Apr 20 15:15:09 FIPS[6086]: *** shell: FIPS Mode = disabled ***

[dbandulas]

APs disconnect and rejoin randomly exact time of 42 sec,( not a reboot) 

[marce1000]

 - Use the following commands to analyze AP (dtls) connection issues :
        show wireless stats ap join summary
        show wireless dtls connections
        show platform hardware chassis active qfp feature wireless capwap datapath statistics drop all
        show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> details
        show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> statistics
        show platform hardware chassis active qfp feature wireless dtls datapath statistics all
        show platform hardware chassis active qfp statistics drop all | inc Global | Wls (Data Plane Statistics – Global Wireless Drops)

  - Also have a checkup-review of your  9800   Controller current configuration with the CLI command : show tech wireless  , have the output analyzed with : https://cway.cisco.com/wireless-config-analyzer
                                         Checkout all advisories!

           Some further useful commands :
    show ap summary | i Number of APs
    show ap uptime | ex ____([0-9])+ day (check on APs reloaded or restarted within 1 day)
   show ap crash
   show wireless stats ap session termination (Check for highest number of events or unexpected events of AP session termination)
   show wireless stats ap history | i Disjoined (Check for disconnect reasons and time of disconnections)
   show ap tag summary | i Yes (Check for APs with misconfigured tags)
   show ap sum sort descending client-count | i __0_ (check for APs with no clients connected)

            And also look at : https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin

  M.

[dbandulas]

Thanks, guys I'm still working on this and will update, thanks for the all support

[Rich R]

Apr 20 15:11:15 kernel: [*04/20/2023 15:11:15.3837] Re-Tx Count=5, Max
Re-Tx Value=5, SendSeqNum=87, NumofPendingMsgs=6
Apr 20 15:11:15 kernel: [*04/20/2023 15:11:15.3837]
Apr 20 15:11:18 kernel: [*04/20/2023 15:11:18.2347] Max retransmission
count exceeded, going back to DISCOVER mode.

You need to check connectivity between AP and WLC to work out why you're getting those retransmissions and timeouts.

"Active version: 17.3.3.26"
That's a really outdated version of software - refer to the TAC recommended versions below and update accordingly.

[dbandulas]

Ap Version 

AP Running Image : 17.6.4.56
Primary Boot Image : 17.6.4.56
Backup Boot Image : 8.10.130.0

C9800:

Upgrade Advisor
	Current Controller version:	17.6.4
	Maximmum possible or Recommended  Network version:	17.6.5
	Maximum Supported Controller Version:	No Current Limit
	Maximum Common Supported APs Version:	No Current Limit
	Maximum Supported version per AP detected model:
	C9115	No Current Limit
	C9105	No Current Limit
	Recommended upgrade path to maximum version:	Upgrade directly to 17.6.x
	Documentation for more details	9800 Recommended Code
		Upgrade Tips and Tricks
		Wireless Compatibility Matrix

[Rich R]

From TAC recommended doc (link below and in the info you just pasted): "Cisco recommends 17.6.5 CCO image for all deployments without these specific IOS APs (1700/2700/3700/1572)."  So start with upgrade to 17.6.5.  You could also consider 17.9.3 but that's not a recommended version yet.

Have you acted on all the relevant warnings from the config analyzer?

But back to my previous point - connectivity between AP and WLC to work out why you're getting those retransmissions and timeouts?

If you're 100% sure no problem on connectivity then look at CPU usage on the WLC.  You haven't mentioned what model of WLC you're using (?) but you need to look at WNCd CPU:
show process cpu platform sorted | incl wncd
If your CPU is badly shared across the wireless management controllers then they can end up dropping traffic and you'll lose APs.  So that then comes to your site tag design which should ensure good load balancing across the processes as per the best practices guide - link below.  Other things like https redirect (if you're using web auth) can also cause high CPU - we had to disable it because it does not scale at all on 9800 (8540 handles it at least 10 times better).

[dbandulas]

AP Name Ethernet MAC Radio MAC AP Up Time Association Up Time
---------------------------------------------------------------------------------------------------------------------------------------------------
TDL-Ground-Artists-Bar a49b.cd2c.9348 e44e.2dca.9a40 15 days 1 hour 48 minutes 50 seconds 15 days 1 hour 45 minutes 58 seconds
TDL-Ground-Bar-Snug a00f.3713.ba04 a00f.373b.cfe0 37 days 17 hours 32 minutes 55 seconds 1 day 20 hours 53 minutes 5 seconds
APC828.E58B.0C14 c828.e58b.0c14 c828.e582.ca20 8 days 18 hours 1 minute 29 seconds 1 day 18 hours 22 minutes 35 seconds
TDL-Ground-Bar-Upper-Behind-Bar a00f.3713.7290 a00f.3739.9440 37 days 17 hours 35 minutes 20 seconds 1 day 13 hours 18 minutes 56 seconds
TDL-Ground-Bar-Upper-Hatch a00f.3713.c1d4 a00f.373c.0e60 37 days 17 hours 32 minutes 32 seconds 20 hours 55 minutes 45 seconds
TDL-Ground-Bar a00f.3713.be58 a00f.373b.f280 6 days 11 hours 0 minute 17 seconds 19 hours 44 minutes 46 seconds
TDL-Systems-Test-9ED0 a00f.3713.9ed0 a00f.373a.f640 2 days 20 hours 16 minutes 29 seconds 18 hours 18 minutes 59 seconds
APA49B.CD2C.985C a49b.cd2c.985c e44e.2dca.c2e0 18 hours 18 minutes 13 seconds 18 hours 15 minutes 23 seconds
TDL-Ground-Promenade-Station-1 a00f.3713.2a8c a00f.3737.5440 37 days 17 hours 40 minutes 10 seconds 16 hours 56 minutes 45 seconds
TDL-Ground-Promenade-Station-4 a49b.cd2c.8364 e44e.2dca.1b20 15 days 1 hour 48 minutes 50 seconds 16 hours 21 minutes 13 seconds
TDL-Ground-Cake-Shop a49b.cd2c.42b8 e44e.2dc8.15c0 16 hours 20 minutes 39 seconds 16 hours 17 minutes 49 seconds
TDL-Ground-Reception-Desk a49b.cd2c.9190 e44e.2dca.8c80 37 days 17 hours 31 minutes 35 seconds 15 hours 16 minutes 58 seconds
TDL-Ground-Promenade-Station-3 a49b.cd2c.99f0 e44e.2dca.cf80 15 days 1 hour 48 minutes 50 seconds 13 hours 15 minutes 16 seconds
TDL-Ground-Bar-Washrooms a00f.3713.74a0 a00f.3739.a4c0 37 days 16 hours 32 minutes 4 seconds 13 hours 1 minute 51 seconds
APA49B.CD2C.8B98 a49b.cd2c.8b98 e44e.2dca.5cc0 1 day 18 hours 51 minutes 23 seconds 11 hours 9 minutes 24 seconds
TDL-Ground-Concierge-Desk a49b.cd2c.943c e44e.2dca.a1c0 88 days 14 hours 32 minutes 47 seconds 9 hours 39 minutes 27 seconds
TDL-Ground-Front-Office a49b.cd2c.9a88 e44e.2dca.d440 86 days 15 hours 24 minutes 37 seconds 58 minutes 4 seconds
TDL-2nd-239 c828.e58b.0a04 c828.e582.b9a0 9 minutes 57 seconds 6 minutes 2 seconds

Highlighted one disconnect and rejoin again (42 sec)

[dbandulas]

Event Previous State Occurance Count
------------------------------------------------------------------------------------
NO_EVENT Unknown 1515
DTLS session closed INIT 7
Reset by API JOIN_PROCESS 1
Reset by API RUN 23
Heartbeat timer expiry RUN 7
Message timer expiry RUN 7

=========================

AP Name Radio MAC Event Time Recent Disconnect Time Disconnect Reason Disconnect Count
---------------------------------------------------------------------------------------------------------------------------------------------------
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 16:36:00 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 16:35:12 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 16:00:05 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 15:59:06 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 15:39:47 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 15:38:57 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 15:01:34 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 15:00:45 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 14:46:22 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 14:45:32 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 13:45:31 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 13:44:38 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 13:09:16 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 13:08:22 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 12:35:30 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 12:34:39 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 11:38:55 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 11:38:00 NA App send req failed 1
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Joined 04/20/23 07:38:40 NA NA NA
TDL-Ground-Promenade-Station-1 a00f.3737.5440 Disjoined 04/20/23 07:37:45 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/19/23 20:13:49 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/19/23 20:12:45 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/12/23 17:57:45 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/12/23 17:56:48 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/12/23 09:26:56 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/12/23 09:26:04 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/12/23 09:02:40 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/12/23 09:01:32 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/06/23 16:47:15 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/06/23 16:46:23 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/06/23 00:51:10 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/06/23 00:50:22 NA App send req failed 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/04/23 11:23:47 NA NA NA
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Disjoined 04/04/23 11:23:15 NA Country changed in AP profile 1
TDL-Ground-Bar-Upper-Behind-Bar a00f.3739.9440 Joined 04/04/23 04:03:58 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 20:30:54 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 20:30:04 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 17:18:41 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 17:17:46 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 17:07:32 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 17:06:39 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 15:28:14 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 15:27:17 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 14:48:11 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 14:47:21 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 14:04:23 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 14:03:27 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 12:23:11 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 12:22:20 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 10:37:04 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 10:36:11 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 07:52:05 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 07:51:16 NA App send req failed 1
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Joined 04/20/23 03:24:46 NA NA NA
TDL-Ground-Bar-Washrooms a00f.3739.a4c0 Disjoined 04/20/23 03:23:55 NA App send req failed 1
TDL-Systems-Test-9ED0 a00f.373a.f640 Joined 04/20/23 15:13:45 NA NA NA
TDL-Systems-Test-9ED0 a00f.373a.f640 Disjoined 04/20/23 15:12:46 NA Tag modified 1
TDL-Systems-Test-9ED0 a00f.373a.f640 Joined 04/20/23 08:13:00 NA NA NA