AP-C9115AXI-E disconnect from c9800 controller and rejoin

[sumithliyanage]

We have a c9800-48 controller connect with 93 of c9115AXI, controller monitor by solar wind, AP goes down randomly and rejoin again without disconnecting the session.

Alert  is similar to below:

The following Access Point went DOWN on 27 October 2023 at 15:33: -

Access Point: TDL-Ground-Front-Office

IP Address: 10.16.23.3

This alert was triggered by the alert named 'PO3 Access Point Changed Status'

=================================================================

after exactly the 10 min

The following Access Point came UP on 27 October 2023 at 15:43: -

Access Point: TDL-Ground-Front-Office

IP Address: 10.16.23.3

===========================================

checking the logs on AP similar to below

Oct 27 11:42:44 kernel: [*10/27/2023 11:42:44.3087] CLSM[AE:2B:C2:8A:72:D7]: Failed adding dms client config
Oct 27 12:02:46 kernel: [*10/27/2023 12:02:46.2741] CLSM[C6:60:F8:E8:DD:F3]: Failed adding dms client config
Oct 27 12:38:16 kernel: [*10/27/2023 12:38:16.7105] DOT11_DRV[0]: off_chhannel RX timeout, ch = 1
Oct 27 13:07:15 kernel: [*10/27/2023 13:07:15.3090] wl: Bad Address
Oct 27 13:42:16 kernel: [*10/27/2023 13:42:16.3467] Re-Tx Count=1, Max Re-Tx Value=8, SendSeqNum=8, NumofPendingMsgs=2
Oct 27 13:42:16 kernel: [*10/27/2023 13:42:16.3467]
Oct 27 13:42:21 kernel: [*10/27/2023 13:42:21.0976] Re-Tx Count=2, Max Re-Tx Value=8, SendSeqNum=9, NumofPendingMsgs=3
Oct 27 13:42:21 kernel: [*10/27/2023 13:42:21.0976]
Oct 27 13:42:25 kernel: [*10/27/2023 13:42:25.8496] Re-Tx Count=3, Max Re-Tx Value=8, SendSeqNum=11, NumofPendingMsgs=5
Oct 27 13:42:25 kernel: [*10/27/2023 13:42:25.8496]
Oct 27 13:42:30 kernel: [*10/27/2023 13:42:30.6016] Re-Tx Count=4, Max Re-Tx Value=8, SendSeqNum=11, NumofPendingMsgs=5
Oct 27 13:42:30 kernel: [*10/27/2023 13:42:30.6016]
Oct 27 13:42:35 kernel: [*10/27/2023 13:42:35.3526] Re-Tx Count=5, Max Re-Tx Value=8, SendSeqNum=16, NumofPendingMsgs=10
Oct 27 13:42:35 kernel: [*10/27/2023 13:42:35.3526]
Oct 27 13:42:40 kernel: [*10/27/2023 13:42:40.1046] Re-Tx Count=6, Max Re-Tx Value=8, SendSeqNum=19, NumofPendingMsgs=13
Oct 27 13:42:40 kernel: [*10/27/2023 13:42:40.1046]
Oct 27 13:42:44 kernel: [*10/27/2023 13:42:44.8566] Re-Tx Count=7, Max Re-Tx Value=8, SendSeqNum=20, NumofPendingMsgs=14
Oct 27 13:42:44 kernel: [*10/27/2023 13:42:44.8566]
Oct 27 13:42:49 kernel: [*10/27/2023 13:42:49.6076] Re-Tx Count=8, Max Re-Tx Value=8, SendSeqNum=21, NumofPendingMsgs=15
Oct 27 13:42:49 kernel: [*10/27/2023 13:42:49.6076]
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Max retransmission count exceeded, going back to DISCOVER mode.
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 20, eleLen = 28, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 464, eleLen = 472, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ...Vendor SubType: AP_CDP_CACHE_PAYLOAD(24) len: 460 vendId 409600
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 20, eleLen = 28, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 3083, eleLen = 3091, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 3083, eleLen = 3091, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 20, eleLen = 28, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 14, eleLen = 22, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 14, eleLen = 22, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 35, eleLen = 43, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ....TLV: TLV_AP_THROUGHPUT_PAYLOAD(745), level: 0, seq: 0, nested: true
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 35, eleLen = 43, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ....TLV: TLV_AP_THROUGHPUT_PAYLOAD(745), level: 0, seq: 0, nested: true
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 104, eleLen = 112, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ....TLV: TLV_INACTIVE_CLIENT_DATA_PAYLOAD(2213), level: 0, seq: 0, nested: true
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 124, eleLen = 132, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ....TLV: TLV_INACTIVE_CLIENT_DATA_PAYLOAD(2213), level: 0, seq: 0, nested: true
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_ECHO_REQUEST, type = 1, len = 0, eleLen = 8, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 464, eleLen = 472, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ...Vendor SubType: AP_CDP_CACHE_PAYLOAD(24) len: 460 vendId 409600
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 20, eleLen = 28, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 863, eleLen = 871, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 38, eleLen = 46, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ....TLV: TLV_ECHOPAYLOAD(42), level: 0, seq: 0, nested: true
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] Dropping msg CAPWAP_WTP_EVENT_REQUEST, type = 34, len = 64, eleLen = 72, sendSeqNum = 24
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3596] ....TLV: TLV_APETHERINTFPAYLOAD(46), level: 0, seq: 0, nested: true
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.3606] Flexconnect Switching to Standalone Mode!
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.6206]
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.6206] CAPWAP State: DTLS Teardown
Oct 27 13:42:54 upgrade: Script called with args:[CANCEL]
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.7076] status 'upgrade.sh: Script called with args:[CANCEL]'
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.7606] do CANCEL, part2 is active part
Oct 27 13:42:54 upgrade: Cleanup tmp files ...
Oct 27 13:42:54 kernel: [*10/27/2023 13:42:54.7806] status 'upgrade.sh: Cleanup tmp files ...'

Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.3295] systemd[1]: Starting dhcpv6 client watcher...
Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.3605] systemd[1]: Stopping DHCPv6 client...
Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.3615] Discovery Response from 10.16.4.250
Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.3625] Discovery Response from 10.16.4.250
Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.3695] systemd[1]: Starting DHCPv6 client...
Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.4105] systemd[1]: Started DHCPv6 client.
Oct 27 13:43:09 kernel: [*10/27/2023 13:43:09.4415] systemd[1]: Started dhcpv6 client watcher.
Oct 27 13:43:18 kernel: [*10/27/2023 13:43:18.6935]
Oct 27 13:43:18 kernel: [*10/27/2023 13:43:18.6935] CAPWAP State: DTLS Setup
Oct 27 13:43:18 kernel: [*10/27/2023 13:43:18.8275] dtls_verify_server_cert: Controller certificate verification successful
Oct 27 13:43:19 kernel: [*10/27/2023 13:43:19.1565]
Oct 27 13:43:19 kernel: [*10/27/2023 13:43:19.1565] CAPWAP State: Join
Oct 27 13:43:19 kernel: [*10/27/2023 13:43:19.1795] Sending Join request to 10.16.4.250 through port 5249
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.4635] systemd[1]: Starting Lighttpd Watcher...
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.4935] systemd[1]: Started Lighttpd Watcher.
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.8215] Sending Join request to 10.16.4.250 through port 5249
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.8255] Join Response from 10.16.4.250
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.8255] AC accepted join request with result code: 0
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.8605] Received wlcType 0, timer 30
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.9025]
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.9025] CAPWAP State: Image Data
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.9025] AP image version 17.6.4.56 backup 8.10.130.0, Controller 17.6.4.56
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.9025] Version is the same, do not need update.
Oct 27 13:43:23 upgrade: Script called with args:[NO_UPGRADE]
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.9475] status 'upgrade.sh: Script called with args:[NO_UPGRADE]'
Oct 27 13:43:23 kernel: [*10/27/2023 13:43:23.9975] do NO_UPGRADE, part2 is active part
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.0035]
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.0035] CAPWAP State: Configure
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.0055] capwapd: reading file /click/nss_lag_control/lacp_state: No such file or directory
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.0055] capwapd: reading file /click/nss_lag_control/capwap_state: No such file or directory
Oct 27 13:43:24 capwapd[5616]: Check lagloadbalance setting flex_mode 1 cfg 0 linkstate 1 ap_type 83
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.7795]
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.7795] CAPWAP State: Run
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.8375] AP has joined controller TDL-WIFICTRL-01
Oct 27 13:43:24 kernel: [*10/27/2023 13:43:24.9835] Flexconnect Switching to Connected Mode!
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.6005] Previous AP mode is 2, change to 2
Oct 27 13:43:25 capwapd[5616]: Check lagloadbalance setting flex_mode 1 cfg 0 linkstate 1 ap_type 83
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.6325] capwapd: reading file /click/nss_lag_control/lacp_state: No such file or directory
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.6325] capwapd: reading file /click/nss_lag_control/capwap_state: No such file or directory
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.8365] Current session mode: ssh, Configured: Telnet-No, SSH-Yes, Console-Yes
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.8365]
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.8365] Current session mode: telnet, Configured: Telnet-No, SSH-Yes, Console-Yes
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.8365]
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.9085] Current session mode: console, Configured: Telnet-No, SSH-Yes, Console-Yes
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.9085]
Oct 27 13:43:25 chpasswd: password for user changed
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.9405] chpasswd: password for user changed
Oct 27 13:43:25 chpasswd: password for user changed
Oct 27 13:43:25 kernel: [*10/27/2023 13:43:25.9595] chpasswd: password for user changed
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1085] systemd[1]: Starting Cisco syslogd watcher...
Oct 27 13:43:26 syslogd exiting
Oct 27 13:43:26 syslogd started: BusyBox v1.32.1
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1735] systemd[1]: Started Cisco syslog service.
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1875] systemd[1]: Started Cisco syslogd watcher.
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1905]
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1905] Same LSC mode, no action needed
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1935] CLSM[00:00:00:00:00:00]: U3 Client RSSI Stats feature is deprecated; can no longer be enabled
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.1975] systemd[1]: Starting ntp file watcher...
Oct 27 13:43:26 ntp_update: NTP: Fri Oct 27 13:43:26 2023 :Can not create ntp process log file.
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.2115] Update NTP source to WLC
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.2245] systemd[1]: Started ntp file watcher.
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5335] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5335] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5355] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5355] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5375] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5375] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5385] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5385] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5405] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5405] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5425] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5425] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5645] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5645] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5655] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5655] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5675] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5675] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5695] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5695] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5715] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5715] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5725] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5725] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5745] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5745] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5765] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.5765] not a valid cache_type in add_cache_param_via_tlv
Oct 27 13:43:26 kernel: [*10/27/2023 13:43:26.9285] Got WSA Server config TLVs
Oct 27 13:43:28 kernel: [*10/27/2023 13:43:28.5245] systemd[1]: Starting Lighttpd Watcher...
Oct 27 13:43:28 kernel: [*10/27/2023 13:43:28.5645] systemd[1]: Started Lighttpd Watcher.
Oct 27 13:43:28 kernel: [*10/27/2023 13:43:28.9725] systemd[1]: Starting nbar watcher...
Oct 27 13:43:29 kernel: [*10/27/2023 13:43:29.0185] systemd[1]: Started nbar watcher.
Oct 27 13:43:29 kernel: [*10/27/2023 13:43:29.4365] systemd[1]: Starting nbar watcher...
Oct 27 13:43:29 kernel: [*10/27/2023 13:43:29.4645] systemd[1]: Started nbar watcher.
Oct 27 13:43:31 kernel: [*10/27/2023 13:43:31.5175] systemd[1]: Starting nbar watcher...
Oct 27 13:43:31 kernel: [*10/27/2023 13:43:31.5465] systemd[1]: Started nbar watcher.
Oct 27 13:43:31 kernel: [*10/27/2023 13:43:31.9585] systemd[1]: Starting nbar watcher...
Oct 27 13:43:31 kernel: [*10/27/2023 13:43:31.9855] systemd[1]: Started nbar watcher.
Oct 27 13:43:32 kernel: [*10/27/2023 13:43:32.4685] AP tag change to TAG-POL-TDH
Oct 27 13:43:32 kernel: [*10/27/2023 13:43:32.5365] flags value is 1 process iot_radio
Oct 27 13:43:32 root: BLE reset lock acquired
Oct 27 13:43:32 kernel: [*10/27/2023 13:43:32.7895] Powering down BLE radio
Oct 27 13:43:34 root: released BLE reset lock
Oct 27 13:43:34 kernel: [*10/27/2023 13:43:34.4715] systemd[1]: Starting nbar watcher...
Oct 27 13:43:34 kernel: [*10/27/2023 13:43:34.5055] systemd[1]: Started nbar watcher.
Oct 27 13:43:53 NCI: CLEANAIR: Slot 0 admin disabled
Oct 27 13:43:53 kernel: [*10/27/2023 13:43:53.0625] set cleanair [slot0][band0] disable
Oct 27 13:43:53 kernel: [*10/27/2023 13:43:53.0645] set cleanair [slot1][band1] disable
Oct 27 13:43:55 NCI: CLEANAIR: Slot 1 admin disabled
Oct 27 13:45:11 sshd[14133]: error: Could not get shadow information for admin
Oct 27 13:45:11 sshd[14133]: Accepted password for admin from 10.16.33.139 port 52761 ssh2
Oct 27 13:45:12 FIPS[14265]: *** shell: FIPS Mode = disabled ***

[Leo Laohoo]

Are these switches APs connected to mGIG-capable ports?

EDIT:  Fixed my mistake (was already sleepy when I typed this last night).

[sumithliyanage]

Yes 1gig fiber connection to core sw

[Rich R]

That's not what Leo asked!  
Are these switches connected to mGIG-capable ports?
mGig is copper so it can't be fibre.
I'd also add - are the APs connected to mGIG-capable ports?

Either way - update to 17.9.4 with SMUs and APSPs installed and see whether you still have the problem for a start.  For info on 17.9.4 read the release notes which Marce provided the link (you should always read the release notes).  And make sure you update the ROMMON before you upgrade the IOS as per requirement.

Always refer to the TAC recommended link below to make sure you're using the latest TAC recommended release.  This also explains why you need to update the ROMMON.

[sumithliyanage]

thanks for the reply,please see the sample configuration and active switch port status

description WIFI AP
switchport trunk native vlan 1505
switchport mode trunk
auto qos trust
spanning-tree portfast trunk
service-policy input AutoQos-4.0-Trust-Cos-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
end

==========================

Hardware is Gigabit Ethernet, address is a84f.b191.be0b (bia a84f.b191.be0b)
Description: WIFI AP
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 68451
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 571000 bits/sec, 603 packets/sec
301976885 packets input, 117040574500 bytes, 0 no buffer
Received 24022987 broadcasts (23550103 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 23550103 multicast, 0 pause input
0 input packets with dribble condition detected
14804072760 packets output, 5055267310397 bytes, 0 underruns
Output 4305142784 broadcasts (9940120233 multicasts)
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[marce1000]

  >...switchport trunk native vlan 1505
       switchport mode trunk
       auto qos trust
      spanning-tree portfast trunk
      service-policy input AutoQos-4.0-Trust-Cos-Input-Policy
      service-policy output AutoQos-4.0-Output-Policy

  Note that when dealing with issues , it's always best to start with the simplest configurations first , in that context for the time being I would remove the qos related directives and check the impact , you also only need trunk mode if the AP must operate in Flexconnect mode , 

 M.

[sumithliyanage]

Thanks for your reply, all APs are flexconnect, let me simplify the configuration and test 

[stuart.pannell]

have you had any luck with this issue, I also have the same problem with Cisco 9115AXi AP's, some will randomly drop during the day. I have looked at layer1 issues and changed cabling out with STP cables, from my perspective they drop when the network is busy, not seeing any packets dropping on interfaces and have qos marking/matching CAPWAP CS6 or COS 3 and prioritising the control traffic through the network. Its not all of the APs that are having issues and could be 2 out of 3 on the same switch. out of hours they appear to be stable. I am using Gig switches not mGig however I have also applied qos to my SSID's to reduce the throughput of the wireless to 400Mb/s. I have swapped one of the 9115AXi with an upgraded 1830 series and this has been solid, is there an issue with the CAPWAP keep alives and control packets? I have TAC looking into the issue, not getting very far with it to be honest.

[Leo Laohoo]

What firmware is the WLC on?

And what APSP have been installed?

[sumithliyanage]

Hi Stuart

still, we are having this issue with model 9115AXi, as suggested here, I look forward to upgrading the controller Firmware, SMU and ISOXE to 17.9.4a. as the controller in the production environment, hasn't got a suitable time slot for downtime, I will update once I complete the process

[stuart.pannell]

I have upgrade the WLC to 17.9.4a and the same for the APSP, still the same. At least I didn’t have any issues with the upgrade this time

[sumithliyanage]

Does this mean have any issues with the previous upgrade ?

[stuart.pannell]

Yes I had some issues, I was upgrading from 17.6.4 to 17.9.3 using the GUI and it failed, had to manually run the upgrade from the CLI. Once the upgrade completed and everything restarted then I hit the Day0 fault where it thinks that it's a new install and you have to set the country code before you can get in via the GUI.

[sumithliyanage]

what is the AP software version after upgrading? 

example : 

From Version                  :  17.9.4.27
To Version                    :  17.9.4.201