2800 AP diassociating to WLC oftenly after upgrade.

[AK002]

Hi, 

Can Someone help me on the Issue in the 2800 Access points which is integrated with 5520 controller. 

All AP's in the WLC randomly re-joining with the controller and giving the below error in the AP logs. Below logs are repeating many times in a day and getting alarm for re-joining. Issue in both Flex and local mode AP's. 

WLC version : 8.5.131.0

Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.8012] Warning, unencrypted data keepalive failed
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.8013] Failed to receive data keepalive
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.8013]
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.8013] Lost connection to the controller, going to restart CAPWAP...
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.8013]
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.8013] Restarting CAPWAP State Machine.
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.9069]
Aug 21 11:06:43 kernel: [*08/21/2018 11:06:43.9069] CAPWAP State: DTLS Teardown
Aug 21 11:06:44 NCI: CLEANAIR: Slot 0 CAPWAP down
Aug 21 11:06:44 NCI: I0: shutdownNci
Aug 21 11:06:44 NCI: CLEANAIR: Slot 1 CAPWAP down
Aug 21 11:06:44 NCI: I1: shutdownNci
Aug 21 11:06:44 kernel: [*08/21/2018 11:06:44.9164] Dropping dtls packet since session is not established. Peer 10.130.208.101-
5246, Local 10.130.20.71-5272, conn (nil)
Aug 21 11:06:44 kernel: [*08/21/2018 11:06:44.9165] Dropping dtls packet since session is not established. Peer 10.1.1.1.-
5246, Local 10.130.20.71-5272, conn (nil)
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6079]
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6079] CAPWAP State: Discovery
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6103] Discovery Request sent to 10.1.1.1, discovery type STATIC_CONFIG(1)
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6105] IP DNS query for CISCO-CAPWAP-CONTROLLER
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6128] DNS resolved CISCO-CAPWAP-CONTROLLER
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6128] DNS discover IP addr: 10.1..1.1
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6157] Discovery Request sent to 10.1.1.1, discovery type STATIC_CONFIG(1)
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6168] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6293] Discovery Response from 10.1.1.1
Aug 21 11:06:49 kernel: [*08/21/2018 11:06:49.6377] Discovery Response from 10.1.1.1
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.0000]
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.0000] CAPWAP State: DTLS Setup
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.7361]
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.7361] CAPWAP State: Join
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.7390] Sending Join request to 10.1.1.1 through port 5272
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.7611] Join Response from 10.1.1.1
Aug 21 11:06:59 kernel: [*08/21/2018 11:06:59.8475] HW CAPWAP tunnel is ADDED
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.8892]
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.8892] CAPWAP State: Image Data
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.9262] do NO_UPGRADE, part1 is active part
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.9300]
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.9300] CAPWAP State: Configure
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.9322] DOT11_CFG[0] Radio Mode is changed from FlexConnect to FlexConnect
Aug 21 11:07:00 kernel: [*08/21/2018 11:07:00.9330] DOT11_CFG[1] Radio Mode is changed from FlexConnect to FlexConnect
Aug 21 11:07:01 kernel: [*08/21/2018 11:07:01.2748] CAPWAP HW tunnel params changed, DELETING the existing
Aug 21 11:07:02 kernel: [*08/21/2018 11:07:02.3321] HW CAPWAP tunnel is ADDED
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5692] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5693] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5693] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5694] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5694] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5695] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:03 kernel: [*08/21/2018 11:07:03.5696] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(
8).
Aug 21 11:07:04 kernel: [*08/21/2018 11:07:04.0322]
Aug 21 11:07:04 kernel: [*08/21/2018 11:07:04.0323] CAPWAP State: Run
Aug 21 11:07:04 kernel: [*08/21/2018 11:07:04.0641] CAPWAP HW tunnel params changed, UPDATING the existing
Aug 21 11:07:05 kernel: [*08/21/2018 11:07:05.1593] AP has joined controller WLC1
Aug 21 11:07:05 kernel: [*08/21/2018 11:07:05.1722] Flexconnect Switching to Connected Mode!
Aug 21 11:07:06 syslog: Password for xxx changed
Aug 21 11:07:06 kernel: [*08/21/2018 11:07:06.0844] Password for xx changed
Aug 21 11:07:06 kernel: [*08/21/2018 11:07:06.9100] save_on_failure is set to 1
Aug 21 11:07:06 kernel: [*08/21/2018 11:07:06.9103] save_on_failure is set to 1
Aug 21 11:07:07 root: SYS-COND: Retrigger fair condition, 6 power
Aug 21 11:07:34 kernel: [*08/21/2018 11:07:34.1928] set cleanair [slot0][band0] enabled
Aug 21 11:07:34 NCI: I0: openSensor(slot=0)
Aug 21 11:07:34 kernel: [*08/21/2018 11:07:34.2123] set cleanair [slot0][band1] enabled
Aug 21 11:07:34 kernel: [*08/21/2018 11:07:34.2318] set cleanair [slot1][band1] enabled
Aug 21 11:07:35 NCI: I0: SensorApp=1.15.4
Aug 21 11:07:35 NCI: I0: SensorHdw=1.2.3.0
Aug 21 11:07:35 NCI: I0: Hardware Radio Band = [4890, 5935] MHz, BW=150625
Aug 21 11:07:35 NCI: I0: Hardware Radio Band = [2400, 2500] MHz, BW=150625
Aug 21 11:07:35 NCI: slot=0 mode=2 chanCnt=1 cw=1
Aug 21 11:07:35 NCI: chans: 36 0 0 0 0 0 0 0 0 0 0
Aug 21 11:07:35 NCI: I0: channel map channels: in=1 cloned=1
Aug 21 11:07:35 NCI: I0: Requesting MonBand [5180, 5180] bw=20MHz/0 ant=0xbc
Aug 21 11:07:35 NCI: I0: Monitoring (cf=5180, span=20), RadioUsage=3%
Aug 21 11:07:35 NCI: I0:     dwell=20000us, update=1000ms, resBW=78126
Aug 21 11:07:35 NCI: CLEANAIR: Slot 0 enabled
Aug 21 11:07:35 NCI: I1: openSensor(slot=1)
Aug 21 11:07:36 NCI: I1: SensorApp=1.15.4
Aug 21 11:07:36 NCI: I1: SensorHdw=1.2.3.0
Aug 21 11:07:36 NCI: I1: Hardware Radio Band = [4890, 5935] MHz, BW=150625
Aug 21 11:07:36 NCI: slot=1 mode=2 chanCnt=1 cw=1

[Sandeep Choudhary]

1. are AP and WLC in same subnet !

2. check your connection between AP subnet and WLC subnet!

Regards

Dont forget to arte helpful posts

[AK002]

Hi Sandeep,

Local mode Ap's in same subnet and Flex Ap's are in different Subnets.. But both are having issues.

Regards,AK

[Leo Laohoo]

What version did you upgrade from? 8.2?
Have you tried rebooting the AP?

[AK002]

Hi Leo,

Upgraded from 8.3.133.0 to 8.5.131.0

Not tried any AP reboot, As we see this issue in all the AP's in the WLC. Most of the AP's are 2800 model.

Regards,

Arun

[pieterh]

check you time sync environment
Aug 21 11:07:06 syslog: Password for xxx changed
Aug 21 11:07:06 kernel: [*08/21/2018 11:07:06.0844] Password for xx changed
may suggest the periodic changing of the key between AP and WLC is not synchronized properly.

[AK002]

Hi, How to do that time Sync. 

All AP's generally in UTC 0 and it will get sync from WLC only. 

Do you need to check show clock/time. 

Regards,AK

[AK002]

Hi, 

WLC NTP server is in sync only. 

Cisco Controller) >show time

Time............................................. Wed Aug 22 11:30:03 2018

Timezone delta................................... 0:0
Timezone location................................ (GMT) London, Lisbon, Dublin, Edinburgh

NTP Servers
    NTP Version..................................     3
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ---------------------------------------------------------------------
       1              0                               10.11.1.2                In Sync              AUTH DISABLED

AP Clock: 

***** show clock *****
*13:04:51 UTC Mon Aug 20 2018

[AK002]

Hi, 

Can anyone please help on this error..

[Leo Laohoo]

Upgrade to 8.5.135.0.

[Vivek_Shukla]

I have exactly same setup and hitting the same bug after upgrade from 8.3.133 to 8.5.131 . I have a TAC case open for the same and as suggested by TAC, upgraded to 8.5.135 .. 

I can confirm upgrading to new code is of NO USE. 

TAC is still investigating but I am planning on removing LAG from the APs to see if this helps with disconnections. 

[Vivek_Shukla]

Arun, have you found any solution to this.

[cae_technology]

Same the TAC is suggested to remove the LAG from the 2800/3800 Model access points. Also try to upgrade to 8.5.135.

Please remove LAG from the AP's in order to stop random channel change in the 5ghz..Also change the AP Re-transmit Config Parameters from 5/3 to 8/5 to stop the AP random disassociation. 

[Vivek_Shukla]

Removed LAG from few APs in our environment yesterday. Since then those APs are stable. As per TAC this issue is resolved in version 8.5.137 which will be release in next month. 

[Vivek_Shukla · Accepted solution]

Just a quick reply for everyone. I removed LAG from all of the APs in our env and since then all APs are stable. No more disconnections or channel changes. :)