Re: AP will lose association with controller and then reassociate.

saleff · ‎11-06-2014

I have 240 AP's on a 5508 controller. The AP's are at 12 different locations and they all stay associated except for one location. This location has 2 AP's and when they lose the association with the controller it is always both AP's at the same time. I have nothing in the logs on the switch they are connected to, the ports on the switch are not dropping. My link to the building shows it is stable, but I still have this problem. They associate with the controller after about 15 seconds.

Log on AP

*Nov 6 13:21:30.035: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 11)
*Nov 6 13:21:30.035: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Nov 6 13:21:30.035: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.50.2.20:5246
*Nov 6 13:21:30.083: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Nov 6 13:21:30.083: %LWAPP-3-CLIENTERRORLOG: LWAPP LED Init: incorrect led state 255
*Nov 6 13:21:30.095: %LWAPP-3-CLIENTERRORLOG: NumOfSlots Mismatch Reinit all Radios config rcb:2 Cfg:0
*Nov 6 13:21:30.095: %CAPWAP-3-ERRORLOG: Binding Config Initialization failed for binding 1

*Nov 6 13:21:30.103: %CLEANAIR-6-STATE: Slot 0 down
*Nov 6 13:21:30.107: %CLEANAIR-6-STATE: Slot 1 down
*Nov 6 13:21:30.107: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*Nov 6 13:21:30.107: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*Nov 6 13:21:30.111: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 13:21:30.135: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 13:21:31.107: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 13:21:31.135: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Nov 6 13:21:31.143: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov 6 13:21:32.127: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 13:21:32.135: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Nov 6 13:21:32.163: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 13:21:32.171: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Nov 6 13:21:32.179: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 6 13:21:33.163: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Nov 6 13:21:33.171: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 13:21:33.199: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 13:21:34.199: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 13:21:40.135: %CAPWAP-3-ERRORLOG: Selected MWAR 'Central 5508'(index 0).
*Nov 6 13:21:40.135: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Nov 6 13:21:30.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.50.2.20 peer_port: 5246
*Nov 6 13:21:30.523: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.50.2.20 peer_port: 5246
*Nov 6 13:21:30.523: %CAPWAP-5-SENDJOIN: sending Join Request to 10.50.2.20
*Nov 6 13:21:30.527: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*Nov 6 13:21:30.527: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.
*Nov 6 13:21:30.527: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Nov 6 13:21:30.527: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.50.2.20
*Nov 6 13:21:30.899: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Nov 6 13:21:30.959: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 6 13:21:31.167: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 13:21:31.531: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller Central 5508
*Nov 6 13:21:31.783: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Nov 6 13:21:31.899: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 13:21:31.991: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Nov 6 13:21:31.999: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov 6 13:21:32.167: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Nov 6 13:21:32.983: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 13:21:33.027: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 13:21:33.035: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Nov 6 13:21:33.043: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 6 13:21:34.027: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Nov 6 13:21:34.035: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 13:21:34.067: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 13:21:35.067: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 13:22:03.635: %CLEANAIR-6-STATE: Slot 0 enabled
*Nov 6 13:22:05.695: %CLEANAIR-6-STATE: Slot 1 enabled
*Nov 6 15:15:10.739: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Nov 6 15:15:10.743: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov 6 15:15:11.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Nov 6 15:15:11.771: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 15:15:12.771: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Nov 6 15:15:29.711: %CLEANAIR-6-STATE: Slot 1 down
*Nov 6 15:15:46.491: %CLEANAIR-6-STATE: Slot 1 enabled
*Nov 6 15:32:26.611: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 38)
*Nov 6 15:32:26.611: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Nov 6 15:32:26.611: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.50.2.20:5246
*Nov 6 15:32:26.659: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Nov 6 15:32:26.663: %LWAPP-3-CLIENTERRORLOG: LWAPP LED Init: incorrect led state 255
*Nov 6 15:32:26.671: %LWAPP-3-CLIENTERRORLOG: NumOfSlots Mismatch Reinit all Radios config rcb:2 Cfg:0
*Nov 6 15:32:26.671: %CAPWAP-3-ERRORLOG: Binding Config Initialization failed for binding 1

*Nov 6 15:32:26.683: %CLEANAIR-6-STATE: Slot 0 down
*Nov 6 15:32:26.683: %CLEANAIR-6-STATE: Slot 1 down
*Nov 6 15:32:26.683: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*Nov 6 15:32:26.683: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*Nov 6 15:32:26.695: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 15:32:26.711: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 15:32:27.687: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 15:32:27.715: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Nov 6 15:32:27.723: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov 6 15:32:28.707: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 15:32:28.715: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Nov 6 15:32:28.743: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 15:32:28.751: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Nov 6 15:32:28.759: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 6 15:32:29.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Nov 6 15:32:29.751: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 15:32:29.779: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 15:32:30.779: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 15:32:56.711: %CAPWAP-3-ERRORLOG: Selected MWAR 'Central 5508'(index 0).
*Nov 6 15:32:56.711: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Nov 6 15:32:47.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.50.2.20 peer_port: 5246
*Nov 6 15:32:47.483: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.50.2.20 peer_port: 5246
*Nov 6 15:32:47.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.50.2.20
*Nov 6 15:32:47.491: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*Nov 6 15:32:47.491: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.
*Nov 6 15:32:47.491: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Nov 6 15:32:47.491: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.50.2.20
*Nov 6 15:32:48.371: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Nov 6 15:32:48.379: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 6 15:32:48.499: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller Central 5508
*Nov 6 15:32:48.555: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 15:32:48.767: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Nov 6 15:32:49.371: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 15:32:49.411: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Nov 6 15:32:49.419: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov 6 15:32:49.555: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Nov 6 15:32:50.403: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 15:32:50.447: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 6 15:32:50.455: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Nov 6 15:32:50.463: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 6 15:32:51.447: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Nov 6 15:32:51.455: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov 6 15:32:51.487: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 6 15:32:52.487: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov 6 15:33:18.559: %CLEANAIR-6-STATE: Slot 0 enabled
*Nov 6 15:33:20.623: %CLEANAIR-6-STATE: Slot 1 enabled

Controller

UP Time

Controller Associated Time

Controller Association Latency

Dhiresh Yadav · ‎11-06-2014

Hi,

*Nov 6 15:32:26.611: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

Looks like it is loosing the heartbeat with the WLC.Just try to increase that value if that helps.

Is proper QOS in place to prioritize Capwap packets.

Regards

Dhiresh

**Please rate helpful posts**

Leo Laohoo · ‎11-06-2014

What firmware is your controller running on?

saleff · ‎11-07-2014

7.6.120.0

Leo Laohoo · ‎11-07-2014

Ok, try upgrading your firmware to 7.6.130.X and see if this issue re-occurs.

Prasan Venky · ‎11-08-2014

Exactly Mr.Leo.

I faced the same issue and after upgrading it to 7.6.130.0. It worked for me.

JC-SE · ‎10-10-2022

We had a similar issue recently. APs kept losing association with the controller and then reconnecting for short periods of time. After doing a packet analysis, I found that there was a device flooding the network with UDP traffic. It was a device with dual NICs. One was supposed to be on our network and the other was supposed to be on a separate network. Once I removed the device from the network our APs were on, the APs were back to functioning normally.