Access Point is unable to connect to WLC on certain network link only (DTLS error)

[bennybyte]

Hi Mobility Community,

let me share one AP connection issue I´m struggeling with, hoping someone has a good idea.

We have one site with couple of different Cisco AP models connecting to a WLC 5508 v5.1.161.0 working fine. Everytime I activate a new redundant WAN link (dark fiber) at this site, all APs are loosing connection to WLC and are unable to connect again. Switching back to prior WAN link let Access Points recover. Symptom is moving with different ports, cables and fiber optics for WAN link and uplinks. Redundant dark fiber is connecting to same switches, same configuration, same network path.

All Access Points and test laptop connected to AP VLAN is getting correct IP and have Gateway and WLC connectivity at all times (on both WAN links, except the STP convergence time about some sec.). DNS is used for WLC discovery. When using redundant WAN link, WLC join stats show "unable to join" for APs with unknown reason.

I first thought about a firewall filtering CAPWAP/DTLS related traffic, but it seems CAPWAP traffic is reaching WLC.

A "debug capwap client info" on AP shows following logs about every 60 sec., endless looping while WLC claims "AP not able to join":

Received CAPWAP_DTLS_SESSION_DELETE_TIMER_EXPIRY Capwap Timer Msg.
Event = CAPWAP_DTLS_SESSION_DELETE_TIMER_EXPIRY(35) State = DTLS Teardown(4).
[CAPWAP] control firewall rule state 2new 0 old 1
[CAPWAP] data firewall rule state 2new 1 old 1

CAPWAP State: Discovery
...

Decode Discovery Response: WLC x.x.x.x(5246) in DiscResp[0]
Total msgEleLen = 108.
Allow expired MIC/SSC
Capwap control packet processed. Freeing packet 0xd80000.
[CAPWAP RX] CTRL: x.x.x.x[5246] -> x.x.x.x[5256] len 124
Received Capwap Control Msg From AC.
Rx unencrypted CAPWAP packet from x.x.x.x
Received Capwap Control Msg.
Control message: length = 124.
Msg Type = CAPWAP_DISCOVERY_RESPONSE(2) Capwap State = Discovery(2).
Discovery Response from x.x.x.x
Decode Discovery Response: WLC x.x.x.x(5246) in DiscResp[1]
Total msgEleLen = 108.
Allow expired MIC/SSC
Capwap control packet processed. Freeing packet 0xd7e000.
Received CAPWAP_DISCOVERY_INTERVAL_EXPIRY Capwap Timer Msg.
Event = CAPWAP_DISCOVERY_INTERVAL_EXPIRY(31) State = Discovery(2).
[CAPWAP] control firewall rule state 3new 1 old 0
[CAPWAP] data firewall rule state 3new 1 old 1

CAPWAP State: DTLS Setup
CAPWAP control packet sent to x.x.x.x

Any ideas?
Thanks
Benjamin

[Leo Laohoo]

The controller is running 5.X.X.X?
Good gosh. Version 5 was toxic. It was the mother-of-all-train-wrecks.

[bennybyte]

Hi Leo,
.... sorry for the typo. It’s 8.5.161.0 of course.... software update was one of the first steps I demanded. :-)

Thanks for the fast reply!

[Rich R]

Is it really a dark fibre or is it actually a switched/routed virtual ethernet connection?
If it's switched or routed I've seen service providers using in-line IDS/IPS which policed the CAPWAP UDP traffic causing connectivity issues for the APs.
Presume you've checked the obvious like all different size pings to make sure it's not an MTU problem or specific packet sizes getting dropped etc?

[bennybyte · Accepted solution]

Just a final update on this: The issue was solved by replacing the 3650 Access Switch on which the Access Points were connected. It turned out that the port-group of ports 1/1/3 and 1/1/4 was buggy, leading to malformated packets.

Thanks again for the suggestions!

Regards

Benjamin