Flexconnect APs stopped joining

[andrew.chappelle]

Hello Community,

A customer of mine has a centralized 2504 WLC with 7.2 code running.  They have 1142N APs deployed locally as well as in remote sites (3) in FlexConnect mode.  For no apparent reason last Thursday all the remote APs disassociated with the controller and could not rejoin.  All the local APs remained up and unaffected.

No changes to the WLAN, LAN, Firewall or MPLS WAN occured to cause this.

The customer opened a TAC case and their determination was that ports 5246-5247 were not getting thru.  When the customer engaged me this morning I had him run a packet capture on the Sonicwall firewall to prove out if the CAPWAP signals were leaving and returning across the WAN.  Sure enough we can see this bi-directional traffic (pic attached).  Also, I had the MPLS provider run a trace at the far end and they see the same traffic leave the remote site. 

And then an odd thing happened; one of the APs at one of the remote sites all of a sudden Joined the controller.  So I tried rebooting the AP that is located in the same office, and it fails to Join.  When I look on the controller under AP Join statistics, the last activity shows the controller receiving a Discovery Request and response is sent, but no further Config Request and response or Join Request and response.

Frankly a little stumped, we are re-engaging TAC, but thought that maybe someone in the community has run across this scenario before. 

I also thought that it may be good to upgrade to 7.4, just on spec.

As always any feedback, advice and comments are appreciated and welcome.

Thanks.

[Scott Fella]

Upgrade isn't always the answer... For example, there is a major bug with v7.4 and 1142's. this would of caused you
More issues because the 1142 would just stop working and you would need to reboot the AP.... This would happen every day or so and it depends so v7.4 should not be used. There is a TAC version of v7.4 that fixes this, but you will need to request that from TAC.

When the APs disassociated from the WLC, did they join another WLC by chance?

Sent from Cisco Technical Support iPhone App

[andrew.chappelle]

As always, thanks for the quick reply Scott.

So, good comment on the upgrade, wasn't aware of that, so I'll shelve that idea for now.  Funny how first instinct is to fix by upgrading.

There's only the one WLC, so no, the APs just stay disassociated.

Funny thing tho, and I'm not sure it would make any difference or how it happened.  One of the APs that can't join, when I do a #show controllers dot11radio1, the country set shows as US, but the country code on the controller is CA for Canada.  Being that both are A-region, I wouldn't think that should make a difference.  I wonder if doing a #world-mode command and force the country code IE to CA would make a difference?

thank you,

Andrew

[Scott Fella]

It can, but don't start changing too much stuff:). You should be able to see in the ap logs if the ap has been denied due to the country code. Maybe just add the country code and see what happens.

Sent from Cisco Technical Support iPhone App

[Scott Fella · Accepted solution]

On thing I would try is to stage a new Ap or take one that is up locally and place it on one of the remote site. If the Ap doesn't join, I bet something is blocking udp 5246 and or udp 5247. The funny thing is that this happened to one of my clients and it was because someone was playing around with ACL's and happened to block these ports. It's worth a shot to see if its the network or not. I'm betting its something blocking somewhere.

Sent from Cisco Technical Support iPhone App

[andrew.chappelle]

I just checked, and that world-mode command isn't even available on the radio interface, so it must only be available for Autonomous APs.  I wondered if that was the case.  I'd love to try the remote AP swap, but the closest remote site is about 2000km away.  may come to that.  I'll see if they have a spare lying around we can try that with.

When I look at the AP Join stats for a failed AP, theres a Discovery Request Received, and a Discovery Response sent, but nothing past that.

When I do a #show ap join stats detailed the reason for failure shows as "Unavailable"

I did confirm that all the working APs have a country code of CA, which makes sense if they just received that from the controller upon joining, so that may just be a rat-hole I'm chasing down.

We do see that traffic leave and come back across the edge firewall, so it must be traversing, just not the full process.  If it was blocked it should be totally blocked, not mid-way thru?

[Scott Fella]

As long as you know the route hasn't changed at all and the provider isn't dropping it in accident, then I really don't know what to say. If you have any old ap sitting around I would use that. It's hard to believe that all of a sudden multiple sites stopped working but the local AP's continued to work. Since nothing CCO aged in the WLC, there is nothing you can do to really have these AP's join again unless you figure out if a local ap that you have joined to the WLC can join when it's at a remote site. That will tell you if udp 5246 and udp 5247 is being blocked/dropped.

Sent from Cisco Technical Support iPhone App

[andrew.chappelle]

So, the final resolution was that the Sonicwall was fragmenting the 5246 packets into 3 parts and only forwarding two of the packets occasionally.  Because it wasn't explicitly dropping the one fragment, it didn't register as a dropped packet.  In the end, we bypassed the Sonicwall (which is being replaced) and sent the traffic directly to the WAN router, and trouble appears to be resolved.  Scott, thank you as always for the feedback and advice, always appreciated.

[Scott Fella]

Thanks for providing the resolution. I too have had issues with them dang Sonicwalls. Oh well at least it's working and you know everything else is fine.

Sent from Cisco Technical Support iPhone App