8510 WLC AP's randomely disconnect and SSID's stop

[leighs]

HI,

I'm really hoping there's a simple solution!!!

I have a pair of 8510 WLC's in HA. Running 8.5.161.0, I've just upgraded to 8.5.171.0 (current "golden Image')

There are 1907 AP's joined most of the time. However, there are 2 issues being experienced randomly;

1. Randomly AP's disappear  off the controller and later rejoin

2. Some of the AP's radios randomly "freeze" or stop broadcasting and clients loose connectivity. then later they may re-enable randomly

I've looked through the logs and see a number of messages concerning no Pre Auth. Then there are others with RX and TX error messages but the code ends up being "Unknown"

Has anyone a similar experience??

[Kasun Bandara]

Hi,

I suggest to upgrade for next version (if any available) or contact TAC.

[leighs]

Hi Kusan,
Thanks a million,
I specifically upgraded to the version for the 8510 that was selected as the Cisco 'golden image' . My understanding is that, that is considered the most stable at this time, or am I missing something??
It's still doing it though. It's a bit of a mission to upgrade as there are 1900 AP's with about 3500 client connected most of the time.
I will raise a TAC with the logs and show tech to see what they come back with
Thanks again for the input
Leigh

[Rich R]

There is a known bug https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv78719 which might be related to the problems you're seeing but currently no fix for that in 8.5  The "fix" in 8.10 is just a workaround which allows the AP to recover automatically after 20 seconds while providing extra diagnostic info for the developers.  Note the workarounds - try disabling MU-MIMO.

Be warned that 8510 is past the end of software maintenance milestone so if they determine it's a new bug you're unlikely to get a fix.  CSCvv78719 might still get fixed in 8.5 for 8540 in which case 8510 might inherit the fix by default.

https://www.cisco.com/c/en/us/products/collateral/wireless/8500-series-wireless-controllers/eos-eol-notice-c51-740222.html

You're correct about 8.5.171.0 being the current TAC recommended release.  They've cautioned against 8.5.182.0 because of a new bug which causes reloads although we've had it in production for 2 weeks so far without seeing any problems.

https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc10

I expect there will be another release soon with a fix for that bug.

[leighs]

Hi,

Thank you for the update.

We'll ook into the feasibility of implementing the workaround in the environment and the client must make a choice.

What we did see though, was that when we failed over the HA pair to the standby unit the situation did improve, but the issue still persists. Not sure why that had any impact at all though?

Leigh

[Leo Laohoo]

I agree with @Rich R about CSCvv78719.  When this bug hits, the AP blackholes everything, including DHCP requests.  

Try rebooting the AP (kill the PoE to the AP) and see if this makes any difference.  

I have seen this happening in 8.10.X.X and the only way is to reboot the APs to get some kind of relief.