Flexconnect on Slow Links

[Mohammed al Baqari]

Hi,

We had a deployment of centralised WLC (earlier 3504) with flexconnect configured for remote sites (centralized auth, local switching, ISE CWA. ISE, WAPs and clients are on same LAN). These sites are connected to the controller using satellite links (latency 700-800 msec). That was working without issues.

We migrated the WLC to 9800-CL on Azure (same latency) and started facing issues with CAPWAP throttling by the controller.  

My questions:

- Are these messages triggered by the latency between WAPs and WLC.?

- Are there any tweaks for timers to overcome these errors? Please suggest.

- In 9800 can we have local auth or it has to be center (this was a must in 3504)?

Here are sample messages:

May  8 09:58:41.373: %CAPWAPAC_SMGR_TRACE_MESSAGE-4-AP_MSG_THRESHOLD: Chassis 1 R0/0: wncd: Warning : Mac: 1cfc.17c6.5440 Session-IP:x.x.x.x[5273] x.x.x.x[5246] Capwap messages are queued for longer than 21 seconds, turning on client throttling. Queued messages : 36

May  8 09:58:58.661: %CAPWAPAC_SMGR_TRACE_MESSAGE-4-AP_MSG_THRESHOLD: Chassis 1 R0/0: wncd: Warning : Mac: 10a8.2980.1da0 Session-IP: x.x.x.x[5275] x.x.x.x[5246] Capwap messages are queued for longer than 20 seconds, turning on client throttling. Queued messages : 26

May  8 09:59:04.104: %CAPWAPAC_SMGR_TRACE_MESSAGE-4-AP_MSG_THRESHOLD: Chassis 1 R0/0: wncd: Warning : Mac: 70b3.1780.37e0 Session-IP:x.x.x.x[5264] x.x.x.x[5246] Capwap messages are queued for longer than 20 seconds, turning on client throttling. Queued messages : 23

[marce1000]

  - I am presuming the APs are still in Flexconnect mode ? Have a checkup review of the 9800-CL configuration with the CLI command show tech wireless ; have the output reviewed with : https://cway.cisco.com/wireless-config-analyzer/

 M.

[Flavio Miranda]

Hello

"- Are these messages triggered by the latency between WAPs and WLC.?"

 If could be. But when we see logs like this "Capwap messages are queued for longer than 20 seconds", also make think about processing and memory. Usually WLC have no problem with CPU and memory so delay on the link can be one possibility.

"- Are there any tweaks for timers to overcome these errors? Please suggest."

 I would take a look on the Link Latency paramenter on the WLC. You may extend the value to the maximum allowed.

"- In 9800 can we have local auth or it has to be center (this was a must in 3504)?"

   Support both

"This document describes how to configure FlexConnect with central or local authentication on Catalyst 9800 Wireles LAN controller."

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213921-flexconnect-configuration-with-central-a.html

[Mohammed al Baqari]

Many thx Flavio. With regards to local authentication, I read this one but I am using MAB for guest portal provided by ISE. The document is describing dot1x only. Is MAB supported using local authentication as well? If there is any document will be g8.

[Flavio Miranda]

Hi

Take a look here, maybe it can help you

 https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213922-configure-mac-authentication-ssid-on-cis.html

[Jolly99er]

What do you mean by "I would take a look on the Link Latency paramenter on the WLC. You may extend the value to the maximum allowed." Can you provide some documentation?

[gnburgos]

Hi Mohammed did you check this BUG https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwh89539

do you fix this issue? i have the same issue with wlc 9800-L-F with differente IOS version like 17.9.3, 17.9.4, 17.9.4a, 17.9.5, 17.3.6 and is the same with al versions massive AP´s disconnections if you have more information let us know please.

regards

[Rich R]

TAC opened CSCwh89539 for me.  They insist that it is cosmetic and not service affecting and that no traffic is actually being throttled.
It has nothing to do with link latency - it starts when the WLC is under heavy load - eg after reload or large number of APs trying to join, and then the affected APs remain in that state forever after with the counter constantly increasing to ridiculously large number of seconds.
With WLC on 17.9.4 + CSCwh31966 & CSCwh87343 SMU + APSP6 the APs have been stable in spite of those log messages which tends to confirm that the messages are only cosmetic.

@gnburgos you'll need to troubleshoot the exact cause of your AP disconnections with TAC - there's a whole lot of things in the setup which could cause this as well as various bugs which could be causing it.  Without knowing any detail of your setup impossible to say.  Use the Config Analyzer (link below) to check your config as a starting point and fix any major issues highlighted in red.  Also check the Best Practices guide (link below).

[gnburgos]

@Rich RThanks for the information about this issue, for us is complicated because when we have the message refer to CSCwh89539 this generate massive ap disconnections we already use the config analyzer and basically follow all the best practices recomended from TAC, by the way last week we upgrade the Rommon. whats happend to you when you got the this messages do you have massive disconnections? or just see the mesage in the log? like a cosmetic message? Please any info that you can share will be useful for us.

Regards

[Rich R]

The bug was opened while we investigated disconnections and other issues with TAC.
But the messages continue even after the issues are resolved so do seem to be only cosmetic.

[gnburgos]

@Rich R  appreciate your information, in my case is different with the message come the ap´s disconnection and the only way to have the ap´s back joined is doing a redundancy force-switchover. and we are not able to take captures because customer dosnt allowed us to have service down. thanks again!!!

regards

[Rich R]

I think the messages are a symptom of the underlying problem rather than the cause.

As I said the messages start when the WLC is under heavy load, so it's that load which is likely causing the problems with the messages and disconnections being the result.

Concentrate on what might be causing the load on the WLC. 17.9.4 + CSCwh31966  & CSCwh87343  SMUs + APSP6 has been stable for us but obviously every network is a little different so it might not be for you.

How many APs and client do you have on the 9800-L-F ?

[gnburgos]

@Rich Rwe have 159 AP's and more less 1100 users, we can see that the controller is healthy.

[Rich R]

So you've been monitoring WNCd and CPU never goes high?