Lower throughput on Flex SSIDs compared to Centralized SSIDs - WLC9800

[Devinder Sharma]

Hello all,

I can use some advice on troubleshooting this issue. I have case opened with support for few days but have not made much progress. I am working on a project with Qty 2 of WLC9800 in HA and 220 AP9115ax.  WLCs are running latest TAC recommended 17.9.2 code. 300 Flexconnect APs on a single site have been supported for some time now.

We are doing some testing with parallel deployment of controllers with existing and with few new APs. Everything is working fine, but we wanted to test the max throughput available (via simple speedtest to Internet). The APs are terminated on catalyst 3850 / 9200 switches which in turn uplink to core 9400 via 10Gig each. The WLCs are connected via a portchannel  of 2 copper ports each to core  1Gig ports. Internet pipe is 2.5Gig commercial service on a 10Gig access fiber.

A guest SSID with PSK is setup for centralized switching and speed tests with APs set up in few locations, when measured with newer AX laptops is close to 750Mbps up and down (and with 200 plus AP1800/2800 still transmitting thru older 5K controllers). The AP9115x are set for best channel bandwidth and settle in 80MHz channels. APs are set up as flexconnect mode.

Then we added a 802.1x SSID in local switching mode with centralized authentication to NPS radius. The speeds achieved on same test clients drop down to 450Mbps.

Then I added a PSK SSID in local switching mode and speeds again on same test clients is close to 450Mbps up and down.

The same AP when sending encapsulated traffic direct to controller thru same uplinks results into 750Mbps speeds, but when the AP has to work to put a tag / untag on locally dropped traffic, it seems to gets bogged down to 450Mbps. Is this expected behavior from a 9115AX AP?

Thanks much in advance.

[marce1000]

 - For starters  review the WLC9800   configuration with the CLI command (on the current active controller) : show  tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.               Checkout all advisories!

 M.

[Devinder Sharma]

Thanks Marce. There were no errors and only 4 or 5 warnings. 2 of them were incorrect as it thinks that it is looking at AireOS (based on proposed commands to fix them) and indicated NTP server missing and HA wireless mobility mac address not configured, while both are correctly configured and NTP associations show synced to public NTP server. Another one was on SI and ARP proxy recommendations. Anyway, no errors were found, and I did not expect them as everything is working fine, but just this bit lower speed on flex SSIDs. 

[Rich R]

Generally speaking the throughput should be the same or better for locally switched because centrally switched is getting CAPWAP tunnelled to the WLC.
But it might be something like TCP MSS adjust making the difference if your locally switched traffic is getting fragmented while the centrally switched isn't.

Get a packet capture of a client on each and see what's different at packet level.  If they're both the same then you might need a TAC case.  Presume you've checked everything else in the path and there are no other differences that could account for that?

[Devinder Sharma]

Hi Rich, the TCP MSS under AP join profile is at default 1250 (used to be default 1363 or something a while ago on AireOS and then it was reduced to 1250 to fix some issues with Intel drivers). So there should be no question in local or CAPWAP tunneled traffic to result into fragmentation and reassembly. I have checked everything else and there is no reason for lower speeds at locally switched SSID, unless as I think it has to do with some extra overhead for AP to incur to tag / untag the traffic. I wont be onsite for couple of days and then see if we can do a packet capture to compare for two scenarios. 

Thanks for your advice. Let me know meanwhile if anything else should be looked into.

[Rich R]

So I still think it could be MSS.  The older AireOS docs (up to 8.4) say "TCP Adjust MSS is supported only on APs that are in local mode or FlexConnect with centrally switched WLANs".  This has been removed from docs since 8.5 and the 9800 docs refer to routers (!!!) but even the later AireOS docs talk about it applying to client traffic through the CAPWAP tunnel so I don't think it is applied to flex local switched traffic.  So I think it's up to you to apply TCP MSS adjust on your locally switched gateway router.  Have you tried that with the same value (1250) as the centrally switched traffic?

[Devinder Sharma]

Hi Richard, this is interesting point researched by you. Even if I were to apply the command ip tcp adjust-mss 1250 under gateway (which will be core switch SVI), this will then equally apply to any wired or wireless traffic. While this adjusts end to end traffic to use lower mss for tcp traffic from client, this will potentially lower overall the achieved speeds with more overhead at least for wired connections, and my understanding is that higher speed consuming traffic will generally be 4K video, which will be TCP (as YouTube uses TCP) and also speedtest.net uses TCP.  Other services might use TCP as well for streaming. Another thing is with MSS to be 1250, the segments larger than this will be dropped if some NIC drivers fail to adjust MSS down to 1250. I will need to test it during maintenance windows when we will install, I would like to hear more if there could be side effects if applied globally to the SVI on core switch. I have never used TCP MSS adjust anywhere lower than 1360 in the past. Thanks again.

[Rich R]

Valid concern as it will affect all traffic but it could also improve performance for all traffic.  MSS is a soft feature at layer 4 - it' not "enforced" anywhere.  It's just a convenient way of telling the endpoints to use an optimal value to avoid fragmentation. If anything ignores it (I've never seen that happen) it won't make any difference - it would just be as if you hadn't adjusted MSS at all.  They won't get dropped - they'll be fragmented - just like they are today which ultimately reduces performance.

Also 1250 is what works best for CAPWAP + other overheads for AP<->WLC transit but you could probably set it higher for the local traffic.  This is where you need packet captures to see what's actually happening.  You could experiment with different values.

[Arshad Safrulla]

Devinder,

Best way to test the speed of your wireless environment is 

1. Isloate the AP registered to 9800 as much as you can (to reduce the interference from other APs), take 2 APs and advertise one SSID locally switched in one AP and centrally switched in other AP. Make sure that these 2 APs are in non-overlapping channels in both 2.4 and 5.
2. Hardcode the channel width (do not depend on dynamic "best" option), depending on the channel width you can create multiple test cases.
3. Upgrade the drivers of the clients connecting to the AP to the latest, if you want try other drivers you may create test cases appropriately.
4. Speedtest shoud be done using iperf where the server is sitting locally connected to your switching infra, not public internet speedtests. Public speedtest servers are just eye candy, my personal opinion is that shouldn't be used to define the performance.

There shouldn't be any overhead on the AP when its on flex mode, unless it is in standalone mode. (even in standalone mode impact should be very minimal)

[Devinder Sharma]

Thanks Arshad. We had already ruled out any bottlenecks on underlying wired network last week with iperf tests from clients wireless vlan to server vlan and we were easily getting 900Mbps up and down on two test devices, Both test devices are very new with latest updates and drivers. Regardless, how can then you explain getting 750 to 810Mbps even with speedtest thru same test client thru same AP but on two different SSIDs, consistently. And RF environment if had any bearning here, will not make one SSID 200Meg faster. And this late afternoon, we did a remote session with TAC engineer who spent an hour and ruled out any configuration or RF issues and he is going to discuss internally and leaning towards same conclusion that I had, that it could be capability of 9115 to handle tagged traffic. With CAPWAP tunnel for centralized SSID, tunnel is pre-established, while for local switched traffic, AP needs to tag . untag on each frame basis. 

[Scott Fella]

The best way is to test apples to apples. Use the same device for testing and test with iPerf running on a wired device. Test with the iPerf server on the same subnet and then on a different subnet. I don’t think 80mhz is something you should use with that many access points but that is just my opinion. 
For another test, can you setup a FlexConnect ap on the 55xx controller and also test with the same end device? 
Once you collect some data, you can even setup an iPerf server in another location to test bandwidth speeds. This way you are not using a Speedtest for your troubleshooting purposes. 

[Devinder Sharma]

Hi Scott, thanks for your inputs as well. I believe, I have answered most of these questions in my response above. And I fully agree to not use 80MHZ width and we were not going to, as 40MHz will be more than enough and we don't want a single client to monopolize too much of Internet pipe anyway. The reason we were doing this in our test environment was to test the limits of the new gear. We were going to slide the max bandwidth from 80 to 40MHz while keeping best as option for channel width. 

[Devinder Sharma · Accepted solution]

Hello All,  I have tried making the same Flex SSID as Centralized SSID and I could immediately see the speeds improve. So that ruled out issues with the SSID configuration itself. I also changed MSS to 1300 at SVI and that overall lowered the speeds by about 100meg average over multiple speed tests, compared to default. There are few old external antenna APs onsite that are being serviced by 5520 controllers, so I have not done any firmware update on these to support 9115 to check the situation on Air-OS. The wireless network has pretty much migrated and we have assumed that 9115 is not capable of providing higher speeds because of tagging / untagging overhead. Thanks again to everyone who helped with their valuable suggestions.

[Rich R]

That doesn't sound right - you should open a TAC case to get them to help you diagnose the issue.

[Devinder Sharma]

of course, that was done several days ago, and they could not figure it out.