2504 running on 8.5.151.0 - bandwidth issues

[danailpetrov]

Hi guys,

I am having some weird problems with my 2504 controller running on 8.5.151.0 

I noticed today that I am unable to achieve more than 30-32Mbps on an individual device. It's like there is a hard-cap that doesn't let the device getting more bandwidth. 

I have played alot, trying to tweak different QoS settings, but apparently 2500 is very limited in configuration of the bandwidth rates, burts and traffic profiles. I was unable to find a way to completely disable the qos for an SSID and just let it be on "best effort". 

My questions are: 

* are there any built-in, default policers for different classes and if so - what are the values? 

* Is there anyway possible I can completely switch off that behaviour (preferably on a per SSID basis?)

* Is that an 8.5.151.x problem because I don't recall having this problem on older versions

[marce1000]

 - What AP-models are you using ?

    M.

[danailpetrov]

Hi, apologies I forgot to mention this - AIR-AP1852I-E-K9. 

All my devices support 802.11ac/Wave2 and I am literally under one of the APs (with RSSI of -44). 

I noticed that my pings to other devices on the network (wired ones) are very strange. They progressively increas from 2-3-5-50-110-160-200-280ms and then normalize .. again back to 1-2ms .. and this is happening every 5-10 sec. It looks like some sort of a buffer / queueing that kicks in :| 

P.s. some screenshots

Wireless status on my macping to a wired device (member of the same VLAN)

[Scott Fella]

Looks like either a nic driver issue, a configuration issue or duplicate address on your network. Have you verified all these?

[danailpetrov]

Hi Scott & thanks for your inputs. 

I have tested with numerous of devices, all of which (for good or worse) are apple. 

The results I observe are identical - I cannot achieve more than 32-35Mbps using a basic "Speedtest.net" test. Now I know this is not the best measurement tool, but I do get a similar results copying (AFP/CIFS) from a server connecte to the same switch where the WLC is connected to and I cannot get more than ~4MBs (32-35Mbps) even there. 

I have two iPads, two phones and three Apple computers (2xMacBook pro's & 1Mac Pro) and they're all having the same results (when it comes to the speedtest). If I connect the device physically to the switch - I get the full bandwidth (CAR) from the ISP. 

It's very weird issue, but I have done 60-65MBs (~500Mbps) using CIFS on same equipment a few months back, with exactly the same config and everything and today - I can barely get 1/20th of that - it's just weird... 

None of the devices (WLC) nor the switch report any CRCs or any other type of errors... 

[Scott Fella]

I have a 2504/5520/9800-L at home an I do testing and never had any issues with throughput. I have around 40+ devices connected and 8 access points I use to test. What I can say is look at your configuration and make sure there are no duplicate address in your network that might overlap with your interfaces on the controller. Or create a test SSID and map that to a different vlan and test again. I would make sure that you test with an open said and psk with bare minimal features enabled.

I also test with iperf3 locally and only reply on speediest to check my internet. Make sure your device is connected on 5ghz and not 2.4ghz or try both and see where its limited. Higher throughput happens on 5ghz with channel bonding of 40mhz or more. You probably don’t want to go more than 40 if you have a dense deployment.

One other test you can do is flexconnect local switching, in which your client traffic will agree the ap and get placed on the switch. This way the traffic is not tunneled back to the controller.

[danailpetrov]

Thanks Scott.

 I have roughly around 42 devices too, but only 5 APs. When you say "look at your configuration" - can you elaborate, which bit would you advise me looking at? Again, I am very confident there is no IP overlapping in the network as there are only a handful of devices configured with static IPs, which are outside the DHCP scope. 

I am not sure what the separate SSID will prove tbh if I don't know what settings I need to tweak. I just tried running iperf and the results are better, but only because I played a bit with the TCP window size (as it is a major factor considering the observed results and RTTs jumping over 500-600 ms). As you may or may not have seen, I am using 80Mhz channels. Flex connect I wouldn't consider using because I roam with some devices and that'll be a problem. 

I find speedtest.net quite useful as it tracks the whole connection - end-to-end and this is something I have always used to measure both - my internet service as well as my wireless one. Check the below picture - it's very clear. Up until recently, I've always got at least 200+ Mbps. Regardless what I did today, I cannot get more than 30-32Mbps. (unless I get on the wire were I get the whole ~215 Mbps).. The only thing I've changed lately was the AirOS. I have tried changing this now to 8.3.150, but it's the same result.. now my problem is I can't remember what I was running on before I upgraded to 8.5.150 so I can get back to it ....

[Scott Fella]

Well, since you are using 80mhz, what does your channel utilization look like? You might want to try 40mhz especially if there are adjacent neighbors with wireless. When I say test with a new SSID, this helps isolate if the issue is with the SSID configuration or maybe with the network. So if you have an open said and only allow 5ghz and on a different subnet, what is your test look like. What happens if you no define that ap on 40mhz, then test with using 2.4ghz only. You should be able to get enough data to see where the performance increases and where it starts to fall. Also when you upgrade, the issue is that it changes some of your settings, so you need to look at your wlan’s and make sure that some of the features didn’t get enabled by itself. Upgrade the device NIC driver also for windows and see if that helps.

Use some free tools to check the channel utilization and see if the channel your ap is on, the utilization is high or not. All this testing is assuming nothing on the network side is the issue.

[danailpetrov]

I only use the 5Ghz on all the devices. The 2.4 is for some IOT stuff that I have around the house and I can't turn it off (not that it is a problem anyway). I haven't tested the 2.4 / multi-channel, but I don't want to dig into this for now. 

I checked a few tools to check for channel overlapping as well as channel utilisation and it's nothing there either! 

As per my original topic, I was wondering if you guys know: 

• are there any built-in, default policers for different classes and if so - what are the values? 
• Is there anyway possible I can completely switch off that behaviour (preferably on a per SSID basis?)
• Is that an 8.5.151.x problem because I don't recall having this problem on older versions.

Also, what is your opinion on this "Best practices" menu. At the moment I am on 29/50, but some of these are not for home users anyway. 

Some of enabled features I have on mine

[Scott Fella]

On your wlan setting under QOS is where you need to look. By default nothing is enabled.

[danailpetrov]

Hi Scott & thanks for your inputs. 

I have tested with numerous of devices, all of which (for good or worse) are apple. 

The results I observe are identical - I cannot achieve more than 32-35Mbps using a basic "Speedtest.net" test. Now I know this is not the best measurement tool, but I do get a similar results copying (AFP/CIFS) from a server connecte to the same switch where the WLC is connected to and I cannot get more than ~4MBs (32-35Mbps) even there. 

I have two iPads, two phones and three Apple computers (2xMacBook pro's & 1Mac Pro) and they're all having the same results (when it comes to the speedtest). If I connect the device physically to the switch - I get the full bandwidth (CAR) from the ISP. 

It's very weird issue, but I have done 60-65MBs (~500Mbps) using CIFS on same equipment a few months back, with exactly the same config and everything and today - I can barely get 1/20th of that - it's just weird... 

None of the devices (WLC) nor the switch report any CRCs or any other type of errors... 

But to answer your questions: 

- Don't think it's a driver issue as the problem doesn't follow an end-device, but it rather follows the "network" (Wireless)

- I am not 100% about configuration issue, but pretty much everything I could have though of - I've checked and it seems to be alright. 

- There is no duplicate of addresses on the network. 

Hope the above makes sense & thanks very once again for your time guys!

[danailpetrov]

Some more data.

I have used another, non-apple computer (IBM Lenovo) which also has 802.11AC capable adapter and here is the results from it too:

[patoberli]

Might be a software bug, maybe not. Have you tried rebooting everything?
The fairly high latency is typically caused by co-channel interference (meaning other wireless clients/networks on the same channel transmitting data).
There is a beta for the next version of 8.5.x in the forum, maybe it's a known issue with the 1815? See here for the release notes and the manual on how to access it:
https://community.cisco.com/t5/other-wireless-mobility-subjects/85mr6-interim-build-availability/td-p/3987047

[danailpetrov]

Hiya,

I have tried numerous of times rebooting the whole setup - nothing helped. I even switched back to 8.3.150 but it was all the same! 

I agree with the interference thing, but I only have a limited amount of software tools to analyse what's on the spectrum. From what I can see there is nothing reported  and I tried (literally) all the possible channels (and getting connected to different APs (different physical locations in the house) and nothing helped!) it's super annoying and I have no idea where to look next :| 

check the below screenshots but I can't find any correlations between latency going up and anything massively changing on the "left hand side" .. :| Is there anyway possible there could be some non-wifi interference on 5Ghz!? I don't know what to think anymore... 

P.s. the latency isn't as bad as it was right now... but I am still unable to achieve more than 35Mbps :| It all leads me to the controller, like there is a queue or a buffer / queueing happening and as soon as this happens, TCP is just ... being TCP, trying to adapt to these latencies and basically dropping the bandwidth. If I play with iperf and increase the TCP window size or simply start whacking UDP traffic - I can eventually get to ~400Mbps, but that's not the point .. (as it is more or less useless If I have this latency issue as any protocol (especially chatty ones such as CIFS/AFP will suffer big time)) 

I really hope that makes sense ... 

[Scott Fella]

There isn’t something that will throttle anything like that. There are bandwidth limits but that isn’t supported on a 2504. Maybe something happened to the WLC like it's going bad, don’t know. I would just rebuild it and see if that helps… I have done so many that rebuilding doesn’t take me long, but that might not be something you might want to do.