<30mbps throughput on 3702 over 5GHz with Mobility Express controller

[Jeff A.]

I'm been working through troubleshooting this issue for the past two weeks and have checked and changed every setting I can think of within the controller without any luck. I have a single 2802 running the current 8.10.130 Mobility Express image acting as a controller with two additional 3702 AP's joined in CAPWAP mode to the controller. I'll also add that I'm still fairly new to the technical side of the Cisco ecosystem so please be patient with me.

When I am connected over 5GHz to the 2802 I have no throughput issues, able to get up to about 600mbps on my devices (iPhones, MacBooks, iPads) and everyone connected is happy. However, on my 3702 the connection starts off alright at about 400mbps and then starts to deteriorate over hours/days until eventually the same devices are getting less than 30mbps total throughput. Re-associating with the network sometimes kicks it back into gear and other times it doesn't.

Funnily enough, I have observed cases where a MacBook associated with the same 3702 is able to pull 300mbps but an iPhone connected was only able to pull 30mbps. Then the next day it was reversed.

I have tried just about every setting under the sun on the Mobility Express controller. Tried to set everything up based on the Cisco Best Practices. Tried the defaults. Tried suggestions in articles floating online. Nothing seems to make a difference to the throughput measured. I even reset the whole system - every AP, and controller, back to defaults – and this helped for a day, but then back to the old antics.

I have confirmed that the switch is providing PoE+ and all the AP's are confirming they are happy displaying Full Power.

I'm not sure where to go from here – there's a couple of posts in these forums and on Reddit with 3702 devices having lower than 300mbps throughput, but that seems to be a different issue than what I'm having. If anyone could shed some light or point me in the right direction to further troubleshoot this weird issue I'd be very grateful.

[Scott Fella]

Let me go over what I have experienced in the past. When I test various codes, I at time also see issues with speed , sometimes good and sometimes bad. I tend not to always check because at the end of the day, speed will differ on what ap I’m connected to and what channel width that ap is also using. So what you can do is try a different code very and see if that helps or not, you can also try to factory reset the 3702 and see if that helps. Since these AP’s are connected and drop traffic right to the connected switch, maybe swap out the patch cable or switch to a different port. The best way to really test is to use Iperf and have the one device running as a server on a wired connection. This way you can see what throughput you are getting from the various AP’s and or location.

[Jeff A.]

Thanks for the insight Scott, but the thing is that I’ve tried all that, and yet the issue remains/returns. It’s something which doesn’t appear to have any consistency which is the most frustrating part, but once it starts I can observe the behaviour on that client for hours.

I have tried all the usual - I’ve adjusted everything I can think of: channels, QoS, fastlane, frequencies, channel width, CleanAir, encryption, application visibility, roaming options, band selection/steering, single/multiple ssid’s, multicast, and the list goes on. I’ve tried a different switch, new cables, even wired directly with patch to the switch and not using building wiring with the same behaviour. 

Throughput testing is done using iperf on a wired server with wireless clients. I can sustain 980mbps with wired clients on the same switch that the AP’s are connected to so that is not a problem.

I am just at a loss as to what else could be causing the issue with the 3702’s that the 2802 is not experiencing.

[Scott Fella]

Different ap models, not all ap’s behave the same. It would be different if you were able to compare multiple 3702’s for example to make sure there is not an issue with the 3702 you have. Making changes to the configuration will not help especially when the issue comes and goes. That is usually an issue with memory leak or hardware.

[Jeff A.]

I understand that different AP hardware will have different performance, but it’s not looking like a CPU or memory issue. Checking memory and CPU on the 3702’s, they’re not maxed out.

If I have two clients attached to a 3702, one of them can get 300mbps over iperf and the other night only get 30mbps or less. Both are connected with a high score and 100% capability.

It’s as if the AP is choosing to favour one client over the others or there’s an issue with splitting up airtime that I can’t get to the bottom of.

Do you have any suggestions as to how I could try to find the cause of the issue?

[Scott Fella]

I would try a different code to be honest.

[Jeff A.]

Sorry, what do you mean by a different code? Do you mean an older controller version or an older AP firmware version?

[Scott Fella]

Controller firmware

[Jeff A.]

I was originally on 8.10.130 and have tried 8.8.130 with the exact same results.

It seemed fine originally after downgrading, but after about 4 hours the issue re-appeared.

Edit: The image did not swap correctly, so when I checked the version while doing more troubleshooting I found it was still running 8.10.130. I’ll try again over the next 24 hours and see how I go.

[Scott Fella]

Just try to downgrade and see. If it still has issues, then a Tac case would be your next option.

[Jeff A.]

Thanks for the suggestion. I’ve confirmed a downgrade to 8.8.130. If that doesn’t help I’ll try going to 8.5.161.

A TAC case is an option, though I was hoping to try and find the cause first.

[Scott Fella]

Like I mentioned previously, things like that just doesn’t go away with a config change. That is usually an issue with the code or the ap itself. Throughput dropping after a few hours especially if you are near or next to the ap is unusually. However, if you do move away and possibly to a different room or floor, it is normal for throughput to drop. I don’t think that is your case.

[Jeff A.]

I have tested with 8.10.130, 8.8.130, and 8.5.161, and all exhibit the same issue after about 12-24 hours of the 3702's uptime.

I'm beginning to think it's not an AP issue since both of my 3702's are having the issue, and am instead leaning to it being an issue with Apple's wireless stack, but again, not sure why it's only occurring on the 3702's and I never see it happen on other networks or AP's.

The other thing I was thinking of trying was to bring up a trial of the Virtual Wireless Controller and see if the issue persists on that. Either way, it's not an issue with the ME code...

[Scott Fella]

Might as well experiment. It’s the best way to learn.

[Scott Fella]

We run 8.5.140.0 and majority of our access points at 3702’s. We don’t run into these issues. However, devices are not left on, but during a users work hours, there has been no complaints. Doesn’t mean it’s not happening, just that no tickets come in stating throughput drops.