Throughput issues - slow upload

[Clutz5250]

Greetings,

I am currently dealing with an issue that I've been working with TAC on (best effort support unfortunately) and the issue is still ongoing. The symptom is a slow upload - ranging from near normal rate (~40 Mbps) to single digits. I cannot see a utilization issue from pcaps. There are bouts of TCP bursts (from the speed tests I reckon).

Here's some of the context around the site.

1. site was recently upgraded to a 50 Mbps circuit
2. site is part of AT&Ts AVPN MPLS service. MPLS QoS is treating all traffic as CS0, which gets treated as CS4.
3. site has multiple VRFs, and multiple customers tied in from different routers. GET encryption is being used on two VRFs.
4. site route has a shaper configured. I have attempted to tune the queues - no luck. Shaper was set at shape average 100 percent, 85 percent. Bandwidth reference has been adjusted to be higher - as to i guess increase other buckets and cope with bursts. 
5. Local switch on one of the customers was upgraded. qos soft queues were increased. 
6. Overall ceiling/range has improved it seems (numbers originally polled were typically lower), but the sudden drops float all the way down to 2 Mbps upload during testing (speedtest.net for example). 
7. Site traffic flows through to our datacenters and goes from there. 
8. Site router is a glorious 2911
9. Drops have been seen on the outgoing interface, and typically (not all the time) align with QoS drops (class default). Drops do not increment regularly, but do seem to increment when speed tests are started.
10. Besides speedtest metrics, user experience is felt. So it isn't an artificial problem the speed tests.
 
My current direction is to push at the provider and have them adjust their QoS profile. I have next to no knowledge as to how MPLS QoS works over such a provider network, but with some other sites reporting similar symptoms, it seems to reinforce my hypothesis this might be the right direction. 

Has anyone dealt with something like this before. I'd be awfully grateful of anyone's input and can provide further information (though sanitized). 

[Leo Laohoo]

---

@Clutz5250 wrote:

but the sudden drops float all the way down to 2 Mbps upload during testing

---

What time of the day when the speedtest drops down to single-digits?

Is there a bigger router to try (for a week)?  The logic behind this is to eliminate the 2911 as a culprit.  

If a bigger router is installed and the issue is still the same then the issue could, potentially, be the carrier.  

And as far as anyone knows when dealing with carriers, they do not perform any troubleshooting and it is up to the customer to provide the proof the issue lies with them.  

[Clutz5250]

"What time of the day when the speedtest drops down to single-digits?"

In and out of production

"Is there a bigger router to try (for a week)?  The logic behind this is to eliminate the 2911 as a culprit."

Unfortunately not at this time. 

[Joseph W. Doherty]

Is this a multi-point topology?

[Clutz5250]

Yes.

[Joseph W. Doherty]

Ah, in that case, a common issue is multiple nodes sending to the same node, concurrently.  What then happens is congestion on the "cloud" provider's link to your site (node) which you cannot "see" and generally, by default, which you cannot manage such congestion.

MPLS "cloud" providers often offer (or did, I'm very dated dealing with such) QoS, although, IMO, weak implementations of QoS (again, remember I'm dated in this area).

Can you mitigate this yourself, doing QoS on just your equipment?  Yes, although it depends on what QoS features your equipment supports and you often need to set aside bandwidth which you otherwise would like to use.  (Cisco's adaptive QoS might be a "good fit", but I've no actual experience using it.)

Typically, if this is the cause of your problem, your "cloud" provider will suggest obtaining more bandwidth, which, indeed, might mitigate the problem (it also mitigates your provider's income problem too).

[Georg Pauwen]

Hello,

post the full running config (sh run) of the router, as well as the output of 'sh interfaces x' where x is the outgoing interface, maybe we can spot something...