Queueing configuration question/problem

[brian.holmes]

We are seeing the latency of our AF31 class traffic increase by ~30ms when the WAN circuit is saturated.  We are using a carrier MPLS service offering with Custom Queueing on the CE router which is Cisco.  We only see this symtom with one carrier and not another so we are suspecting there are some differances in how they configure the queuing.

The carrier we see this issue with has:

random-detect dscp 26   720   960   10

while the other leaves this at the default.  Could this larger queue depth be part of the problem?  Since the Custom Queue for AF31 has plenty of reserved bandwidth when the circuit is max out, it does not make sense why the router would be needed to queue up the AF31 traffic and add latency.

Any ideas?

[Mohamed Sobair]

Hi,

A larger queue-limit will certainely increase latency when congestion occurs on the WAN. If I observed latency increase on the WAN, I would definately decrease the number of packets in the queue depth of the custom queue.

The default queue depth is normally measured based on the availble memory resources of the router to be allocate for each queue. If it provide good result with the default queue-depth, I wouldnt recommend altering this value unless on Special WAN interfaces where its documented to decreae the queue-limt without increasing it.

HTH

Mohamed

[Joseph W. Doherty]

Disclaimer

The Author of this posting   offers the information contained within this posting without   consideration and with the reader's understanding that there's no   implied or expressed suitability or fitness for any purpose.    Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind.  Usage of   this posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

There's insufficient information to offer much in the way of advice or suggestions.

Generally, if there's traffic based congestion, it often first forms at the most common bottlenecks, WAN cloud ingress or egress.

Sometimes WAN cloud vendor will define queues too deep so as to not drop any customer packets, although doing so can create excessive queuing delay.

Given RED drops starting at 720 packets, at first glance, many might assume this is excessive, but since you didn't note the link's bandwidth, it may be just fine or even too shallow.

[brian.holmes]

This is a 6Mb Multilink interface and the reserved bandwidth for AF31 is no where near exhausted even during circuit saturation.

The carrier is stating that WRED is being performed before the packets hit the custom queue which seems like the reverse of how things should be handled.

Sent from Cisco Technical Support iPhone App

[Joseph W. Doherty]

Disclaimer

The Author of this posting    offers the information contained within this posting without    consideration and with the reader's understanding that there's no    implied or expressed suitability or fitness for any purpose.     Information provided is for informational purposes only and should not    be construed as rendering professional advice of any kind.  Usage of    this posting's information is solely at reader's own risk.

Liability Disclaimer

In    no event shall Author be liable for any damages whatsoever  (including,   without limitation, damages for loss of use, data or  profit) arising  out  of the use or inability to use the posting's  information even if  Author  has been advised of the possibility of such  damage.

Posting

I too am unaware of how you do WRED except on an egress queue.

For 6 Mbps (4x T1s?), WRED set to start at packet 720, would mean up to 719 packets, @ up to 1500 bytes (discounting L2), would be about 8,628,000 which could lead to queue delay of over a second; seems excessive.  Even dropping all subsequent packets, queue depth could extend to 960 packets.

Surprised to read vendor is using CQ, would have expected CBWFQ.

Again, though, sometimes to find certain errors needs stress testing.  For example sending a 6 Mbps test stream in/out of site.  Shouldn't be any drops.  Push test stream to 10 Mbps, and then you can further confirm 6 Mbps delivered, and vary traffic rates in different classes, to confirm they get their reservations.

Sometimes I've filled pipes, if their QoS model supports it, with scavenger traffic.  Pipe fills but other traffic, if within their class bandwidth reservations, should see little difference.  If they do, then the question, to vendor, why?

[brian.holmes]

Thanks all for the feedback.

We have the carrier engaged, but they are giving the standard "it is working per design" answer.  From reading a QOS book from Cisco Press, they say that WRED should be used in the Core of the network and not the edge.  Would a CE router be considered Core or Edge?  It think the E stands for Edge but maybe they mean the user edge.

[Mohamed Sobair]

Brian,

Good Point,

As declare in the Documentation, the Core means where you apply your QoS policy. Normally you classify and Mark at the Edge of the Network and you apply your QoS policy at the Core/Egress outbound direction.

Here, your CE is actually the Edge and Core where you need to classify and mark as well as Apply your QoS towards the Provider Cloud. Wred is designed

to prevent congestion dropping unlikely TCP packest from the Queues to avoid congestion on the WAN.

So, You have no worries having WRED on your CE router.

Regards,

Mohamed

[paolo bevilacqua]

brian.holmes wrote:
Thanks all for the feedback.
We have the carrier engaged, but they are giving the standard "it is working per design" answer.  From reading a QOS book from Cisco Press, they say that WRED should be used in the Core of the network and not the edge.  Would a CE router be considered Core or Edge?  It think the E stands for Edge but maybe they mean the user edge.

That is correct, WRED should not be used at the network Edge (if used at all), and I think it only causes confusion and unnecessary processing in your situation.

Waht you want is priority queueing and WFQ, nothing else.

[Mohamed Sobair]

Paolo,

What unnecessary Processing and confusion here???

I dont agree with you on this point, the OP can still use Wred in his case to avoid congestion on his WAN, look at his post earlier, he declared that Telco only honoring AF11 for him and he needs end to end QoS by Marking critical traffic using AF11, while dropping unnecessary TCP flows from the Aggressive Custom Queue (Which already designed and implemented) in Telco.

So Still, he has no worries to implement it for the default class or low priority traffic to avoid congestion on his WAN Link for the Priority traffic.

I hope this clarifies my point here.

Regards,

Mohamed

[paolo bevilacqua]

I stand by my point, WRED it's an useless technique that doesn't solve anything in this case, it was invented many years ago to solve well different problems.

In low bandwitch applications, all one can really to do is to have one or two special classes of traffic using priority and percent, an hope for the best.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind.  Usage of  this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever  (including, without limitation, damages for loss of use, data or profit)  arising out of the use or inability to use the posting's information  even if Author has been advised of the possibility of such damage.

Posting

Again, if the problem is within the provider's network, they often will say "it is working per design" until you can demonstrate to them it isn't.  Once spent 3 months arguing with tier 1 WAN vendor who also said there's no problem on our side.  (However there was.  Turned out to be buggy firmware on a MetroE line card.  It was type of situation where it worked correctly 97% of the time.)

The QoS book isn't too wrong about where WRED might be best used.  The reason they mention using it in the core, WRED is relatively light weight on resource usage and works better against many flows.

CE would be the customer's edge which has the link to the provider's edge, PE.

[brian.holmes]

I notice the one vendor has:

random-detect exponential-weighting-constant 9

while the other has

random-detect exponential-weighting-constant 0

How might this relate to the latency increase during saturation?

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the  information contained within this posting without consideration and with  the reader's understanding that there's no implied or expressed  suitability or fitness for any purpose.  Information provided is for  informational purposes only and should not be construed as rendering  professional advice of any kind.  Usage of this posting's information is  solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

That parameter determines how quickly WRED responds to instantaneous queue size changes.  I recall, but haven't checked, that 9 is the default and 0 causes "real-time" queue size tracking.  Regarding this parameter's impact relative to latency, closer tracking to "real-time" would sooner drop packets than leave them in the queue, i.e. less queuing latency.  (BTW, instantaneous tracking works against one of the things WRED is trying to accomplish, which is dealing with longer term congestion, not bursts.)