QoS: Difference between bandwidth and priority commands.

[NetworkingGeek1]

Hello experts, I'm trying to understand difference between 

bandwidth 

and

 priority 

commands for QoS. It's written that with

priority

command, traffic from LLQ can't exceed allocated bandwidth during congestion, but can exceed if there no congestion. For 

bandwidth

command, traffic can exceed allocated bandwidth during congestion. But my question is, how traffic from non LLQ can exceed the allocated bandwidth, if there is congestion anyway? I mean how does it make sense, if I understand meaning of "congestion" correctly, it means that interface is busy and all queues are full. How come there is free bandwidth during congestion? Maybe I don't understand correctly what congestion means in QoS.

Please help me to understand this.

[Joseph W. Doherty · Accepted solution]

"Maybe I don't understand correctly what congestion means in QoS."

Probably true.  ; )

BTW, my definition of congestion is whenever a frame is sent to an interface and it cannot be immediately transmitted, i.e. it needs to be queued.  Also, BTW, congestion, alone, is not always adverse to you traffic's service needs.

"For 

bandwidth 

command, traffic can exceed allocated bandwidth during congestion."

Bandwidth 

command doesn't actually allocate bandwidth, it guarantees that bandwidth as always being available for that traffic.

". . . traffic from LLQ can't exceed allocated bandwidth during congestion, but can exceed if there no congestion."

Correct, but this because it applies (I believe) to traffic in the actual LLQ queue.  If there, it's subject to the LLQ policer.  If not there, it can use all the available bandwidth.

One key thing you might be unaware of, an interface has a hardware FIFO queue, which Cisco calls its tx-ring.  That's the queue that directly feeds the interface (egress).  When an interface starts to back up, queue into the tx-ring, only when tx-ring fills, does the overflow go into the software managed queues, i.e. like those in a CBWFQ service policy.  (This is how LLQ class traffic can be forwarded, without being policed, because it never overflowed into the LLQ.)

BTW, you cannot bypass using the tx-ring, but you can minimize its size.  Doing so, often make something like a service policy much more effective.

[NetworkingGeek1]

@Joseph W. Doherty  Thank you very much for the reply. Let me explain what I understood from your post, please correct me if I'm wrong:

1. If there is no congestion, then there is no queuing. In that case all traffic is FIFO, no matter if QoS configured or not.
2. There is hardware queue, which is always FIFO and doesn't have any CBWFQ or LLQ, it only has FIFO. Only when traffic overflows hardware queue and software queue comes into play, CBWFQ and LLQ start to work.
3. During congestion traffic in LLQ can't exceed bandwidth which was configured in

priority 

command. This traffic can exceed it, only when there is no congestion and all traffic forwarded according to FIFO through hardware queue.
4. During congestion traffic in CBWFQ, in contrast to LLQ, can exceed guaranteed bandwidth, which was configured in

bandwidth 

command.

My original question was: How traffic in CBWFQ can exceed minimum guaranteed bandwidth during congestion, if there is already congestion? I mean, if there is congestion, then no free bandwidth is available. If I understand it correctly: Congestion doesn't mean that all queues are filled up, for example, it can be that one type of traffic need all the bandwidth and other queues are empty and it also considered to be congestion and in that case traffic which needs all the bandwidth, will take it from other queues. Am I correct?

[Joseph W. Doherty]

#1 Correct.

#2 Correct.

#3 Not quite correct.  If you need to queue, you have congestion.  But there are two queues, hardware and software.  Traffic queued in just the hardware queue, does not trigger CBWFQ functions.  Traffic queued in the software queue(s) do trigger CBWFQ functions.  Believe implicit LLQ policers only triggers where there's traffic queued in the LLQ.

#4 Correct, if there's excess/available bandwidth.

"My original question . . .  Am I correct?"

See if the following helps . . .

Consider a policy with two classes, A and B.  Class A is allocated 25%, Class B is allocated 75%.

At the moment there's no Class B traffic, but Class A "offers" 110%.  What it will obtain is 100%, and there's congestion, as the offered rate exceeds interface bandwidth.

Next Class B wants to use 25%.  As it's guaranteed 75%, it gets its 25% and now Class A gets 75% (although it still wants 110%).

So, two examples of Class A getting more than its guaranteed bandwidth, when there's congestion.

Next Class B wants 80%, but as Class A still wants 110%, it gets its guaranteed 75% and Class A now only gets its guaranteed 25%.

Next Class A drops to only wanting 21%, which allows Class B to now get 79%.

Next Class A drops to only wanting 15%, which allows Class B to now get its desired 80% and now 5% of the bandwidth is not used.

[NetworkingGeek1]

@Joseph W. Doherty  Thank you so much for this explanations. It really helped. And the last question, I didn't get regarding this point: "#3 Not quite correct..." I think I told the same as you, let me rephrase myself.

During congestion: Traffic in LLQ can't exceed bandwidth which was configured in

priority

command. Because there is traffic policers which triggers if traffic in LLQ and wants to exceed bandwidth configured in

priority

command.
If there is no congestion: Traffic which supposed to be in LLQ (But not queued. Because if there no congestion, then there are no queues) can get all bandwidth which is available at the moment.
Additionally: If there is traffic only in hardware queue, then there are no any LLQ and CBWFQ queues. Only FIFO. This is considered to be not a congestion. If traffic overflows the hardware queue, it's going to software queue and this is considered to be congestion. In that case LLQ and CBWFQ queues comes into play.

Is that correct?

[NetworkingGeek1]

@Joseph W. Doherty   Thank you so much for this explanations. It really helped. And the last question for this topic, I didn't understand regarding this point: "#3 Not quite correct..." I think I told the same as you, let me rephrase myself.

During congestion: Traffic in LLQ can't exceed bandwidth which was configured in

priority

command. Because there is traffic policers which triggers if traffic in LLQ and wants to exceed bandwidth configured in

priority

command.
If there is no congestion: Traffic which supposed to be in LLQ (But not queued. Because if there no congestion, then there are no queues) can get all bandwidth which is available at the moment.
Additionally: If there is traffic only in hardware queue, then there are no any LLQ and CBWFQ queues. Only FIFO. This is considered to be no congestion. If traffic overflows the hardware queue, it's going to software queue and this is considered to be congestion. In that case LLQ and CBWFQ queues comes into play.

Is that correct?

[Joseph W. Doherty]

On point 3, the answer revolve around what's congestion, which I defined as whenever an interface needs to queue packets.

If I understand what you've been writing, it appears you consider there's no congestion when none of the software queues (including LLQ) have any packets.  However, if there are packets in the hardware tx-ring, there's congestion.

As to whether LLQ implicit policer polices or not, I believe, is strictly tied to whether there are packets in the LLQ.

There are "corner cases" understanding the interaction of these two different queues, is crucial, to understanding LLQ policing behavior.

Consider a burst of data packets that almost fills the tx-ring.  Then a few VoIP packets are queue behind them.  Then another burst of data packets arrives, which not only fills the tx-ring, but overflows into the software queue(s).  The VoIP packets sitting in the TX-ring are not limited by the LLQ policer, i.e. they will transmit at full interface speed.

Conversely, a bunch of data packets arrive, filling the TX-ring, but don't overflow it.  Then VoIP packets arrive, which do overflow the TX-ring, and are queued in the LLQ queue.  As the interface drains the TX-ring, it will pull packets from the software queue(s), in this case LLQ.  However, as those packet's are dequeued, they are now policed.

So with point 3, it simply is whether there are packets in LLQ or not.

BTW, there used to be only one (newer IOSs support two) LLQ, although multiple LLQ classes can be defined, each with a different policed rate.

[NetworkingGeek1]

@Joseph W. Doherty   So, let me correct myself:

If there are packets in hardware queue (tx-ring), it considered to be as a congestion, but if packets didn't overflow hardware queue and therefore didn't go to the software queue, then there are no any type of LLQ and CBWFQ queues. Only FIFO. And in this case, any type of traffic, (including traffic which supposed to be in LLQ), can exceed bandwidth which was configured for them.

So, basically QoS comes into play only when traffic overflows the hardware queue (tx-ring) and going to the software.

If there are no packets in the software queue, but there are packets in the hardware queue (tx-ring) it's still a congestion. Even though it's considered to be a congestion, in this case packets forwarded according to FIFO.

Is that correct?

Thank you for the patience.

[Joseph W. Doherty]

"So, basically QoS comes into play only when traffic overflows the hardware queue (tx-ring) and going to the software."

Correct, although this is for software based routers.  Switches do their QoS in hardware.

"If there are no packets in the software queue, but there are packets in the hardware queue (tx-ring) it's still a congestion. Even though it's considered to be a congestion, in this case packets forwarded according to FIFO."

Correct.

"If there are packets in hardware queue (tx-ring), it considered to be as a congestion, but if packets didn't overflow hardware queue and therefore didn't go to the software queue, then there are no any type of LLQ and CBWFQ queues. Only FIFO. And in this case, any type of traffic, (including traffic which supposed to be in LLQ), can exceed bandwidth which was configured for them."

Correct, that concerning bandwidth (which doesn't limit) and priority statements.  However, you can set an explicit policer or shaper, and that could limit bandwidth even when there's not physical port congestion at all.