C9200 Shaper Command Failing

[xzevallos]

I have a site with an ISP router and a C9200 switch attached to it.  The switch connects to various edge devices, such as FWs and SDWAN devices.  I want to limit the download bandwidth to each device, so that one device does not consume the ISP bandwidth for traffic from the other devices.  For this, I configured egress shaper commands on each port to the devices.  As an example, below is the config for the port toward the SDWAN device:

interface GigabitEthernet1/0/3

 switchport access vlan 510

 spanning-tree portfast

 service-policy output Shape-VERSA-ISP-1

policy-map Shape-VERSA-ISP-1

 class class-default

  shape average 350000000

However, we are having a lot of packet drops on the traffic egressing this port toward the device, and users are experiencing intermittent application slowness.  According to our OSS system, the bandwidth on this port is far below the 350 Mbps shaper limit.  We are experiencing the same situation for all the other edge devices.  See the output below:

sh policy-map inter gi 1/0/3 output

GigabitEthernet1/0/3

  Service-policy output: Shape-VERSA-ISP-1

    Class-map: class-default (match-any) 

      0 packets

      Match: any

      Queueing

      (total drops) 34657812

      (bytes output) 19699360753

      shape (average) cir 350000000, bc 1400000, be 1400000

      target shape rate 350000000

What am I doing wrong? 

[Elliot Dierksen]

The first thing I see is that you have a measurement internal (Tc) that is very high. Looking back at some really old frame relay traffic shaping configs in my archive, here is an example.

map-class frame-relay P1536-C256
 frame-relay cir 1536000
 frame-relay bc 15360
 frame-relay be 0
 frame-relay mincir 256000
 frame-relay adaptive-shaping becn
 frame-relay fecn-adapt
 frame-relay priority-group 1

The target that was the accepted standard for voice was a Tc of 10ms. Why this matters is that once the token bucket it used, it will wait until the measurement interval has ended before it attempts to send any more traffic.Here is a link that talks about the newer methods of traffic shaping.

https://community.cisco.com/t5/switching/traffic-shaping-understanding-auto-generation-of-tc-bc-values/td-p/1221642 

[xzevallos]

Hello Elliot,

If I use the formula Tc = Bc / CIR and use the values shown in my output, I get 4ms.  Is that too high?

To change the Tc value, I need to change the Bc value.  But under the policy-map command, I don't see any command to change this value.  How can I change it?

[Joseph W. Doherty]

". . . I get 4ms."

I recall the Bc value is in bytes, and if so, Tc would be 32 ms.

"How can I change it?"

From a quick skim of 9200 QoS configuration documentation, it may not be possible.  (BTW, as you didn't identify the specific 9200 model or what its running, cannot be certain about the shaping options.)

[Joseph W. Doherty]

"What am I doing wrong?"

Possibly a misunderstanding of "how things work".  (I have in mind a deep understanding of various data transfer protocols, e.g TCP.)

If I correctly understand what you're trying to accomplish, possibly a policer would be "better" than a shaper in this situation.

"According to our OSS system, the bandwidth on this port is far below the 350 Mbps shaper limit."

Measured on what time scale?

Again, assuming I understand what you're trying to accomplish, it's very, very difficult to achieve, especially on a router or switch.  Years ago, there were some 3rd party devices (e.g. Packeteer) that were designed for this kind of situation, but even those weren't perfect.

[Joseph W. Doherty]

Thinking some more about your policy, if the device supports (?), something like this (?) might help:

class-map match-all TCP
 match protocol tcp !if this isn't support, might use an ACL

policy-map Shape-VERSA-ISP-1-RED
 class TCP
  bandwidth percent #
  random-detect
 class class-default
  bandwidth percent #

policy-map Shape-VERSA-ISP-1
 class class-default
  shape average 350000000
  service-policy Shape-VERSA-ISP-1-RED

The logic behind the above, preclude synchronized tail drop of the shaper's FIFO queue and also try to get TCP flows to slow before FIFO queue overflow.

There's more that might be done, and possible should be done (e.g. buffer tuning; beneficial if dealing with microbursts), with deeper analysis.

[xzevallos]

Hi Joseph,

I think your random-detect suggestion is a good idea, so I will try it.  

Could you explain why policing would be better than shaping?  I always thought policing was stricter on bandwidth control hence more packet drops.

[Joseph W. Doherty]

"Could you explain why policing would be better than shaping?"

In this case for two reasons.

First, I assume you want to inform upstream sender to slow its transmission rate ASAP (to limit itself to the available bandwidth you're allowing).  As a policer will usually drop sooner than a shaper, the sender might slow down sooner.

Second, if the shaper is using a single queue, it can lead to global synchronized tail drops.  (Normally, I would suggest FQ for a shaper, but doubt that's an option on this switch.). If we knew your shaping drops were due just to microbursts, I would stay with a shaper and increase queue depth, but we don't know that and the switch's buffer resources are probably most limited on a 9200.

Although a policer is set to drop sooner than a shaper, in some cases, overall you achieve more throughput.  (BTW, same logic for using RED.) Whether that's the case here, insufficient information to say for sure.

Also BTW, many assume zero drops is the ideal.  For TCP it is, but only when receiver's RWIN exactly corresponds with at-the-moment BDP.  In the real world zero drops usually indicates RWIN too small.  Drops indicate RWIN too large.  For the latter, we want to maximize "goodput", i.e. minimum number of drops to achieve maximum possible data transfer.

I initially suggested a policer might be better because I didn't realize the 9200 supported a hierarchical policy with a parent shaper.  From further reading its QoS docs, my second suggestion is a more refined version of the same logic.  Still, cannot guarantee it will perform better.  Again, trying to manage upstream bandwidth is problematic.  Products like the earlier mentioned Packeteer had other techniques not available on Cisco routers or switches.

[xzevallos]

Thanks for your detailed explanation.  I think the most important first step is to configure WRED before I try policing, to avoid the TCP synchronization problem. If I enter random-detect by itself, I don't think it will do any random drops.  In my lab switch, I configured a test policy with the random-detect command and applied it to an interface.  Please see the attached output (I couldn't paste it here because the values get misaligned).    

Where you see 100/100, my interpretation is that it means nothing is being randomly dropped.  So I would need to enter thresholds such as the following command:

random-detect precedence 0 percent 50 80

Here, precedence value is 0 (assuming all ISP traffic is marked 0), minimum queue length before random drops are applied is when the queue is 50% full, and maximum random drops occurs when the queue is 80% full or more.  I checked the output again and this time 100/100 was changed to 50/80, so I think my interpretation is correct.

The problem is that I don't know what are good starting points for the threshold values, and how to monitor WRED to optimize these values.  Could you advise?

By the way, there is no way to configure Bc and Be values for the shaper command on this platform, but they are available with the police commands.  

[Joseph W. Doherty]

"By the way, there is no way to configure Bc and Be values for the shaper command on this platform, but they are available with the police commands."

A policy shaper is a late addition to low end Catalyst switches.  As a switch uses dedicated hardware, possibly the hardware only supports a single Bc/Be value.

"I think the most important first step is to configure WRED before I try policing, to avoid the TCP synchronization problem."

I suggested a policer to also avoid TCP synchronization problem, but, hopefully, WRED would be better for that purpose.

"If I enter random-detect by itself, I don't think it will do any random drops."

You need WRED to "see" a queue, before it will drop packets, and then there's also a sliding probability of the drop chance, low when queue is small.

Also, you policy applied WRED in the same class as the shaper.  It's not documented, to my knowledge, applying WRED in the same class as the shaper.  Generally, actions against the shaper's queue are performed in a child policy, as I presented earlier.  Further, my suggestion broke TCP into its own class, in the child policy, for WRED application.

You still have not identified the IOS being used.  Looking at the documentation for the latest IOS, the default min value, for IPPrec 0 should be 50%.

"Where you see 100/100, my interpretation is that it means nothing is being randomly dropped."

Actually, no, at 100% there should be random drops.  When you exceed max, all packets are dropped.  (Usually the drop probability is 10% at 100%.)

Yes, you want to set you min lower than 100%.  As to what value you should use, generally, you find in by trial and error.  You want to avoid buffer exhaustion drops, and also, if possible, WRED drops beyond max.  You also want min to be no lower than necesary.

As you only expect to have one class of traffic (IPPrec 0), your max should be 100%.  Again, experiment with different min values, perhaps starting at 50%.  If drops beyond 100%, lower min value.  If no drops beyond 100%, increase min value.  (I suggest making min value changes by halves.)

[xzevallos]

I will apply your suggested solution, and configure random-detect at 50/100 to start.  Please explain how I could go about checking your statement: "If drops beyond 100%, lower min value.  If no drops beyond 100%, increase min value."  

[Cristian Matei]

Hi,

   First thing you've got to fix your QoS policy, as you're not matching traffic. Never performed QoS on Cat 9K, so I suggest to take a look on platform capabilities first: 

https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKENS-2096a.pdf

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9000/white-paper-c11-742388.html

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2024/pdf/BRKTRS-3090.pdf

Moreover, usually, what you're trying to accomplish is never performed at the layer 2 level. Why don't you better do it at next-hop level, Versa, as it does support QoS.

Best,

Cristian.

[Joseph W. Doherty]

"Moreover, usually, what you're trying to accomplish is never performed at the layer 2 level."

Oh?  You've never seen QoS on a pure L2 (or L3) switch using L2 frame CoS?

Or, do you mean dropping frames, which contain (TCP) packets cannot "influence" sender's transmission rate?

Or, do you mean something else?  For example something you've (personally) have never seen done?

"First thing you've got to fix your QoS policy, as you're not matching traffic."

What specifically are you referring to?

[Joseph W. Doherty]

"Could you explain why policing would be better than shaping?"

Unsure, as I don't have a 9x00 to look at all its stats options.  (Again, without you providing platform/IOS, don't even know what Cisco documentation to review.)

What to look for are drop stats higher than WRED drop stats.

[xzevallos]

The platform is C9200L-24T-4X, IOS is 16.12.04.

I'll let you know in a few days if the solution works.  Currently there is no service-policy applied as a temporary measure, so we have to find a time window to minimize user complaints.