cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
cancel
1126
Views
3
Helpful
41
Replies

Cisco 9300 queues and output drops

FuadG
Level 1
Level 1

Hello everyone,

I am trying to troubleshoot VOIP related problems when voice during calls is intermittent

The phones are connected to separate switches and there is Cisco 9300 in between those switches.

I have read through this article:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-switch/216236-troubleshoot-output-drops-on-catalyst-90.html#anc13

And increased qos queue-softmax-multiplier 1200 

Now drops are not increasing anymore in this table, but TH0/TH1/TH2 are still increasing

AQM Global counters
GlobalHardLimit: 5176 | GlobalHardBufCount: 0
GlobalSoftLimit: 14672 | GlobalSoftBufCount: 0

----------------------------------------------------------------------------------------------
High Watermark Soft Buffers: Port Monitor Disabled
----------------------------------------------------------------------------------------------
Asic:0 Core:1 DATA Port:15 Hardware Enqueue Counters
-------------------------------------------------------------------------
Q Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
(Count) (Bytes) (Bytes) (Bytes)
-- ------- -------------------- -------------------- --------------------
0 0 15626884274 32907700571 2516556011
1 0 0 0 7226027781
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
Asic:0 Core:1 DATA Port:15 Hardware Drop Counters
-----------------------------------------------------------------------------------------------------------
Q Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
(Bytes) (Bytes) (Bytes) (Bytes) (Bytes)
-- -------------------- -------------------- -------------------- -------------------- --------------------
0 14586413831 614569504 0 0 0
1 0 0 15293510335 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0

Can this queuing affect voice traffic? 

41 Replies 41

To your first question, the later Cisco switches usually have a default active QoS config, where different CoS or ToS markings are equally divided between egress queues, each with equal resources, but possibly with different drop thresholds.  Whether your traffic hits a drop threshold would depend on how your traffic is tagged, how much traffic you have, and the device's defaults.

I presume if you check similar ports on the same switch, or like model switches, you'll also see different port utilization stats too.  ; )

To your second question, yes, you might increase a buffer allocation, but it's also a QoS command.  Consider you would change it to reduce drops, i.e. it impacts the quality of service.

Might it help?  Possibly.

Understand, often QoS is a zero sum game.  One pie slice size increase is less available for all the possible pie slices.

Ideally, though, you take advantage of excess resources, those not actually needed.  E.g. you slice a pie into 8 slices, but 1 of your expected 8 guests is a no show.  Knowing that, you can dole out more pie to one or more of your guests.

Possibly you have a guest of honor who gets their choice for their pie slice size and everyone else gets to divide up the remainder.

When we adjust resources, we often do so at the expense of some traffic to the benefit of other traffic.

To your first question, the later Cisco switches usually have a default active QoS config, where different CoS or ToS markings are equally divided between egress queues, each with equal resources, but possibly with different drop thresholds.  Whether your traffic hits a drop threshold would depend on how your traffic is tagged, how much traffic you have, and the device's defaults.

I checked all ports on 2 switches (different models) and on all of them only Queue 3 Weight 2 has drops. Does it have anything to do with CS3 Qos marking that is configured on IP Phones by default?

Is it possible to check the drop threshold on specific ports and increase it somehow?

"Does it have anything to do with CS3 Qos marking that is configured on IP Phones by default?"

It may, although VoIP traffic tends to usually use relatively little bandwidth.  Other traffic might be marked CS3 and other traffic might have a different marking but directed to the same place.  There should be a way to query the device what traffic get directed there.

"Is it possible to check the drop threshold on specific ports and increase it somehow?"

Usually yes, but I suggest a useful policy be implemented on all ports unless your exceptional policy is only applied to exceptionally few ports.

Do you use multi Gige for ingress and one gige for egress?

You need to add PO I think for this issue

MHM

Hello,

Topology is like this:

Cisco Phone 1 - Switch A - 9300 Switch - Switch 8 - Cisco Phone 2

All uplinks are Gig. Cisco 9300 has other switches connected to it as well

What is PO?

PO meaning port channel' 

You need to faster transfer traffic before it full Queue.

MHM

Well, according to Solarwinds, these Gig interfaces are hardly getting 100mb/s of traffic in Gig uplink, do you think PO will help?

100 mbps maybe from traffic drop and drop in queue.

So sure try use PO abd check status.

Hope this help you in your issue

MHM

Sorry, did not understand you comment about 100mbs

For 100 mbps first check if interface work as 1 Gbps or 100 mbps

Do show interface 

Show interface status 

MHM


@FuadG wrote:

Well, according to Solarwinds, these Gig interfaces are hardly getting 100mb/s of traffic in Gig uplink, do you think PO will help?


Unfortunately, that's a very, very (very) common misinterpretation of interface stats.

Laugh - so ponder on why a gig interface hardly ever even showing 10% utilization drop packets?  If that doesn't seem to make sense - exactly!  Are those stats wrong?  Probably not.  Again, often they are misinterpreted (or misunderstood).

Hello @FuadG ,

you should implement QoS on all switches in the path as suggested by @Joseph W. Doherty .

>> Cisco Phone 1 - Switch A - 9300 Switch - Switch 8 - Cisco Phone 2

You need to find out what kind of switches are Switch A, Switch B and their OS system version.

Ideally , you should be able to call also without QoS applied if the network load is low but you may have production data traffic and without QoS you cannot avoid issues like big data packets being served = put on the wire before the small delay sensitive VOIP packets. This can create delay variance = jitter that impairs voice quality.

Hope to help

Giuseppe

 



@Giuseppe Larosa wrote:

you should implement QoS on all switches in the path as suggested by @Joseph W. Doherty .


I don't believe I actually explicitly suggested that, although I may have inadvertently implied that. ; )

As to whether QoS is actually needs to be configured depends on whether it actually is also needed and can deliver the results needed.  I.e. often it's not actually needed to be configured end-to-end; still, though, it's very advisable to do so.

That said, the "book" suggests that it should be, and the "book" recommendation is good because only with QoS configured end-to-end can you provide end-to-end service guarantees.  Also, networks, and their usage, can change.  If you don't have end-to-end QoS, what works fine today, might not work well tomorrow.

So am I now, recommending end-to-end QoS?  Well . . .

QoS has its pros and cons.  Cons include, QoS has its own "care and feeding" requirements.  Further, depending on your needs, a one size fits all QoS model might not work for you.

So, what I actually often recommend with QoS, is a case by case analysis of its RoI.  If some cases, usually like WANs, QoS often has a great RoI.  In some other cases, usually like LANs, other approaches might have a better RoI.

Also, understand, you can mix and match.  For example, a complex QoS model for WAN interfaces, and a simple QoS model for LAN interfaces.


@Giuseppe Larosa wrote:

Ideally , you should be able to call also without QoS applied if the network load is low but you may have production data traffic and without QoS you cannot avoid issues like big data packets being served = put on the wire before the small delay sensitive VOIP packets. This can create delay variance = jitter that impairs voice quality.


There's much truth in the forgoing, but "low" traffic levels don't guarantee VoIP success and conversely "high" traffic level don't guarantee VoIP problems.  Also "big" data packets aren't, in themselves, even jumbos, necessarily (or in modern networks often) adverse to "small" VoIP packets.

Although often traffic utilization level and/or packets sizes seem to go hand-in-hand with VoIP issues, there are other variables that matter that are often not considered.

Simple example:

On a 64 Kbps link, it takes 187.5 ms to serialize a single 1518 byte Ethernet frame. Consider, the usual maximum bound for VoIP latency is 250 ms, and 150 ms or under is recommended, 187.5 ms isn't particularly good.  This is an example of the "big" packet issue, Giuseppe mentions.

For low bandwidth interfaces, Cisco had/has a Link Fragmentation and Interleaving (LFI) feature.

So if 1500 is borderline bad, a 9000 byte jumbo must be out of bounds, correct?

Yes it would be on a 64 Kbps link, but what if used on a gig link?

Then its serialization delay is 72 Āµs.  As VoIP latency tolerance hasn't changed, 72 Āµs isn't much of a concern.

What impacts VoIP quality is packet loss, overall latency and (as mentioned by Giuseppe) jitter, but problem causes are a bit more than just packet sizes (which is a factor) or average load utilizations, the latter usually measured in lots of seconds to minutes.  With VoIP, we need to understand, and allow for, "things" happening in milliseconds.

In other replies, I've remarked that Cisco's QoS information has greatly improved, over the years.

In Cisco's Technote, Troubleshoot Output Drops on Catalyst 9000 Switches, refer to the last section which shows how drop issues cannot be seen even at 1 second intervals, but can be seen at 1 ms intervals.

Thank you for detailed explanation

When I use this formula:

Percentage of output drops = Total output drops / Total output bytes X 100 I get an extremely low value.

Max Jitter on the web interface of the phone is between 1-10

Drops count are no longer increasing on 9300 switch in the table from my initial post, but Enqueue numbers are increasing constantly - is it something abnormal or it is as it should be?

If one side use 100 M and other use 1 G

Then sure there is drop

Some interface not support autonego you need to hardcoded speed. 

MHM

Review Cisco Networking for a $25 gift card