Cisco 9300 queues and output drops

[FuadG]

Hello everyone,

I am trying to troubleshoot VOIP related problems when voice during calls is intermittent

The phones are connected to separate switches and there is Cisco 9300 in between those switches.

I have read through this article:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-switch/216236-troubleshoot-output-drops-on-catalyst-90.html#anc13

And increased qos queue-softmax-multiplier 1200 

Now drops are not increasing anymore in this table, but TH0/TH1/TH2 are still increasing

AQM Global counters
GlobalHardLimit: 5176 | GlobalHardBufCount: 0
GlobalSoftLimit: 14672 | GlobalSoftBufCount: 0

----------------------------------------------------------------------------------------------
High Watermark Soft Buffers: Port Monitor Disabled
----------------------------------------------------------------------------------------------
Asic:0 Core:1 DATA Port:15 Hardware Enqueue Counters
-------------------------------------------------------------------------
Q Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
(Count) (Bytes) (Bytes) (Bytes)
-- ------- -------------------- -------------------- --------------------
0 0 15626884274 32907700571 2516556011 
1 0 0 0 7226027781
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
Asic:0 Core:1 DATA Port:15 Hardware Drop Counters
-----------------------------------------------------------------------------------------------------------
Q Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
(Bytes) (Bytes) (Bytes) (Bytes) (Bytes)
-- -------------------- -------------------- -------------------- -------------------- --------------------
0 14586413831 614569504 0 0 0
1 0 0 15293510335 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0

Can this queuing affect voice traffic? 

[darren-oconnor]

hi @FuadG - what is your telephony platform, cloud or on-prem?  What QoS do you have configured end to end (phone through to telephony)?  What are the voice related issues you're experiencing?  Are the calls inbound, outbound, internal (IP to IP), landline, mobile etc etc?

[FuadG]

Hello,

On Prem CUCM, we have no qos configured. Voice is intermittent, like being dropped at some point and then comes back. Calls are internal, between 2 internal cisco ip phones 

[Joseph W. Doherty]

What queue are your VoIP packets directed to?

[FuadG]

We are not using qos at the moment, how can I check which queue are voice packets being directed to?

[Joseph W. Doherty]

First, you need to determine what, if any, L2 CoS or L3 ToS markings are set on VoIP phone.  Then you need to examine each transit switch's config.

BTW, do the VoIP phones have their own network jack, or do the VoIP phones also a PC connected to them?  Are there dedicated VoIP VLANs?

With QoS, usually you can insure excellent VoIP, regardless of what else the network is being used for.  (Even something like the bandwidth "hungry" G.711 codec only uses about 100 Kbps, so that much bandwidth can usually be provided.)

[FuadG]

Hello Joseph,

Cisco phones are connected through PC and there is a dedicated Voip VLAN.

Seems like this issue only affects Cisco 7811 and 7821 models, as 79xx are working fine. It also seems like there is a burst of traffic happens that affects voice calls. So when call is made again in 2-3 sec, the issue disappears. 

[Joseph W. Doherty]

---

@FuadG wrote:

Cisco phones are connected through PC and there is a dedicated Voip VLAN.

Phone are connected through PC?  Are your sure it's not that the phone connects to the wall jack and the PC connects to the phone?

So the edge ports that have the phone PC combos are configured with voice VLAN?

---

@FuadG wrote:

Seems like this issue only affects Cisco 7811 and 7821 models, as 79xx are working fine. It also seems like there is a burst of traffic happens that affects voice calls. So when call is made again in 2-3 sec, the issue disappears. 

---

I vaguely remember, one shop I was in had some early Cisco VoIP phones, don't recall the model, but recall the had FE interfaces, and we found they couldn't provide 100 Mbps to the PC.  (We replaced them after Cisco said they couldn't correct the issue.)

Don't recall issues being reported about VoIP issues, but if they could not support the downstream PC at 100 Mbps, I would wonder what impact 100 Mbps being sent to or from the PC might have on VoIP quality.  Interesting you mention a burst of traffic impacting the voice call.

If the VoIP phones, themselves are the issue, at least with high bandwidth transit PC data, possibly a simple fix, only use those model VoIP phones connected to a dedicated voice port.  Otherwise, replace them.

[FuadG]

@Joseph W. Doherty I am sorry, my mistake, PC's are connected through phones of course

Yes, using dedicated ports or replacing the phones sounds like a plan, I was just trying to understand, what could be the root cause of this issue, since these phones were working fine for 3-4 years now 

[Joseph W. Doherty]

"I was just trying to understand, what could be the root cause of this issue, since these phones were working fine for 3-4 years now"

Possibly some new or revised app?  Something that has increased port traffic?

[FuadG]

I did a little bit of testing now

So I connected 7811 phone and a PC behind it. I started Speedtest on the PC and Total Output Drops started to increase rapidly on the switch port. While speedtest was running, I was talking to this phone and there were no voice issues. I was also pinging the phone and time increased from 1ms to 5-9ms intermittently

Then I disconnected the phone and connected PC directly to the switch, initiated speedtest and Total Output Drops started to increase again.

These results leave me clueless - seems like these drops do not necessarily affect voice quality issues 

[Joseph W. Doherty]

"These results leave me clueless - seems like these drops do not necessarily affect voice quality issues"

That can be perfectly normal if your speed test traffic and VoIP traffic are in different queues.  That's how QoS insures voice quality.

[FuadG]

But we dont have Qos configured currently

[Joseph W. Doherty]

---

@FuadG wrote:

But we dont have Qos configured currently

---

I didn't see your reply until after posted my prior reply.

In it:". . . the later gen Cisco switches (starting with 3650/3850?) actually implement a simple QoS model by default."  This includes the 9K series, I believe.  Your OP stats show at least two queues with non-zero counters.

[FuadG]

Hello @Joseph W. Doherty 

I am still investigating this behaviour. I have upgraded 7811/7811 firmware, now waiting for results, so far no Output Drops, but still monitoring

One curious question, I have noticed that only Queue 3 Weight 2 drops are incrementing. I checked on different switch models, and only this queue and weight is getting drops. There are even other topics related to Output Drops here in this forum and over the internet and all of them shows particularly this Queue and Weight drops. What is so specific about this queue? 

Interface Gi1/0/25 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 8568
Queue 4
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 5
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 6
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 7
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0

I saw suggestions to increase the buffer size for this queue without implementing Qos with this command:

mls qos queue-set output 1 buffers

Is it something that can help?