Prioritization of Dante PTP and Audio Traffic with Auto QoS (mls qos)

hakan.topcu · ‎06-18-2024

Hello all,

urgent help needed: we recently set up a temporary infrastructure for a broadcast environment with several Cisco Catalyst C3560CX access switches connected with gigabit uplinks to a C9300-48UB as a central switch. We work there with Dante audio workstations and we have massive problems with audio dropouts. Colleagues report a production-critical condition. They suspect that PTP oder Audio communication is disrupted when larger audio editing projects are saved to the database at the same time.

In its network design guidelines, Dante recommends activating QoS on all switches with the following settings:

Priority - Usage - DSCP

High - PTP - CS7/56

Medium - Audio - EF/46

Low - Reserved - CS1/8

https://usa.yamaha.com/products/contents/proaudio/docs/dante_network_design_guide/201_qos.html

I understand that implementing QoS should take a larger context into account, but we have a temporary and almost isolated environment and a production-critical condition, so let's assume this is purely about optimizing Dante traffic.

How would one set up Auto QoS or does anyone have already experience with activating Auto QoS for optimizing Dante traffic?

C3560CX, Version 15.2(7)E
C9300-48UB, Version 17.09.04a

Many thanks in advance

hakan.topcu · ‎06-24-2024

To share my findings with you.

After activating "mls qos" on the C3560CX switches, I configured "trust dscp" on all access-port and uplink interfaces.

With the default configuration, the switch configures four egress queues with a guaranteed bandwith of 25%. Priority queue is not activated with the default configuration but queue 1 is limited to 25% per default, but that's still 250 Mbps and a lot for audio.

Because of the default DSCP to Queue mapping, Dante PTP (CS7) will be mapped to queue 4 and Dante Audio (EF) will be mapped to queue 1. Default traffic is mapped to queue 2.

Queue / Weight / DSCP / Application
1 / 25 / 40-47 / Dante AUDIO EF=46
2 / 25 / 0-15 / Default
3 / 25 / 16-31 / ---
4 / 25 / 32-39, 48-63 / Dante PTP CS7=56

This mapping seems not to be optimal for Dante, because PTP should get the highest priority. But because PTP and Audio each receive 250 Mbps guaranteed bandwidth with this configuration, we decided not to manipulate the mappings.

Statistics showed traffic corresponding to the DSCP values on the incoming interface and on the outgoing queues. So far so good, but colleagues reported problems with distorted audiostreams. This problem was gone, as soon, as I deleted the "trust dscp" on the interfaces, what means, that all traffic was simply enqueued to queue 2 with no problems.

Something seems not to work with the default configuration or with auto qos in general. Because we are talking about only a very few audiostreams that were running with no or less competeting traffic during the test, the bandwidth should not be an issue. Unfortunately, we have not yet been able to determine the cause, because the tests had to be stopped due to productive operation.

Joseph W. Doherty · ‎06-24-2024

"This problem was gone, as soon, as I deleted the "trust dscp" on the interfaces, what means, that all traffic was simply enqueued to queue 2 with no problems."

But MLS QoS was still enabled, correct? If so, that's a curious result. Curious it's better using Q2 alone rather than using all 4 egress queues. (BTW, when QoS is disabled, there's only one egress queue, but when using just one of four egress queues, buffers are reserved per egress queue, so often drops, for the latter, increase.)

Additionally, ". . . with no problems.", then you have a usable solution, correct?

"Something seems not to work with the default configuration or with auto qos in general."

The default includes QoS disabled, correct? Again, using a single egress queue or just one of four egress queues often have different behaviors because of buffer allocations.

Auto QoS, though, well, when you enable it, it makes lots (and lots) of changes, and implements whatever Cisco felt Auto QoS should be at the time that particular IOS was released. It's probably one of the later 12 (logical) class models mapped into the 4 egress queues. After Auto QoS activation, it can be modified. How suitable that model is, for your needs, cannot easily say.

BTW, if you enable Auto QoS, recall in also creates interface ingress policies, which may DSCP values differently than the default DSCP mappings.

Also BTW, glancing though other Dante documentation they consider QoS a must for 100 Mbps, but often optional on gig unless there a lot of other non Dante traffic, is there?

hakan.topcu · ‎06-25-2024

@Joseph W. Doherty wrote:
The default includes QoS disabled, correct?

I mean the default "mls qos" configuration. I did not reset the whole qos configuration when the problem occurred, but just the trust status on the interfaces. Therefore I said that all traffic goes through q2 now.

@Joseph W. Doherty wrote:
other Dante documentation they consider QoS a must for 100 Mbps,

As already explained, I do not believe in a bandwidth problem, because all relevant ports are gigabit.

In our current production environment we use about 30-40 dante devices connected to multiple C3560CX 8port switches, which are connected to a central C9300-48UB switch. After we had the initial problems with the audio dropouts, we moved the database servers from the small switches to the C9300-48UB central switch and applied qos to all relevant ports on the central switch. This seems to have solved the worst problems.

When we tested the Auto QoS configuration on the C3560CX 8port switches, audio was distorted (not dropouts!) when interfaces are set to trust dscp. We tested a signal, that went from one port to another port on the same switch with a 48k stereo signal (about 3-4 Mb/s). As soon as I deleted the trust dscp, the signal was clean. PTP status on the devices was "green".

Unfortunately I was not able to did more tests. But I suspect that there is a little latency or jitter problem, when PTP and Audio is scheduled to different queues on the C3560CX. But this is only a guess

Joseph W. Doherty · ‎06-25-2024

Still a tad unclear.

If the 3560CX operates as the earlier 3560s, the initial default is no QoS. From that you could just enable QoS or Auto QoS. The latter also enables QoS but with many and extensive QoS configuration modifications. Once made there was no simple way to unmake them. In other words, there are two different default QoS configurations. So, what's unclear which was being used.

However, if there's no congestion, whatsoever, one would expect no significant difference between QoS disabled, default QoS or default Auto QoS.

There is another difference between disabled QoS and enabled QoS. The former totally ignores ToS while the latter will set ToS to zero unless trusted or there's an ingress policy.

Your theory that enabled QoS, with trust, is causing additional latency/jitter seems plausible. Normally if that is happening, it wouldn't be enough to impact typical application traffic, but you're working with PTP. Interestingly, the Catalyst 9k series is the only Catalyst series I found described as being PTP switches.

Unless I misunderstand, you have a running configuration that doesn't cause an issue, correct? If so, what the "ask"? Perhaps just confirmation of your theory?

From all described, possibly disabling QoS would work for you too, offering, possibly, less processing latency, increased single queue resources and without resetting ToS.

hakan.topcu · ‎07-01-2024

@Joseph W. Doherty wrote:
Unless I misunderstand, you have a running configuration that doesn't cause an issue, correct? If so, what the "ask"? Perhaps just confirmation of your theory?
From all described, possibly disabling QoS would work for you too, offering, possibly, less processing latency, increased single queue resources and without resetting ToS.

Since my last change, by deleting the trust settings on all interfaces but leaving global "mls qos" enabled, no complaints were heard from the colleagues. In so far, there seems to be no issue with the running configuration. Enabling/disabling "trust" worked like a switch on the audio quality: trust on = distorted, trust off = clean.

I assume that it will work with the default configuration (qos disabled) on the C3560CX as well. But we decided to leave it like that for now, because we have already been in a productive environment. We were able to solve the initial problems with audio dropouts using QoS on the central Switch c9300 and doing some small topology changes.

Joseph W. Doherty · ‎07-01-2024

Firstly, thanks for the update.

Secondly, glad you have found a working config.

Lastly, your results are surprising unless, perhaps, you still have the remnants of Auto QoS configuration changes.