A few additional comments on Jens' additional comments. (If uninterested in deeper QoS concepts, please ignore.)
Marking traffic is only the first step as marking doesn't prioritize traffic by itself. You also need to implement rules based on this marking, e.g. putting the EF-class into the priority queue or reserving a certain amount of bandwidth on some links. So marking is only the basis to implement the appropriate actions.
To be clear, using ToS markings, alone, is a very worthwhile goal, but it's not an absolute requirement (in the context of Jens's paragraph, his "marking is only the basis", I believe is not intended to imply that).
ToS marking is for efficiency. It's probably much less resource intensive to process a packet based on a ToS tag than subjecting it to deep packet analysis, such as using NBAR as in OP.
One risk of not performing a deep packet analysis, all the time, you don't insure the packet is appropriately marked. I mean, programmers would never do something like mark all their traffic with a DSCP EF tag; cough, cough - laugh. The general recommendation is, once you rely on ToS tags alone, it should be behind a trust boundary, i.e. where you know the tag has been validated and can be trusted.
For example, possibly all the traffic coming from the cloud voice provider, in OP, is already marked DSCP EF. If it is, and if you trust them, no need to use NBAR nor remark. Or, all traffic is marked IPPrec5. Again, if you trust that marking, you might remark as DSCP EF, and not use NBAR. (BTW, ToS tags can be changed in-transit.)
In general, you need to implement QoS on all your devices across your network, i.e. end-to-end from the entry point into your network up to the end user. Just a single switch on the way that is not or incorrectly configured for QoS can render all your efforts useless. QoS needs careful planning and implementation, otherwise it can make things even worse.
In general, the recommendation is to use QoS end-to-end. Also, indeed, just one transit device, without the "correct" QoS policy might render other QoS ineffective, or overall worse, not necessarily all traffic being handled by QoS, but that's possible.
However, the corollary is, possible QoS isn't actually needed on any or all devices. The end-to-end recommendation, "insures" you shouldn't be caught by surprise. So, again, is QoS truly "needed" on every device, end-to-end, no, but it's a good recommendation.
Finally, you may not need to implement QoS at all if all your links have sufficient bandwidth and you do not face any bottlenecks. OoS kicks in e.g. if links get congested but otherwise you may not notice any difference with or without QoS.
Definitely, QoS isn't always "needed", but in my experience, terms like "sufficient bandwidth", "bottlenecks" or even "QoS", are not deeply (correctly?) understood.
BTW, some QoS features kick in, without congestion, such as shaping and policing.
A real world example, decades ago, I supported a 6513 that had 11 (6196) line cards, each with 96 10/100 Mbps PoE ports, supporting VoIP phones with downstream PCs. It's been so long, I forget what the dual Sups were in that 6513, possibly the original Sup32, but let's assume there was but just one sup using just one 10g uplink.
Often, it's assumed if there's no oversubscription, there's no need for QoS.
Well, a 10 g port, has the same bandwidth as 100 100 Mbps ports or a 1,000 10 Mbps ports. So, in theory, if we only enabled 1,000 of our 1,056 10/100 ports, and ran all at 10 Mbps, there's no oversubscription, so no need for any QoS, right?
Well, suppose the very, very (very) unlikely event happens that all thousand 10 Mbps ports transmit into the switch, all with a destination reached on the 10g uplink. Even the 10g port can only transmit one frame at a time, so we need to queue 999 frames, right? If we don't we'll drop frames, right? So, is this a QoS consideration?
If only the VoIP phones were being used, and even if they were using (bandwidth hogging) g.711, which uses less than 100 Kbps of bandwidth, our total bandwidth consumption would be only be 100 Mbps for the 1,000 phones. A tiny fraction of a 10g port. So, there no "congestion", but a moment ago, we had to possibly queue 999 frames!
BTW, my personal definition of "congestion" is whenever you want to transmit a frame/packet and you cannot immediately do so, you have congestion.
When you do have congestion, how long you need to additionally wait, or whether you're dropped is what's impactful.
Low bandwidth average utilization stats don't always indicate all's good. Conversely, high (even 100%) bandwidth average utilization utilization stats don't always indicate there's a problem.
So, I fully agree with Jens' "QoS needs careful planning and implementation"! However, additionally, you also need to truly understand what specific kinds of traffic truly need, to work acceptably, and how different QoS features support you in accomplishing your QoS goals.
(Oh, and if anyone is wondering, for the above 6513, most of the ports were enabled, all were running at 100 Mbps, with VoIP and data traffic, and there may only have been two gig uplinks. So, there were QoS policies on this device; to assure VoIP service needs.)
