Does you VoIP traffic use the ISP path?
Where does your VoIP traffic flow?
If across your "WAN", that could be the trouble spots, again, as WANs tend to often had much less bandwidth.
BTW, even with 100% continuous bandwidth consumption, with QoS, such consumption can be a non-issue. For example, years ago, we had a couple of WAN links that ran at 100% all day long, during business hours. Ran all kinds of traffic across those links. QoS insured traffic like VoIP got the bandwidth it needed. (Incidentally, what caused the links to be so saturated, an application was installed to do constant backups of user PCs. Within QoS, that traffic only received left over or unused bandwidth, but that's what drove the links to 100%.)
Conversely, low average bandwidth consumption and/or no drops doesn't guarantee you don't have an issue because of the adverse impact of one traffic type to another during "microbursts". Usually you have sporadic issues with some "sensitive" traffic kinds (like VoIP), but often this is "invisible" to common network monitoring.
From decades of experience, if your VoIP traffic is crossing WAN links, I would analyze those first. Unfortunately, many network engineers (i.e. your WAN management provider) are not used to dealing with performance at the millisecond level, so often it's a uphill battle with them until you rub their nose in the problem. SLA tests can be useful to document the issue.
Often when a performance issue is recognized, the typical offered solution is, you need more bandwidth. Truthfully, you may, but many times QoS is a "better" solution. (Understand, if your bandwidth is such that no link can be oversubscribed, you're golden. But, that's very, very unusual. For example, if you have a switch with 48 ports, the uplink would need 48x the bandwidth of one of those ports to not be oversubscribed. Consider doing more of that at every level.)
Also, unfortunately, a WAN provider is often used to offering (selling) more bandwidth, not usually experienced in better management of your bandwidth.
One example I used to do, to demonstrate the "power" of QoS, I would setup a pair of routers, back-to-back, using a T1 serial, and ping and/or telnet between them. All fine.
I would then connect a host to one router, and blast 2 Mbps of traffic across the 1.5 Mbps T1. Pings, if they got across at all, had horrible response time, telnet became practically unusable.
Then, I activated FQ on serial interface, i.e. added just one interface command. Ping and telnet performance almost returned to the performance before I started the additional traffic stream, which was still running the link at 100% with a huge drop rate.
What I showed, was unlikely to happen real world, but the performance improvement that QoS provided, also works at the millisecond level, and all it took was one interface command. Of course, if I replaced the 1.5 Mbps with a link that could handle 3 Mbps or better (BTW, 2 Mbps or so, still very likely to see issues), should also restore ping/telnet performance. However, typically increasing WAN bandwidth is rather expensive compared to adding one additional interface configuration statement.
