Hello!
Another thing that can increase your latency is queue length. If you have the line fully utilized and the interface cannot process all necessary packets, they will wait in a queue. If this queue is filling in, and you have no policy apply to it, it will start to tail drop packets, so a packet that is dropped has to be retransmitted, adding latency.
If you have packets with EF and there is congestion, then this packets will have priority in the queue and they will be send before other packets (regular priority). Usually Voice packets are marked with EF bit.
Cheers,
Calin