VoIP latency, is just overall delay (which for VoIP, there are acceptable limits - delay, shouldn't matter to NTP).
VoIP jitter is variable, which matters both to VoIP and NTP.
Going sideway a moment . . .
Years ago, I read a book about the development of maritime chronometers (clocks), knowing the (accurate) time was/is crucial for (accurate) celestial navigation. It was so important:
In 1714, the British government offered a longitude prize for a method of determining longitude at sea, with the awards ranging from £10,000 to £20,000 (£2 million to £4 million in 2023 terms) depending on accuracy.
The interesting thing about "accuracy", the chronometers, didn't need to have the "correct" time, but they needed to be very consistent in the time they might gain or lose, every day. I.e. if they had a consistent daily gain/lose, one could adjust for the change and then have accurate time.
I mention the above, because with NTP, overall latency, per se, isn't a problem, it's how variable measurements are, both the variable latency between the NTP server and client, and also whether you can tell there's a constant gain/loss of the client's clock (much like the marine chronometer) vs. the actual time.
Anyway, if we go to ntp.org, we find:
5.1.1.2 How will NTP use a reference clock?
A reference clock will provide the current time. NTP will compute some additional statistical values that describe the quality of time it sees. Among these values are: offset (or phase), jitter (or dispersion), frequency error, and stability. Each NTP server maintains an estimate of the quality of its reference clocks and of itself.
5.1.2.1 How is Time synchronized?
Time can be passed from one time source to another, typically starting from a reference clock connected to a stratum 1 server. Servers synchronized to a stratum 1 server will be stratum 2. Generally the stratum of a server will be one more than the stratum of its reference.
Synchronizing a client to a network server consists of several packet exchanges where each exchange is a request and reply pair. When sending out a request, the client inserts its own time (originate timestamp) into the packet being sent. When a server receives the packet, it inserts its own time (receive timestamp) into the packet, and the packet is returned after putting a transmit timestamp into the packet. When receiving the reply, the receiver will once more log its own receipt time to estimate the travelling time of the packet. The travelling time (delay) is estimated to be half of “the total delay minus remote processing time”, assuming symmetrical delays.
Those time differences can be used to estimate the time offset between both machines, as well as the dispersion (maximum offset error). The shorter and more symmetric the round-trip time, the more accurate the estimate of the current time.
5.1.3.1 How accurate will my Clock be?
For a general discussion see Section 3. Also keep in mind that corrections are applied gradually, so it may take up to three hours until the frequency error is compensated (see Figure 5.1a).
Figure 5.1a: Initial Run of NTP
The final achievable accuracy depends on the time source being used. Basically, no client can be more accurate than its server. In addition, the quality of the network connection also influences the final accuracy. Slow and non-predictable networks with varying delays are bad for good time synchronization.
A time difference of less than 128ms between server and client is required to maintain NTP synchronization. The typical accuracy on the Internet ranges from about 5ms to 100ms, possibly varying with network delays. A survey by Professor David L. Mills suggests that 90% of the NTP servers have network delays below 100ms, and about 99% are synchronized within one second to the synchronization peer.
With PPS synchronization an accuracy of 50μs and a stability below 0.1 PPM is achievable on a Pentium PC running Linux. However, there are some hardware facts to consider. Judah Levine wrote:
In addition, the FreeBSD system I have been playing with has a clock oscillator with a temperature coefficient of about 2 PPM per degree C. This results in time dispersions on the order of lots of microseconds per hour (or lots of nanoseconds per second) due solely to the cycling of the room heating/cooling system. This is pretty good by PC standards. I have seen a lot worse.
You might also find of interest, figure 3 in: https://blog.meinbergglobal.com/2021/02/25/the-root-of-all-timing-understanding-root-delay-and-root-dispersion-in-ntp/
