Latency per se will not have a negative impact on the VoIP traffic, unless it is (from end-to-end, i.e., the IPS latency + the "normal" network latency for the link) fairly extreme.
The usual killer is extreme variances in the latency ... usually handled by the "deJitter buffer" on the phones.
Good Luck
Scott