I use Verizon as my SIP trunk provider to my nyc cube (Version 15.0(1)M2,).
Everything was great clean call, but the 2nd week of April, my NYC office started projecting static and pops and clicks into 800 conference calls hosted by either Verizon conferencing or any other audio confernce provider when running via their cube.
The office side of the call can't "hear" the static, but others on the conference can.
Same with inbound calls from other analog sources, or cell phones to NYC cube. Callers report static, clicks and pops. I was told that even "on net" voip calls produce "static" by the Vz VoIP tech.
Nothing changed in my infrastructure back in April. I know that Verizon is making massive changes in NYC in the financial district due to all the flooding with Sandy.
Vz VoIP team reports some Jitter and Latency, and they can hear the static on the captures, yet can't isolate it.
My connection is a MPLS Ds3 that runs 20 feet on BNC directly into a VZ slick that is fiber all the way across the the street to the CO.
If I "fail" my NYC cube and force the SIP trunks and calls to process via my FL cube (over my MPLS circuit) NO static!
Can jitter and latency cause pops and clicks?
This static is so bad on 800#'s it almost sounds like a bug zapper if processed out my NYC cube. Run it out of FL cube, no issues.
OK can't help then. I wondered whether you had a G729a/b mismatch. I've seen (heard) symptoms a bit like this in this situation, especially after a hold/offhold.
Packet loss and jitter on a G.711 call would either be periods of silence, "tin man", "sounds like I'm in a tunnel". Static and popping are analog symptoms and there are only two places that could be introduced on your side:
This leaves only the phone but I seriously doubt ~800 phones had a hardware failure the same week! Also, if it were the phone you would have still heard it when using the Florida-based CUBE. If you're doing address hiding on CUBE it would only be changing the headers of the packet, not the RTP payload.
If I were in your shoes I would grab a packet capture from CUBE's perspective and play the call out in Wireshark. If you hear the problem then you have a local equipment failure, almost certainly a DSP. If you don't hear it, share the capture with Verizon and tell them to figure it out.
Guess, what... VERIZON PROBLEM!
After verizon time and time again kept telling me it was something on MY network. We finally got the correct group to do a packet capture on my Cube Packets "entering and exiting their SBC".
And...this analysis of packets revealed, NO EF marked on my networks outbound voice traffic entering their SBC! My edge router managed my VZ is marking all voice traffic outbound from my Cube to their SBC "EF". But "somewhere" past the VZ PE router and before it gets to their SBC, my EF gets stripped off!. However, when their SBC sends traffic back to my Cube, the traffic once again is EF.
Hats to the Vz group that finally discovered this. There are so many groups in Vz and none have total visability from end-to-end, its a miracle they found this. Only took 4 weeks to find it.
Now that it is found, they don't know "where" to fix it...yet! So, the saga continues.
I'm glad to hear that you're almost fixed.
Here's a Cisco white paper on some examples of call quality problems ...