I setup a new office in Los Angeles and since there's only a few people there we couldn't warrant a PRI. Their calls to the PSTN are going out through a PRI in the Irvine office. Los Angeles has a 2821 ISR router with the usual DSCP 46 QoS enabled that we have in all our other offices. Bandwidth is not the issue either as they have a multilink T1 into the Century Link MPLS provider. Phones are 7962 and register with a CUCM 8.6. THere's
WS-C3750X switch there that uses the auto macro feature to enable auto QOS for cisco phones.
Below is an audio sample that one of our staff there recorded. It sounds a little similar to Cisco's examples at the link below although I'm not really sure which one. One of our staffers in that office describes it simply as "Tron voice". At this point I'm not sure if this issue happens with on-net G.729 calls but it definitely happens with some of the G.711ulaw calls.
Can I hear your expert opinions on this? I'd also like to request some troubleshooting debug methods that you think might allow me to zero in on where the issue occurs. Thank you!!
We had this problem in our environment.
We had a source of audio (a third party bridge), that was periodically having some performance issues due to a database configuration problem. During the database queries in the software there was an usually long delay, and some of the RTP streams would be stopped and then restarted after 5-10 missed packets as a result. When the RTP stream was restarted after a certain minimum threshold, the RTP header "marker" bit would be set to identify it as a new/restarted RTP stream, but the RTP SSRC tag would not change because it was technically the same RTP stream as it was before.
This happened fairly often under high load on this bridge, and the users would hear pops and clicks, and other audio quality issues. However, "Tron" audio was rare, and when it did happen, it only affected our 7962s. (We also have a large amount of 7940s, 7960s, 791xs, 7937s, etc; no reports of Tron audio issues on any of those phones.) "Tron" would only happen to one participant at a time, and the rest of the bridge users would hear no problems, even on the same call. If the user waited 90-120 seconds, it would resolve itself.
A captured network stream would play just fine in Wireshark, but the phone is playing out this garbage Tron audio anyway; so the problem seems to be related to the phone audio playback. (DSPs, firmware, whatever handles this.)
The vendor for the 3rd party bridge told me that Cisco phones are known to have issues with RTP marker bits without an associated changed SSRC.
Anyone that has this issue can turn on Span to PC port on the phone, then set a filter for RTP markers in Wireshark on the attached PC. If you see these marker packets you should investigate the source of that RTP stream for performance or DSP issues.
Note that if you use Wireshark, you might need to find the raw UDP and "Decode As" RTP. You should expect to see some frames at the beginning of a call when you use the "rtp.marker==1" filter. Also expect to see them at DTMF events, etc. But... they should not happen with no explanation.
In our case, we were able to correct the database issue and then return the 3rd party bridge to normal performance. As I mentioned above, the issue has not happened since.
Hope this helps someone.