DTMF Digits Sending Incorrectly?

[pdavapark]

So I've been through with wringer with WebEx support on this and am drawing a blank... We recently (After getting hit with large overage charges on our toll-free usage) began asking all users to switch to calling into the WebEx meetings using the toll number provided by WebEx. About a week after doing this, 4 or 5 users have come mentioning issues where their digits entered for the meeting ID are read back incorrectly by WebEx which leads them to being kicked off the call and unable to join their meeting. This ONLY happens if they call using the toll number - call in using the toll-free and everything is perfectly fine. It also appears this MAY be tied to only one meeting, but I haven't confirmed that quite yet for sure.

This sounds like a DTMF signaling issue, because random digits in their string will be duplicated, similar to an old bug I read about, but we're on IOS 15.04 on our 2911 GW which is well past when that bug had been fixed.

I submitted a support ticket to WebEx and they can't reproduce the problem, but it seems to be popping up pretty frequently for at least all users getting into this one specific meeting whether calling from a desk phone (7941/8945 SCCP or Jabber Windows client). All calls exit the same SIP GW from our datacenter, but the fact this can ONLY be reproduced when calling WebEx's toll-free number tells me it may be something on their end.

The only dtmf related config on our GW at the moment is dtmf-relay rtp-nte on all of our dial peers. I was tempted to play with some of these settings, but seeing as this has never been an issue in the 18 months+ since implementing our CUCM it seems strange to believe it could be something on our end. Only way to know for sure seems like it would have to be a packet capture and debug on GW at the time of joining the meeting to see what's really being sent?

Thoughts? 

[Charles Hill]

Many moons ago, I experienced a similar issue, twice and with two different customers.  After months of troubleshooting we finally determined the dsp farm was causing the issue.

Second trouble, DSP farm was the first thing we looked at.  School of hard knocks. :)

Hope this helps.

Please rate helpful posts.

Thanks.

[pdavapark]

Something configuration related, firmware on the card, or an actual faulty component? Funny you say that because our last seemingly impossible to find issue was dsp farm related as well.

[Charles Hill]

Bad DSPs.  Both times.

Trouble was dtmf digits were being duplicated or missed.

[kdotten36]

This is happening to us now but not specific to Webex, it's any random outbound call.  A user tries to press 1234 for example and what's read back is 11122233333444 or some variation. 

DSP farm eh?  I'll look into it, but do you have any further advice on your resolution?  Been playing with my dtmf relay settings for the last couple hours but I can't reproduce it on any calls yet.

[pdavapark]

I finally yesterday was able to reproduce this a lot and got some good packet captures of the duplicates to finally see it's definitely on our end. The two Wireshark screenshots of are of the same capture, just one of them being on the inside leg of my call and the other on the outside leg (After being through the DSP). You can see each DTMF is being duplicated, and further down that call was the ending pound sign being tripled or quadrupled actually. Sometimes all digits get it, other times only one or two, other times it works A-OK. I now have a TAC case open for this and am started to believe it's a faulty DSP card, especially since it seems to be higher utilization on the CUBE that causes this, which has two DSPs, a PVDM3-16 and -64, and I'm wondering if only calls being coded on one of the DSPs are getting this issue, hence it being so intermittent.

[kdotten36]

I'm seeing 16 packets for each key press.  As I understand, it's supposed to be 7 or 8 for redundancy and a duplicate for each tone over 50ms long.  I'm just looking at a "debug voip rtp session named-event" log from the router though.  I dialed back in and just tapped the buttons quickly and checked the trace and am only getting twelve packets.  Maybe the timing is off or something.  

[ansonantoo]

May be this would be same case with my scenario as well...user was opting call back option after login to webex and he received the call on VoIP. After that he got IVR saying press 1 to join the meeting. This DTMF wasnt passing. All other inbound and outbound DTMFs are working fine except this WebEx incoming IVR call. I couldnt simulate the issue and still issue remain unsolved..

[pdavapark]

Very interesting, makes me wonder if WebEx is still the cause of the problem somehow. We call into other teleconferencing systems every day without issue, and have never once seen this issue on the WebEx toll-free number. Inbound DTMF works fine as well with customers calling into our UCCx ACD.