Double Dialing

[Stan Coleman]

UPDATE 03/02/2017 01:49 PM Central: Time between dialing is 60 seconds not the 1 second I posted. Patrick Sparkman very eloquently corrected me in his post.

We have an issue were Conductor will call a Participant twice in the same conference. The first call gets connected and shows up as Ad-Hoc in TMS. Depending on how the receiving codec is setup the second call either never connects or it connects a second time. All our endpoints are setup to not except a second call when in a call so it doesn't cause us a lot of issues. However we have several outside organizations that we connect to that DO allow multiple incoming calls at the same time. For those units the second call connects as well as the first call. The second call that connects also connects as Ad-Hoc and the system starts dialing a third time. The multiple connections go on and on until either we reach maximum connections on the endpoint or someone gets tired of all the high pitched feedback and calls us.

This issue appears to be completely random. We can go for days or weeks with no issues and then have a day full of them. All our calls are scheduled and the issue doesn't always effect the same conference in the reoccurring series. However we have seen cases where it will be specific to a reoccurring scheduled call as well. We are using our system to connect Classrooms together. The classes are schedule for 16 weeks in advance so the reoccurring scheduled calls are very large. Classes that happen 5 times a week for 16 weeks generate a large number of conferences. We have had it happen when we load down the bridges with 6-8 conferences running an average of 6 endpoints a piece and when we only have 2 conferences on the bridge. Load doesn't appear to be part of the issue. All conferences do start at the same time. It's not uncommon to have 6 conferences all start at the exact same minute.

When you look in the VCSc log you will see that there are two requests to add the endpoint separated by about a second.

2017-03-01 08:00:14	SIP (INVITE)	10.xxx.xxx.xxx	sip:Greeley/W_LSRM200_211@xxx.xxx.xxx.211	Found	View
2017-03-01 07:59:14	SIP (INVITE)	10.xxx.xxx.xxx	sip:Greeley/W_LSRM200_211@xxx.xxx.xxx.211	Found	View

This may be a poor example because its an H323 call. You'll just have to trust me that it happens on both H323 and SIP calls.

Each call to the participant has it's own Call Serial Number and Tag. Other than that the logs in VCSc look identical for both calls. All the same rules are applied to both calls. If the call goes outside our network then there are two more calls listed on VCSe with the exact same time stamps as VCSc.

It has happened when we connect to our in-house SX80's, C60's and when we connect to Lifesize and Polycom units.

Our setup is TMS, Conductor, Three MCU5320's, Three VCSc and one VCSe. Really need the Communities help on this one as I have had a TAC case open for months on this with no resolution. TMS,Conductor the MCU's and VCS's are all on the same subnet and in the same rack.

[Stan Coleman]

Originally the Connection Timeout for Scheduled Calls was set to 60 second and we had Double Dialing.

Connection Timeout for Scheduled Calls and Allocation (in seconds):

We decreased it from 60 to 30 seconds and we still have Double Dialing.

The interesting part of the log to me is the two lines that say 'Ad hoc call added: omitted-by-bridge'. Those are not Ad-Hoc calls those are two Participants that were apart of the originally scheduled call. For some reason TMS doesn't recognize those two Participants as having belonged to the conference and marks them as Ad-Hoc and then goes on to call the same Participants again. That's when we get the Double Dialing. One connection marked Ad-Hoc and another one with the same name that TMS then goes on to call again.

3/17/17 9:59:02 AM    ConferenceInfo        CTC    Instance: Ad hoc call added: omitted-by-bridge
3/17/17 9:59:02 AM    ConferenceInfo        CTC    Instance: Ad hoc call added: omitted-by-bridge

[fwadmin]

Has a resolution been found? I am experiencing the exact same issue.

[Stan Coleman]

No, no resolution has been found. It happens less when we aren't running as many conferences. This summer we have only had two occurrences since May. However we aren't running even half as many conferences over the summer as we would this fall or did this past Spring.

We have looked in a lot of areas but plan on revisiting QoS settings as a possibility. We do know that the first call in a Double Dial is missing the ACK from the End Point. However even with a missing ACK the call connects. Somehow TMS doesn't seam to believe it is the End Point it request to connect and marks it as an incoming Ad-Hoc Call. Then TMS goes on to dial the End Point again. 

[Stan Coleman]

Update: Still working on a solution.

[Wayne DeNardi]

I have seen some Double Dialling issues previously when there were some issues with the Search Rules and Transforms on the VCS.  

[Stan Coleman]

I have had several Cisco techs look at and walk through all the search and transform rules. At this point I think it highly unlikely that it's a dial plan issue.

We did discover though that TMS was roughly 45 seconds behind CTC, and VCS units. TMS runs on Windows 2012 R2 which syncs with time.windows.com by default. CTC and VCS units have a setting for NTP. So while TMS was syncing with Microsoft, CTC and VCS units were syncing with mytime.mycompany.com. To make a long story shorter I figured out how to get the Windows server to sync with mytime.mycompany.com. Now that they are on the same time we haven't had any Double Dials.

Warning. In the past we have gone several weeks after we changed a setting without a Double only to watch it come back. Aronicly time will only tell if the changes to Time work.

[Stan Coleman]

At the risk of jinking myself, it finally appears we MAY have some movement on the issue. We changed the DNS server settings to a different set. I'm not a DNS expert, the way I understand it is the current set can lookup codecs and TMS faster than the old set.

Fingers CROSSED. Thursday will be one week with the codecs on the new servers not having any Double Dials.

[CGL]

Hi Stan, 

We are having this same issue at the moment. 

Did the DNS change and/or NTP change actually fix the issue?

Thanks 

[Stan Coleman]

Oddly, the DNS change was the most promising. Have to admit it made no sense to me but at this point, anything is worth a shot. The DNS change seemed to work when we had two-thirds of the codecs changed. Ran for weeks with no issues so we changed the last third only to watch the problem come back. 

It seems, in our case, to be dependant on load. In the summer we don't even run half the load we do in the winter. Had maybe a handful of Doubles at the beginning of Summer, then nothing afterword. 

If its anything like the past we will be able to resume testing again in a couple of weeks.

[Wayne DeNardi]

Rather than having all your devices query NTP off the Internet, set your Windows Domain Controller to get its time from the Internet, then set all your endpoints (and other devices on your LAN) to use the Domain Controller as their NTP source.