CUBE Gateway - strange delays on outbound calls / mystery debug message

[TONY SMITH]

Hi,

On Friday the site reported delays on outbound calls, reporting something like 90 seconds from when they dialled to when the call started to ring.  Tracing it down the delay was in the CUBE gateway, the gateway receiving the Invite and sending back Trying almost immediately, then sitting on he call for a long time before finally sending the outbound Invite, after which operation was normal.

During that waiting time the debug showed repeated messages like this ...

1171886: Jun 12 11:15:59.862: //241053/673072000000/CCAPI/cc_api_get_fork_stream_ids:
   Unable to get stream ids
1171887: Jun 12 11:15:59.862: //241054/673072000000/CCAPI/cc_api_get_fork_stream_ids:
   Unable to get stream ids
1171888: Jun 12 11:15:59.862: //241065/8A5B21800000/CCAPI/cc_api_get_fork_stream_ids:
   Unable to get stream ids
1171889: Jun 12 11:15:59.862: //241075/ACED3A800000/CCAPI/cc_api_get_fork_stream_ids:
   Unable to get stream ids
1171890: Jun 12 11:15:59.862: //241078/B4ACDF000000/CCAPI/cc_api_get_fork_stream_ids:
   Unable to get stream ids

Filtering for just my test call id it showed this logged these messages at around 1 second intervals during the delay/

Has anyone seen this message or can shed any light on what it means?  That's the only message that hints at an error, and in fact are the only debugs for that call id during the wait.

What's odd is that the fault affected both his live gateways.  That maybe suggests its something in the way that CUCM presents the call that causing the issue.  I'm not aware of any changes on the gateways immediately prior, or anything relevant on CUCM.  The only oddity prior to the delay is that the CUCM->CUBE Invite in my test call was DO, whereas the trunks are both set to EO.

Even more oddly this morning the problem seems to have cleared.  And the debugs for a test call show EO from CUCM.

Any thoughts welcome.  There is a Cisco TAC case but the engineer said they'll need some time to investigate.

[Jonathan Schulenberg]

Is CUBE forming calls to a recording server? If it was and the test calls were DO it’s possible the recording logic isn’t able to build an EO to the recording server.

[TONY SMITH]

Thanks.  There's no call recording, that's why it's weird to see references to call forking.

I've done some sanity checking, definitely no configuration changes were made to the gateways, and I am confident they wouldn't have been reset or anything as inbound calls are very business critical.  Changes could have been made to Callmanager, there was an unrelated issue with a different SIP trunk that's not part of the normal RL/RG for outbound calls.  Possibly something was done that affects ALL CUCM SIP trunks.  But I don't know where to look for that.

And even more strangely the problem has now gone away.  I looked at the SIP messages for a few calls this morning, and all show only 11 or 12ms between CUCM->CUBE Invite and CUBE->ITSP Invite.

[husali]

Hi Tony,

The cc_api_get_fork_stream_ids is a function that is executed for call recording, signaling forking, or to generate call stats, which is the case here. You will see it with "debug voip ccapi inout" and is just an informational message.

It is in no way the cause of the call setup delay you have experienced, which could be for many reasons.

Regards,

Hussain

[TONY SMITH]

Sorry I forgot that I hadn't updated the post.  It turned out to be DNS issues.   Although the gateway has a hard coded entry for the service provider proxy, the only device that it ever talks to, for some reason the gateway was trying to resolve a couple of SRV records with variations on that proxy's name.  For example if the proxy was "proxy.provider.com" then it was trying first for "_sip._udp.proxy.provider.com" type 33.  Repeating with additional domain suffixes.  In normal operation these would return immediate "not found" responses from DNS and the gateway goes on to use the hard-coded straight host entry.

During this fault the site had issues with DNS, so these attempts to resolve the SRVs were timing out.

Resolution is to remove the hard coded host entry and change the proxy references to explicit IP address. I am not sure why that wasn't used originally.  Some of the other SIP parameters need to use DNS names to match the SIP realm, but these don't affect IP communications as that is purely to/from the provider proxy.

[husali]

Glad to know the issue has been resolved and the root cause has been determined.

Regards,

Hussain