Odd issue with C40s - sometimes they appear to disconnect inbound calls from an MCU, then call back.

[Chris Swinney]

Hi all,

C40 software - various but last recorded issue TC6.3.1

Calls are H.323 using the E.164 for dialling.

We have noted a really odd issue with some of our C40's. These units are being dialled directly from an MCU (I believe a Codian 8510 - these are not directly under our control so this is just the likely MCU) and what appears to be happening is that soon after the call is placed from the MCU to the C40, the call get disconnected but then somehow the C40 creates an outbound call to the MCU's indicated E.164. At this point the user sees the auto attendant and has no clue what to do. The MCU in the meantime tries to re-establish the call, but as a call is already in operation on the C40, it only gets connected as a audio call, resulting in even more confusion for the user.

You can see from the call history both inbound and outbound calls from and to the MCU.

1. Firstly, is there a setting I am missing somewhere in the C40 that tries to allow it to reconnect to a disconnected inbound call?
2. Has anyone else seen similar issues?

I have had a quick look at the logs and the first thing that pops out is that there is an NTP issue with this particular unit. In the event.log file, I can actually see log entries jump to January 04, then back to June 23 (today). The NTP server in the system status is show a blank and an "Unknown" status. Even so, I'm not sure if this would be the cause of the issue.

I need to wait for calls to finish and I will reboot and potentially upgrade the device, but I have to say this one stumps me.

Cheers

Chris

[Wayne DeNardi]

Hi Chris,

I'm seeing something that is similar, but I'm only seeing it happen for EX90s (my MXPs, SXs, and C60s are all working fine).

When I place a call to the MCU, using H.323, from the EX90, the MCU tries to call the EX90 back, and the EX90 then gets the option to merge the two calls (the one it placed to the MCU, and the one the MCU then calls it back on for some unknown reason).

Placing the call using SIP doesn't have this problem.

I'm currently working with the TAC (SR 630767151) on this issue and will be doing some more troubleshooting with them on Thursday (they already have all the logs from the VCS, MCU, etc.).

We've tried upgrading and downgrading the MCU to see if that helped (versions 4.4 and 4.5) but the behavior has stayed the same.  We're running TC7.1.3 or TC7.1.4 on the EX90s experiencing the issue - and the same software version on the C60s and SX20s behaves fine.

Our VCSes are running X7.2.3 - what version are you running on your VCSes?  We've only seen this behavior recently (since the VCS Upgrade for Heartbleed), so if you're running the same version, that may help track down the problem?

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Chris Swinney]

Hey Wayne,

Our issue has been very sporadic and I don't think it is tied to the upgrades on the VCS. Yes, they were moved to x7.2.3, but we have the odd record of this maybe happening way back before x7.2.3 would have been installed.

My initial though was to point the finger at the MCU, but I now think that this might be a CODEC issue. 99% of our call are made via an MCU and are directed outbound to the CODEC - of course all the user normally has to do is unmute the microphone. I haven't seen this issue happen on any other unit apart from these C40's (we don't have any higher C series CODEC out in the wild), but what I really can't understand is, should the CODEC (or in your case MCU) even call back the dialling party if the call is disconnected? However, I have asked for the MCU model and software revision just in case.

As an aside, I checkout some of the other C40s that have in the past seem to show this odd behaviour and each of them seemed to show a problem with NTP. Whilst an NTP server was assigned, the NTP status was "unknown" - multiple reboots and subtle changes did not resolve the issue. I therefore ran an upgrade last night on all C40 to get them onto TC7.1.4 and this seem to magically fix this issue. However, we can't run TC7.x at the moment as this breaks our Extron control system (!) so I then downgraded them to TC6.3.1 again (but at least this gives us the new self signed cert as well). Once back on TC6.3.1, things are now working as expected with all NTP server now synced on all C40s. The only slight glitch is that one of the C40s, in the System Status --> NTP Sever --> Address field is blank, although the current address does show the correct resolved address and it is synced.

This may all be erroneous, but I thought worth mentioning.

Cheers

Chris

[Wayne DeNardi]

Hi Chris,

From some of the troubleshooting I did with Cisco TAC yesterday, we managed to replicate your issue in our environment as well as the one we were having.  Getting the MCU to call an endpoint made a double call.

This only appears to happen for endpoints that are defined on the MCU, and whos endpoint name on the MCU is the same as their H.323 ID.  If we rename the endpoint on the MCU to something else, the double calls don't seem to occur any more.

Example:  Endpoint H.323 ID is "CAA-Support", endpoint defined on MCU with the same name "CAA-Support" - I get the double call - if I rename the endpoint definition on the MCU to "Endpoint-CAA-Support" it all works as it should.

So it definitely looks like a MCU bug, not an endpoint one.

The TAC engineer took a whole bunch more logs to take back and look at further.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Chris Swinney]

Hi Wayne,

Interesting, although I'm not to sure if this is 100% applicable to us, however, as I am not in control of the MCU's, I don't know how they are set up to operate although I don't think endpoints will be defined on the MCU. Still, it may well be worth looking at and I will pass this info on to those who know better than I.

As far as I understand it, the conferences are created using a custom interface and database housing the endpoint information, which then issues API calls to the MCU. I'm not sure what information gets passed to the MCU but the end result is an H.323 call placed from the MCU --> End Point using the E.164 number of the endpoint.

The odd thing is that I can see an outbound call initiated on the End Point (well at least according to the End Point call history) which would indicate a call being placed in the opposite direction to where it was initiated (originally MCU --> End Point). Are you saying that you can replicate this very case, or that you see the MCU established two calls from it to the endpoint?

I am still waiting for the guys who look after the MCUs to see if they can get the logs to show what happened from the MCU's point of view. Hopefully this will help explain things further.

I may well raise our own TAC case (unfortunately I can't do it from this account) but we have issues the Cisco at the maintenance contract on the End Point at the moment. I have been assured that this is being sorted out.

Cheers

Chris

[Wayne DeNardi]

Hi Chris, it may not have been exactly the same problem as yours, but was replicated with the MCU placing the calls.  When dialling up an endpoint, the MCU would place a second call to it, and one of them only connected as an audio call.

The TAC engineer has managed to replicate the issue in the lab, but only when the endpoint is specified in a MCU user account.

We're going to do some more testing on Monday morning.

Wayne

[Wayne DeNardi]

Hi Chris,

Further testing and confirmation of the issue (for our problem, but yours may be related) was done this morning.

The issue was having E.164 telephone number and video endpoint details in the "user" record on the MCU.

Removing those has stopped the audio only and/or duplicate calls/call-back issues we were having.

These settings are mentioned in the help, but wasn't 100% clear to what was required and someone had configured them all previously, which possibly hadn't become an issue until one of the newer firmware releases, but could have been there for a while - we're not sure... but, configuring them as per the help (ie, for a Video User: leave all fields blank, for a Telephone user: fill in the E.164 Telephone Number field - and optionally fill in the video source - but with a completely different device if required - but usually leave as <none>).

So, perhaps it's worth checking with your MCU people how they have configured it, particularly the settings for any defined users.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Nick Halbert-Lillyman]

Just to add my experience - we've had what seems like an identical issue to the one Wayne described, only with TP server (versions ranging from 2.2 to 3.1).  It only happens every few months so I haven't really been able to gather logs etc, but the symptoms sound exactly the same (initial call connects, then a second audio-only call connects also).  It was all on C-Series codecs, scheduled via TMS.

[Wayne DeNardi]

Hi Nick,

It would be worth you checking the users configured on your TPS to see if any of them have the E.164 and/or video endpoint details configured similar to the MCUs... if the TPS is doing a similar thing with those settings, that may be the source of your problem.

Note: it was only happening on our EX90s as those were the ones the users had assigned - we've since replicated the issue with other model endpoints so it was definitely not restricted to the one model endpoint or software version.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Nick Halbert-Lillyman]

Hi Wayne,

There weren't any pre-configured users/endpoints on the TP server, but some of the endpoints were listed under other conferences (usually permanent conferences) as "previous participants" so maybe there was a "user record" as per your previous post - if it happens again I'll definitely look at this, thanks.

I should also note that I've seen it happen using both E164 dialling and SIP, but they were always dual registered endpoints (essentially we turned off H323 dialling for scheduled conferences between the first and last time the issue occured, but endpoints still had an H323 ID + E164).