Echo cancellation delay warning after upgrading to TC7.1.x

[Jens Didriksen]

Getting a very annoying echo cancellation delay warning on some systems after upgrading these to TC7.1.x (all systems now on TC7.1.3), see screenshot:

seeing similar warning on some SX20, C40 and C60 after upgrade, and here is the interesting bits;

• none of the systems use HDMI audio
• all of the systems showing this warning use external DSPs for audio; Nexia and Tesira, all using multiple microphones and speakers.
• downgrading system to TC7.0 or lower, permanently clears the warning
• doing a factory reset clears the warning for a short time only.

We do not use HDMI audio on any of our systems, and this warning does not show up on any of the systems not using an external DSP.

It's more an annoyance than anything else as it has absolutely no impact on audio quality etc, but would be nice if I could turn this off somehow - any ideas? (oh, and there's no option to do this using admin)

/jens

[Wayne DeNardi · Accepted solution]

Hi Jens,

We've seen the same here on quite a few of our endpoints.

I've logged a couple of calls about this and the answer so far has been to request the password for the "remotesupport" user through the TAC (with the token generated from the endpoint) and then to log in and issue some commands to turn the EcReferenceDelay to 0 (and set it to Fixed):

1) SSH in to codec and log in as remotesupport

2) Type 'tsh' and press enter

3) Use the following two commands:

• sys-ttpar set-litr Audio/Configuration/EcReferenceDelay[1]/Mode[1] Fixed
• sys-ttpar set-int Audio/Configuration/EcReferenceDelay[1]/Delay[1] 0

This has removed the warning message and resolved some of the echo issues we experienced on our endpoints.

I still see it as a workaround as the issue wasn't there prior to TC7.1.1 and there's obviously something new in those releases that have caused this to occur.  I believe there is a bug open for this, although I don't have permission to see it: CSCul15840

I've also been told that the Delay will be reset to 0 on SX20 endpoints when disconnecting and reconnecting the microphone with TC7.2 (which hasn't been released yet).

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Jens Didriksen]

Thanks Wayne,

Yeah, I suspected I would have to use the new "remotesupport" account to fix this, but thought I'd check here first before heading off to TAC.

It's not really an issue for us (more an annoyance), as we don't have any echo issues at all; audio is perfect. Just weird it's only happening with all of the systems using external DSP and not with any of the others.

I believe TC7.2 is due out end of this month by the way - all going well.

/jens

[Wayne DeNardi]

Yep.  Same with ours.  Nexia and Audia DSPs on ours - and never had HDMI output (it's DVI on those systems affected).

I can also do it on a SX20 by unplugging the HDMI output cable - the endpoint then complains about the added delay on the HDMI audio, but doesn't tell me that the output cable is no longer connected.

And this is supposedly "normal" functionality

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Jens Didriksen]

"...unplugging the HDMI output cable - the endpoint then complains about the added delay on the HDMI audio, but doesn't tell me that the output cable is no longer connected."

Gold!  - Hope the developer(s) reads this...

/jens

[Jens Didriksen]

So here we go again :(

Thought I'd do the right thing and opened a case with TAC to get the password for the Remote Support User account so I could do what's needed. Yeah, well - the TAC engineers response was "we do not give these passwords to the customers - you'll have to set aside time for a webex session"! WTF - seriously?

Come on Cisco, move this stuff into the admin area so we don't have to go through this b/s. :(

/jens

Please rate replies and mark question(s) as "answered" if applicable.

[Bjørn Winsvold · Accepted solution]

We are currently working on a modified text for this warning to hopefully make it less confusing. The key point in this warning message is that we have detected a very high latency in the "acoustic" loop from loudspeaker output to microphone input. This latency is introduced out of our control either in the loudspeaker path or in the microphone path and gives a not optimal user experience for real-time communication.

The most normal use case is a codec connected to a TV monitor via HDMI. TV monitors may introduce  100 - 200 ms latency. We recommend to set the TV monitor in PC or Gaming mode in order to reduce the latency to an acceptable level.

Jens, in you case it is a bit different since you are using an external DSP. I assume that the acoustic echo canceller is done in the external DSP. If that is the case, you have to turn off the echo-canceller in the codec in order to not get any echo or other related issues:

xConfiguration Audio Input Microphone 1 EchoControl Mode: Off

If you do that, you will also get rid of the annoying warning.

I suggest you try this configuration:

• xConfiguration Audio Input Microphone 1 EchoControl Mode: Off
• sys-ttpar set-litr Audio/Configuration/EcReferenceDelay[1]/Mode[1] Auto ( = default)
• sys-ttpar set-int Audio/Configuration/EcReferenceDelay[1]/Delay[1] 0

You should not experience any echo issues after doing this. Can you please try out this configuration and post your feedback here? That would be very helpful.

Thanks,

Bjørn

[Jens Didriksen]

Hei Bjørn,

"The most normal use case is a codec connected to a TV monitor via HDMI"

Yes, and we do use HDMI for video in our meeting rooms, but that's for video only - we don't use HDMI for audio anywhere at all as we always use external, powered speakers, and the warning does not pop up on any of those systems, but there's no external DSP in play either, so;

"I assume that the acoustic echo canceller is done in the external DSP. If that is the case, you have to turn off the echo-canceller in the codec in order to not get any echo or other related issues:"

Yes, we use a relatively large number of microphones and speakers with these sysrems, so the Nexia handles the lot - and we have not experienced any echo, or other audio related issues, audio is perfect - despite what the warning is telling us.

"The key point in this warning message is that we have detected a very high latency in the "acoustic" loop from loudspeaker output to microphone input."

I suspect that is caused by the DSP - which causes the codec to "spit the dummy" - which is interesting since there's no such issue in versions prior to TC7.1.1 - so you guys must have introduced a new "feature" in TC7.1.1 

I'll open a case with TAC tomorrow so I can obtain a token to access the "remoteuser" account and let you know what happens.

mvh jens

[Wayne DeNardi]

My setup is similar to Jens'.

Our codecs are fed by external DSP devices that perform the microphone mixing - we however let the codec do the echo cancelling, and prior to TC7.1.1 didn't have any issues - it's only since TC7.1.1 that we've had these error messages, and in some cases, the additional delay it has magically decided to add to the systems has caused problems.  Setting the delay to fixed/0 as per my other post has resolved the audio issues and everything behaves as it did prior to TC7.1.1.

We've been running these devices (predominantly C60s) like this since TC3.0 (2009) and TC7.1.1 and above are the first releases we've had this sort of issue with.

Again, similar to Jens, we don't have our displays on the HDMI output, the screens are all on the DVI output.  The audio out from the codec is via the RCA connectors which then feed back in to other audio components (switchers/dsp/etc).

The error message specifically mentions HDMI audio output which is completely irrelevant in this configuration, so either the error message itself is misleading, or there's something else going screwy in the newer software releases adding delay where there was none previously.

PS It's becoming a pain to have to request the passwords for the remotesupport user for each and every one of the codecs as we upgrade them to then be able to turn the delay off - surely there can be a better way.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Bjørn Winsvold]

Hi Wayne.

Interesting to read you experiences. I would like to really understand your setup in order to be able to help you.

First a correction:

"the additional delay it has magically decided to add to the systems has caused problems"

This is not correct. We do not add any extra delay. We detect the delay in the acoustical loop - that is the delay from output of the codec to the input of the codec. So if the external DSP introduce some delay it may cause trouble for the build-in echo canceller since it is not aware of the external delay (if the delay is above 50 ms). So if we detect a delay we can compensate for the external delay in the AEC in order to have good echo canceller performance.

As I mention earlier, we realize that the text in the delay warning is confusing and we are correcting that for the next release.

You are using an external microphone mixer in combination with the AEC in the Cisco codec. Are you using a "fixed mixer" or is it a "smart mixer"?

If you are using a "smart mixer", echo issues may occur since the mixer will introduce a lot of echo path changes causing the AEC to re-adapt to the echo path changes. So by using an external mixer in combination with the build-in AEC, the mixer has to be a fixed mixer in order to work well with the AEC.

You say you started to experience echo issues with TC7.1.1 and had no issues before that. That is really strange, because the automatic delay detection has been there since TC6.3. The only new thing in TC7.1.1 is this "delay warning" where the goal is to make the end user aware of non-optimal real-time user experience both on audio and video (typically where the codec is connected to a consumer TV via HDMI). The behavior with respect to AEC should be the same in TC6.3 and TC7.1.1. What was the previous TC version you were running without echo issues?

Bjørn

[Wayne DeNardi]

Hi Bjørn,

One of the jobs I logged with the TAC was SR 630108859.  Please feel free to contact me directly outside of the forums for more info.

Wayne
 

[Jens Didriksen]

Yup, that worked quite nicely, thank you very much. :)

Would've been nice if these settings were accessible through the admin account though, instead of having to go through TAC for every single system displaying this warning.

/jens

[Bjørn Winsvold]

Good.

Regarding the need for request remotesupport password I totally agree with you. But unfortunately that is not my responsibility so I can not help you there :-(

[Jens Didriksen]

So now I'm finding myself having to downgrade a system to get root access, re-set the error, upgrade again and hoping the upgrade won't reset the change I made - all because Cisco TAC refuses to create a remote user support password because there is no service contract, or a service contract is not associated with my Cisco i.d., brilliant. :(

How about moving these command options in under admin?

/jens

[Wayne DeNardi]

+5 to this suggestion.  It's a major pain even when you do have a service contract.

Another suggestion I had made a number of time previously was to have an automatic way of generating the password from the token, something like an additional page on the licencing portal whcih those of us with service contracts have access to so we could get the password relatively easily without having to involve the TAC.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.