Tandberg Content Server disconnected 3 times from 2 Video conferences

[morpb1939]

I had two seperate VCs being recorded at the same time. The MCU stated that there was network connection error for the TCS on all 3 disconnects. I was able to re-dial the TCS from the bridge. Spoke with one of our communication engineers and there were no network issues during the time frames when the disconnects happened. I looked at the Windows 2003 box that the TCS resides on and there  are three events (Source: TCSCE) that happened at the exact time the TSC disconnected.

vice.exe exited for unknown reason, restarting.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Vice.exe is the Streaming/Archiving Engine and it is located in Files\TANDBERG\ContentServer\ContentEngine

The server has been rebooted and subsequent tests have been successful. Looking for explanation as to what happened and how to prevent future occurrences. 

Both calls were bridged calls running on a Codian MSE 8510.

Sodtware versions

TCS: S4.1

Codian MSE 8510: 4.3(2.18)

C60: TC5.1.5.297625

[Magnus Ohm]

Hi Brian

First of all I would recommend you to upgrade your TCS to S5.1 and see if the problem still exists as there have been many bug fixes since 4.1 (assuming the unit is covered and you have access to a release key). Does the TCS in fact receive any media at all, since if the call connects and the TCS does not get any media it will disconnect by design, logs will tell you if this is the case.

You can also try to run the Repair on the TCS which in many cases resolves small issues including issues with the vice.exe being unstable.

Just go to control panel, find the content server software, click Change. Then choose the repair option and follow the wizard. The clue here is was it working fine before? Is there any errors in the logs that indicated the reason for disconnect? TCS will disconnect in a number of scenarios for example does it have enough space?

/Magnus

[Jens Didriksen]

We are still seeing this issue on all of our three 2nd gen TCS, all running 5.3.3 :( - we'll make a decision as to whether to upgrade to 3rd gen or change to something completely different, i.e. Echo 360, MediaSite etc, early next year. Chances are we'll be dropping the TCS.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

[Patrick Sparkman]

Jens -

We've had a TCS since back in the TANDBERG days when it was a 1st Gen appliance and again a 2nd Gen appliance, we've since migrated them to VMs running the same specs as a 3rd Gen appliance.  We've had some issues here and there, but most of them could be explained. What does the Content Engine log say for the conferences that got disconnected?

[Jens Didriksen]

Yeah, we've been running TCS since they were first released by TANDBERG, I first came across them when they were still Ectus. :)

Anyways, it's the old "vice.exe exited for unknown reason, restarting" issue, happens now and then with all three. Running repair at regular intervals seems to keep them going, mind you, all three are pretty much running at full capacity most of the time, so they're rather busy, but I can't recommend upgrading to gen 3 as we need a solution we can use to capture lectures anywhere, campus wide. We also use Camtasia Relay, but are now in the process of re-evaluating lecture capturing technologies as a whole.

/jens

[Patrick Sparkman]

Maybe something to do with Windows, an update or GPO that might be affecting vice.  We record almost all day everyday, but have never came across that issue.  We've looked at various lecture recording solutions ourselves, and the ability to record using video conference hardware that is in place within the classrooms is a bit plus for us, though we're so committed to what we've recorded over these few years, don't see us changing unless something major comes along.

Wish you luck!

[deeptiswain]

Agreed Patrick,

Lately I have seen many customers in Educational institutions using TCS extensively for recording lectures, creating video catalogs etc. For example, I came accross one customer, who use the TCS so extensively that, they exhaust 1 TB of storage almost every 15 days..!! They kind of host their TCS in a data centre and different colleges use it to record their lectures. But they never complained about the vice.exe issue. 

For Jens's case, the event logs are not going to give a detailed description about the failure. We need to read the event logs along with the TCS logs. If we dont see anything from them, then we will need to turn on some detailed audits on the TCS till the issue is reproduced.

[deeptiswain]

Hi Jens,

vice.exe process drives the content engine. I usually did not see many of these "vice.exe exited for unknown reason" much. The TCS is actually pretty stable in version 5.3.2 and in the 3rd generation iteration. Have you checked the event logs (system and application) after you encountered this issue? Also, please make sure to keep the server updated with all the required and Cisco recommended MS patches and updates. You can find them in the following links:

http://www.cisco.com/c/en/us/support/conferencing/telepresence-content-server/products-field-notices-list.html

http://www.cisco.com/en/US/products/ps11347/prod_bulletins_list.html

If you open a TAC case with Cisco, they may provide you with some debug methods, which you can run on the server and it will help a lot in determining the root cause of this issue. 

Regards,

-Deepti

[Jens Didriksen]

Thanks Deepti, the "vice.exe exited for unknown reason" is from the event logs. We forwarded a bunch of logs to Cisco a long time ago when we first encountered this issue, but that came to nothing.

It's not too bad if/when they are not running at full capacity for hours on end. I'll be able to take closer look at it now that the semester is over and they're not in use as much.

/jens

[Wayne DeNardi]

Hey Patrick,

A little off-topic, but we're about to migrate our clustered applicance Gen 2 TCSes (S5.3.2) to Virtual "3rd Gen" specs (S6.1) on our own hardware.  Did you do a Physical to Virtual migration, or just stand new ones up side by side with the old ones?  If you did the P2V migration, did you use the Cisco TCS Migration tool, and if so, were there any gotchas in the process as this tool/migration is what we're currently planning to do?

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Patrick Sparkman]

We did do a physical to virtual migration, there is a TCS migration PID (R-VMTCS-MIG-K9=) that you order to obtain the VM files (OVA, install scripts, and data migration tool).  One order for each TCS, this is so each instance of the old version of TCS is migrated to the VM in Cisco's database, however I'm still working through that as they still show us as running 2nd Gen servers and not VMs.

We used the migration tool provided in the VM files to migrate the TCS data, the tool will then put the TCS it migrated the data from into a state that will make the TCS inoperable (doesn't matter because you're migrating to a new server anyway).  The current supported method is to install the VM on the current hardware, 2nd or 3rd Gen server.  Spec-based installs are right around the corner from what I've been told, if you choose to go that route though.  More details on the install and migration can be found in the TCS VM Install Guide.

We came across a laundry list of gotchas throughout our install, as we were pretty much one of the first ones if not the first to run through the install literally the following Monday after FCS on Friday, and actually had Cisco update there install guide as it was missing important information such as the the SP1 requirement and some other steps or images that were incorrect.

If you haven't already installed the OVA and Windows 2008R2 Standard SP1, I wouldn't install all Windows updates past SP1.  The TCS installer will throw errors because the updates install newer versions of items than what the TCS will reference.

As far as the scripts, make sure you go into the properties and unblock them, or else they'll error out when you try to run them.

That's all I can think of at the moment.  Any issues, I'm glad to help, just send me a message.

[Wayne DeNardi]

Thanks Patrick,

We've already ordered the Migration tool, just haven't got around to the final planning and scheduling of the "doing it" bit.

Thanks for the heads up on some of the issues - we'll keep them in mind.

Wayne

[Patrick Sparkman]

No problem, it's a pretty straight forward install actually.  Glad to help anyway I can, TAC and the TCS developers helped us, as I had a case open from day 1 of our migration just in case.

Here is the bug on the TCS scripts (CSCuq47613) for reference.

Also 1, if you ever have to do a repair or anything with add/remove programs, you must be logged in as the local administrator, and not via a domain account or other account as the installer will error out and fail to run.  This is a security thing, since you're running Windows 2008R2 from what we were told.. and it's by design.

Also 2, if you run a cluster, the TCS Wizard will not be able to create the Microsoft loopback adapters (CSCuq65423), so you'll have to create them yourself (VMWare KB 1004779).

[Wayne DeNardi]

Thanks for all the info Patrick.  That's a great help!

The hint on the cluster adapters will be helpful too, as yes, we do have our current ones in a cluster and intend to do so after their migration too.  I understand we have to break the cluster before the upgrade and data migration anyway.

Wayne

[Adam Wamsley]

Hey Brian,

How many live recordings are you doing at once?

Vice.exe

• Content Engine
  
• Can cause TCS to drop from call
  
• Generation 1 Hardware vice.exe gets taxed out in conjunction with live recordings causing vice.exe to shutdown (dropping calls). This is better with S5 because it will do live recordings at a lower resolution.
  

Adam