Solved: Unable to upgrade C-series from TC6.0 to TC6.x through TMS

Jens Didriksen · ‎04-22-2013

I first noticed this issue when scheduling end-points in TMS to be upgraded from TC6.0 to TC6.01 as I mentioned in the thread below:

https://supportforums.cisco.com/message/3897394#3897394

However, thought it was more appropriate to create a separate thread for this.

TMS version 14.1.1

Upgrade C-series from TC5.x to TC6.x through TMS works just fine.

Upgrade C-series from TC6.0 to either TC6.01 or TC6.1, or from TC6.01 to TC6.1 fails.

TMS upgrade status shows, in most cases, upgrade stuck at 60%, and in some cases 5%.

Upgrading endpoint directly through web gui from TMS itself server works fine, as does upgrading direct from my workstation and/or laptop.

End-points include C20, C40, C60 and C90 - haven't dared to try the EX90s as yet as these executive systems.

Will open a TAC case when back at work in the morning.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Kjetil Ree · ‎04-22-2013

Hi Jens,

TMS uses completely different upgrade mechanisms for TC software depending on the software it is already on. That is why you see a different result for endpoints that are on TC < 6.0.

With the new upgrade mechanism introduced in TMS 14.1 and TC 6.0, we put most of the responsibility on the endpoints. Your first troubleshooting step should be examining the endpoints logs. If everything looks good there, have a look at a Wireshark trace and see if TMS sends the correct URL to the endpoints, and check if the endpoint actually can reach that URL.

Regards,

Kjetil

View solution in original post

Chris Swinney · ‎06-12-2013

Hey Jens,

FWIW, here is an extract from the EventLogAll of a C40 CODEC:

Jun 12 13:04:51 ppc appl: 2451.95 TMS I: provision(): http request url/https://IP_Address_Of_TMS/tms/public/external/management/systemmanagementservice.asmx/796 bytes
Jun 12 13:04:51 ppc appl: 2452.04 TMS I: provision(): tms query succeeded, length 1404
Jun 12 13:04:51 ppc appl: 2452.05 TMS I: executor(): Found heartbeat rate 0
Jun 12 13:04:51 ppc appl: 2452.08 TMS I: executor(): cuil-processed buffer of 607 bytes
Jun 12 13:04:51 ppc appl: 2452.08 TMS I: provision(): ok ! heartbeat 0 ! upgrade yes ! 0 files ! 0 docs
Jun 12 13:04:51 ppc appl: 2452.08 PROV I: provision(): software upgrade url='http://IP_Address_Of_TMS/' release-key='1TC006-1-*******'
Jun 12 13:04:51 ppc appl: 2452.15 HTTPFB I: Queue::removeAllByCollectionId: 3
Jun 12 13:04:52 ppc appl: 2452.38 PROV I: HttpDownloader notifyFailure [s:3] [url:http://IP_Address_Of_TMS/] [c:403]
Jun 12 13:04:52 ppc appl: 2452.39 PROV ERROR: HttpDownloader(this=0x22a88988)[c:403]: failed with [5](No error)
Jun 12 13:04:52 ppc appl: 2452.39 PROV ERROR: HttpDownloader(this=0x22a88988)[url:http://IP_Address_Of_TMS/]
Jun 12 13:04:52 ppc main: SWUpgrade: Failed [c:403] 'http://IP_Address_Of_TMS

It seems to me that TMS is provisioning the wrong information to the CODEC by telling it to download software from the ROOT of TMS. Of course, the CODEC not going to find any software there and this also has authentication and SSL applied. The only section in TMS that I can find that seems like it should be the information provided to the CODEC is in "Administrative Tools --> Configuration --> Network Settings --> URL Where Software Packages Can Be Downloaded" - the location of which on our install points to "http://FQDN_Of_TMS/tms/public/Data/SystemSoftware" - which I have enabled Anonymous Authentication (disabling others) and removed SSL.

In addition, TMS cannot (at this present time) provision the C series CODECs in the same way that it does for Jabber Video and the E series endpoints, and we don't utilise CUCM. I can't find any "software" URL or Upgrade options in the TMS Configuration Template under 'Cisco TelePresence Group Series' options. If I browse to the "Configuration --> System Status --> Provisioning" section of the CODEC, I can see the "Upgrade Status URL" - which points to the root of TMS server - but I see no option to set this anywhere.

Just as a test, I added a virtual folder in IIS that linked "http://FQDN_Of_TMS/tms/public/Data/Software"

with the location of the FTP folder as specified in TMS at "Administrative Tools --> Configuration --> General Settings --> FTP Software directory", but this didn't work either.

I have noted that TC6.1.2 has just been release, but this is only a maintenance release and does not seem to contain any fixes with regard to this issue. I think I shall also open a TAC case.

Chris

Message was edited by: Chris Swinney Added paragraph re-provisioning and templates

View solution in original post

Kjetil Ree · ‎06-12-2013

Hmmm.... TMS isn't using the URL Where Software Packages Can Be Downloaded folder when telling TC software >= 6.0 endpoints where to go for upgrades, TMS builds the full URL itself ( if I remember correctly). This seems to have failed somehow in your case, which very well could be a TMS bug.

Please go ahead and open a TAC case.

-Kjetil

View solution in original post

Kjetil Ree · ‎06-13-2013

@Jens:

Can you check whether the "Software FTP directory" is a location that can be reached by IIS? Unless you do some IIS magic of your own, it must be a subfolder of C:\Program Files (x86)\TANDBERG\TMS\wwwTMS\. If it isn't, I see the same symptoms as Chris sees when I try to upgrade from TC 6.1.1 to 6.1.2.

@Chris:

When you created the virtual folder, are you sure you gave the user IIS uses the necessary read permissions? There are a lot of things that can go wrong when you do this yourself. What happens if you move the Software FTP Directory back to a subfolder of IIS, e.g. the default value of C:\Program Files (x86)\TANDBERG\TMS\wwwTMS\Public\data\SOFTWARE?

Regards,

Kjetil

View solution in original post

Chris Swinney · ‎06-14-2013

Wohoo - sorted. I'm actually at home at the mo but it seems as though the upgrade has now been downloaded to the CODEC and the CODEC is requesting info from the correct location.

Kjetil - indeed we have had TMS for many years, continuously upgrading the DB from version to version. We originally ran TMS on Windows 2003 32 bit, but have recently upgraded to Windows 2008 R2 which is of course, 64 bit.

The ONLY thing in the "C:\Program Files\TANDBERG\.." is the Software tolder, which is indeed the location where TMS Tools pointed to. I removed the virtual folder in IIS that pointed to this folder and instead created a "Software" folder under "C:\Program Files (x86)\TANDBERG\TMS\wwwTMS\Public\Data\". I relied on the inherited permissions for the folder creation. Update the location in TMS Tools, restart all services..... and all looks OK.

In fact, as I am finishing this post I have just checked the test C40 CODEC, and it has updated successfully.

Many thanks for the pointers along the way. Jens - I hope this goes someway to resolving your issue.

Chris

View solution in original post

Kjetil Ree · ‎04-22-2013

Hi Jens,

TMS uses completely different upgrade mechanisms for TC software depending on the software it is already on. That is why you see a different result for endpoints that are on TC < 6.0.

With the new upgrade mechanism introduced in TMS 14.1 and TC 6.0, we put most of the responsibility on the endpoints. Your first troubleshooting step should be examining the endpoints logs. If everything looks good there, have a look at a Wireshark trace and see if TMS sends the correct URL to the endpoints, and check if the endpoint actually can reach that URL.

Regards,

Kjetil

Chris Swinney · ‎04-22-2013

Jens, It would be interesting to find out what you find out here. We had issues downgrading CODECs from TC 6 back to TC5 via TMS. Only tried 2 so far but both failed via TMS (just timed out) and had to be downgraded manually.

We had issue with TC6.01 on a C40 what has dual display via a matrix - we lost one screen and sound. Reverting back to TC5x resolved the issue. We have been asked to upgrade to TC6, reset and delete the config and reconfigure from scratch but our Cisco partner, but we would wanted to wait for TC 6.1 to see if this resolved the issue. This fix would be impractical for all our endpoints.

Jens Didriksen · ‎04-23-2013

Opened TAC case; 625653937 - and TAC engineer has replicated the problem in their lab.

Bug report is being created by TAC and passed up the chain; CSCug63269

Work-around: upgrade end-points manually.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Jens Didriksen · ‎05-09-2013

Was advised by Cisco yesterday that the bug has been closed by the "end-point people" as they can't see a problem - been asked to open a new TAC case if/when it happens again.

I have, meanwhile, upgraded all of our C-series manually, so guess it'll have to wait until the next TC6 release, yes I know TC6.1.1 is out, but can't see any point upgrading to this after reading the release notes.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

simonf_13 · ‎05-21-2013

We are having a similar issue. We have tried manually upgrading our C40's from Ver 5.1.6 to version 6.0.1 and had problems with one of the large wall mounted display LCD screens not working properly

.

In the first room we did this in the scenario was this:

When in a call, the LCD screen which normally shows data (PC images,etc) defaulted to be a black image when no data was being sent to it. Giving the impression the screen wasn't working! not good for the users in the room.

When not in a call this screen defaulted to camera view. Trouble is most people use the room when it IS in a call due to calls already being connected when they arrive in the room, so when they swithch on the 'system' (screens) in the room (Extron touch screen controlled) one of the screens appears not to be working. This will lead to unecessary phone calls to myself as technical support. We downgraded and all worked fine.

The second room was more problematic:

Similar to room one except that the screen affected was the one showing the main camera, and most importantly it wasn't switching on at all. We double checked that there were no faults in the room and it was definatly down to the upgrade. We downgraded and all worked fine.

We did update the Extron touch screen programming to tie up with the new release of the Codec to ensure that wasn't going to cause any issues.

We have now left this our supplier to log our three rooms as support request to resolve this issue ASAP as we are having issues with the cameras, so we would like to install the update.

Jens Didriksen · ‎05-22-2013

Thanks for your reply Simon, but this is a completely different issue concerning TMS not being able to upgrade the end-points from TC6.0 to a higher version.

I am aware of the "display issue" you mention, however, this is not an issue for us as we don't use self-view, but "DualPresentationOnly" in all of our rooms with dual monitors or projectors.

As far as the camera issue is concerned, I have had the same issue in the past where only one of the cameras was upgraded - fixed that by disconnecting the daisy chain, upgraded and reconnected the daisy chain.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

juriss · ‎10-15-2014

Ok, I know this is an old thread.... but very good info

My question is.. Can TMS 13.2.2 be used to upgrade a TC endpoint that is running TC7.1.1...

We have plans to upgrade to TMS 14.... however we need to upgrade about 350 systems to TC7.2.1 for shellshock/Bash

Is it that it works better with TMS 14 than TMS 13 or does it NOT work with TMS 13

If TMS 13 does not work we will just do the upgrades manually again.. like we did with Heartbleed

Thanks

Tom Kalikajaros · ‎05-22-2013

Hi Jens,

Only time we have had an issue with TMS delivered upgrades failing came back to a corrupt copy of the upgrade on the TMS Server. Our TMS Admin deleted the upgrade and redownloaded to resolve.

I am not sure if there were any logs on TMS side that our Admin used to confirm it was corrupt.

Cheers

Tom

Jens Didriksen · ‎05-22-2013

Yeah I've had that problem on a couple of occasions in the past, but this is completly different.

I can upgrade the end-point manuall through the web gui of the end-point using the same TC6.x package TMS is using, but TMS is unable to upgrade any end-point already running TC6.x to a higher version.

I'll wait until TC6.2 is out before opening a new TAC case.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Chris Swinney · ‎05-23-2013

Or downgrade for that matter.

Chris Swinney · ‎06-07-2013

Hey Jens,

Just out of curiosity, do you know if the software download folder in TMS (shown in Administrative Tools --> Configuration --> General Settings --> FTP Software directory (on ours this is C:\Program Files\TANDBERG\TMS\wwwTMS\Public\data\SOFTWARE\ ). is actually correct when relating to HTTT(S) requests? This folder for us is not alterable via the TMS web interface, but does NOT map to an HTTP web folder. This is the folder where TMS automatically downloads software updates to

The setting in "Administrative Tools --> Configuration --> Network Settings --> URL Where Software Packages Can Be Downloaded" shows "http://FQDN/tms/public/Data/SystemSoftware" which actually relates to "C:\Program Files (x86)\TANDBERG\TMS\wwwtms\public\Data\SystemSoftware" - subtle different to above. This folder actually contains no software that the CODEC could download. I also assume that like upgrading jabber clients, the folder has to be standard HTTP not HTTPS.

However, it is true that from TC 6.1 TMS will no longer automatically downloads updates - and I'm assuming that this will hold true for all further update across the board such as VCS's and other hardware. Meaning that you must manually download software updates to the latter folder - and probably when we can downgrade once we have upgraded. I haven't tested this yet so it will have to wait until next week.

Chris

Jens Didriksen · ‎06-07-2013

Hmm, interesting point, and yes, I did notice the empty folder when scrummaging around in the server, would be nice to know the difference between the two. I'll take another look at that when back at work Tuesday.

Bit confusing though since upgrading from TC5.x to TC6.x works fine, whereas from TC6.0 to TC6.x doesn't, so some changes have been made that my TMS doesn't like.

TC6.2 should be out very soon now, so I'll open a TAC case then, and now that the Cisco SUS backend issue has been fixed I know that side of things won't confuse the issue.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Chris Swinney · ‎06-08-2013

Hey,

Am I correct in thinking that the way the TC endpoint obtains software via TMS has changed in version 6 such that:

In version 5 and previous,TMS effectively "pushed" the software to the CODEC. This would mean that as long as the software resides in a folder on the TMS server that TMS knows about (I'm assuming here this is the "FTP Software folder" mentioned above, although I'm not saying that FTP is the protocol actually used), the software can be pushed out to the CODEC without the CODEC even knowing about the location of the software folder on TMS
In version 6. The CODEC now is told that a new software version is available, but it is down to the CODEC to "pull" the software from TMS. This means that the CODEC must be able to access a folder on TMS via an HTTP link (the"URL Where Software Packages Can Be Downloaded" mentioned above). Of course, TMS Auto software update is putting the software in the FTP folder location, NOT the new Software Packages location.

So whilst an upgrade from version 5 to 6 will work just fine (as the software is being pushed), additional upgrades once the CODEC in on version 6 (or even downgrades) will not work as the HTTP link points to a different physical empty folder on TMS - the CODEC cannot simply not find and software.

Just a thought....

Chris

Jens Didriksen · ‎06-08-2013

Yeah, that's how I understood Kjetils comment re changes implemented in TC6.0 and TMS 14.1 onwards, however, I've tried putting the software package in the other folder as well, but didn't make any difference.

Won't really know what's going on until I take the end-point root log during upgrade attempt when TC6.2 is released.

/jens

Please rate replies and mark question(s) as "answered" if applicable.