UPGRADE EX, C SERIES WITH TMS 15.1.0

[Chet Cronin]

I am running TMS 15.1.0 and have started using the upgrade feature.  I am able to upgrade my MXP's with no issue but can't get the EX or C series to upgrade from TC7.3.3.   I have the release keys for TC7.3.5 but when I try to run an upgrade to either the EX60 or C90 it will not work.  they are on TC7.3.3 and I have tried it without adding the key and also with adding the key.  I have SSH open to watch for any activity on the codec but nothing ???  I did perform an upgrade on the EX60 locally from the system itself and it was successful so the file is a good .pkg ...

Are their any settings on the EX and C to make sure of ??? If so what ???

[Wayne DeNardi]

You need to make sure that TMS can communicate with the endpoint, and the endpoint can communicate with TMS.  If the endpoint is unable to contact TMS (due to a firewall, or proxy server, or something else in the way of http/https traffic) then it will be unable to download the software.

Also be aware that these software images are also quite large (250-300Mb) so can take a while to download on a slow link.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Chet Cronin]

Wayne,

Yes the TMS appears to be communicating with the EP and the EP seems to be communicating with TMS ...

I did notice when I ran a compare between the server and EP that for provisioning settings

I have

External

GET

TMS

Selected.

There was a difference in that the Server had AUTO, POST, TMS so I changed so the server now sees External, GET and TMS  one the EP and the Server, but still nothing ???

I am not running provisioning on the VCS or have TMSPE installed and don't use provisioning feature on my set up.  The MXP's upgrade with no issues.  It's just the darn EX, C that will not work.

[Patrick Sparkman]

If you log into any of the endpoints and check the provisioning status, what does it say?

[Chet Cronin]

I downloaded a known good .pkg and have upgraded a couple systems already EX, and C series by logging directly into the system and not using TMS.  

When I try to upgrade others and execute the upgrade from TMS it will just sit at 0 percent status for over an hour as an example.  I remote into the EX and nothing is happening and I also SSH into the EX prior to starting the upgrade and still nothing is happening. 

Not like the MXP's which I have had virtually no issue with them... YA TANDBERG :) :) ...

Provisioning is turned OFF and still the same negative issue.

Provisioning is turned on and TMS is selected and set to POST and still nothing ...

I am not sure Provisioning is even a factor as all the reading I am doing on CISCO site indicates provisioning is just for config, and FindMe.  Nothing about upgrading at all.

I assume the upgrade is happening via the port 443 as SNMP isn't used on our systems.

Http and telnet are OFF. 

[Wayne DeNardi]

If it stays at 0%, then that's a sign that TMS hasn't been able to communicate with the endpoint to tell it it has an upgrade to request.  From the TMS server, are you able to access the web interface of those endpoints?

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Chet Cronin]

Wayne,

Agree that seems to be the case .... Yes I am able to access the endpoint web interface from TMS.

[Wayne DeNardi]

I'd suggest doing a wireshark capture on your TMS server and capture any packets between TMS and the endpoint to see if there are any unexpected responses in the traffic between the two.  The inability to do the upgrade via TMS is usually related to something on the network that is affecting the traffic flow between the two (firewall, packet inspection, WAAS, proxy servers, NAT, etc).

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Chet Cronin]

No change ... I am running a Windows Server 2008 R2 with SQL 2008R2 and TMS Version 15.1.0.  I can upgrade MXP's all day long with no issues from TMS and any EX or C series I try fails.   I ran wireshark and didn't see anything out of the norm. 

I did turn on logging on one system and am receiving the following error:

TMS ERROR: http (this=0x4a89aa78): failed with <operation timed out> (Timeout was reached)

TMS I: provision(): tms query FAILED, Length 0

TMS I: provision(): http request url/https://(domainname)/tms/public/external/managment/systemanagementservice.asmx/772 bytes

NOTE: Kept repeating until I turned off log output.

Can you explain the actually upgrade process when the TMS initiates the upgrade to an endpoint?  Like when a call set up happens?  Is the upgrade happen only via port 443?   I can browse to an Endpoint from the server with no issue.  I see that TMS is talking with the endpoint when I do a refresh from TMS and see IDLE status when it's not in a call.  There are no firewalls involved with the systems I am trying to upgrade from the TMS server.

[Wayne DeNardi · Accepted solution]

A few more things that I can think off off the top of my head that you can check/try:

Do your endpoints have valid DNS servers defined on them that can resolve the domainname of your TMS server? 

If they don't, TMS will be telling them to request a software upgrade, but the endpoint will never be able to download the software if it can't resolve the correct domain name/URL.

If that's OK, then you can try turning the extended logging on on the endpoint and initiating an upgrade and see if you can see the actual upgrade message hitting the endpoint in the packet capture it performs.

You can also look in the endpoint log files, you should also see something like this (just captured on one of mine):

2016-04-18T10:19:57.911+09:30 ppc appl[2863]: 4585179.57 TMS I: provision(): ok ! heartbeat 0 ! upgrade yes ! 0 files ! 0 docs
2016-04-18T10:19:57.912+09:30 ppc appl[2863]: 4585179.57 PROV I: provision(): software upgrade url='https://yourTMSserver/tms/public/data/SystemSoftware/s52000tc7_3_5.pkg' release-key='1TC007-1-FFFFFFFF'
2016-04-18T10:19:57.948+09:30 ppc appl[2863]: 4585179.60 PROV I: Starting normal download
2016-04-18T10:19:57.949+09:30 ppc appl[2863]: 4585179.60 PROV I: Starting download from https://yourTMSserver/tms/public/data/SystemSoftware/s52000tc7_3_5.pkg to /upgrade/pkg.tmp
2016-04-18T10:19:57.951+09:30 ppc appl[2863]: 4585179.61 PROV    HttpDownloader(this=0x232c6c60): initialized

Another thing you can check is the HTTPS VerifyServerCertificate setting on the endpoints (under Configuration, Network Services, HTTPS).  If it's On, change it to Off, just in case one of your server certificates is invalid or expired and try again.  If that works, then you know a setting that can make it work, and/or that you need to fix your certificates.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Chet Cronin]

Wayne,

I think I figured out the problem ... The FQDN wasn't correct.  Once I changed it and ran the upgrade process it pushed the firmware.  I am doing a couple more just to make sure.

By the way I have a new problem.  I noted on a couple systems I uploaded from WEB Remote that they get hung on the last part of the upgrade i.e. When it installs hooks ... I have to restart the system???    What might cause that???

[Wayne DeNardi]

I have one EX90 that still does that as well, every time we upgrade it, it needs to be manually rebooted.  I suspect it's faulty hardware on that one as all the others are fine, and others that did have the issue that we've since RMAd are now all fine too.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.

[Wayne DeNardi]

Have you actually downloaded the TC7.3.5 software and copied it up to the TMS server?  Since TC6.1.9, TMS no longer automatically downloads the software itself and needs to have it manuallly uploaded.  See this document for details.

Are you able to share a screenshot of the Activity Status screen or the endpoint logs to help us see where the failure might be occurring.

Wayne
--
Please remember to rate responses and to mark your question as answered if appropriate.