10-25-2012 08:03 PM - edited 03-18-2019 12:02 AM
Hi,
Our client has TMS 13.1.1 running on windows 2003 R2 Standard with Windows SQL 2008 clustered.
For the last 2 1/2 years the automatic software update feature fails more often than it succeeds.
The issue has been present across several 12.X versions of TMS and now also on 13.1.1.
There have been several previous Tandberg and TAC cases in the past that have been unable to resolve this.
Eventually the issue goes away for a couple of months and then comes back again.
The client has tried with and without using a proxy (currently removed), and has put static firewall rules for port 443 for the TMS server and the URL IP.
It has failed the last 10 times in a row. Opening the url from the server works fine and the switchport is clean.
The client does have a large amount of endpoints (over 2,500) and I'm wondering if this could be related.
Below is a sample error from the scheduler logs.
Does anyone have any ideas?
08:57:49,387 [Event830906] WARN Tandberg.TMS.Service.SystemUpgrade.SystemUpgradeService - Invalid response from upgrade service, no software found for system: 5056
09:01:58,361 [Event830906] ERROR Tandberg.TMS.Service.SystemUpgrade.UpgradeCheckEventHandler -
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
--- End of inner exception stack trace ---
at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at Tandberg.TMS.Service.SystemUpgrade.SoftwareUpgradeWebService.getSoftwareUpgradeHwsnMac(VideoSystem[] systems, String clientID)
at Tandberg.TMS.Service.SystemUpgrade.SystemUpgradeService.GetSoftwareUpgrades(VideoSystem[] systems, SoftwareUpgradeWebService webService, String tmsSerialNumber)
at Tandberg.TMS.Service.SystemUpgrade.SystemUpgradeService.CheckForSystemUpdates(IList`1 tmsSystems)
at Tandberg.TMS.Service.SystemUpgrade.UpgradeCheckEventHandler.Start(BackgroundEventBase bgEvent)
09:01:58,361 [Event830906] ERROR Tandberg.TMS.Service.BackgroundEvent.BackgroundEventObject -
System.Exception: Update check failed. Please see system logs for details. Note that TMS must have internet access to contact the software update service.
at Tandberg.TMS.Service.SystemUpgrade.UpgradeCheckEventHandler.Start(BackgroundEventBase bgEvent)
at Tandberg.TMS.Service.BackgroundEvent.BackgroundEventObject.BackgroundEventProc()
10-25-2012 11:20 PM
Hi David
This may relate to a proxy issue or something else. Try to disable the automatic software update check and re-enable it. This will set a new timestamp for the update check (and also the user, since usually its the system user that executes this by default) and it will start about 3 minutes after you re-enabled it. If it is still unsuccessfull do the same process again and grab a wireshark trace from he TMS server while the check is running, do not filter on anything. Then open a TAC case or analyze the trace for anything that should not be there ("407 Proxy authentication required" in particular).
If you get a 407 it hits a proxy and this is most likely the issue. If it succeeds, monitor it the next 2 days and see if it fails again on the automatic launch.
/Magnus
10-28-2012 06:01 PM
Hi,
Thanks for your recommendation Magnus,
The software check has been disabled and reenabled many times in the past to no avail.
We're holding off doing it againet just yet as it deletes all of the logs.
The proxy has also been removed and readded many times without a change of behaviour.
The proxy is currently removed so there is NO proxy!
We're noticing in the logs if usually fails around the same time about 9 minutes in as per below.
When it does succeed it takes about 15 min but we see several end points failing consistently everytime (up to 19 now and increasing). I am going to check some of these manually.
This consistency in failed time suggests to me it is unlikely a connectivity issue. I'm am wondering whether it has something to do with the number of endpoints that need their SN checked (over 2,500) or related to some of the endpoints failing? (or the number that are failing)
22/10/2012 8:52 AM - Event executed by XXXXXXXXXXXX
22/10/2012 8:52 AM - Software update check started. Connecting to the update service.
22/10/2012 8:54 AM - Update check error for system XXXXXXXXXX.
22/10/2012 8:54 AM - Update check error for system XXXXXXXXXXX
22/10/2012 8:55 AM - Update check error for system XXXXXXXXXXXX
22/10/2012 9:01 AM - Update check failed. Please see system logs for details. Note that TMS must have internet access to contact the software update service.
22/10/2012 9:01 AM - The event failed to complete. Details: Update check failed. Please see system logs for details. Note that TMS must have internet access to contact the software update service.
I've just opened a new TAC case so will see if TAC have any recommendations other than a wireshark trace!
Thanks!
11-01-2012 07:43 AM
Did you get a resolution to this yet? I am having the exact same problem....
11-01-2012 04:50 PM
Not yet!
Working with TAC who assure me that 3000 + endpoints are ok and that the individual failing endpoints is not related or a cause of failure.
Plan to get wireshark traces of working and non working to compare and client database to load in their lab!
Please let me know your scenario/symptoms/versions/ect as TAC believe my client is the only one having this problem?
03-05-2015 09:10 AM
I see this post is 2 years old now - did you get a resolution? We recently upgraded from TMS 13.2.1 to 14.5.0 and are now seeing the update check fail every time it runs. If anyone else reading this can comment, does the automated process use the TMS-configured proxy settings and the manual update check use the browser-configured proxy settings? The manual check works every time.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide