TMS No response/No HTTP response for an endpoint

Anthony Thomson · ‎02-19-2014

I've just upgraded a client's TMS to 14.3.2 (and migrated to Windows Server 2008 R2/SQL 2008 in the process), and now one of their systems continually shows a System Status of "No response" or "No HTTP response". It did not do this under TMS 13.2.2 on Win2003/SQL 2005.

Now, I've searched the forums, and found some postings that say this has to do with the BITS service, and web proxies, and the fix is to manually configure the BITS services to use No Proxy. This has had no effect.

Given that there is only a single endpoint having this problem, and given that this single endpoint has significant communication lag since the connection is over a satellite connection (500-600ms), I suspect that this may be more of a timeout issue more than anything. However, I don't know if or how I can change a parameter that may be affecting this. I had thought I had seen a posting somewhere that concerned this, but I am unable to find it now.

Any help is appreciated. Thanks.

Patrick Sparkman · ‎02-19-2014

Hello Anthony -

You can set TMS to not update system status, by changing the "Update System Connectivity for Systems" setting in Configuration > Network Settings within TMS.

Specify whether Cisco TMS will change a system's connectivity status (for example: Reachable on LAN, Behind Firewall) if it detects that this has changed.

Automatic: Cisco TMS will change the system's connectivity status as it detects it.
Manual: Cisco TMS will not change any system's connectivity status.

If set to Manual, administrators will have to change a system's connectivity status themselves by going to Systems > Navigator > select a system > Connection tab > System Connectivity.

Anthony Thomson · ‎02-19-2014

The System Connectivity setting isn't the issue though. Not sure how this helps.

I'm concerned with the System Status: Idle/In Call/No Response/No HTTP Response/No SNMP Response.

Patrick Sparkman · ‎02-19-2014

Can you confirm your TMS is reachable from the endpoint's network. What type of endpoints are we talking about? The newer endpoints C-Series/EX/E20 etc us 80 and 443 to communicate to TMS. Some older systems use SNMP 161/162 (can't remember which port in particular).

Anthony Thomson · ‎02-19-2014

It's an MXP endpoint. It is able to ping the TMS server successfully, with round trip times 500-600 ms.

xstatus feedback shows "status='Synced'".

It is an older build of software, due to the issues of trying to upgrade it over the satellite link. Only F6.4, and since EoL now, no way to upgrade it to F9.3.1.

Patrick Sparkman · ‎02-19-2014

MXP endpoints use SNMP to communicate it's status to TMS.

Btw, there are security advisories out for the MXP, C-Series/EX, E20 and VCS software that will allow you to upgrade to either the most recent major release of that software or very close.

http://www.cisco.com/en/US/products/csa/cisco-sa-20110831-tandberg.html

http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130619-tpc

http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20140122-vcs

Steve Kapinos · ‎02-19-2014

When you goto the system in system navigator, and use 'edit settings' or 'force refresh' or goto the 'connection' tab, is the system able to be displayed without reporting no connection?

(this tests the HTTP or HTTPS to the device in the context of the logged in user)

If not, you need to confirm the network connectivity between TMS TO the device over HTTP and/or HTTPS. The timeouts under Administrative Tools -> Configuration -> General Network settings wil be in play here. I would not recommend raising these high tho as you will impact nearly all communication with these settings.

If you can do that, and the device drops to no response on its own after a period of time, that means the Discovery Scanner can not reach the device, or TMS failed to communicate with the device with background tasks.

That is when the proxy reference comes into play, because those are handled under the security context of the services, not the logged in user. This only matters when proxies that require authentication are in play in the network.

For an MXP system, TMS knows it supports SNMP, so normally it would try to use SNMP in the quick system scanner. I know for sure it would skip SNMP if you had 'secure-only' enabled in TMS, but I think it will follow if you disable SNMP in the endpoint as well (turn off the SNMP network service).

If you look at the history log for the device (again under system navigator) you can get a hint of when it's connection status is changing

Anthony Thomson · ‎02-20-2014

"When you goto the system in system navigator, and use 'edit settings' or 'force refresh' or goto the 'connection' tab, is the system able to be displayed without reporting no connection?

(this tests the HTTP or HTTPS to the device in the context of the logged in user)"

Sometimes; usually not.

"If not, you need to confirm the network connectivity between TMS TO the device over HTTP and/or HTTPS. The timeouts under Administrative Tools -> Configuration -> General Network settings wil be in play here. I would not recommend raising these high tho as you will impact nearly all communication with these settings."

I raised these; there seemed to be some response, as I could do a Force Refresh now.

"If you look at the history log for the device (again under system navigator) you can get a hint of when it's connection status is changing".

The History shows a long list of alternating OK/No HTTP Response/No SNMP Response. The "No SNMP Response" portions appear to only be from 2 or more days ago though.

TMS told me to look at the log-web.txt file due to the issues, and what I see there are connection attempts to the endpoint over HTTPS (GetDocument Failed. URL: https://192.168.x.x/putxml, base exception: Unable to connect to the remote server) , even though HTTPS was not enabled on the endpoint and TMS is not configured for Secure-Only device communication. So, I enabled HTTPS on the endpoint (and left HTTP enabled), and now TMS reports this:

CommunicationFailureHTTPException: hostAddress: 192.168.x.x, description: GetDocument Failed. URL: http://192.168.x.x/getxml?location=/Status/, base exception: httpResponse was 302 redirect

It's no longer trying to connect to HTTPS, but it gets redirected to the HTTPS page, and doesn't appear to like it.

Steve Kapinos · ‎02-20-2014

TMS tries HTTPS first.. then uses HTTP when talking to these devices. It's changed over different releases how it deals with affiniity for the protocol... but the mindset is by checking HTTPS first, it uses the more secure protocol and can cope with the endpoint changing it's available services.

You shouldnt't be getting a 302 redirect by enabling HTTPS - are you sure you rebooted the endpoint after changing the services? Are you sure the 302 redirect wasn't from ANOTHER device in the path?

If your attempts at force refresh are intermitant that would infer timeouts.

But you shouldn't be getting the 'tms encounterd an error' page purely based on timeouts. Timeouts would be handled under normal exception handling and the page would just say 'could not connect to...' kind of messages. This also points to something more than timing delays being in the mix.

I think for better speed, the HTTPS would probably be better since it's what TMS tries first. Your 302 issue points to something else maybe being in the mix.

Focus on TMS being able to reliably reach the endpoint first.. (force refresh is a simple test for this). If your results are intermitant, I would run wireshark on the box while you are doing this, so you can see what the real responses to the queries are.

TMS 13.x to current is a significant jump in the product and in what has changed... but I can't think of changes in the way things are done that would cause these kinds of symptoms. I'd wager there is another change invovled (like switching servers, network location, etc) that is involved, and not just the TMS version.

Anthony Thomson · ‎02-21-2014

Steven Kapinos wrote:

You shouldnt't be getting a 302 redirect by enabling HTTPS - are you sure you rebooted the endpoint after changing the services? Are you sure the 302 redirect wasn't from ANOTHER device in the path?

Well, if I try to connect to the web interface of the codec from the TMS server, I don't get errors. It's slow, but no errors.

Steven Kapinos wrote:

Focus on TMS being able to reliably reach the endpoint first.. (force refresh is a simple test for this). If your results are intermitant, I would run wireshark on the box while you are doing this, so you can see what the real responses to the queries are.

TMS 13.x to current is a significant jump in the product and in what has changed... but I can't think of changes in the way things are done that would cause these kinds of symptoms. I'd wager there is another change invovled (like switching servers, network location, etc) that is involved, and not just the TMS version.

Okay, well I'm going to have to put this on hold for a while. Thankfully it's just a single endpoint. I'm on training next week, and then March insanity begins (i.e. I'm really busy)...could be the end of March before I have time for this. Thanks for the pointers, and I'll get back to this.

Nikita Shrivastava · ‎02-21-2014

Hi Anthony,

What is the software version on the MXP.

Can you please share the xconfig from the MXP.

Thanks.

Anthony Thomson · ‎02-21-2014

F6.4. It's never been upgraded beyond that due to the difficultly of trying to do so over a satellite link.