No HTTP response (TMS 14.3.1)

[Chet Cronin]

Any ideas???

I am running TMS on Window2008R2 64Bit.  TMS version 14.3.1.

Getting errors on my 70 systems connected "No HTTP response" ????

Called our network and firewall folks and we confirmed when forcing update from the TMS to the endpoint there is communications.  They notice the server is sending resets on port 80 ???  Appears the server (TMS) doesn't like what the endpoint is sending for a reply to the query???

Community string is other than public and this issue began a few hours ago ... Nothing changed that we know of other than a server restart which I do weekly?? 

Checked services and they are started. 

[Steve Kapinos]

If it says 'No HTTP response' that is TMS trying to hit it via the web protocols... community strings are not applicable.

First, ensure you can initiate TMS talking to the device via the TMS GUI.  Goto a device in System Navigator, and do a 'force refresh' when viewing the settings.  If the page refreshes without any errors, TMS can talk via HTTP or HTTPS (assuming a Cisco endpoint) to the endpoint.  Yes, it will use both ports 443 (HTTPS) and port 80 (HTTP).

The status of the device should also change to reflect it can communicate to the device.  (Idle, InCall, etc)

If that device then later drops to 'No HTTP Response' without any change to the endpoint... and you can do a force refresh and it changes again...  This means TMS could not talk to the device during some background task.  Tasks that happen in the background run in a different security and user context than actions taken by you navigating in the TMS GUI.

If it works when you are browsing the TMS GUI, but not when the background tasks do it... this is normally a proxy configuration issue.

Check with your network people if a proxy is required to communicate between TMS and the endpoints.  Ensure authentication is not required.. if so, ask that TMS be added as an exclusion to the proxy authentication requirements.

If the proxy is required, the windows server may not be configured to use the proxy for service accounts (this is similar to the problem tha impacts Windows Updates). 

Open a command prompt

>netsh winhttp show proxy

It will show if a proxy is configured at all..   You can configure your 'internet options' proxy settings in Windows/IE (which configure it for YOUR user.. not all users on the box) and then import those settings into winhttp

>netsh winhttp import proxy source=ie

[Chet Cronin]

Thank you for the response.

We have verified port 80 and 443 are not blocked.  No poxy being applied or required.

What we do see is over 70 systems (MXP series EP's) are the issue. 

There are five systems set to Idle at the moment and they are C-Series Codecs.

What is interesting is that All the systems effected have HTTP disabled and HTTPS enabled.

Yet TMS is looking for HTTP responses and what looks like not port 443 communications between the server and the EP's.

The Idle ones all have port 80 and 161 turned off yet TMS seems to not have issues with them on 443???

These systems are in numerous geographic locations. 

Is like TMS doesn't want to communicate with the EP's on port 443 anymore and it defaulted to HTTP responses only????

[Steve Kapinos]

Chet Cronin wrote:
 Thank you for the response. 
We have verified port 80 and 443 are not blocked.  No poxy being applied or required. 
What we do see is over 70 systems (MXP series EP's) are the issue.  
There are five systems set to Idle at the moment and they are C-Series Codecs. 
What is interesting is that All the systems effected have HTTP disabled and HTTPS enabled. 
Yet TMS is looking for HTTP responses and what looks like not port 443 communications between the server and the EP's. 
The Idle ones all have port 80 and 161 turned off yet TMS seems to not have issues with them on 443??? 
These systems are in numerous geographic locations.  
Is like TMS doesn't want to communicate with the EP's on port 443 anymore and it defaulted to HTTP responses only???? 

Understand TMS will still try HTTP and HTTPS at times regardless of the endpoint's settings.  This is because TMS is doing fallback after it can not reach the device on HTTPS to account for situations where the endpoint's services might have changed or the device type might have changed. 

There is no way in TMS to force HTTP only, so there is no worry that it's fallen into such a state.

You should try a 'force refresh' from System Navigator for a system in the condition and see if TMS can communicate to the device or not.  If it fails and drops to the Connection tab for the system, TMS can not reach the device on the supported protocols.

I would repeat the test while running wireshark on the TMS server and watching the communications to the IP in question. 

Ensure you don't have 'secure-only mode' enabled and 'verify certificates = yes' enabled if you have not properly deployed signed certitificates - that is the only case where HTTPS connections would be rejected by TMS if HTTPS were enabled on the device.  But this combination should give you a 'no HTTPS reponse' error message, not 'no HTTP response', so I doubt that is your issue.

[Chet Cronin]

Thank you all for response.  I ended up created a SEV2 TAC and working with the LAB to recreate the issue. 

Also found out that the C-Series and EX-Series and MX Series are all using HTTP/HTTPS only now.  The MXP must use SNMP along with HTTP or HTTPS.  Also problem now looking at SNMP being the culprit on the server ...

[Michael Jensen]

Chet:

We have the same issue here, running TMS 14.2.2 and MXP units on 9.3.1.  Reports "No HTTP Response" but units are registered to VCS, can make and receive calls and have no firewall issues.

We also have opened up a TAC ticket, and performed a WireShark packet capture and submitted to TAC.  Their initial analysis showed HTTP connection to the endpoint, with a re-direct to HTTPS, and that was followed by TMS timing out.

We have 140 or so C series codecs all functioning normally.

I look forward to hearing how your TAC case proceeds.

[fahad ck]

By default, the Cisco TMS installer will set up HTTPS for web communication, offering to create a self-signed certificate if the administrator does not provide one. For improved security, we strongly recommend using valid certificates signed by a certificate authority.
Note that you must create a regedit entry of TLS 1.2 and TLS 1.1 for Client and Server to enable TLS 1.2 and TLS 1.1 communication between windows server and Cisco TMS.

if  this software version TC7.3.6 - it Discontinued support for TLS 1.0

Cisco TelePresence Endpoints running TC7.3.6 only support TLS version 1.1 and 1.2 due to security concerns with TLS version 1.0. Please note that this may affect communication with servers that only support TLS version 1.0. If TMS is running on a Windows server that only has TLS version 1.0 enabled by default (i.e. Windows Server 2008 R2) it may cause connection problems when theendpoints upgraded to TC7.3.6. Make sure TLS 1.2 or 1.1 is enabled on the server before upgrading to TC7.3.6. Older browsers may not be able to reach the endpoints web interface on HTTPS if the browser only supports TLS 1.0.

[Steve Kapinos]

Chet Cronin wrote:
 Thank you all for response.  I ended up created a SEV2 TAC and working with the LAB to recreate the issue.  
Also found out that the C-Series and EX-Series and MX Series are all using HTTP/HTTPS only now.  The MXP must use SNMP along with HTTP or HTTPS.  Also problem now looking at SNMP being the culprit on the server ...

SNMP is expected to be on for MXP systems, but TMS can work with reduced functionality if SNMP is turned off on the endpoint.  And if SNMP were the problem, it would not say 'No HTTP response', it would say 'No SNMP response'

You should go through the troubleshooting I outlined prior

1 - Make sure you can do a 'force refresh' against the endpoint from the System Navigator.  If you can't do that, proceed no further and focus on HTTPS or fallback to HTTP connectivity between TMS and the endpoint.

2 - If you can do a force refresh, and it later changes to 'no http repsonse' on its own, this means the background services could not reach the endpoint.  Check the History log of the endpoint in system navigator to see when the problem happens.  If you can do it via the GUI, but the background services can not, this is a authenticated proxy issue on the network or a proxy setting on the server.

The #1 step is always to check 'can I do a force refresh' from System Navigator.  That isolates the problem between 'all connections from TMS' vs 'background services connecting from TMS'

[Chet Cronin]

Roger that on all.  We can do a force refresh with no issue.  Working with your team on the CNS and ITS side they noted the MXP series EP's do use both HTTP or HTTPs and SNMP even when SNMP is turned off on the EP.  Something about the firmware and that some data is transferred via SNMP.  Our issue we are now looking at is the SNMP version the MXP's are running i.e. V1 or V2.  FIPS equires using V3 and we think that is the cultprit.  The CISCO LAB and us are checking. 

I have a server that was on the network running Windows2008R2 and TMS 14.3.1 and the windows security shows FIPS disabled and when that was online a short time ago the EP's were find.  The current online servers all have FIPS endabled and now the problem ...

Not saying that is the cause but we are checking it out.

[Steve Kapinos]

Chet Cronin wrote:
 Roger that on all.  We can do a force refresh with no issue.  Working with your team on the CNS and ITS side they noted the MXP series EP's do use both HTTP or HTTPs and SNMP even when SNMP is turned off on the EP.  Something about the firmware and that some data is transferred via SNMP.  Our issue we are now looking at is the SNMP version the MXP's are running i.e. V1 or V2.  FIPS equires using V3 and we think that is the cultprit.  The CISCO LAB and us are checking.  
I have a server that was on the network running Windows2008R2 and TMS 14.3.1 and the windows security shows FIPS disabled and when that was online a short time ago the EP's were find.  The current online servers all have FIPS endabled and now the problem ...
Not saying that is the cause but we are checking it out. 

If FIPs is enabled on the Windows Server, TMS will react by not doing anything that requires MD5.  FIPS only will not impact TMS SNMP behavior on the server.  Tho honestly none of our clients enable FIPS and still use SNMP.  SNMP is disabled because v1 and v2 are not secure and policy dictates they be disabled.  Those customers usually using FIPS would be running Secure-Only mode in TMS.   If you are doing other customizations like enabling FIPS on the server, I'd be looking at what other type of security lockdowns or AV are in place.

SNMP has nothing to do with 'No HTTP response' status.

Since you can do a Force Refresh fine... then look at the history log of the endpoint (in the System Navigator view of the system.. in the log tab) and see when the status is changing. If it's changing by the network service, that means the background services could not connect to the device.  Methods, which were tested the same when you did a 'force refresh'.

This means the problem is some process or security implementation is preventing the Windows Services (which run under a different security context) from reaching those systems.

I am a senior member of the development team that created TMS and also the one who wrote all the policies and methods for our DoD certifications.  You can believe I am well versed in these topics

[Chet Cronin]

Steven,

Thank you for the detailed response.  Really appreciate it.  Worked with Brian and David last night... Still can't fix the the issue.  Would be nice to know what triggered the error ... the TMS has been running like a champ for months.  We just can't figure out what triggered the problem.  Brian did mention something called Digest???  Only reason we were focusing on SNMP as that on one of the old server it was disabled and on the production one it's enabled.  I was under the impression that SNMP was used which is why we were focusing there.  Your comments now put us in the dark again ....  Can you explan more about what exactly the difference is with MXP not working and the C & EX series with no issues????  That might help us localize the issue.  Was hoping CISCO LAB could re-create our problem.  I have logs and such but they are only available on to the CNS team. 

Thank you again. 

Also I think I might have confused someone.  The force refresh is just that it refreshes with the same error.  Sorry.

[Steve Kapinos]

When FIPS mode is enabled in windows, MD5 is disabled on the server.  HTTP Digest authentication uses MD5, so Digest is no longer a valid form of authentication for the server when talking to devices over HTTPS/HTTP

This means your run of the mill system will only be left with basic (clear text) authentication available for Web connections.  This means you should ONLY use SSL  to avoid having your users or systems authenticate using basic in the clear across the network.  HTTPS should be on, and HTTP off on all of your endpoints if you have FIPS mode enabled in TMS.  At that point, I would also recommend using 'secure-only mode' in TMS as well to force HTTPS use only.

A simple way to test if digest is part of your problem is... just disable authentication on one of the endpoints to see if the problem goes away.  Blank out the systemunit password on the MXP endpoint, and the endpoint will stop prompting for web authentication. If that works, you know there is some interop issue between the server and device when digest auth is not available.

If the problem is visible during 'force refresh' - meaning you do it, and TMS drops to the connection tab and says it can not communicate with the system... then TMS can not establish a HTTPS or HTTP connection to the device.  A packet capture (just run wireshark on TMS while you try to do a force refresh) is the most direct way to narrow the problem.

Things like authentication issues, connection resets, etc would all be pretty easy to identify in a packet trace.

Why MXP vs TC?  They are different endpoints.. different servers.. etc.  It's an important differentiation, but it alone doesn't narrow things down - it just elminates other potential server wide issues.