06-28-2012 07:36 PM - last edited on 03-25-2019 09:03 PM by ciscomoderator
Migrated TMS 13.2 with TMSPE from physical server to VM 2008 R2, which went fine, however, VCS-C is now giving me HTTP exception status for all provisioning services; users, phonebooks and devices.
Status: failed
Response: HTTP Exception
Reason: (400) HTTP Error: WWW-Authenticate
And yet, TMSPE diagnostics in TMS shows no alarms; it's all green.
TAC is working on it though.
Anyone seen this before ?
/jens
Solved! Go to Solution.
07-27-2012 05:04 AM
Thanks for the clarification Jens And I'm talking internally to all those involved here since it appears we have 'several hands in the kitchen' (as well as different kitchens) concernng this one...meaning let's use the official channels for troubleshooting this one and we'll simply update this thread with what the outcome was...agreed?
02-06-2013 03:19 PM
Hi Guys,
After about 4 hours on a case, I have figured out what is causing this issue. I am able to replicate this in the lab as well.
This issue is what is sent in the 401 Unauthorized from TMS. The working message stream is as follows:
1. VCS sends 4 GET messages to TMS (1 for each TMSPE service)
2. TMS sends a 401 Unauthorized with the following 3 lines:
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
WWW-Authenticate: Basic
3. VCS sends 4 GET messages with a selected authentication method
The failed message stream is as follows:
1. VCS sends 4 GET messages to TMS (1 for each TMSPE service)
2. TMS sends a 401 Unauthorized with the following 3 lines:
WWW-Authenticate: Basic
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
3. VCS doesn't send anything
Notice how Basic authentication is listed first. When the VCS receives the 401 it doesn't send the GET messages at all. This is the issue at hand. The VCS is expecting Basic to be listed in the 3rd spot. You can still change the order of NTLM or Negotiate by following the steps in my previous post.
To change the order of the Basic message you need to change the order in which it lists in the modules portion of IIS.
From what I have found you will most likely have to modify the applicationHost.config file. In the GUI you can move this up using the modules section but I have found that the items are locked and are unable to move in the GUI.
The file is located in C:\Windows\System32\inetsrv\config
To fix it I have written the following steps below:
1. Backup the applicationHost.config file
2. Stop the WWW Publishing service on the Windows server
3. Edit the applicationHost.config file by opening it in Notepad
*WARNING: This file contains a lot of information that pertains to IIS. If the file is not modified correctly you might have to end up uninstalling and re-installing everything.*
4. Search/Find the string "
You should find only two of them
Look for the one that looks like this:
5. Look below the modules tag and find the line with:
6. Move the above line in step 5 line anywhere above the following two lines still underneath the
If the BasicAuthentiacationModule line is below the WindowsAuthenticationModule lines in the file, the Basicauthentication line in the 401 from TMS will be at the top causing failure.
7. Save the file which is located in C:\Windows\System32\inetsrv\config
8. Start the WWW Publishing service
9. Disable and re-enabled the services on the VCS
Let me know if you have further questions.
Chad
06-29-2012 01:37 AM
Hey Jens
TMSPE diagnostics does not check the VCS communication errors its only checking health on the TMS server it self that is why you don't see any errors here. You can have all sorts of communication errors to your VCS and the diagnostics will still stay green since the TMS is healthy.
I suspect this to be an IIS issue so I would check the authentication methods in IIS. I found a case showing this error and it was resolved by turning of windows authentication in IIS and turning it back on again... Is the TMS in a domain and the username is typed in with DOMAIN\username in the VCS provisioning configuration (via TMS)?
/Magnus
06-29-2012 01:50 AM
Hei Magnus,
Aha, that clears up the diagnostics side of it.
TMS is in domain, but I created a local admin user account on the server which also has full site admin rights in TMS, this worked fine with the old server.
Just changed the hostname to reflect the TMS VM hostname, so it's hostname\username, and this is present everywhere it should be; TMS provisioning extension settings page in TMS and also in the VCS provisioning configuration.
By the way, the original tmsng db was copied across to an external VM SQL server, and original tmspe db was copied across to the VM TMS server itself.
I'll take a look at the IIS next, thanks for the tip.
/jens
07-26-2012 03:01 AM
Just an update.
Still having the http exception issue with TMSPE, been working with TAC for quite a while now trying to resolve it.
So far no luck, case has now been escalated to R&D so hopefully we can get to the bottom of this.
/jens
07-26-2012 05:29 AM
Hi Jens,
This has reached my level and although I don't want to troubleshoot here and also within our internal tools, I responded to the TAC engineer handling the case (Deepti) with basically what Magnus as stated above. For example, is the domain part being provided in the user account you are utilizing in all locations, e.g. CISCO\daleritc? So has to see if this is just on the TMS side, can you change it to another account (that is in the TMS Site Admin group) in the provisioning extenstion settings page successfully? In addition, I assume your not using HTTPS on the Cisco TMS Connection on the provisioning extension settings page?
And the provisioning extension log is clearly stating you have an authentication issue:
Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials
org.apache.commons.httpclient.auth.InvalidCredentialsException: Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials
And with regards to the db locations now, Deepti has told me that both the tmsng and tmspe db are now on the same external SQL server. Can you please confirm, although I don't believe the location of the dbs is the root of your problem.
And finally, was there an OS difference from the physical server to the VM server? Are the TMS and TMSPE installed to the Default Website on the new server? Are there any other applications or something on the new server that may be effecting IIS securities? For example, if you look at the TMSAgent virtual folder, are both basic and windows authentication enabled? No IIS re-directs or anything like that going on?
Let Deepti know I've posted here
07-26-2012 10:41 PM
Hi Dale,
Old, physical server, where everything worked fine; OS Win 2003 32bit, SQL 2005, both tmsng and tmspe as well as TMS itself sat on this one server.
New VM TMS server; Win 2008 r2 64bit, SQL 2008 r2 svc pack 1 - only TMS runs on this server now.
External SQL server now housing both tmsng and tmspe; OS Win 2008 r2 svc pack 1, SQL 2008.
Nothing else runs on either server.
Domain part included, local user created on TMS server with full site admi rights and authenticating with hostname\username.
Changing credentials to an existing ad account with full admin and site admin rights, makes no difference.
HTTPS not used on the provisioning extension page.
Folder authentications thoroughly examined by 3 different Cisco engineers, so I assume these are correct.
Also got a surprise call from Andrew Bell and Michael McGary this morning, and spent some on Webex with them going through quite a few things. They did add this to the case notes, so might be best to check those too.
cheers jens
07-27-2012 02:55 AM
"New VM TMS server; Win 2008 r2 64bit, SQL 2008 r2 svc pack 1 - only TMS runs on this server now."
So where is the TMSPE app now, if only TMS run on this server now?
I'll cross ref with Michael and Andrew as well as the case notes.
rgds,
Dale
07-27-2012 04:57 AM
Sorry Dale, should have said:
"New VM TMS server; Win 2008 r2 64bit, SQL 2008 r2 svc pack 1 - only TMS and TMSPE app runs on this server now that tmspe db has been moved to external SQL server".
/jens
07-27-2012 05:04 AM
Thanks for the clarification Jens And I'm talking internally to all those involved here since it appears we have 'several hands in the kitchen' (as well as different kitchens) concernng this one...meaning let's use the official channels for troubleshooting this one and we'll simply update this thread with what the outcome was...agreed?
07-27-2012 05:14 AM
Absolutely, I only updated this thread to keep others who might have a similar problem, or encounter a similar problem in the future, informed of the progress - so if someone should Google TMSPE http exception error...well
cheers jens
08-13-2012 02:58 AM
I just Googled for TMSPE HTTPException and got to this thread - I have this issue after migrating to TMSPE from TMS Legacy Mode under the Phone Book Service Status:
(404) No phonebooks found for entity 041ae628-c4be-4395-8fdb-aa390775ea91
The other services seem to work fine, in my case only the Phone Book Service seems to be affected.
What was the resolution to your issue?
08-13-2012 03:15 AM
Hi I do not believe you see the same issue as Jens here. The phonebook service is just letting you know that you don't have any phonebooks to synchronize yet.
If you go into Manage Phonebooks and select a phonebook (i.e the Provisioning Phonebook) and then click the "Access control" tab. Select the group you want to be able to access this phonebook and save. Then try to do a full synch with the VCS again and see if this resolves your issue.
Best Regards
Magnus Ohm
08-13-2012 03:21 AM
Completely concur with what Magnus says Two completely different issues (400 vs 404). Magnus was just quicker with the keyboard than I And yes, it is basically telling you that you don't have any PBs to sync just yet.
08-13-2012 04:17 AM
Magnus, Dale thanks for the input, you are right it was a Phone Book issue.
08-24-2012 06:47 PM
Marked another reply as "correct answer" since I can't mark my own post as such, but here it is:
Correct Answer:
I'm pleased to say our httpe exception issue has now been resolved.
Uninstalled TMSPE
Uninstaled TMS
Rebooted server
Uninstalled NET framework 4
Uninstalled IIS
Rebooted server
Downloaded and re-installed NET framework 4
Rebooted server
Re-installed TMS 13.2.1 and allowed it to configure IIS
Re-installed TMSPE
Rebooted server
And it now works as it should - big "thank you" to everyone involved; I know there was a lot of people in Cisco looking at this and how to resolve it - which was really appreciated.
I am now going to have an excellent week-end - and I wish you all the same.
----------------------------
PS
I understand my particular issue is related to NET framework 4 and IIS which is installed when building Microsoft 2008R2 servers, Cisco is talking to Microsoft re this. If any Cisco engineers wants to take a look at the case notes; TAC case ref is SR 622269725.
---------------------------
Update 1:
Even though the above solved the communication issue between VCS-C and TMS, phonebook provisioning still didn't work properly as only Provisioning phonebook would be exported to VCS-C, not any of the other ones, so:
re-installed TMSPE and allowed it to create and configure a new database located on the local TMS server, instead of on the external SQL server.
Re-created user directory and device templates.
I now have full provisioning again of all TMS phonebooks (not only Provisioning phonebook), users and devices.
---------------------------------------
Update 2:
Allowed Windows update to download and install 9 Net4 framework security updates resulting in loss of phonebook synchronisatiIon.
I took a snapshot of the server prior to installing these updates, re-applied this snapshot, and all services again working properly.
Windows update has now been turned off on this box and currently running Net4 v4.0.30319 - again this is a 2008 R2 server.
/jens
Message was edited by: Jens Didriksen - included additional information.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide