cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
10316
Views
20
Helpful
35
Replies
Jens Didriksen
Engager

HTTP Exception after migrating TMS with TMSPE

Migrated TMS 13.2 with TMSPE from physical server to VM 2008 R2, which went fine, however, VCS-C is now giving me HTTP exception status for all provisioning services; users, phonebooks and devices.

Status: failed

Response: HTTP Exception

Reason: (400) HTTP Error: WWW-Authenticate

And yet, TMSPE diagnostics in TMS shows no alarms; it's all green.

TAC is working on it though.

Anyone seen this before ?

/jens

Please rate replies and mark question(s) as "answered" if applicable.
2 ACCEPTED SOLUTIONS

Accepted Solutions

Thanks for the clarification Jens And I'm talking internally to all those involved here since it appears we have 'several hands in the kitchen' (as well as different kitchens) concernng this one...meaning let's use the official channels for troubleshooting this one and we'll simply update this thread with what the outcome was...agreed?

View solution in original post

Hi Guys,

After about 4 hours on a case, I have figured out what is causing this issue. I am able to replicate this in the lab as well.

This issue is what is sent in the 401 Unauthorized from TMS. The working message stream is as follows:

1. VCS sends 4 GET messages to TMS (1 for each TMSPE service)

2. TMS sends a 401 Unauthorized with the following 3 lines:

    WWW-Authenticate: Negotiate

    WWW-Authenticate: NTLM

    WWW-Authenticate: Basic

3. VCS sends 4 GET messages with a selected authentication method

The failed message stream is as follows:

1. VCS sends 4 GET messages to TMS (1 for each TMSPE service)

2. TMS sends a 401 Unauthorized with the following 3 lines:

    WWW-Authenticate: Basic

    WWW-Authenticate: Negotiate

    WWW-Authenticate: NTLM

3. VCS doesn't send anything

Notice how Basic authentication is listed first. When the VCS receives the 401 it doesn't send the GET messages at all. This is the issue at hand. The VCS is expecting Basic to be listed in the 3rd spot. You can still change the order of NTLM or Negotiate by following the steps in my previous post.

To change the order of the Basic message you need to change the order in which it lists in the modules portion of IIS.

From what I have found you will most likely have to modify the applicationHost.config file. In the GUI you can move this up using the modules section but I have found that the items are locked and are unable to move in the GUI.

The file is located in C:\Windows\System32\inetsrv\config

To fix it I have written the following steps below:

1. Backup the applicationHost.config file

2. Stop the WWW Publishing service on the Windows server

3. Edit the applicationHost.config file by opening it in Notepad

*WARNING: This file contains a lot of information that pertains to IIS. If the file is not modified correctly you might have to end up uninstalling and re-installing everything.*

4. Search/Find the string "" without quotes

   You should find only two of them

   Look for the one that looks like this:

  

       

           

5. Look below the modules tag and find the line with:

6. Move the above line in step 5 line anywhere above the following two lines still underneath the tag.

  

  

If the BasicAuthentiacationModule line is below the WindowsAuthenticationModule lines in the file, the Basicauthentication line in the 401 from TMS will be at the top causing failure.

7. Save the file which is located in C:\Windows\System32\inetsrv\config

8. Start the WWW Publishing service

9. Disable and re-enabled the services on the VCS

Let me know if you have further questions.

Chad

View solution in original post

35 REPLIES 35
Magnus Ohm
Cisco Employee

Hey Jens

TMSPE diagnostics does not check the VCS communication errors its only checking health on the TMS server it self that is why you don't see any errors here. You can have all sorts of communication errors to your VCS and the diagnostics will still stay green since the TMS is healthy.

I suspect this to be an IIS issue so I would check the authentication methods in IIS. I found a case showing this error and it was resolved by turning of windows authentication in IIS and turning it back on again... Is the TMS in a domain and the username is typed in with DOMAIN\username in the VCS provisioning configuration (via TMS)?

/Magnus

Hei Magnus,

Aha, that clears up the diagnostics side of it.

TMS is in domain, but I created a local admin user account on the server which also has full site admin rights in TMS, this worked fine with the old server.

Just changed the hostname to reflect the TMS VM hostname, so it's hostname\username, and this is present everywhere it should be; TMS provisioning extension settings page in TMS and also in the VCS provisioning configuration.

By the way, the original tmsng db was copied across to an external VM SQL server, and original tmspe db was copied across to the VM TMS server itself.

I'll take a look at the IIS next, thanks for the tip.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Just an update.

Still having the http exception issue with TMSPE, been working with TAC for quite a while now trying to resolve it.

So far no luck, case has now been escalated to R&D so hopefully we can get to the bottom of this.

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Hi Jens,

This has reached my level and although I don't want to troubleshoot here and also within our internal tools, I responded to the TAC engineer handling the case (Deepti) with basically what Magnus as stated above. For example, is the domain part being provided in the user account you are utilizing in all locations, e.g. CISCO\daleritc? So has to see if this is just on the TMS side, can you  change it to another account (that is in the TMS Site Admin group) in the provisioning extenstion settings page successfully? In addition, I assume your not using HTTPS on the Cisco TMS Connection on the provisioning extension settings page?

And the provisioning extension log is clearly stating you have an authentication issue:

Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials

org.apache.commons.httpclient.auth.InvalidCredentialsException: Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials

And with regards to the db locations now, Deepti has told me that both the tmsng and tmspe db are now on the same external SQL server. Can you please confirm, although I don't believe the location of the dbs is the root of your problem.

And finally, was there an OS difference from the physical server to the VM server? Are the TMS and TMSPE installed to the Default Website on the new server? Are there any other applications or something on the new server that may be effecting IIS securities? For example, if you look at the TMSAgent virtual folder, are both basic and windows authentication enabled? No IIS re-directs or anything like that going on?

Let Deepti know I've posted here

Hi Dale,

Old, physical server, where everything worked fine; OS Win 2003 32bit, SQL 2005, both tmsng and tmspe as well as TMS itself sat on this one server.

New VM TMS server; Win 2008 r2 64bit, SQL 2008 r2 svc pack 1 - only TMS runs on this server now.

External SQL server now housing both tmsng and tmspe; OS Win 2008 r2 svc pack 1, SQL 2008.

Nothing else runs on either server.

Domain part included, local user created on TMS server with full site admi rights and authenticating with hostname\username.

Changing credentials to an existing ad account with full admin and site admin rights, makes no difference.

HTTPS not used on the provisioning extension page.

Folder authentications thoroughly examined by 3 different Cisco engineers, so I assume these are correct.

Also got a surprise call from Andrew Bell and Michael McGary this morning, and spent some on Webex with them going through quite a few things. They did add this to the case notes, so might be best to check those too.

cheers jens

Please rate replies and mark question(s) as "answered" if applicable.

"New VM TMS server; Win 2008 r2 64bit, SQL 2008 r2 svc pack 1 - only TMS runs on this server now."

So where is the TMSPE app now, if only TMS run on this server now?

I'll cross ref with Michael and Andrew as well as the case notes.

rgds,

Dale

Sorry Dale, should have said:

"New VM TMS server; Win 2008 r2 64bit, SQL 2008 r2 svc pack 1 - only TMS and TMSPE app runs on this server now that  tmspe db has been moved to external SQL server".

/jens

Please rate replies and mark question(s) as "answered" if applicable.

Thanks for the clarification Jens And I'm talking internally to all those involved here since it appears we have 'several hands in the kitchen' (as well as different kitchens) concernng this one...meaning let's use the official channels for troubleshooting this one and we'll simply update this thread with what the outcome was...agreed?

Absolutely, I only updated this thread to keep others who might have a similar problem, or encounter a similar problem in the future, informed of the progress - so if someone should Google TMSPE http exception error...well

cheers jens

Please rate replies and mark question(s) as "answered" if applicable.

I just Googled for TMSPE HTTPException and got to this thread - I have this issue after migrating to TMSPE from TMS Legacy Mode under the Phone Book Service Status:

(404) No phonebooks found for entity 041ae628-c4be-4395-8fdb-aa390775ea91

The other services seem to work fine, in my case only the Phone Book Service seems to be affected.

What was the resolution to your issue?

Hi I do not believe you see the same issue as Jens here. The phonebook service is just letting you know that you don't have any phonebooks to synchronize yet.

If you go into Manage Phonebooks and select a phonebook (i.e the Provisioning Phonebook) and then click the "Access control" tab. Select the group you want to be able to access this phonebook and save. Then try to do a full synch with the VCS again and see if this resolves your issue.

Best Regards

Magnus Ohm

Completely concur with what Magnus says Two completely different issues (400 vs 404). Magnus was just quicker with the keyboard than I And yes, it is basically telling you that you don't have any PBs to sync just yet.

Magnus, Dale thanks for the input, you are right it was a Phone Book issue.

Marked another reply as "correct answer" since I can't mark my own post as such, but here it is:

Correct Answer:

          

I'm pleased to say our httpe exception issue has now been resolved.

Uninstalled TMSPE

Uninstaled TMS

Rebooted server

Uninstalled NET framework 4

Uninstalled IIS

Rebooted server

Downloaded and re-installed NET framework 4

Rebooted server

Re-installed TMS 13.2.1 and allowed it to configure IIS

Re-installed TMSPE

Rebooted server

And it now works as it should - big "thank you" to everyone involved; I know there was a lot of people in Cisco looking at this and how to resolve it - which was really appreciated.

I am now going to have an excellent week-end - and I wish you all the same.

----------------------------

PS

I understand my particular issue is related to NET framework 4 and IIS which is installed when building Microsoft 2008R2 servers, Cisco is talking to Microsoft re this. If any Cisco engineers wants to take a look at the case notes; TAC case ref is SR 622269725.

---------------------------

Update 1:

Even though the above solved the communication issue between VCS-C and TMS, phonebook provisioning still didn't work properly as only Provisioning phonebook would be exported to VCS-C, not any of the other ones, so:

re-installed TMSPE and allowed it to create and configure a new database located on the local TMS server, instead of on the external SQL server.

Re-created user directory and device templates.

I now have full provisioning again of all TMS phonebooks (not only Provisioning phonebook), users and devices.

---------------------------------------

Update 2:

Allowed Windows update to download and install 9 Net4 framework security updates resulting in loss of phonebook synchronisatiIon.

I took a snapshot of the server prior to installing these updates, re-applied this snapshot, and all services again working properly.

Windows update has now been turned off on this box and currently running Net4 v4.0.30319 - again this is a 2008 R2 server.

/jens

Message was edited by: Jens Didriksen - included additional information.

Please rate replies and mark question(s) as "answered" if applicable.
Create
Recognize Your Peers
Content for Community-Ad