Very slow performance after Active Directory integration

[cmartin]

Hi all.

We did a CallManager 4.1(3)sr1 with Active Directory 2000 integration. All seems to be right, but we realized that CallManager access and navigate into administration pages were very slow when you login with a user / password stored in AD (obviously, this user has total rights in MLA); when we use CCMADministrator user the navigation is normal (¿?). But this is not the only issue that we realized. When we try to access to the corporate directory trough a phone (7912 or 7940, we have several models and always the same), the query take 21 seconds to return any results (one o twenty of them, the time is always the same). Strange, isn't it?

Trying to solve the question, we run a sniffer in CallManager to show the packets, an we discovered the next traces:

first, CallManager send a LDAP request to AD with the query we did through the phone (sometimes even we can capture the packets for the bind process, but still work the same if we catch them or not). The AD answers inmediatly (0.1 seconds) with the right information back and then appear the traditional ACK packet to finish the request. Great.

All these proceses take 0.2 seconds but... no result appears on phone's screen.

20 seconds after the last packet, CallManager start again a bind process against AD, the bind process return a succesful binding operation and CallManager send two qestions more, with the same parameters that it use the first time, but varying the search base, from the specified users search base (that was used in the first query) to the "Configuration" CN (under root AD schema) the first query, and to "Schema" under the "Configuration" CN the second one. AD return no matches both, and CallManager send the ACK. These two searches and all the proceses associated least 0.2 seconds, and after these proceses, the results from the first query appears on phone's screen.

If all these searches were placed one after the other, the user query from the phone should be take less than a second, but, due to this strange delay of 20 seconds after the firt query, the directory search is too long. We really think that this is the same problem we have when trying to navigate administration web pages using a AD user,that is, querys between CCM and AD.

Our questions are, does anybody know why CallManager waits so many time (20 secs) between the first query to AD and the other two querys?

And, of course, does anybody know any kind of workaround to solve this issue?

Thank you very much.

Regards.

[smalkeric]

You will find this URL helpful:

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/4_0/sys_ad/index.htm

[kthorngr]

Typically these types of delays are due to problems with DNS or problems with referrals that are returned from the LDAP server. Many times what happens is the AD server that CCM is pointed to will respond with a referral. CCM will attempt to contact the server(s) listed in the referral. DNS issues and/or network issues can result in the CCM not being able to contact some servers which in turn causes delays in the responses.

If you are using Ethereal to look at the packet captures you can use this filter to see if there are any connection issues:

tcp.flags.syn && tcp.port == 389

Also, see if there are any DNS issues.

The CCM is not going to delay sending requests. The delays are caused by the information not being returned and CCM is waiting on a timer to expire for the request.

HTH,

Kevin

[cmartin]

Thank you both for your interest in solving this issue.

We were working with serveral possibilities without any kind of possitive results so we decided to open a TAC case, because is very strange. Maybe the problem goes through our AD server, because we installed an IPCC in other MCS but integrated with the same AD and we have the same delay problem (¿?). The traces showed the same time waiting "something" from AD.

As I said, is very strange, and we can't discard any theory (DNS issue is also been investigated).

In the very moment we get the solution, we share it here with all of you.

Thanks.

P.D.: Of course, any new ideas are welcome ;-)

[lnthompson]

Even once you iron out the DNS issues I think you will be disappointed in the performance. We had a TAC case with similar issues on this.

First we did find DNS issues. Do a nslookup for each and every zone:

Example

company.com

DomainDNSZones.company.com

subdomain1.company.com

DomainDNSZones.subdomain1.company.com

...

..

.

If any of those DNS queries return and address that is not a valid server then you will run into problems. We basically had 2 servers that had ethernet interfaces that weren't properly disabled and 169.x.x.x addresses were being resolved for those domains.

Anyway, to make a long story short. EVEN AFTER all those issues were resolved, performance was better, but still unacceptible in my opinion.

I have started to look into "indexing" the LDAP directory. I have also read that if you do LDAP queries against the global catalog server on port 3268 that you won't see any referrals, but I haven't been able to confirm Cisco's support on this. http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/TechRef/8196d68e-776a-4bbc-99a6-d8c19f36ded4.mspx

Maybe between the two of us, we can find a resolution.

[kthorngr]

Yes, you can search a GC against port 3268. If the attributes being searched are replicated to the GC then the GC will simply return the attributes requested instead of returning a referral. I have done this many times for customers where referrals were an issue.

Is there a possiblity that you can change the search base to reduce the scope of the search? This might help reduce or eliminate referrals.

I am not familiar with indexing the LDAP and whether that would help reduce referrals.

How long does a search take now?

Have you compared the time to a search tool, such as ldp.exe, to compare the search result times?

Make sure to set the search scope to subtree. If the searches are comparible but the CCM search takes signifigantly longer then we should take a look to see what is happening. Could be a CPU issue on the CCM server or something with our search algorithm. I have not seen any performance issues with our search algorithm though.

Kevin

[lnthompson]

How do you make sure that the appropriate attributes are replicated to the GC?

I can't reduce the scope because we are a centralized callmanager implementation with multiple subdomains.

I am not familiar with using the ldp.exe tool. can you show me a sample syntax?

When I "search" for a user it only takes about 5-6 seconds. When a user tries to login to CCMUser OR I am updating the user device assosciations it can take 30secs.

[kthorngr]

I am not terribly familiar with AD administration but this article indicates using the Active Directory Schema snap-in:

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/ServerHelp/8ac658b0-199c-47df-ac2b-ef9cb56fa7f0.mspx

Please refer to your AD documentation for the details. There apparently is some trade off between the attributes beign replicated and the performance of the AD.

You can see all of the attributes that CCM is requesting by looking at packet captures for a user search, CCMUser login and device association. To make it easier you might just want to try it and look at the packet captures to see if referrals are returned.

A google search for ldp.exe yields many results with this being one of them:

http://support.microsoft.com/?kbid=224543

You can look at a packet capture from the CCM for the particular search filters that are being used. You won't be able to simulate the CCMUser login or the device associations but you should be able to see how long it takes for a search result.

One other suggestion would be to look through the paket captures from the CCM for the CCMUser and device associations to see where the delay is and determine if the problem is with delays in the response or delays in CCM processing the response and sending out the next request packets.

Kevin

[lnthompson]

Thanks for your help. I'll keep looking into it. There is definatley a direct relation to the time it takes and referrals. I did a packet capture and it appears that a number of referrals are being sent out.

I'll look into the documents you mentioned and get back.

Thanks again.

[lnthompson]

After changing our CM to LDAP integrate on port 3268 to the global catalog server our queries are lightning fast!

However, when we try and update attributes on a user we get the following error:

Error

The following error occurred while trying to load the requested page.

Could not update user. 1

________________________________________________________

Error No: -1009

Error Description:

(Once we get this little problem working) I was also wondering if it is possible to do this same sort of LDAP query with no referals on the Cisco Conference Connection. The LDAP settings in the Applicaiton Administration page don't allow for 3268 as the LDAP port.

[cmartin]

Hi.

We had a similar trouble with users when we tried to integrate our CCM with LDAP. You should "use" the user in Cisco CallManager before attempting to change somethig, that is, change something in CCM user configuration (like CTI integration alowed check box, for instance). We discovered that CCM only creates Cisco user ID in LDAP attributes when the user is changed via CCM administration first time, and CCM needs this ID for managing the user.

Maybe this should solve tour problems. It worked for us!

Bye.

[lnthompson]

I appreciate the response. Unfortunately, I get the same error when I try to enable the CTI Allowed checkbox or Call-Park Retrieval checkbox or the device association.

I guess I am going to have to open a case on this one. I am pretty sure it has to do with LDAPing on the different port.

[cmartin]

Hi all again!

We are still looking for a solution.

We have built a new enviroment (not in production, of course) in order to reproduce the issue. One CallManager 4.1(3) in a MCS-7825 with 2 Gb of RAM and SO version 200.2.7a, IP 10.10.66.6, integrated with our corporate AD 2000, IP 10.10.68.2.

Of course, we have the same troubles. This time we have near 40 seconds of delay between the first request and the final response (great!, isnt't it?).

So we decided to show you the captures from ethereal and wait for any suggestions from you.

Any ideas?

Thak you very much.

[kthorngr · Accepted solution]

Took a look at the traces you supplied. I can see that the process takes around 40 seconds to complete. The traces are filtered so you won't be able to see everything that is happening. If you would like please make another capture with no filters. Please email me directly the trace file so I can open it in Ethereal. My email is kthorngr@cisco.com.

In the trace UserSearchAD.txt I a search request in packet 1 and a search response with a search reference in packet 2. Packet 3 is a TCP Ack from the CCM for packet 2.

After this there is a 42 second delay, then the CCM creates a TCP connection to the AD server to continue the search due to the search reference. It would be interesting to see what is happening in that 42 seconds. An unfiltered packet capure will show this.

Kevin

[cmartin]

Hello all.

First, I would like to thank Kevin his great efforts and his unestimated help.

We just solve our problem with this delay. And Kevin show us the track to the solution.

Kevin was right when, after look carefully the traces I sent to him, suggested us to look for the DNS entries and what servers was trying our CallManager to contact during the time that we thought CallManager did nothing: 10.10.68.7 and 10.10.64.117.

The problem was that our DNS server had these two IP addresses as first and second option for the LDAP IP search, so CallManager was trying to connect them before reaching the right server. Once we erased these entries, the delay disappeared and the performance was right.

Kevin, again we must thank you all your efforts and your key help in resolving our trouble.