Cisco CUCM 10.5 - LDAP auth & CTI

[IngoJuergensmann]

This is a crosspost from https://communities.cisco.com/thread/50881 - I believe it is better suited to be discussed over here...

Maybe someone can confirm this:

When changing LDAP auth user search base DN in CUCM 10.5.2 this results in loss of CTI functionality in Cisco Jabber 10.6.x, when logging into Jabber after the change has been made.

Changing LDAP auth user search base DN back to its original value and restarting Cisco Jabber will make CTI work again.

More detailed information:

We are about to set up a new site office or more precise: a new subsidiary company. This subsidiary company will be located outside of the current search base in our AD.

Current user search base for LDAP authentication is:

OU=User,OU=company_DE,DC=company,DC=de

The new location would be:

DC=company,DC=de

After this change has been made, the user is still able to login in Cisco Jabber as well as in CUCM Self Care site. I take this as confirmation that authentication is working with that new LDAP auth user search base.

The problem is: when the users will sign into their Cisco Jabber client at next opportunity, they will experience that CTI will not work anymore. The users are listed as "Active LDAP Synchronized User".

The corresponding error in Cisco Jabber is:

Deskphone - Unhealthy

Status: Not connected

Protocol: CTI

Reason:Connection error. Ensure the server information in the Phone Services tab on the Options window is correct. Contact your system administrator for assistance.

When changing search base DN back to its original setting, CTI is working again when the user is logging in again.

This behaviour is reproducible and it doesn't matter whether you restart CTI Manager in Serviceability or not.

My undestanding, however, is: changing the user search base DN for LDAP auth to a higher entry should not affect CTI at all. The user DN should remain the same, which is confirmed by users being able to login in Cisco Jabber as well as in CUCM itself, for example the self care portal.

Is there a workaround possible? Or even a solution?

[Aaron Harrison]

Hi

It's been a year or so since I deployed Jabber in anger... however I recall that generally the CTI Manager service is a little flakey when it comes to LDAP. If the response from LDAP for auth does not come back quick enough, it times out.

The solution is to eliminate delays in auth. The simplest way to do this is to point LDAP-auth on CUCM at the GC (3269) rather than 389, obviously ensuring that you are pointing it at servers that are configured as GCs.

If you've done that already, set up your LDAP auth for non-SSL, then packet capture it and see where the delays happen. It can be a million things - from DNS delays, to 'ghost' DCs left in AD that weren't cleaned up, to lots of other stuff.

Aaron

[IngoJuergensmann]

Hi Aaron!

Thanks for answering! We are already using a non-SSL connection to our AD on port 389. I don't think that response times are an issue here, because there's not much outside of the original search base DN.

[Aaron Harrison]

Hi

I say 'non-SSL' as that allows you to troubleshoot delays with LDAP. It's OK to use SSL generally.

Use of 3269 avoids some lookups with 389, it's a common fix so that would be a good starting point to test.

And, of course... this is guesswork based on experience. If you want to know why it doesn't work, you look at the logs. Logs on the client, and CTIManager logs on CUCM.

Aaron

[IngoJuergensmann]

Hi Aaron!

a) I tried to change search base DN from OU=User,OU=company_DE,DC=company,DC=de to OU=company_DE,DC=company,DC=de which results in a still working CTI with Cisco Jabber. When shortening DN to DC=company,DC=de it stops working. 

b) I'm attaching SDL log and some output from PRT. 172.25.10.139 is the IP of the PC running Cisco Jabber. Hope this helps and someone can find something... 

(I've already opened a SR with our Gold Partner so that a TAC case can be opened this way...)

[Aaron Harrison]

Hi

In your log I see an attempt to connect to 389:

01397644.043 |21:37:11.341 |AppInfo  |Authenticating with SSL not enabled (ldap://ldap.aida.de:389) 

Followed by lots of rebinds every few seconds:

01397644.074 |21:37:11.647 |AppInfo  |cisco_ldap_rebind:enter
01397644.075 |21:37:11.647 |AppInfo  |cisco_ldap_rebind:calling simple bind.
01397644.076 |21:37:11.988 |AppInfo  |cisco_ldap_rebind:enter
01397644.077 |21:37:11.988 |AppInfo  |cisco_ldap_rebind:calling simple bind.

Then it stops

01397692.000 |21:37:21.095 |SdlSig   |CtiHandlerShutdownRequest              |shutting_down                  |CTIHandler(2,200,22,1147)        |CTIHandler(2,200,22,1147)        |2,200,13,1150.5^*^*                      |[R:H-H:0,N:0,L:1,V:0,Z:0,D:0] mShutdownReason=2 mSequenceNumber=0 UserPkid=
01397692.001 |21:37:21.095 |AppInfo  |CQBEBuilder::BuildQbeMessage(): objectID=132
01397692.002 |21:37:21.095 |AppInfo  |CTIHandler::OutputQbeMessage: TcpHand=[2:200:13:1150] QbePref={0x0xf1383e9c,0x1c} pQbeMsg=0x0xf1383ea4 qbeMsgSize=0x1c tmpLen=0x24 msgSize_=0x24
01397692.003 |21:37:21.095 |AppInfo  |[CTI-APP] [CTIHandler::OutputCtiMessage      ]     CTI   ProviderClosedEvent    (  reason=2)
01397692.004 |21:37:21.095 |AppInfo  |[CTI-INFO] [CTIHandler::QBE_ProviderCleanup]     sent ProviderSubscriptionUnRegNotify for user juergensmann.ingo
01397693.000 |21:37:21.095 |SdlSig   |CtiUnsubscribeRegUnRegNotificationInd  |ready                          |CTIDeviceRegManager(2,200,24,1)  |CTIHandler(2,200,22,1147)        |2,200,13,1150.5^*^*                      |[R:N-H:0,N:1,L:1,V:0,Z:0,D:0]  CTIHandlerPid=(2,200,22,1147) LoginUserId=juergensmann.ingo

Have you tried my suggestion of using 3269 yet?

[IngoJuergensmann]

Yes, today I've tested port 3269 as well. It could only connect with SSL checked on, but CTI didn't work as well.

Then I tried to connect to port 3268, without SSL, and CTI is working. According to our Windows sysadmins it shouldn't be possible to authenticate against a GC, but logging into Jabber as well as CUCM does still work.

The servers are still the same, but use a different port. Personally I see no reason why it shouldn't work with port 389 as well, giving a timeout instead. We have now a TAC SR open and I'll report back any findings from the TAC to here.

[Aaron Harrison]

Hi

Sounds like your AD folks need to go back to Microsoft school ;-)

Indeed, it was 3268 I meant to say but it has been some time since I've thought about this.

At any rate, it CAN work over 389, using 3268 just proves that the delay originates with your AD.

The difference between 389 and 3268 at a basic level is that the GC/3268 service contains a subset of AD attributes for EVERY object in the forest.

A normal DC contains EVERY attribute for every object in the domain.

Using 3268 means that for basic stuff the GC doesn't have to refer you to other DCs or domains.

The best way to troubleshoot this (and this is what Cisco will eventually ask, unless they just fob you off back to your MS guys with a 'it's AD timing out' conclusion) is to do a packet capture on your CUCMs for all DNS and LDAP (389/3268) traffic. 

That will show you CUCM resolving DCs, connecting to them, and you will see (in plain text) the referrals and so on. You'll probably see things like DNS responses timing out, or connection attempts to DCs that have long been removed but not properly cleaned up. Feel free to post or PM me the trace if you need help diagnosing it.

Regards

Aaron

Please rate helpful posts, and mark questions 'answered' when appropriate to highlight useful content.

[IngoJuergensmann]

Hi Aaron!

Also the change looked promising yesterday, this morning all users were unable to login. So I reversed the change back to its original setup.

We are investigating this now with a Cisco TAC engineer and I expect to have a Webex today on this topic... I'll keep you updated...

[IngoJuergensmann]

Ok, we had a Webex with the Cisco TAC engineer on Friday and he found out that the LDAP server is giving a login error when using the shorter LDAP auth search base DN. 

Why this is happening, is a different story, because when using a LDAP browser it is working. Maybe the LDAP user we are using in CUCM do not enough priviledges to access this higher level in LDAP subtree. We have to investigate this with our Windows admins next week. 

Other reasons that came to mind are: 

• CUCM LDAP auth search base needs to start with OU=...,
• CUCM LDAP auth search base cannot be shorter than the LDAP directory search base
• in both cases: CUCM does something different (comared to LDAP browser) in setting up the LDAP connection.

In any case: there's no quick fix or workaround - other than moving the subtree for our new office (and independent company) within our existing and working subtree. That's nothing completely and awfully bad, but it's somewhat contradicting the design goal to separate both companies as much as possible in our LDAP/Active Directory. I think we have to live with that, because the decision that needs to be taken (moving company inside others companys LDAP subtree (clean design) vs. working setup) will persist. 

Here is what the TAC engineer found out: 
---------------------------------------------------------------------------

The first difference in LoginCheckRequest in CTI are lines

03514824.073 |14:29:30.121 |AppInfo  |LDAP Search for User base: 'OU=User,OU=company_DE,DC=company,DC=de'
03531822.073 |14:36:43.462 |AppInfo  |LDAP Search for User base: 'DC=company,DC=de'

Then in working scenario there is
03514824.074 |14:29:30.124 |AppInfo  |LDAP Search complete. Code: 0
And after that there is process of getting control over Phone

Which is missing in nonworking. We are receiving different message from LDAP in nonworking

As response for
LDAP     229         searchRequest(3) "OU=User,OU=company_DE,DC=company,DC=de" wholeSubtree

Based on this that is LDAP, it is responding in different way.

[Bennett Yeung]

Hi IngoJuergensmann,

Any update. I have the same issue here.

[IngoJuergensmann]

No, no news yet. We have still the SR with TAC open, but there's no process here and I'm quite busy with a large project to keep this going together with our Gold Partner. 

We had to move the user group inside the AD anyway because there were other problems with this user group being outside of the existing company structure in the AD. These problems resulted in not being able to logon to the Wifi network and such. 

[Jakub Stroinski]

For me it was a certificate issue. The self-signed certificate with root wasn't good enough it seems for CUCM so we bought a SSL certificate for the machine providing AD and it worked right away... What a huge waste of time but everything is working beautifully now.

[IngoJuergensmann]

Well, SSL certs are a mess, really. Anyway, we are not using LDAPS but only the plain LDAP with no SSL certs involved. However I think for Windows this seems to work now, but on Mac we still have the issue that LDAP queries are not working properly. The TAC engineer was unavailable for some time and is expected to have returned yesterday, so maybe I get some updates the other days...

[Jakub Stroinski]

Any updates? Same issue here