"Error connecting to the local LDAP server" (OpenDS&TMS13.2.1)

[Martin Koch]

Did anyone else see this error, when accessing the (Legacy) Provisioning Directory in TMS 13.2.1.

It was upgraded from TMS13.1.2 earlier which worked like a charm. After the upgrade everything looked ok.

Error connecting to the local LDAP server The specified base DN cannot be found, please enter a valid base DN. Will try to reconnect in 5 seconds...

The "Verify that OpenDS is available." and index diagnostics on the local agent show: "Not completed within the timeout of 90 seconds."

If I restart (the started) OpenDS service it looked ok before, but it stopped working again afte some time

Some other things in addition:

OpenDS itself seems to answer, I can see in the problematic state that port 389 is open and I can connect with an ldap

browser to it, but it only shows the root. After restarting it also shows the dc=provisioning.

Last time the restart of opends showed this symptom in the provisioning directory:

[UNKNOWN MODEL/VERSION:] upfront all parameters.

After a complete TMS reboot it shows up fine again.

TMS is running on Win2003 with an express database on a virtual machine with 4gb ram, latest win updates

present and Java says its: build 1.6.0_33-b03

Any clue what could be the cause? If I see it happen again I guess I will open a TAC case for it.

Martin

[Magnus Ohm]

Hi Martin

I have seen this before

I mean it might not be the same thing that happened to you here. But at least in the case I saw this there was basically no content under the dc=provisioning. For some reason all the information was "unviewable". I could see the base dn in the opends control panel but it seemed totally corrupted. The affected had imported over 30 000 users and tried to restore the backup containing all these users to a clean opends install. Then this issue occurred. A reboot did not help in our case. But we where planning to upgrade to TMSPE anyway and that is working and everyone is happy

So unfortunately we did not come up with a root cause of this.

/Magnus

[Martin Koch]

Now this tms fails with an ajax error.

Our TMS 13.2.1 seems to run very unstable (13.1.x worked like a charm)

It is connected to a clustered and a non clustered VCS.

TMS is running Win2003 with the latest upgrades, VCS are all X7.1

The symptom we experience is now that the opends process on the TMS

Seems to die. I was not able to bring it up by restarting the OpenDS and TMSAGent services,

But a complete reboot of the TMS brings it back to live.

OpenDS currently runs for aprox a day and then the same failure occurs.

The Provisioning directory will first show just an empty outline of the frames and then later

“the Ajax request failed”

An TMS Agent Diagnostics already fails on the second test “Verify that all OpenDS database indexes are installed.”

With a 90 sec timeout.

Going to escalate this thread to TAC

[daleritc]

Hi Martin,

You can surely escalate to TAC but I'm fairly certain I know what their answer is going to be and that's "Upgrade/migrate to TMSPE"

However, I can tell you that there have been no changes between 13.1.x and 13.2.1 when it comes to Legacy TMS Agent. Therefore, what you may be experiencing is data corruption/instability within the OpenDS db...meaning what's changing here is the amount of data in the db and the state it's in. And my 'educated guess' here is that db is attempting to re-index itself which is probably consuming memory on the TMS box and probably why it's falling over. And obviously a reboot of the server clears this memory but then re-indexing starts all over again and chomps up the memory again. BTW,  The following was in white paper called "Why upgrade to TMSPE?":

Re-indexing of the OpenDS database on the replicating nodes, which could also cause memory stress on the

replicating nodes and in some cases require restarting OpenDS on some or all the replicating nodes.

rgds,

Dale

[Martin Koch]

Hi Dale!

I think the whitepaper is very one dimensional and does not reflect issues and disadvantages of the TMSPE.

One of the major issues is the lack to change or add user based configurations. TMSPE would force you

to do all of that on a group level, but this does not scale at all.

Especially for endpoints (ex90, ...) which you provision you might need user dependent settings.

Also the way how the different templates have to be configured is bad. You might end up having way to

many templates and they are not really easy to keep in sync (you cant for example auto inherit settings

or copy settings to multiple templates).

All over, great thought about removing opends. The current result is questionable and at least, I am far

away from happy.

[Justin Ferello]

Martin,

You do bring up some very valid points.  I must admit, the individual user configurations have so far prevented us from moving to PE in house.  We have people all over the US, with different time zones, different speed requirements etc.

Thanks,

Justin

[daleritc]

Hi Justin/Martin,

Removing single user configurations from TMSPE was a 'well discussed' decision within the engineering team during TMSPE development and the decision to remove it was simply because we didn't see many customer utilizing it...albeit from yourselves obviously...and those that had them (e.g. we had several large customer dbs in house) didn't know they had them, e.g. created by accident. In addition, we also felt that in a large environments, there was no easy way to properly manage or view them appropriately. For example, and in Legacy TMS Agent, there is no way in the UI to 'see' what users have single user configurations unless you 'drill down' to each and every user, i.e. not very convenient.

And to use another example, this is also somewhat like user rights in a Windows domain environment...meaning you can give single users access to a specific directory or file but MS recommends using 'groups' when setting securities within the file directory structure since its difficult/challenging to manage/view it per user vs. groups in AD. Therefore, and although in the Windows example you can still do individual user accesses, the TMSPE engineering team made the decision not to based on what I said above. Now this doesn't mean we can't do it again but other than yourself and Justin...who I would call 'uber-users' of Legacy TMS Agent...we haven't had anyone else formally or informally asking for it again since we released this past March. In fact, most customers are loving it and simply utlize groups for any special users, e.g. test users group.

rgds,

Dale

[Justin Ferello]

Dale,

Thank you for the explanation, as usual you have been very helpful.  With any new software change, there will always be growing pains.  We just need to bite the bullet and create a ton of templates   I think I was holding off to see if this would change, thinking that user defined configs would be missed by all, apparently I was wrong

Thanks,

Justin

[Martin Koch]

Hi Dale, again I can follow up the thoughts behind it but do not see that we are alone.

Further I say that the problem with the "accident options" is just shifted in the other direction.

For me a way to have solved that would be a table view of options/settings like you for example can

get with the old style system navigator. Then you could list for example all set up options, sort them by

for example "public in bandwidth" and see if its an inherited or custom set up value.

I got the suggestion from TAC to create a group for each user, ... no comment on that.

Just picture you would have to have 500 groups and 500 templates, how do you even think

of keeping trac of that? And just picture you have to change the sip server address for all.

To keep track of mismatched settings in between different templates will become even more

"accidental"

If you run a setup with no special needs TMSPE will work fine. If you need to fine tune something

you will hoplessly get lost.

[daleritc]

Hi Again Martin,

I never said nothing could be resolved And wasn't totally insinuating you guys were the only ones. I'm just stating the facts regarding the decision around single user configurations in TMSPE and this was based on feedback and as seen in databases recieved from some of our larger legacy TMS Agent customers. And your db may well have been one of them but you know what they say: Majority Rules

In any case, and again because I do consider you guys 'uber-users' of the provisioning solution, this 'change' was going to be a bit hard to swallow. However, I can't totally see the picture your creating, i.e. 500 groups with 500 templates. For example, a group will more than likely be associated with a particular VCS or VCS cluster...meaning it would be rare IMHO that the SIP Server Address (VCS or VCS cluster) would actually change, but if did, then you'd just change it for the entire group since this is where they would provisioning and registering, but yes, you'd need to keep track of which group is associated with which VCS or VCS cluster.

As far other settings, it was the opinion of engineering that these settings would normally 'effect' a group of people vs. one individual, e.g. bw limitations. And for testing purposes, a testing group could be created where one could simply move the user into so has to test settings, etc. For example, in TMSPE, we now have the ability to move manual created users from one group to another. For users imported via AD, then AD search rule changes would need to be made on that group so has to move those kind of users to the test group.

cheers,

Dale

[Martin Koch]

Just picture, you have the demand of:

* having users with different bandwidth settings (lets say 4 groups 512, 1 and 2 mbit and one for crappy dsl ones with 512/2048)

* have the demand for 4 time zones (then we are already have 16 device templates)

* then auto answer on or of (32)

* two different VCSe (64)

Before that I had one base setting on the root directory and then for groups where they needed something

different it was changed and some things which are a user and not really a gourp based setting

(like autoanswer) are simply set on the user level.

and so on, could do the same for jabbervideo as well or other example. you can not tell me that this will scale well,

and as its not possible to easy sync in between templates a signle change for things which are common on

all templates (lets say the phonebook uri) would require to access 64 templates and change it manually.

Like I said, I agree that some might got lost of what or even that settings existed on a user based level,

but what we have now generates the same issue (things getting out of sync and multidimensional) but just from the

other side.

Like I said, some list view of the settings would have been more useful, ...

To be honest, without proper user based settings I do not see a way to properly work with JabberVideo via TMS.

Instead of havong more options I seem to run into more and more limitations over time.

An other option would be for me to move more towards Lync

[Justin Ferello]

Martin,

These are excellent points. These are the same questions we have been asking ourselves.

Thanks,
Justin

Sent from Cisco Technical Support iPad App

[daleritc]

Yes, all good points Martin and excellent feedback But you can't tell me that having single user configurations scales any better Meaning it's not an easy challenge either way. But you could possibly reduce the number of templates by not starting at Root. For example, treat each group like an 'island' per VCS or VCS cluster...meaning you could create sub-groups under those groups for specific config settings per that group. And I agree that even doing it this way you can still get multiple templates under those groups and sub-groups, but it should or may be more managable. And again, and as I said earlier, I never said nothing couldn't be resolved

[kevin.chateau]

Hello,

We have the same problem with TMS 13.1.2 and a VCS Cluster;

it happened after a loss of connection with the VCS;

The java process OpenDS then exceeds 768Mo of memory defined in the environment variable OPENDS_JAVA_ARGS Xmx;

we have increased the value (*2)

[daleritc]

This will surely stop the OpenDS from timing out but will decrease perfomance on the TMS server since it's plausible it may need to constanly swap out the memory between physical memory and the paged file...meaning you need to have a big enough paged file to hold the memory. And obviously not something we would recommend but doable if your hardware can take it as explained, and I wouldn't go over anything than this increased value, i.e. *2.

And I would definitely still consider an upgrade/migration to TMSPE over this

rgds,

Dale