Re: VCS : directory servise database is not running

Oleksandr Yurchenko · ‎11-22-2011

Hi,

I received this warning message for VCS Control. (The directory servise database is not running :state – new : action -).

What does it mean?

I do not see other problems. What affects for this error. Can I manually start the service?

Thanks

Br. Oleksandr

awinter2 · ‎11-23-2011

Hi Oleksandr,

the directory service not running would affect provisioning.

The easiest way to resolve this issue would be to reboot the VCS, as that should clear this error.

If the error persists after the reboot, you probably want to open a TAC case to get this sorted out.

Regards

Andreas

Bryan Birchfield · ‎11-28-2011

We had this issue perpetually with x5 and it did not go away until I did a complete factory reset and upgrade to x6. I worked extensively with tac, but they had no answers.

Sent from Cisco Technical Support iPad App

Yorktel UK Support Team · ‎02-03-2012

Im having exactly the same problem. One day our Movi users could not login and found this error in VCS.

It was running X7.0.1 so we upgraded to the latest firmware X7.0.3 as suggested in this forum, but it has made no difference.

I have since downgraded it to X6 and back to X7, performed a xCommand DefaultValuesSet 3, removed it from TMS, purged it from TMS - but still the VCS reports the same error;

Directory service database failure The directory service database is not running Raised Warning Restart the system 2012-02-03 13:51:28 1 15016

In SSH if I login as root and run 'cluster' and run (1) Check the TMS agent status of this VCS, it returns:

.------------------------------------------------------------------------.

| Checking the state of TMS Agent |

'------------------------------------------------------------------------'

Could not connect to ###.###.###.###:4444. Check that the server is running and

that the provided credentials are valid.

Error details:

javax.naming.CommunicationException: Connection timed out [Root exception is

java.net.ConnectException: Connection timed out]

ERROR

I have now raised a TAC request, as I have no idea what is causing the issue?

Oleksandr Yurchenko · ‎02-03-2012

The reboot (not restart) VCS is fixed my problem. This message is occure after shutdown and power on VCS. Im obtain this message for VCS for X.5, X.6, and X7.

best regards

Oleksandr

cfiestas · ‎02-03-2012

IF rebooting the VCS doesnt fix or correct the issue, simply upgrade your VCS to the current version

For example if you have 6.1 vcs simply upgrade to 6.1

Enjoy

Cesar Fiestas

pdinapoli · ‎02-03-2012

Just back in December while on a customer install - my VCS Control lost its mind and I had this same exact scenario with the directory database.

In my case - I had an extremely cranky customer and an agressive timeline - so I immediately opened a TAC case and the engineer for my case had me up and running in less than 30 minutes:

This is what he recommended:

1/ SSH to the VCS and login as 'root'.

2/ execute; 'tmsagent_destroy_and_purge_data'

*** Note: The tmsagent_destroy_and_purge_data command may need to be ran two

or three times. If it fails simply execute the command again until it passes.

and the output:

-----------------------------------------------------------------------

-.

| Preparing to delete all TMS Agent data on this VCS

|

'-----------------------------------------------------------------------

-'

WARNING: Before running this procedure:

1. On the VCS, create a TMS Agent backup file.

2. If the VCS is clustered or replicates with TMS: on TMS,

go to the TMS Agent tab for this VCS and ensure "Enable TMS Data Replication" is disabled.

After running this procedure:

3. If "Enable TMS Data Replication" was disabled in step 2 above, then re-enable it.

WARNING: All TMS Agent Provisioning and FindMe data on this VCS will

be destroyed.

This procedure resets the TMS Agent LDAP and Replication passwords on

this VCS back to their factory defaults.

Have you performed steps 1 and 2, and are you sure you want to

proceed? (Y/N)y

Destroying and purging (this may take several minutes)

successfully acquired opends lock

Starting opends

Installing OpenDS

Removing old temporary files

Extracting files..

Installing directory server..

Applying configuration changes

Configuring opends after upgrade

Rebuilding indexes

Stopping provisioningd

Starting provisioningd

2011-12-16 01:24:29,189907933: Initializing provisioning

OK

~ # restart

Stopping warningled

Service vmtoolsd is not running

Stopping inactived

Stopping acpid

Service importcontrol is not running

Service winbindd is not running

Stopping hwstatus

Stopping apache2

Stopping crashreportd

Stopping clusterd

Stopping b2bua_launcher

Service telnetd is not running

Stopping phpsessionmonitor

Service pshelld is not running

Stopping proxy-registrationd

Service provisioningserver is not running

Stopping provisioningd

Service phonebookserver is not running

Stopping licensemanagerservermain.py

Stopping policyservermain.py

Stopping credentialmanagerservermain.py

Stopping opends

Stopping tandberg

Stopping managementframework

Stopping tmpstored

Stopping crashmonitord

Stopping clusterdb

Stopping clocksyncd

Stopping updatesw

Stopping logrotated

Stopping sysmonitor

Service snmpd is not running

Stopping sshd

/sbin/restart: line 24: 20483 Terminated $i stop

Stopping setkey

Stopping racoon

Stopping ntpd

Stopping LCDd

Stopping syslogd

Stopping dnsmasq

Stopping network interfaces

Stopping packagesd

Flushing firewall rules..complete

Stopping iptablesd

Stopping fail2ban

Shutdown successful

Stopping requestd

Stopping harddisks

Paul

Yorktel UK Support Team · ‎02-03-2012

I have tried that, but it does not resolve the issue. The problem looks to be a problem with the database itself (OpenDS) as when I try to restart the service it errors out (see the very bottom of the below log)

~ # tmsagent_destroy_and_purge_data

.------------------------------------------------------------------------.

| Preparing to delete all TMS Agent data on this VCS |

'------------------------------------------------------------------------'

WARNING: Before running this procedure:

1. On the VCS, create a TMS Agent backup file.

2. If the VCS is clustered or replicates with TMS: on TMS, go

to the TMS Agent tab for this VCS and ensure "Enable TMS

Data Replication" is disabled.

After running this procedure:

3. If "Enable TMS Data Replication" was disabled in step 2 above,

then re-enable it.

WARNING: All TMS Agent Provisioning and FindMe data on this VCS will be destroyed.

This procedure resets the TMS Agent LDAP and Replication passwords on

this VCS back to their factory defaults.

Have you performed steps 1 and 2, and are you sure you want to proceed? (Y/N)Y

Destroying and purging (this may take several minutes)

cat: /var/run/opendsd.pid: No such file or directory

kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

/tandberg/app/opends/uninstall: line 17: ${LOG_FILE}: ambiguous redirect

successfully acquired opends lock

Starting opends

Installing OpenDS

Removing old temporary files

Extracting files..

Installing directory server..

Applying configuration changes

Configuring opends after upgrade

Rebuilding indexes

~ # restart

Stopping warningled

Service vmtoolsd is not running

Stopping inactived

Stopping acpid

Service importcontrol is not running

Stopping winbindd

Stopping hwstatus

Stopping apache2

Stopping crashreportd

Stopping clusterd

Stopping b2bua_launcher

Service telnetd is not running

Stopping phpsessionmonitor

Service pshelld is not running

Stopping proxy-registrationd

Service provisioningserver is not running

Stopping provisioningd

Service phonebookserver is not running

Stopping licensemanagerservermain.py

Stopping policyservermain.py

Stopping credentialmanagerservermain.py

Stopping opends

ERROR: Cannot establish a connection to the Directory Server 127.0.0.1:4444.

Verify that the server is running and that the provided credentials are valid.

Details: Connect Error

Stopping tandberg

Stopping managementframework

Stopping tmpstored

Stopping crashmonitord

Stopping clusterdb

Stopping clocksyncd

Stopping updatesw

Stopping logrotated

Stopping sysmonitor

Stopping snmpd

Stopping sshd

/sbin/restart: line 24: 27134 Terminated $i stop

Stopping setkey

Stopping racoon

Stopping ntpd

Stopping LCDd

Stopping syslogd

Stopping dnsmasq

Stopping network interfaces

Stopping packagesd

Flushing firewall rules..complete

Stopping iptablesd

Stopping fail2ban

Shutdown successful

Stopping requestd

Stopping harddisks

/tmp: 18069m 18232m 24616m

Stopping sysctl

Stopping hostname

Stopping dbus

Starting dbus

dbus[27624]: nss_ldap: failed to bind to LDAP server ldap://dc1.firstconnections.local: Can't contact LDAP server

dbus[27624]: nss_ldap: could not search LDAP server - Server is unavailable

Starting hostname

Starting sysctl

Mounting harddisks

Setting up swapspace version 1, size = 10010128 kB

fsck 1.41.12 (17-May-2010)

e2fsck 1.41.12 (17-May-2010)

/dev/sdb2: clean, 1715/29310976 files, 3813855/58605120 blocks

Starting requestd

Starting fail2ban

2012-02-03 16:03:54,111 fail2ban.server : INFO Starting Fail2ban v0.8.4

2012-02-03 16:03:54,112 fail2ban.server : INFO Starting in daemon mode

Configuring iptablesd

Applying system generated IPv4 rules

Applying system generated IPv6 rules

Launching iptables configuration daemon

iptables initialization completed

Starting packagesd

Bringing up network interfaces

Waiting for network readyness...

Starting dnsmasq

Starting syslogd

Starting LCDd

Starting LCDd client

Starting ntpd

Starting racoon

Starting setkey

Starting sshd

Starting snmpd

Starting sysmonitor

Starting logrotated

Starting updatesw

Starting clocksyncd

Starting clusterdb

Starting alarms

Starting crashmonitord

Starting tmpstored

Starting managementframework

Starting tandberg

successfully acquired opends lock

Starting opends

OpenDS failed to start

Starting credentialmanagerservermain.py

Starting policyservermain.py

Starting licensemanagerservermain.py

Starting provisioningd

2012-02-03 16:05:34,713838011: Initializing provisioning

Starting proxy-registrationd

Starting pshelld

Starting crl_updater

Starting menumaker

Starting phpsessionmonitor

Starting b2bua_launcher

Starting clusterd

Starting crashreportd

Starting apache2

Starting hwstatus

Service winbindd already running. Not starting.

Fri Feb 3 16:05:39 GMT 2012 S81winbindd startup Failed!

Starting acpid

Starting inactived

Starting vmtoolsd

Starting warningled

~ # /etc/init.d/S77provisioning restart

Stopping provisioningd

Starting provisioningd

2012-02-03 16:08:41,322088467: Initializing provisioning

~ # /etc/init.d/S77provisioning restart

Restarting opends

ERROR: Cannot establish a connection to the Directory Server 127.0.0.1:4444.

Verify that the server is running and that the provided credentials are valid.

Details: Connect Error

~ #

Yorktel UK Support Team · ‎02-03-2012

Usually the following sequence also helps to resolve these types of issues;

1. SSH to the VCS Control, log in as root.
2. Execute the following command:

/etc/init.d/S77provisioning stop

3. Wait for 1 minute, then execute the following commands:

/etc/init.d/S76opends stop
/etc/init.d/S76opends start
/etc/init.d/S77provisioning start

4. Once the provisioning service has started successfully, go to the TMS server, go to systems navigator, open your VCS Control and go to the 'Clustering' tab.
5. Un-check the 'Enable provisioning' checkbox and press save. Wait for the process to finish.
6. Re-check the 'Enable provisioning' checkbox and press save. Wait for the process to finish, this might take a couple of minutes.

However in this case I get the following;

~ # /etc/init.d/S77provisioning stop

Stopping provisioningd

~ # /etc/init.d/S76opends stop

Stopping opends

[03/Feb/2012:16:20:41 +0000] category=JEB severity=NOTICE msgID=8847402 msg=The database backend userRoot containing 0 entries has started

ERROR: Cannot establish a connection to the Directory Server 127.0.0.1:4444.

Verify that the server is running and that the provided credentials are valid.

Details: Connect Error

successfully acquired opends lock

Starting opends

Service opends already running. Not starting.

~ #

Looks a bit more complicated than I have usually seen..... RMA?

Yorktel UK Support Team · ‎02-03-2012

If you try "telnet xxx.xxx.xxx.xxx 4444" you should also usually get a connection established to prove connection to the database.

However in my case, I dont get connection - reiterating that the OpenDS database is not running as it should be.

Not sure how to resolve this one, waiting to hear back from Cisco TAC ...

Rejohn Cuares · ‎02-03-2012

Hi First Connection Support Team,

Were you able to fix your problem? If so could you please post the fix? Thank you.

Please rate replies and mark question as "answered" if applicable.

David Anstee · ‎02-10-2012

No we were not able to fix it in the end. Spent over 6 hours on a WebEx session with Cisco TAC over 2 days, but unable to solve the issue.

In the end we RMA'd the Cisco VCS Control - and now its working again.