LMS 3.2 (Solaris) db/backup problem

[Andrey.Gulenko]

Hi team!
I have a LMS 3.2 on Solaris 10, and I have some problems after trying to install the last few DevicePackage:
1) several processes are not up, then I tried to restart DaemonManager from the console:
bash-3.00 # /etc/init.d/dmgtd stop
Daemon Management is not running.
INFO: Stopping DBEngine processes registered to Daemon Manager
WARNING: Daemon Manager could not stop DBEngine processes.
WARNING: Please run the command "/opt/CSCOpx/bin/dbstop.pl all"
WARNING: to stop DBEngine processes manually.
bash-3.00 # /opt/CSCOpx/bin/dbstop.pl all
Error retrieving dsns Can not open directory /opt/CSCOpx/backup/manifest;
bash-3.00 #
bash-3.00 # ps-ef | grep CSCOpx
casuser 28450 1 0 14:02:43? 3:04 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28587 1 0 14:05:51? 13:38 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28465 1 0 14:02:51? 1:27 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28298 1 0 14:01:59? 30:15 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
    root 605 21740 0 17:37:32 pts / 1 0:00 grep CSCOpx
casuser 28319 1 0 14:02:23? 0:29 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28439 1 0 14:02:35? 1:06 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28309 1 0 14:02:19? 29:07 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28361 1 0 14:02:31? 2:25 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
casuser 28480 1 0 14:02:59? 0:38 /opt/CSCOpx/objects/db/bin64/dbsrv10-x tcpip (HOST = localhost; DOBROADCAST = NO; Ser
bash-3.00 # kill -9 28450 28587 28465 28298 28319 28439 28309 28361 28480
bash-3.00 # ps-ef | grep CSCOpx
    root 607 21740 0 17:38:29 pts / 1 0:00 grep CSCOpx
bash-3.00 # /etc/init.d/dmgtd start
INFO: backed up /var/adm/CSCOpx/log/daemons.log to /var/adm/CSCOpx/log/daemonsbackup.log
Daemon Management 3.9 starting. Log = (syslog) /var/adm/CSCOpx/log/dmgtd.log
bash-3.00 #

2) SelfTest Information:

database.pl
WARN No Data Source Found, No Database Self Test is performed

3) Directory "manifest"  - not exist (/opt/CSCOpx/backup/manifest).

How to fix the database?

How to rebuild the directory "manifest" (/opt/CSCOpx/backup/manifest)?

Can some one help me?

[Nael Mohammad]

Does the /opt/CSCOpx/backup/manifest exists and is it readable?

What happens you attempt to manually start the backup from the CLI after stopping LMS and killing running processes using "/opt/CSCOpx/bin/perl /opt/CSCOpx/bin/backup.pl /opt/CSCOpx/backup /var/tmp/backup.log"?  If the backup fails, post the backup.log from the previous backup attempt.

If the above fails, try the following steps:

1. Stop the LMS Daemon Manager using "/etc/init.d/dmgtd stop".

2. Do a "ps -ef | grep CSCOpx" and kill any process that is running.

3. Go to /opt/CSCOpx/databases directory and locate the subdirectory for each database.

4. Remove the transaction log file from each db directory. (i.e. "mv /opt/CSCOpx/databases/cmf/cmf.log /var/tmp"). Do this repeatedly for each db directory.

5. Restart the services LMS Daemon Manager and post the dmgtd.log, daemons.log, and daemonsbackup.log if the above does not resolve the issue.

[Andrey.Gulenko]

Thanks Nael for the answer!

1) Directory "manifest" does not exist.

2) In the directory there is a several previos successful backup`s, but last backup - fail. (backup.log - in attachment).

3) Flowing issue has not solved a problem.

Files dmgtd.log, daemons.log, and daemonsbackup.log - in attachment.

In addition I have attach transaction log files from each db directory.

I expect your next helps actions.

[Nael Mohammad]

Looks like the manifest directory is in fact missing.  When attempting to create the backup, ensure its outside the /opt/CSCOpx directory to something like /opt/backup.  What syntax are you using to start the backup?

[Andrey.Gulenko]

Backup it is created in the catalogue/opt/CSCOpx/backup (other catalogue does not exist).

bash-3.00# pwd
/opt/CSCOpx/backup
bash-3.00#
bash-3.00# ls -la
total 6
drwxrwxr-x   3 casuser  casusers     512 Oct  4 10:03 .
drwxrwxr-x  33 casuser  casusers    1024 Oct  6 11:00 ..
drwxr-xr-x  11 casuser  casusers     512 Sep 10 02:32 13
bash-3.00#

Backup it was always created in an automatic mode at night, means CommonServices (look in an attachment a screenshot) are thus used.

Transaction logs DB`s to return in an initial place?

[Nael Mohammad]

The fact that the manifest folders are missing is the problem. You need to restore this directory from a last known good bakcup of the server or get a working copy of the manifest folder from another server that has the exact same version and applications and place it back in its orignal location. If you don't have a backup, you must uninstall LMS completely and reinstall from scratch.

Afterwards the bakcup should run fine. Why was the manifest folder removed?

[Andrey.Gulenko]

Thank you for your answer!

Ok.

There are 2 successful backups. Should I use are any special options/keys in recovery process (especially for folder "manifesto")?

Directory ("manifest") was mistakenly removed when i move the all exist backups to a new file system.

[Nael Mohammad]

no special options needed, just ensure the directory contains the orignal folder name. Glad to hear its working.

[Andrey.Gulenko]

I try to start the recovery procedure, the result is the following:

bash-3.00#
bash-3.00# ls -la  /opt/CSCOpx/backup/
total 8
drwxrwxr-x   4 casuser  casusers     512 Oct   8 11:41 .
drwxrwxr-x  33 casuser  casusers    1024 Oct  8 11:26  ..
drwxr-xr-x  11 casuser  casusers     512 Aug 27 02:33 11
drwxr-xr-x  11  casuser  casusers     512 Sep 10 02:32 13
bash-3.00#
bash-3.00#  /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl -d  /opt/CSCOpx/backup/11
******************************************************************************************

Restore started at : 2010/10/08 11:58:21
Please  see '/var/adm/CSCOpx/log/restorebackup.log' for status.

ERROR: Cannot find the generation in the backup  directory [/opt/CSCOpx/backup/11].
       Verify the backup directory and the  generation
bash-3.00#
bash-3.00# /opt/CSCOpx/bin/perl  /opt/CSCOpx/bin/restorebackup.pl -d  /opt/CSCOpx/backup/13
******************************************************************************************

Restore started at : 2010/10/08 11:58:25
Please  see '/var/adm/CSCOpx/log/restorebackup.log' for status.

ERROR: Cannot find the generation in the backup  directory [/opt/CSCOpx/backup/13].
       Verify the backup directory and the  generation
bash-3.00#
bash-3.00# cat  /var/adm/CSCOpx/log/restorebackup.log
************************************************************
Restore  from backup '/opt/CSCOpx/backup/11' started at: 2010/10/08 11:58:21

************************************************************
Restore  from backup '/opt/CSCOpx/backup/13' started at: 2010/10/08 11:58:25

bash-3.00#

Perhaps it was because of the lack of directory "manifest" in the backup?

I found a similar discussion here.

There are still some ways to restore it, or need to do reinstall?

[Nael Mohammad · Accepted solution]

If you don't define the generation to restore from, the restore scripts looks for the latest which is associated with "0".  Like the discussion stated, rename the directory to 0 or define the generation to use.

   To run the restore command, the command syntax is:

   /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl <-d BackupDirectory> [-gen GenerationNumber]  [-t TempDirectory] [-help]

   BackupDirectory : Directory where the backup archive is present

   GenerationNumber: Generation to be restored

   TempDirectory   : Temporary directory for the restore program

                     Default TempDirectory for this restore program:

                     /opt/CSCOpx/tempBackupData

                     Use -t TempDirectory to define your own temp directory

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   To see a list of the Backed Up generations available, use the following command line:

   /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl -h -d BackupDirectory

And then restore from the list of available backup generations.

[Andrey.Gulenko]

The result is negative:

bash-3.00#
bash-3.00# ls -la
total 10
drwxrwxr-x   5 casuser  casusers     512 Oct 11 10:12 .
drwxrwxr-x  33 casuser  casusers    1024 Oct 10 21:00 ..
drwxr-xr-x  11 casuser  casusers     512 Oct 11 10:19 0
drwxr-xr-x  11 casuser  casusers     512 Aug 27 02:33 11
drwxr-xr-x  11 casuser  casusers     512 Sep 10 02:32 13
bash-3.00#  /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl -h -d /opt/CSCOpx/backup/

   Restore Program Help

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   To run the restore command, the command syntax is:

   /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl <-d BackupDirectory> [-gen GenerationNumber]  [-t TempDirectory] [-help]

   BackupDirectory : Directory where the backup archive is present

   GenerationNumber: Generation to be restored

   TempDirectory   : Temporary directory for the restore program

                     Default TempDirectory for this restore program:
                     /opt/CSCOpx/tempBackupData

                     Use -t TempDirectory to define your own temp directory

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   Number of B/U generations kept are set on the 'Schedule Backup' HTML page.
   The most recent scheduled B/U generation will be that with the highest number.
   To restore a particular generation, use the command line option '-gen #'
   where # is a generation number from the list below.

   Generation 0 holds the most recent B/U executed from the 'Backup Data Now' page.
   To restore generation 0, use the '-gen now' parameter.
bash-3.00#
bash-3.00# /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl -d /opt/CSCOpx/backup/ -gen 11
******************************************************************************************

Restore started at : 2010/10/11 10:38:57
Please see '/var/adm/CSCOpx/log/restorebackup.log' for status.

ERROR: Cannot find the generation in the backup directory [/opt/CSCOpx/backup/].
       Verify the backup directory and the generation
bash-3.00# /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl -d /opt/CSCOpx/backup/ -gen now
******************************************************************************************

Restore started at : 2010/10/11 10:39:02
Please see '/var/adm/CSCOpx/log/restorebackup.log' for status.

ERROR: Cannot find the generation in the backup directory [/opt/CSCOpx/backup/].
       Verify the backup directory and the generation
bash-3.00# /opt/CSCOpx/bin/perl /opt/CSCOpx/bin/restorebackup.pl -d /opt/CSCOpx/backup/ -gen 13
******************************************************************************************

Restore started at : 2010/10/11 10:39:10
Please see '/var/adm/CSCOpx/log/restorebackup.log' for status.

ERROR: Cannot find the generation in the backup directory [/opt/CSCOpx/backup/].
       Verify the backup directory and the generation
bash-3.00#

I found directory "manifest" on the old server, it has been installed LMS 3.0 - can I use it? (current version of LMS 3.2)

There are still some methods or need to do to reinstall?

[Nael Mohammad]

No it can't be used, it must be the same version exactly. Meaning same LMS 3.2 installation, same applications installed, same application versions. If the previos restore is from LMS 3.0 and you are trying to run this on LMS 3.2, that will not work. Your only recourse is to restore from a last known good backup that  is identical to your existing server or re-install.

Post the '/var/adm/CSCOpx/log/restorebackup.log' but in the interest of time, if you deleted the previous directory and restored from a different version, just re-install LMS.

[Andrey.Gulenko]

Ok. Clearly ...
I try to recover from the directory "manifest" (old version LMS 3.0) - restore was successful, but some applications LMS 3.2 does not work correctly.

Currently, I see only one way to restore - reinstalling.

p.s. I do not understand why I can not restore the current version of LMS 3.2 of the last successful backups.

p.p.s. This particular mechanism of backup? (thus, no directory "manifest" can not restore the application, even from a successful backup?)

[Nael Mohammad · Accepted solution]

As stated previously, you can't use the LMS 3.0 manifest directory on a LMS 3.2 system then run a backup and try to do a restore from the successful backup of LMS 3.2.  Also when doing a restore, you need to ensure that you have the same applications installed on the new server.

The backup and restore works across any version assuming the directory structure of each respective system has not been altered in anyway and if it has, reinstall the LMS application.