LMS 3.x: Alternative to Job Browser?

[yjdabear]

I set the RME Job Purge policies so none of the RME jobs are supposed to be kept after 180 days. For example, Credential Verification Jobs are set to be kept only for the last 30 days.

Credential Verification Jobs

Enabled

30

1195

Dec 17 2009 17:50:00

Daily

Yet, I just noticed numerous "crud" from years past, even after a "Purge Now" for anything past the last 180 days.

1195.499

CDAPurgeJob

Failed

Periodic

CDA Purge Job

At 17:50:00 EDT daily, startin g 08 Oct 2008

admin

Oct 08 2008, 17:50:00

Oct 08 2008, 17:50:17

This brings me to my multi-folded questions: Is there any alternative (say a CLI interface) to using the Job Browser GUI at http://cwk:1741/cwhp/jobBrowser.do which is more agonizing than watching paint dry when simply trying to check off a bunch of stale jobs for deletion manually? How else could I ensure effective purging of old LMS jobs, given the Job Purge policies do not seem to get enforced effectively?

[Joe Clarke]

No, there is no one CLI interface to list jobs like you see in the GUI job browser.

The best way to see if jobs are completing successfully is to look at the purge job log.  Chances are you may be seeing a memory issue if you have a lot of old jobs.  The log (or the daemons.log on Solaris) may indicate an OutOfMemoryError if this was the case.

[yjdabear]

I was just about to post that I found the following in my LMS notes:

cd NMSROOT/bin
cwjava -cw NMSROOT com.cisco.nm.cmf.jrm.jobcli

Is the above for NetConfig jobs only (I'm guessing)?

When I ran it, I got:

nms-(/opt/vgi/csco/CSCOpx/bin)->./cwjava -cw /opt/vgi/csco/CSCOpx com.cisco.nm.cmf.jrm.jobcli
ld.so.1: cwjava: fatal: relocation error: file /opt/vgi/csco/CSCOpx/lib/jre/lib/sparc/libzip.so: symbol JNU_GetStringPlatformChars: referenced symbol not found
Killed

As far as the job purges, here's what I see in crijobpurge.log:

[ Mon Dec 14  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,sun.misc.Launcher$AppClassLoader@a9c85c
[ Mon Dec 14  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,
[ Mon Dec 14  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ReportJobM
anager,purgeReportJob,962,List of jobs purgeReportJob ofcom.cisco.nm.rmeng.cri.R
eportJobManager []
[ Mon Dec 14  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ReportJobM
anager,purgeReportJob,963,No of jobs purgeReportJob ofcom.cisco.nm.rmeng.cri.Rep
ortJobManager 0
[ Mon Dec 14  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ReportJobM
anager,purgeReportJob,965,Empty job list returned from jrm
[ Tue Dec 15  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,sun.misc.Launcher$AppClassLoader@a9c85c
[ Tue Dec 15  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,
[ Tue Dec 15  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ReportJobM
anager,purgeReportJob,962,List of jobs purgeReportJob ofcom.cisco.nm.rmeng.cri.R
eportJobManager []
[ Tue Dec 15  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ReportJobM
anager,purgeReportJob,963,No of jobs purgeReportJob ofcom.cisco.nm.rmeng.cri.Rep
ortJobManager 0
[ Tue Dec 15  18:45:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ReportJobM
anager,purgeReportJob,965,Empty job list returned from jrm

And criarvpurge.log:

[ Mon Dec 14  18:50:01 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,sun.misc.Launcher$AppClassLoader@a9c85c
[ Mon Dec 14  18:50:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,
[ Mon Dec 14  18:50:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.inventory.reports.job.IRArchiveImpl with period(days) 30
[ Mon Dec 14  18:50:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.inventory.r
eports.job.IRArchiveImpl
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.sa.ui.reports.datagenerators.SyslogReportArchiveImpl with period(days) 30
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.sa.ui.repor
ts.datagenerators.SyslogReportArchiveImpl
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.bugtool.BTArchiveImpl with period(days) 30
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.bugtool.BTA
rchiveImpl
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.cctool.cconx.server.CCArchiveImpl with period(days) 30
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.cctool.ccon
x.server.CCArchiveImpl
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.cas.ui.reports.datagenerators.ChangeAuditReportArchiveImpl with period(days)
30
[ Mon Dec 14  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.cas.ui.repo
rts.datagenerators.ChangeAuditReportArchiveImpl
[ Tue Dec 15  18:50:01 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,sun.misc.Launcher$AppClassLoader@a9c85c
[ Tue Dec 15  18:50:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.common.VerifyR
MELicense,validate,?,
[ Tue Dec 15  18:50:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.inventory.reports.job.IRArchiveImpl with period(days) 30
[ Tue Dec 15  18:50:02 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.inventory.r
eports.job.IRArchiveImpl
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.sa.ui.reports.datagenerators.SyslogReportArchiveImpl with period(days) 30
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.sa.ui.repor
ts.datagenerators.SyslogReportArchiveImpl
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.bugtool.BTArchiveImpl with period(days) 30
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.bugtool.BTA
rchiveImpl
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.cctool.cconx.server.CCArchiveImpl with period(days) 30
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.cctool.ccon
x.server.CCArchiveImpl
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,146,calling purge archive api from class class com.cisco.nm.rme
ng.cas.ui.reports.datagenerators.ChangeAuditReportArchiveImpl with period(days)
30
[ Tue Dec 15  18:50:03 EST 2009 ],INFO ,[main],com.cisco.nm.rmeng.cri.ArchivePur
ger,purgeArchive,148,Purged successfully by class com.cisco.nm.rmeng.cas.ui.repo
rts.datagenerators.ChangeAuditReportArchiveImpl

The only OutOfMemoryError I find right now is this:

daemons.log:Using JAVA_OPTS:        -Xmx1024m -Xminf0.1 -Xmaxf0.1 -XX:MaxPermSize=256m -XX:PermSize=256m -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dlog4j.ignoreTCL=true -Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB -Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton -Djava.security.auth.login.config=/opt/CSCOpx/MDC/tomcat/conf/jaas.config

This probably isn't relevant. I don't seem to have any log entries in daemons*log during the Job Purges--17:55 - 18:55. I'll keep an eye on them.

[Joe Clarke]

The jobcli command does not allow you to list jobs.  But it looks like you may have some kind of shared object problem.  The JNU_GetStringPlatformChars symbol should be in NMSROOT/lib/jre/lib/sparc/libjava.so.  Perhaps you have one of the various LD_ envvars set so that this library is not properly resolved.

The log for the CDA purge will be in the directory pertaining to the CDA Job Purge ID.  They may not show any useful info though.  Last time I had to troubleshoot a purge case, I had to write some debugging code to figure out it was a memory problem.

[yjdabear]

Here's my seemingly awfully dated libzip.so. Makes me wonder if it's from LMS 3.x at all, or a residue from LMS 2.6.

-rwxr-x---   1 casuser  casusers   72736 Nov  9  2006 libzip.so

Could jobcli be used to delete say job "1195.499"?

[Joe Clarke]

The libzip.so on LMS 3.2 should be:

-rwxr-x---   1 casuser  casusers   72736 May 27  2008 libzip.so

[yjdabear]

Rats. How did that happen? I have lots of them. I'm on LMS 3.1:

ls -al /opt/vgi/csco/CSCOpx/lib/jre/lib/sparc
total 27526
drwxr-x---   8 casuser  casusers    2048 Oct 22  2008 .
drwxr-x---  16 casuser  casusers    1024 Oct 22  2008 ..
-rwxr-x---   1 casuser  casusers   20932 Nov  9  2006 awt_robot
drwxr-x---   3 casuser  casusers    1024 Nov 21  2008 client
-rwxr-x---   1 casuser  casusers    5284 Nov  9  2006 gtkhelper
drwxr-x---   2 casuser  casusers      96 Oct 22  2008 headless
-rwxr-x---   1 casuser  casusers    7528 Nov  9  2006 jexec
-rwxr-x---   1 casuser  casusers     689 Nov  9  2006 jvm.cfg
-rwxr-x---   1 casuser  casusers   52176 Nov  9  2006 libJdbcOdbc.so
-rwxr-x---   1 casuser  casusers  976632 Nov  9  2006 libawt.so
-rwxr-x---   1 casuser  casusers  356064 Nov  9  2006 libcmm.so
-rwxr-x---   1 casuser  casusers  172600 Nov  9  2006 libdcpr.so
-rwxr-x---   1 casuser  casusers   23840 Nov  9  2006 libdeploy.so
-rwxr-x---   1 casuser  casusers   14088 Nov  9  2006 libdt_socket.so
-rwxr-x---   1 casuser  casusers  435988 Nov  9  2006 libfontmanager.so
-rwxr-x---   1 casuser  casusers  259672 Nov  9  2006 libhprof.so
-rwxr-x---   1 casuser  casusers   73592 Nov  9  2006 libinstrument.so
-rwxr-x---   1 casuser  casusers   12580 Nov  9  2006 libioser12.so
-rwxr-x---   1 casuser  casusers   57528 Nov  9  2006 libj2pkcs11.so
-rwxr-x---   1 casuser  casusers    5336 Nov  9  2006 libjaas_unix.so
-rwxr-x---   1 casuser  casusers  154884 Nov  9  2006 libjava.so
-rwxr-x---   1 casuser  casusers   41860 Nov  9  2006 libjava_crw_demo.so
-rwxr-x---   1 casuser  casusers   67736 Nov  9  2006 libjavaplugin_jni.so
-rwxr-x---   1 casuser  casusers  235580 Nov  9  2006 libjavaplugin_nscp.so
-rwxr-x---   1 casuser  casusers    2216 Nov  9  2006 libjawt.so
-rwxr-x---   1 casuser  casusers    8768 Nov  9  2006 libjdgaSUNWafb.so
-rwxr-x---   1 casuser  casusers    8312 Nov  9  2006 libjdgaSUNWcg6.so
-rwxr-x---   1 casuser  casusers    8768 Nov  9  2006 libjdgaSUNWffb.so
-rwxr-x---   1 casuser  casusers    6440 Nov  9  2006 libjdgaSUNWm64.so
-rwxr-x---   1 casuser  casusers  285036 Nov  9  2006 libjdwp.so
-rwxr-x---   1 casuser  casusers  184352 Nov  9  2006 libjpeg.so
-rwxr-x---   1 casuser  casusers   11000 Nov  9  2006 libjsig.so
-rwxr-x---   1 casuser  casusers  288600 Nov  9  2006 libjsound.so
-rwxr-x---   1 casuser  casusers   15228 Nov  9  2006 libjsoundsolmidi.so
-rwxr-x---   1 casuser  casusers 6934636 Nov  9  2006 libjvm.so
-rwxr-x---   1 casuser  casusers   24348 Nov  9  2006 libmanagement.so
-rwxr-x---   1 casuser  casusers 1076440 Nov  9  2006 libmlib_image.so
-rwxr-x---   1 casuser  casusers 1862556 Nov  9  2006 libmlib_image_v.so
-rwxr-x---   1 casuser  casusers   69948 Nov  9  2006 libnet.so
-rwxr-x---   1 casuser  casusers   27144 Nov  9  2006 libnio.so
-rwxr-x---   1 casuser  casusers    1844 Nov  9  2006 librmi.so
-rwxr-x---   1 casuser  casusers   42772 Nov  9  2006 libsaproc.so
-rwxr-x---   1 casuser  casusers    7580 Nov  9  2006 libsunwjdga.so
-rwxr-x---   1 casuser  casusers   82932 Nov  9  2006 libunpack.so
-rwxr-x---   1 casuser  casusers   61596 Nov  9  2006 libverify.so
-rwxr-x---   1 casuser  casusers    7372 Nov  9  2006 libxinerama.so
-rwxr-x---   1 casuser  casusers   72736 Nov  9  2006 libzip.so
drwxr-x---   2 casuser  casusers      96 Oct 22  2008 motif21
drwxr-x---   2 casuser  casusers      96 Oct 22  2008 native_threads
drwxr-x---   3 casuser  casusers    1024 Oct 22  2008 server
drwxr-x---   2 casuser  casusers      96 Oct 22  2008 xawt

[Joe Clarke]

Should be:

/opt/CSCOpx/lib/jre/lib/sparc
lib/jre/lib/sparc 15:25 nms-cricket marcus %118> ls -al .
total 29699
drwxr-x---   8 casuser  casusers      51 Jun 25 21:39 ./
drwxr-x---  16 casuser  casusers      36 Sep 14 15:42 ../
-rwxr-x---   1 casuser  casusers   20932 May 27  2008 awt_robot*
drwxr-x---   3 casuser  casusers       8 Nov 19 15:13 client/
-rwxr-x---   1 casuser  casusers    5284 May 27  2008 gtkhelper*
drwxr-x---   2 casuser  casusers       3 Jun 25 21:39 headless/
-rwxr-x---   1 casuser  casusers    7276 May 27  2008 jexec*
-rwxr-x---   1 casuser  casusers     689 May 27  2008 jvm.cfg*
-rwxr-x---   1 casuser  casusers   52176 May 27  2008 libJdbcOdbc.so*
-rwxr-x---   1 casuser  casusers  976632 May 27  2008 libawt.so*
-rwxr-x---   1 casuser  casusers  355936 May 27  2008 libcmm.so*
-rwxr-x---   1 casuser  casusers  172600 May 27  2008 libdcpr.so*
-rwxr-x---   1 casuser  casusers   23840 May 27  2008 libdeploy.so*
-rwxr-x---   1 casuser  casusers   14088 May 27  2008 libdt_socket.so*
-rwxr-x---   1 casuser  casusers  503440 May 27  2008 libfontmanager.so*
-rwxr-x---   1 casuser  casusers  259672 May 27  2008 libhprof.so*
-rwxr-x---   1 casuser  casusers   73592 May 27  2008 libinstrument.so*
-rwxr-x---   1 casuser  casusers   12580 May 27  2008 libioser12.so*
-rwxr-x---   1 casuser  casusers   57516 May 27  2008 libj2pkcs11.so*
-rwxr-x---   1 casuser  casusers    5336 May 27  2008 libjaas_unix.so*
-rwxr-x---   1 casuser  casusers  154900 May 27  2008 libjava.so*
-rwxr-x---   1 casuser  casusers   41860 May 27  2008 libjava_crw_demo.so*
-rwxr-x---   1 casuser  casusers   67736 May 27  2008 libjavaplugin_jni.so*
-rwxr-x---   1 casuser  casusers  236316 May 27  2008 libjavaplugin_nscp.so*
-rwxr-x---   1 casuser  casusers    2216 May 27  2008 libjawt.so*
-rwxr-x---   1 casuser  casusers    8768 May 27  2008 libjdgaSUNWafb.so*
-rwxr-x---   1 casuser  casusers    8312 May 27  2008 libjdgaSUNWcg6.so*
-rwxr-x---   1 casuser  casusers    8768 May 27  2008 libjdgaSUNWffb.so*
-rwxr-x---   1 casuser  casusers    6440 May 27  2008 libjdgaSUNWm64.so*
-rwxr-x---   1 casuser  casusers  285036 May 27  2008 libjdwp.so*
-rwxr-x---   1 casuser  casusers  184352 May 27  2008 libjpeg.so*
-rwxr-x---   1 casuser  casusers   11000 May 27  2008 libjsig.so*
-rwxr-x---   1 casuser  casusers  288600 May 27  2008 libjsound.so*
-rwxr-x---   1 casuser  casusers   15228 May 27  2008 libjsoundsolmidi.so*
-rwxr-x---   1 casuser  casusers 6962384 May 27  2008 libjvm.so*
-rwxr-x---   1 casuser  casusers   24348 May 27  2008 libmanagement.so*
-rwxr-x---   1 casuser  casusers 1076440 May 27  2008 libmlib_image.so*
-rwxr-x---   1 casuser  casusers 1862556 May 27  2008 libmlib_image_v.so*
-rwxr-x---   1 casuser  casusers   70076 May 27  2008 libnet.so*
-rwxr-x---   1 casuser  casusers   27156 May 27  2008 libnio.so*
-rwxr-x---   1 casuser  casusers    1844 May 27  2008 librmi.so*
-rwxr-x---   1 casuser  casusers   42764 May 27  2008 libsaproc.so*
-rwxr-x---   1 casuser  casusers    7580 May 27  2008 libsunwjdga.so*
-rwxr-x---   1 casuser  casusers   82932 May 27  2008 libunpack.so*
-rwxr-x---   1 casuser  casusers   61596 May 27  2008 libverify.so*
-rwxr-x---   1 casuser  casusers    7372 May 27  2008 libxinerama.so*
-rwxr-x---   1 casuser  casusers   72736 May 27  2008 libzip.so*
drwxr-x---   2 casuser  casusers       3 Jun 25 21:39 motif21/
drwxr-x---   2 casuser  casusers       3 Jun 25 21:39 native_threads/
drwxr-x---   3 casuser  casusers       7 Jun 25 21:39 server/
drwxr-x---   2 casuser  casusers       3 Jun 25 21:39 xawt/

[yjdabear]

The /opt/CSCOpx/lib/jre/lib/sparc contents are consistent across all three of my LMS 3.1 servers. I'll upgrade one of them to LMS 3.2 when I get a chance, to see if the SOs get updated or not. The byte sizes of libzip.so of your LMS 3.2 and my LMS 3.1 are the same though, so there might be something else other than stale lib files.