12-15-2009 05:27 PM
Hi,
We have recently starting getting this error message appearing on one of our Windows CiscoWorks servers. This server runs:
It is also the Master server in our environment.
The message started appearing after our OS Support team installed a round of MS Patches. I am wondering if this has started because they did not shutdown the CiscoWorks services properly first, using the net stop crmdmgtd command.
They have since removed the patches but the problem still exists.
Looking through the forums, I have noticed that other people have had similar problems in the past, though with slightly different scenario. I think I have checked all the possible solutions given, but still no joy.
The hostname has been changed since installation, but that was a couple of years ago - not recently. I have rechecked all the files that were involved in the rename and they still have the new name in them.
D:\CSCOpx\bin>pdshow CTMJrmServer
Process= CTMJrmServer
State = Administrator has shut down this server
Pid = 0
RC = 0
Signo = 0
Start = 16/12/2009 12:29:01 p.m.
Stop = 16/12/2009 12:31:01 p.m.
Core = Not applicable
Info = Server started by admin request
The CTMJrmServer.lof file contains:
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Dec 16 12:24:51 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Dec 16 12:24:52 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Dec 16 12:25:30 NZDT 2009 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Dec 16 12:25:30 NZDT 2009 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,<init>,66,Null JRM. Exiting
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Dec 16 12:27:00 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Dec 16 12:27:01 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Dec 16 12:27:32 NZDT 2009 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Dec 16 12:27:32 NZDT 2009 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,<init>,66,Null JRM. Exiting
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Dec 16 12:29:01 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Dec 16 12:29:03 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Dec 16 12:29:33 NZDT 2009 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Dec 16 12:29:33 NZDT 2009 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,<init>,66,Null JRM. Exiting
Needless to say that restarting the services, and the server have not had any success.
Anything else I can try?
Regards
Jeff
Solved! Go to Solution.
01-23-2010 09:30 AM
Yes, do the master first, then update the slaves with the master's new hostname. Don't forget to reaccept the SSL certificates on the slaves.
12-15-2009 05:40 PM
Post the jrm.log, the output of the hostname command from the DOS prompt, and the output of "pdreg -l CTMJrmServer".
12-15-2009 07:26 PM
Thanks for your response.
The output of the pdreg -l CTMJrmServer is:
D:\CSCOpx\bin>pdreg -l CTMJrmServer
Process = CTMJrmServer
Path = D:\CSCOpx\bin\cwjava.exe
Flags = -cw D:\CSCOpx -cw:jre lib/jre -cp:p MDC/tomcat/webapps/rme/WEB-INF/classes,MDC/tomcat/webapps/rme/WEB-INF/lib/ctm.jar,MDC/tomcat/webapps/rme/WEB-INF/lib/log4j.jar com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer geb
Startup = Started automatically at boot.
Dependencies = RMEDbMonitor jrm TomcatMonitor RMECSTMServer
The output of the hostname command is:
D:\CSCOpx\bin>hostname
hp1813
The Jrm.log file is attached.
12-15-2009 08:39 PM
CTMJrmServer still thinks that the server's hostname is geb. Was this the hostname that was changed a while back, or was the server's name changed again?
12-15-2009 10:36 PM
The server was originally built as HP1813 by the OS Team.
For various reasons, our team didnt want to call the device that, they wanted to call it GEB. This has caused more problems than it has solved as you can imagine.
So it was renamed after installation to GEB. Now by renaming, I probably mean aliasing. The computer name is still at HP1813 - so that the OS Team can manage and maintain it. We have DNS entries that refer to GEB and point to HP1813. The SSL Certificate is in the name of GEB.
There have been no changes regarding the naming of this server since installation a couple of years ago (that I know of ).
Should I go through the process that I followed for the renaming back then, again?
Regards
Jeff
12-15-2009 11:58 PM
LMS is very particular about the hostname. If you have changed the LMS files to point to geb, then I highly recommend you change the server's actual hostname to geb. After you do this, reboot, then see if CTMJrmServer comes up.
12-16-2009 01:05 PM
I will see if we can get this organised - if only for a test, and not permanently.
Thanks
Jeff
12-17-2009 01:44 PM
We have renamed the server, and it appears to be working ok.
So, given that we have to rename the server back to its other name, we will have to work on changing the name ion CiscoWorks from GEB back to its other name.
I presume that I can follow the process I used to rename CiscoWorks originally, but in reverse. I will also have to generate some more SSL certificates in the new name.
Are there any issues that you can think of in doing this?
Regards
Jeff
12-17-2009 06:02 PM
You will need to manually update the registry for CTMJrmServer to change geb to the actual hostname. This is not properly documented in the hostname change guide. But other than that, you shouldn't run into any issues.
01-19-2010 04:49 PM
Thanks for your reply. The holiday break is over, and it is back to work!
I will start the rename process off next week. Just two further questions....
Since we have 3 servers, with GEB the master one, should I shut down the others while doing the rename on GEB first, or will it not really matter?
I presume it would be better to do GEB before doing the other two?
Regards
Jeff
01-23-2010 09:30 AM
Yes, do the master first, then update the slaves with the master's new hostname. Don't forget to reaccept the SSL certificates on the slaves.
01-26-2010 05:38 PM
I have organised for the server to be renamed back to its original name.
I have gone through the steps to remove any references to GEB in the CiscoWorks application on the master.
It took me a couple of goes to get the new certificate to be recognised, but managed that. Also had to disable, then reenable the SSO master role to get that bit working.
Unfortunately, I am still getting the orginal error again. The error I get now for the CTMJrmServer is :
H:\>pdshow CTMJrmServer
Process= CTMJrmServer
State = Administrator has shut down this server
Pid = 0
RC = 0
Signo = 0
Start = 27/01/2010 2:28:42 p.m.
Stop = 27/01/2010 2:30:43 p.m.
Core = Not applicable
Info = Server started by admin request
H:\>
Below are the last entries in the CRMJrmServer.log file:
org.omg.CORBA.BAD_PARAM: Invalid GIOP Proxy ior: org.omg.CORBA.INV_OBJREF: Invalid object ref returned by URL resolver: exception com.inprise.vbroker.URLNaming.ReqFailure {
java.lang.String reason="java.io.FileNotFoundException: D:\CSCOpx\www\classpath\gatekeeper.ior (The system cannot find the file specified)"
} minor code: 0 completed: No minor code: 0 completed: No
at com.inprise.vbroker.firewall.FirewallLifeCycleInterceptor.getMechanismValue(FirewallLifeCycleInterceptor.java:139)
at com.inprise.vbroker.firewall.FirewallLifeCycleInterceptor.getMechanismValues(FirewallLifeCycleInterceptor.java:106)
at com.inprise.vbroker.firewall.FirewallLifeCycleInterceptor.create(FirewallLifeCycleInterceptor.java:59)
at com.inprise.vbroker.interceptor.ChainPOALifeCycleInterceptor.create(ChainPOALifeCycleInterceptor.java:22)
at com.inprise.vbroker.poa.POAImpl.
at com.inprise.vbroker.poa.POAImpl.create_POA(POAImpl.java:727)
at com.cisco.nm.util.OrbUtils.createChildPOA(OrbUtils.java:486)
at com.cisco.nm.util.OrbUtils.createPersistentPOA(OrbUtils.java:477)
at com.cisco.nm.util.OrbUtils.createDefaultPOAs(OrbUtils.java:401)
at com.cisco.nm.util.OrbUtils.initMyORB(OrbUtils.java:381)
at com.cisco.nm.util.OrbUtils.initORB(OrbUtils.java:354)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.getJRM(JobManagerExt.java:95)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.main(CTMJobManagerServer.java:34)
java.lang.Exception: com.cisco.nm.util.OrbUtils::Unable to initialize default POAs...
at com.cisco.nm.util.OrbUtils.initMyORB(OrbUtils.java:385)
at com.cisco.nm.util.OrbUtils.initORB(OrbUtils.java:354)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.getJRM(JobManagerExt.java:95)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.main(CTMJobManagerServer.java:34)
[ Wed Jan 27 13:05:46 NZDT 2010 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Jan 27 14:28:43 NZDT 2010 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Jan 27 14:28:43 NZDT 2010 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Jan 27 14:30:09 NZDT 2010 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Jan 27 14:30:09 NZDT 2010 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,
I dont understand the missing gatekeeper.ior file, as it does exist. Although as part of the steps I was following removed this file, it might have been created after this attempt to run.
The second entry in the log was when I manually tried to start the CRMJrmServer process.
Must be something else to change, but not sure what. Any further ideas welcome.
Also noticed that some of the other processes have not started.... Ive attached the output of pdshow.
Regards
Jeff
01-26-2010 10:51 PM
Well cant say I know exactly what I did, but the master server started respoding and behaving properly!!
I did notice that the jrm server was initialising for a while, which prevented other services from running, but that eventually worked. I did another restart and everything started working. Hopefully this will continue beyond the next restart.
I did notice while reading through the forums about jrm initializing, that this could be caused by anti virus products being able to scan the CSCOpx directory.
In our case this does happen, so I have requested that this directory, and all subdirectories, be excluded from any scanning. Hopefully that will help with the performance.
Once the master server was done, the others where pretty easy. I will check everything is still behaving itself tomorrow morning.
Thanks for all your help Joe. Much appreciated.
Regards
Jeff
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide