jrm "Waiting to initialize" state in LMS 3.2

[csschaeffer]

About 1-5 times per week I will find the jrm process showing as "Administrator has shut down this server". I have written a script that 2/day checks the status of the jrm process and if it is down, restarts it (pdexec jrm). About 25% of the time the jrm process will then show a status of "waiting to initialize". The only way I have found to recover it to shut down CiscoWorks and restart it (takes about 25 minutes). Any idea why it would shutdown or how to get it restarted after the initializing state?

-Craig

[Martin Ermel]

on what platform is LMS running?

if it is windows, check if "Windows Remote Management" service is running (run "services.msc" to open the service console) - if it is running, disable it and restart dmgtd

if this does not help, please provide the follwing info:

what are the server spec and the license type of LMS?

are there any other applications running on the server?

is there anti-virus software installed? - if yes, make sure that it does not do an on-access scan on the CSCOpx directory (Installation directory of LMS)

was the hostname of the server changed recently?

please post the jrm.log and a pdshow

[csschaeffer]

The "Windows Remote Management" service is disabled. I finally got an exception implemented to exclude the CSCOpx directory from virus scans. I took a look at the jrm.log file, but it is 142MB in size. I have stopped jrm/renamed jrm.log/started jrm fresh and will take a look when the issue happens again. Server is a VMware virtual running on a vSphere VMWare server. Windows Server 2003 R2 SP2, shows 2 Intel Xeon 3GHz CPUs, 6.67 GB RAM (w/physical address extension).Not sure what specifically you are asking for re license type, but server/admin/licensing shows:

	Name	Version	Size	Status	Expiration Date
1.	CM	5.2	1500	Purchased	Never
2.	DFM	3.2	1500	Purchased	Never
3.	IPM	4.2	1500	Purchased	Never
4.	RME	4.3	1500	Purchased	Never
5.	VNM	1.0	Unlimited	Purchased	Never

The server seems to run at 100% CPU a lot of the time. Up until several months ago we were running w/3GB of memory, but it was upped to 7GB. It seems to use whatever memory we throw at it.The VMWare team is constantly sending me alerts from their console. Hopefully eliminating the virus scans will help. I will post the jrm.log file if the 'waiting to initialize' issue happens again.

[Martin Ermel]

I asked for the license size but should have better asked for the number of managed devices (or devices in DCR)  ... :-)

to get an idea of the server load (also the number of configured jobs could be of interest)

also check the windows event log for entries around the time the issue occurs. When jrm fails, do a "pdshow" in a DOS Box and check the "Stop" time for jrm; also save the complete pdshow in file and post it;

as the jrm.log is text file it can be easily compressed before posting it.

Ask the VMWare team if they have provided the CPU and RAM statically to the server - if it is dynamically this could be an issue. If the changes for the virus scanner did not solve the problem, configure fixed CPU and RAM for testing this point.

In any case, configure a fixed size of swap (page file) twice the RAM size.

Only do one step after the other and see if one influences the current behavior.

[csschaeffer]

  I just increased the paging file to 14GB (2x RAM) and rebooted. The jrm process is in a 'waiting to initialize state'. The VMWare team has confirmed that CPU and RAM are static, not dynamic. I have attached the jrm.log file and the output from a pdshow. Any insight to the issue would be appreciated

[Martin Ermel]

was the hostname of the server changed recently?

is LMS integrated with ACS?

- if not, reset you System Idendity User

    Common Services > Server > Security > System Identity Setup

what is the NIC configuration of the server? is it 1 NIC, or Teaming, or...?

a note to the server spec: with a 1500 license LMS should run on 2 CPU dual core - I am not sure if this is an issue here...

I think jrm must be up and running in 30 sec and it could be that the amount of scheduled jobs slows this down...

is this the complete jrm.log ?

[csschaeffer]

The hostname has not changed.

We are not integrated with ACS

The server has 1 NIC

We have a 1500 device license, but are currently managing about 250 routers and 750 switches.

The jrm log should be complete. I stopped CW, renamed jrm.log. We then increased the system paging file and rebooted. After jrm went into a waiting state, I submitted the current jrm.log file.

[csschaeffer]

After about 3 weeks of stability (after bumping swap file size, removing the CW dir from antivirus checks), my jrm waiting to initialize issue is back. It occurred after a reboot. I have stopped/started CW several times, rebooted but it never gets out of the waiting to initialize state. I have attached a jrm.log file. Any thoughts on what I can check or fix?