Solved: LMS 3.2 and Apache

sbrox · ‎10-28-2010

Hi

I have installed LMS 3.2 on a 64 bit Windows Server 2003 R2 Standard edition with Service pack 2

The main problem is that the apache service will not start.
It seems like all the other ciscoworks services starts, but not the apache services.
How do I solve this problem?

Martin Ermel · ‎11-04-2010

according to your netstat output, it seems that Tomcat (TCP/9009) takes more than 20 min to start. If that is the case, this is too long for TomcatMonitor to wait for and thus it will not be started -causing other dependent processes (like Apache) to fail also.

I guess your virus scanner is the culprit of this issue. Just for a short test disable it and try to stop and start LMS (in a DOS box: net stop crmdmgtdand when it has finished, net start crmdmgtd)

If LMS is working now, you should configure the virus scanner to not touch the CSCOpx directory (at least for online scans).

edit:

Joel, I dropped in and didn't read the whole thread in detail, sorry - but you clearly put the finger on this before and I agree with you that this is the most likely cause

View solution in original post

Joel Monge · ‎11-04-2010

Yes, Apache depends on Tomcat. As I mentioned before, AntiVirus/security software is the most likely culprit here. Since you cannot disable it due to security reasons, you need to setup an exclusion in McAfee for on-access scanning of the CSCOpx folder and all its subfolders.

In fact, I was just about to update this thread since just finished working with another customer with same issues and McAfee was at fault. After adding the exclusion daemons came back up fine. You need to make sure this exclusion still exists just before you restart daemons since it can get removed without noticing. You also need to make sure the exclusion remains throughout a reboot so the problem does not happen again after reboot.

This is documented here for your reference:

https://supportforums.cisco.com/docs/DOC-9132

Mermel:

No worries, your help and anyone elses is always appreciated and we both came to same conclusion.

View solution in original post

Joel Monge · ‎10-28-2010

In DOS, do:

pdshow > c:\pdshow.txt

Post this file. Go to Start and click on 'Run...'. Type in "services.msc" and post the startup type of the CiscoWorks related processes. This has happened since install or was it working before? Did you change the hostname at any point?

sbrox · ‎10-29-2010

Hi Joel,

This is new installation and it has been problems with apache from day one.

We have done some changes with the hostname but the cw2000 application has been uninstalled and we have done a new installation

Joel Monge · ‎10-29-2010

Both TomcatMonitor and Apache are down which will cause many other processes to go down with them. Try this:

1. Shutdown daemons: net stop crmdmgtd.

2. In DOS go to CSCOpx\bin path and run:

perl hostnamechange.pl -ohost  -nhost

3. Enter the correct hostname as seen in a "hostname" output on DOS.

4. Restart daemons: net start crmdmgtd. Go to CSCOpx\objects\dmgt\ready, once the daemon lock is gone, see if Apache and TomcatMonitor are now up.

sbrox · ‎11-01-2010

Hi,

I have some problems with the "hostnamechange.pl" command.

Is it possible to list out the oldhostname?

Joel Monge · ‎11-01-2010

You could try looking at the oldest LMS installation log (Ciscoworks_install_xxx) under your C drive, ie:

COMPUTERNAME=LMS-26-WIN

sbrox · ‎11-02-2010

I have checked the installation log and the only differense I found was that the old hostname was in upercase and the new hostname from the DOS command "hostname" was in lowercase and I then ran the command "hostnamechange.pl" but apache still does't start.

Is it a tomcat problem or some sort of conflict with other running services?

Any other suggestion please?

Joel Monge · ‎11-02-2010

Post these:

- CSCOpx\log\TomcatMonitor.log

- CSCOpx\MDC\Apache\log\error_log

- CSCOpx\MDC\Apache\log\mod_jk.log

- CSCOpx\MDC\etc\regdaemon.xml

- Output of "hostname" from DOS

sbrox · ‎11-02-2010

I have deleted the logfiles and rebooted the server so that all the logfiles should be new and fresh. None of the Apache logfiles was created after the restart. Hope you will find the necessary info in the posted files.

Joel Monge · ‎11-02-2010

Create dummy D:\Program Files\CSCOpx\MDC\Apache\log\error_log and D:\Program Files\CSCOpx\MDC\Apache\log\mod_jk.log files and then restart daemons. Then they should contain some information we can use. Since the daemons will be down, lets take this chance to try all this:

1. Stop daemons:

net stop crmdmgtd

2. Remove all server.* under D:\Program Files\CSCOpx\MDC\Apache\conf\ssl.
3. CD to D:\Program Files\MDC\Apache in DOS and issue:

perl NMSROOT\MDC\Apache\ConfigSSL.pl -disable
perl NMSROOT\MDC\Apache\ConfigSSL.pl -enable

4. Fill out the information.

5. Issue "netstat -anob > C:\netstat.txt" and post this file.

6. CD to D:\Program Files\CSCOpx\MDC\Apache and issue:

apache.exe -t -d D:\Progra~1\CSCOpx\MDC\Apache

Post output.

7. Restart daemons:

net start crmdmgtd

If still no go, post:

- CSCOpx\MDC\tomcat\logs\stdout.log

- CSCOpx\MDC\tomcat\logs\stderr.log

- CSCOpx\MDC\Apache\log\error_log

- CSCOpx\MDC\Apache\log\mod_jk.log

- Any Event Viewer related errros

- Output of pdreg -l Apache

sbrox · ‎11-03-2010

I have done task 1 to 7 and the everything worked but when I restarted the server the same problem was back with apache not running.
It seems for me that if the server has been up and running for a few hours and I stop and start the deamons and I have a lucky day, everything work ok.
The Deamon Manager will not start Apache if the system is recently booted

---Apache\log\error_log is still empty but I hope you will find some interesting things in the other log files

Joel Monge · ‎11-03-2010

If I understood correctly, when you start daemons it works, but if you reboot problem comes back. If that is so, it would seem some other application is causing a conflict with LMS. Turn off any AntiVirus/security software and make sure DEP is set to Windows essential programs only.

You forgot the "netsat ouput" which would show what is using which ports. Can you please confirm that if you do "net stop crmdmgtd - net start crmdmgtd" Apache comes up, and then if you reboot it does not?

What other applications are installed on this server? Also post the CSCOpx\lib\classpath\md.properties.

sbrox · ‎11-03-2010

I have changed the DEP to "Windows essential program only"

And I can confirm that if I wait a few hours and do a net stop crmdmgtd - net start crmdmgtd Apache comes up if I'm lucky.

Our wintel team has installed some tools they need for management of the server and nothing else is installed as far as I know. The server is running McAfee antivirus

Joel Monge · ‎11-03-2010

Are those the only errors you see in the Event Viewer? Do not seem to be related to this problem. Post the CSCOpx\MDC\Apache\conf\httpd.conf and a screenshot of all installed Apps along with all contents under CSCOpx\MDC\Apache\logs. Are you able to temporarily disable AntiVirus and security software and reboot to test?

sbrox · ‎11-04-2010

Yes those are the only errors I can see in the Event Viewer

Because of politcal reasons it is dificult for me to disable the AntiVirus

I have also posted the output from the netstat command 10 , 20 and 40 minuttes after a reboot. Hope you can see something interesting in those files.