on 12-30-2010 08:55 AM
This is a basic guide on the initial workflow one can follow to troubleshoot a process down situation focused mostly on Windows.
Basic Workflow
1. Check your Windows startup services:
CiscoWorks ANI database engine: Manual CiscoWorks Daemon Manager: Automatic CiscoWorks RME NG database engine: Manual CiscoWorks Tomcat Servlet Engine: Manual CiscoWorks Web Server: Manual CWCS Cmf database engine: Manual CWCS rsh/rcp service: Automatic CWCS syslog service: Automatic CWCS tftp service: Automatic DFM dfmEpm database engine: Manual DFM dfmFh database engine: Manual DFM dfmInv database engine: Manual IPM NG database engine - Manual
These should always remain this way and it is best to avoid stopping/starting any of these directly from the Windows GUI. To restart the LMS daemons always use:
net stop crmdmgtd
net start crmdmgtd
2. Make sure DEP is set to Windows Essential Programs only (right click My Computer > Advanced > Performance Settings > Data Execution Prevention) and your anti-virus has an exclusion for the LMS install directory and all its subfolders.
3. Make sure you have enough SWAP space assigned as per the LMS documentation. For most LMS Windows installations, 4GB is enough, but it can go higher. This is specially important in Windows 2008 since it will default to automatic SWAP managing.
4. Use the DOS pdshow command to determine exactly where the problem is. To understand the processes and what they do, you can download the following LMS Daemon Cheat Sheets.
Understanding the LMS processes
The first thing to check is the process dependencies. For example, if Tomcat is down, a multitude of other processes will be down since they are dependent on Tomcat. If this is the case, you want to focus on troubleshooting why Tomcat is down. Next step is to check the log files for the particular process and ports. To check the port availability one can use:
netstat -anob
This will produce such an output:
Active Connections
Proto Local Address Foreign Address State PID
TCP 127.0.0.1:9007 0.0.0.0:0 LISTENING 6060
[tomcat.exe]
TCP 127.0.0.1:9009 127.0.0.1:1768 ESTABLISHED 6060
[tomcat.exe]
If a different application other than Tomcat is listening to ports 9007 and 9009, this would be an indication of a problem. Port conflicts are usually caused due to a third party application occupying ports LMS needs or security software. One common case is CSCte49301, where a Windows patch causes a port conflict between RM and EssentialsDM.
Next step is to enable debugging if the process has a debug option and restart the daemons to collect the log files. It might be a good idea to clean the logs for easier viewing using:
logrot_trunc C:\Progra~1\CSCOpx\MDC\tomcat\logs\stdout.log
One can also restart only the process for troubleshooting purposes to get output on the logs:
pdterm tomcat
pdexec tomcat
Some processes also have a configuration file which one can check to see if there is any corruption or any changes that might be impacting.
Common Issues
System Error: The Daemon Manger cannot bind to port:42340
This could be an indication of CSCtd18854 which affects Windows 2008 installations.
If this is required for the daemons to start, it means something or someone is changing the casuser password in the SAM database or the security policies. This should be checked with your Windows admin.
NOTE: NMSROOT stands for your installation path. By default this would be C:\Program Files\CSCOpx.
Adding to Joels' guide for those needing to do the same from a soft appliance installation:
For Soft Appliance version you can start by checking the status from the basic cli. If you need to restart a specific process and don't want or need to restart the whole application ("application start LMS"), you need to drop into the OS shell and issue the pdexec <process name> command.
Here is an example (using pipe to include to abbreviate output and the pdexec command to show verbose process status at the OS cli):
SecLab-LMS/admin# show application status LMS | i Apache
Apache Running normally 5674
SecLab-LMS/admin# shell
Enter shell password :
Starting shell...
[SecLab-LMS/root-ade ~]# pdshow Apache
Process= Apache
State = Running normally
Pid = 5674
RC = 0
Signo = 0
Start = 04/23/13 08:03:20
Stop = Not applicable
Core = Not applicable
Info = Web Server initialized ok.
[SecLab-LMS/root-ade ~]#
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: