CiscoWorks "Server Not Ready" Admin Web Page

[rzurawski1153]

When trying to login to Admin webpage we received a page stating "Server Not Ready"

Please wait..... System is still coming up. You will be redirected to login page soon

with an OK button. Clicking button reloads page and seems to have no effect.

Looking at Windows event viewer I see no errors/warnings.

Looking at log files contained within Ciscoworks folders I can see the syslog file growing and a JAVA error in the SMTP.log

and error in TomcatMonitor.log of TomcatMonitor: non-handled Msg from dmgtd:1111708500

I have download the CiscoWorks Troubleshooting PDF with no success or related steps to troubleshoot.

Any suggestions for troublshooting?

[yjdabear]

The message itself is not indicative of any "trouble". In fact it's expected, especially if you haven't waited "long enough" (5-30 mins) depending on how big your server is, hardware spec-wise. On Solaris, one can simply watch for the disappearance of "dmgt_start_lock" in /opt/CSCOpx/objects/dmgt/ready as the sure sign that LMS has become "ready". You could do a Find on Windows in the location where CiscoWorks LMS is installed for such a "ready" folder and wait for the disappearance of "dmgt_start_lock", before attempting to log in to the web GUI. Other files that may make an appearance there are "EDS-GCF" and "NameServiceMonitor", AFAICC.

[rzurawski1153]

Thanks for the feedback. That message has

been there for a few days now. Today I left the browser window

open for close to two hours.... message has not gone away.

Going back to what happened last. For some reason, the syslog.log file stopped cleaning itself up and deleting old entries. The hard disk ran out of space. The syslog was zipped, a new one created, windows server rebooted and haven't been able to login to admin page since.

[yjdabear]

Ah well, that's real trouble then. Can you post the following?

1. Full output of "c:\progra~1\CSCOpx\bin\pdshow" or wherever "pdshow" is.

2. Versions of your CWK LMS components. Since the web GUI is inaccessible, another way is to post the following files:

NMSROOT\setup\rme.info, cm.info, cmf.info, cdone.info, dfm.info, cvw1.info, nmim.info, ipm.info, etc.

3. The following log files:

NMSROOT\MDC\tomcat\logs\stdout.log
NMSROOT\MDC\tomcat\logs\stderr.log
NMSROOT\MDC\Apache\logs\error.log
NMSROOT\NameServiceMonitor.log
NMSROOT\logs\TomcatMonitor.log

4. NMSROOT\lib\classpath\md.properties

5. Version of the Windows server.

Also, do you have good CiscoWorks LMS backups (scheduled via Common Services - Server - Admin - Backup), if any?

[rzurawski1153]

I zipped them

except for NameServiceMonitor.log

RME 4.0 patch 5

CM  4.0

CMF 3.0 patch 6

CDone 3.0 patch 6

DFM 2.0 patch 6

CiscoView 6.1 patch 5

It is Windows 2003 SP2 server.

[yjdabear]

It looks to me that your LMS 2.6 went down on Mar 5 around 8:53am, presumably due to the disk space issue. It came back up around 8:58am. There's another LMS startup on Mar 9, but that time Tomcat failed to log anything to stdout.log which remains completely empty 'til now.

There's nothing alarming in TomcatMonitor.log to me.

The only thing that's a little troubling in Apache's error.log is the following change about ApacheCore.dll, but I'm not sure that's the main issue.

[Thu Mar 04 16:06:26 2010] [error] Apache.exe caused a EXCEPTION_ACCESS_VIOLATION in module ApacheCore.dll at 001B:6FFA8159, ap_ctx_get()+9 byte(s)
[Thu Mar 04 16:06:27 2010] [notice] Initializing etag from d:/cscopx/mdc/apache/logs/etag-state

[Fri Mar 05 10:01:45 2010] [error] Apache.exe caused a EXCEPTION_STACK_OVERFLOW in module at 001B:6FFB54F7
[Fri Mar 05 10:01:47 2010] [notice] Initializing etag from d:/cscopx/mdc/apache/logs/etag-state

Is there an stderr.log-OLD-1 from Mar 5?
Is NameServiceMonitor.log empty?

What does EDS.log contain for between Mar 5 - 9?

In Windows' Services Control Panel, are there any CiscoWorks services set to "automatic" startup, other than CiscoWorks Daemon Management, RCP, RSH, TFTP, and IPM?

[rzurawski1153]

There is no file NameServiceMonitor.log that I can find?

There is an CampusOGSServer.log that is about 70 MB and has errors related to SQL queries.

attach are the other two stderr.logs.

Windows Services status

CW ANI db engine   started     manual

CW Daemon MGR   started     auto

CW RME NG DB     started     manual

CW Tomcat Servlet  started     manual

CW Visibroker         started     manual

CW Web Server       started     manual

CWCS Cmf DB        started     manual

CWCS rsh/tcp serv  NOT STARTED manual

I started this process myself and was able to launch the admin login page but was unable to login. Going to the server console I noticed about 30 cwjava.exe processes running and windows explorer would not come up. I restarted the server. I will updated as it restarts and I look around some more.

recent contents of EDS.log file

[Tue Feb 16 14:31:18 CST 2010] Manager initComplete
[Tue Feb 16 14:31:18 CST 2010] Logger initComplete
[Tue Feb 16 14:31:18 CST 2010] Compiler available
[Tue Feb 16 14:31:18 CST 2010] Loader initComplete
[Tue Feb 16 14:31:18 CST 2010] Atom initComplete
[Tue Feb 16 14:31:18 CST 2010] Factory initComplete
[Tue Feb 16 14:31:19 CST 2010] Filter Repository initComplete
[Tue Feb 16 14:31:19 CST 2010] Event Repository initComplete
[Tue Feb 16 19:12:33 CST 2010] Manager initComplete
[Tue Feb 16 19:12:33 CST 2010] Logger initComplete
[Tue Feb 16 19:12:33 CST 2010] Compiler available
[Tue Feb 16 19:12:33 CST 2010] Loader initComplete
[Tue Feb 16 19:12:33 CST 2010] Atom initComplete
[Tue Feb 16 19:12:34 CST 2010] Factory initComplete
[Tue Feb 16 19:12:34 CST 2010] Filter Repository initComplete
[Tue Feb 16 19:12:34 CST 2010] Event Repository initComplete
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at com.inprise.vbroker.util.DaemonThread.start(DaemonThread.java:13)
at com.inprise.vbroker.orb.ThreadPool.doWork(ThreadPool.java:248)
at com.inprise.vbroker.orb.TPDispatcherImpl.dispatch(TPDispatcherImpl.java:51)
at com.inprise.vbroker.orb.SocketSCM.newConnection(SocketSCM.java:93)
at com.inprise.vbroker.orb.SocketSCM$ListenerThread.run(SocketSCM.java:36)
at java.lang.Thread.run(Thread.java:479)
[Fri Mar 05 08:59:04 CST 2010] Manager initComplete
[Fri Mar 05 08:59:04 CST 2010] Logger initComplete
[Fri Mar 05 08:59:04 CST 2010] Compiler available
[Fri Mar 05 08:59:04 CST 2010] Loader initComplete
[Fri Mar 05 08:59:05 CST 2010] Atom initComplete
[Fri Mar 05 08:59:05 CST 2010] Factory initComplete
[Fri Mar 05 08:59:05 CST 2010] Filter Repository initComplete
[Fri Mar 05 08:59:05 CST 2010] Event Repository initComplete
[Fri Mar 19 17:07:54 CDT 2010] Manager initComplete
[Fri Mar 19 17:07:54 CDT 2010] Logger initComplete
[Fri Mar 19 17:07:55 CDT 2010] Compiler available
[Fri Mar 19 17:07:55 CDT 2010] Loader initComplete
[Fri Mar 19 17:07:55 CDT 2010] Atom initComplete
[Fri Mar 19 17:07:55 CDT 2010] Factory initComplete
[Fri Mar 19 17:07:55 CDT 2010] Filter Repository initComplete
[Fri Mar 19 17:07:56 CDT 2010] Event Repository initComplete

[rzurawski1153]

Server was rebooted.... which it has been a few times since this problem started.

I was able to login to CiscoWorks admin site.

I ran selftest - it appears that all checks passed.

Service CWCS rsh/tcp serv is not started and I did not have to manual start it.

There are 34 instances of CWJAVA.EXE running in task manager?

So, at this point, the issue of the "server not ready" has decide to fix itself. I will be keeping an

eye on it plus try to fix the syslog.log not cleaning up?

[yjdabear]

Good news. Can you find a NameServiceMonitor.log file (on D: drive, IIRC) now? Does NMSROOT\tomcat\logs\stdout.log have anything (non-empty) now?

As far as the syslogging problem is concerned:

1) You may want consider relocating the syslogs to another drive other than C: or D: (where your LMS installation is), so the former's growth doesn't impact the OS or LMS availability. Here's the utility that helps with the relocation: NMSROOT\bin\perl NMSROOT\bin\syslogConf.pl

2) To keep the size of the syslog files themselves under control, regardless of where they live, I think the best advice for LMS 2.6 is to follow the advice in the following post by Joe regarding the use of "logrot.pl" (NMSROOT\bin\perl NMSROOT\bin\logrot.pl -c)

https://supportforums.cisco.com/message/664653#664653

and when you upgrade to LMS 3.1 or later, there'll be a GUI-equivalent to handle the syslog rotation, as explained here:

https://supportforums.cisco.com/message/664649#664649