Re: Device update never finishes completely

smandrapa · ‎01-05-2011

CiscoWorks LMS 3.2 is installed on 64-bit Windows server 2008 Standard, Service Pack 2 .

When doing Device update, it never finishes completely – CiscoWorks start page can not be opened. Result is screen ”503 Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later”.

We have noticed that CiscoWorks ANI Database engine is not automatically started. We also suspect that some other tasks can not be finished sucessfully because of this problem (like sending syslog email notifications, Common Services backup failed).

Our solution is to either reboot server or manually restart services.

What could be a problem with Device update?

Joel Monge · ‎01-05-2011

It may be that you did not wait long enough. A device update needs to shutdown the processes, update packages and then restart daemons. This can take around 30 minutes depending on the environment. I suggest you do your device updates using CLI as this will actually show you what is happening. For example, to update your RME device packages do:

PSUCli.bat -p rme -d -dst c:\psu_download -all

This will download the packages into C:\psu_download\rme. Then install this way:

PSUCli.bat -p rme -install -src c:\psu_download\rme -all

This will show you when the processes are being shutdown, packages are being installed and when the processes are being brought up until completion. Other things to keep in mind are, always use these commands to re-start LMS daemons:

net start crmdmgtd

net stop crmdmgtd

Do not use the Windows services to do this or change their startup types. Make sure you have Windows DEP set to essential programs only and your anti-virus on-access scanning has an exclusion for the NMSROOT (by default C:\Program Files\CSCOpx) directory and all its folders. These may be causing problems when the processes need to stop or start during the device update. Also make sure your Windows Services are set as shown on this DOC.

smandrapa · ‎01-17-2011

Thank you Joel.

We have done Device Update few times on this system, and it never finished completely. Updates are being downloaded and installed - and all processes except ANIdb are being restarted. Couple of times we waited whole night before rebooted server or restarted services (through net stop and net start commands).

After your advices we checked and configured DEP and Antivirus as you told (Program Files\CSCOpx and subfolders and also psu_download folder are excluded from scanning).

In next day or two we are planning to apply available updates by using PSUCli.bat scripts.

Is it possible to turn on some additional debuging on LMS, or to collect some log files during this process ? How can we collect some additional info to be used for troubleshooting purposes and for solving this issue?

Thanks in advance !

Joel Monge · ‎01-17-2011

If I understand correctly, we are seeing two problems here. First, when you do a device update the daemons never come back up forcing you to restart the server to be able to login. If you have followed the previous suggestions and this still occurrs, then you are probably looking at CSCtg76942. The workaround for now is to restart the daemons. This bug will be fixed in the upcoming LMS 3.2 SP release in March.

The other problem is that ANIDbEngine does not come up after you restart the daemons, although all other processes do. This could be due to a number of different reasons including even database corruption. If so, please open a separate thread to look into why the ANIDbEngine never comes up after a process restart.

smandrapa · ‎01-26-2011

Hello Joel,

In a last 10 days we had opportunity to execute Device update - a) using PSUCli and - b) using GUI.

a) When doing Device update using PSUCli, everything works fine, first download, then installation. Services were being stopped, packages installed and services started completely (you can see attached file). There was no issue!

b) After few days, when doing Device update using GUI, we had the same problem like before - services were being automatically stopped, packages installed and all services EXCEPT ANI were started. After few hours waiting, ANIDB engine were still down. We then rebooted server, after that, all services were started correctly.

So, there is only one problem - ANIDB Engine service does not start after Device Package update when update is done USING GUI. After restarting server, everythyng works fine (ANIDB is up and running).

Do you still think we are facing CSCtg76942 bug?

Joel Monge · ‎02-01-2011

Please post these files:

- NMSROOT\MDC\tomcat\logs\stdout.log

- ANIServer.log, ANIDbEngine.log, ani.log (under NMSROOT\log)

- NMSROOT\lib\classpath\com\cisco\nm\xms\psu\conf\psu.properties

The fact that only ANIDbEngine does not come up, does not seem to completely match this bug. It might be a different problem.

smandrapa · ‎03-01-2011

Here are asked files...

Joel Monge · ‎03-01-2011

It does seem you are hitting the mentioned bug. This will be fixed in the upcoming LMS 3.2 SP release in March 15th. This fix will increase the interval between the daemon shutdown and restart, which is related to the issue you are seeing.

smandrapa · ‎03-21-2011

Thanks Joel.

We are still waiting for mentioned SP.

I suppose that it will be posted on Service support/Download Software page for Products>Network Management and Automation>...>CiscoWorks LMS3.2>CW LMS Patches ...

Joel Monge · ‎03-22-2011

Yes, it should be posted there. It seems there has been some delay with the release but it should be coming out soon. I do not have an exact date at the moment.

Joel Monge · ‎03-29-2011

LMS 3.2.1 is now available:

http://www.cisco.com/cisco/software/type.html?mdfid=282635181&flowid=16561&softwareid=280775102