CiscoWorks won't start after LMS upgrade 3.0 -> 3.2 ERROR 502 Bad Gateway in Portal

[e.gayosso]

I upgraded CiscoWorks LMS from 3.0 to 3.2 today.  Everything worked prior to the upgrade.

The upgrade was successful, with no errors, and all services started.

Can anyone help, and check everything and I see no problem. The system creates the file MDCSupportInformation.zip me clean without any content.

I add the installation log file;

And tell them that after upgrading to 3.2, you carge the following patches:

LMS321-win-CSCtr77570-0.zip,

cm5.2.1-nt-CSCtw88763-1.0.zip,

lms3.2.1-win-CSCts95764-0.zip,

lms3.2.1-win-CSCts82530-0.zip

in that order.

He added the logs generated by the LMS services Tomcat and Apache

[ngoldwat]

Hi,

Here are the various errors/warning I found while looking through the attached logs.

WARNING:The application that you are installing requires new tasks to be

WARNING:registered with ACS. If you have already registered this application

WARNING:with ACS from another server, you do not need to register it again.

WARNING:However if you re-register the application, you will lose any custom

WARNING:roles that you  had created earlier for this application  in ACS.

WARNING:Some features in ACS may not work correctly on Dual-stack servers.

Warning: System memory is less than the minimum requirement.This may affect performance.

Warning: Current swap space is less than the minimum requirement. This may affect performance.

WARNING: The available RAM is 4086 MB which is lesser than the required RAM of 4096 MB. User chose to continue the installation

WARNING: Domain Name is not found from TcpIp parameters, setting PX_FQDN same as PX_HOST

INFO: User Selected to Reboot the Machine

Installation Completed at 16:38:06

Nov 23, 2012 7:59:11 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run

SEVERE: Caught exception (java.lang.NoClassDefFoundError) executing org.apache.jk.common.ChannelSocket$SocketConnection@d888c7, terminating thread

[warn] RSA server certificate CommonName (CN) `LOND-CW-MDF-1' does NOT match server name!?

[Fri Nov 23 19:19:46 CST 2012] CMFX - fileLoc is \C:\PROGRA~1\CSCOpx\lib\classpath\md.properties

[ Fri Nov 23 19:22:22 CST 2012 ] TomcatMonitor After gettingAJPPort

[ Fri Nov 23 19:22:22 CST 2012 ] TomcatMonitor After gettingAJPPort : ajpPort = 9009 and host =LOND-CW-MDF-1

Cookie value is invalid

LTGateway.run(): connection to LWMS failed, trying again,  Nov 23 19:22:28

You have a BUNCH of the ChannelSocket$SocketConnection exceptions.

The next step for you:

1. Check the running processes via cmd line (pdshow -brief).
2. Check that all processes are running as expected.  If they are then take a look at what is keeping ports open on your box (netstat -anob).  From above, the reason may be an application occupying a port LMS needs.  Also check the Windows error logs.
3. Check the following services, they should look like:

Ciscoworks ANI database engine :Manual

CiscoWorks Daemon Manager :Auto

CiscoWorks RME NG database :Manual

CiscoWorks Tomcat Servlet engine : Manual

CW VisiBroker Smart Agent :Manual

CiscoWorks Web Server :Manual

CWCS Cmf database engine :Manual

CWCS rsh/rcp service :Auto

CWCS syslog service :Auto

CWCS tftp service :Auto

DFM dfmEpm :Manual

DFM dfmFh :Manual

DFM dfmInv :Manual

   4.  Log in as the local Administrator (not Domain).

• Stop daemons (net stop crmdmgtd)
• CSCOpxMDC\Apache\perl ConfigSSL.pl -disable
• CSCOpx\MDC\Apache\perl ConfigSSL.pl -enable (Answer when prompted)
• navigate to ..\NMSROOT\setup\support
• Execute resetCasuser.exe
• Start Daemons (net start crmdmgtd) and wait 20 minutes.
• Check if condition persists.

   5.  Check the output of  ..\NMSROOT\MDC\Apache\bin>pdreg -l Apache

A couple other things:

1. Set the server name to be the server name with the domain as well.
2. Set the primary DNS suffix to match
3. Modify md.properties & Registry & set PX_FQDN to the above.

- HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet

- HKEY_LOCAL_MACHINE\SOFTWARE\Cisco\ResourceManager\CurrentVersion\Environment

Thanks.

[e.gayosso]

Hi ngoldwat;

I followed your advice and not you see success, you add the results (step1.txt,step2.txt,step3.jpg,step4.txt) and continues the error "502 Bad Gateway".

Not understand the following:

1. Set the server name to be the server name with the domain as well.
2. Set the primary DNS suffix to match
3. Modify md.properties & Registry & set PX_FQDN to the above.

- HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet

- HKEY_LOCAL_MACHINE\SOFTWARE\Cisco\ResourceManager\CurrentVersion\Environment

Could you give me more details so you can apply it to the server

I appreciate your advice ...

Greetings.

[ngoldwat · Accepted solution]

Hi,

There are many processes that are not running as seen in your pdshow output.  Most times this is related to permissions.  Is your machine part of a windows domain?  If yes, then perhaps a group policy was pushed that breaks permissions on your box.  Did you look at the Windows Event Logs?  Has a program been recently added to your server?

You can try shutting down all other processes such as anti-virus (be sure they stay off - disabled) and then restarting via net stop crmdmgtd / net start crmdmgtd.

I also noticed this:

-----UHIC ununregistration

Can't load 'C:\PROGRA~1\CSCOpx/bin/perlDbEncrypt.dll' for module perlDbEncrypt: load_file:The specified module could not be found at

C:\PROGRA~1\CSCOpx\objects\perl5\lib/DynaLoader.pm line 230.

at C:\PROGRA~1\CSCOpx/lib/perl/db/Cisco/DbUtils.pm line 72

Compilation failed in require at C:\PROGRA~1\CSCOpx/lib/perl/db/Cisco/DbUtils.pm line 72.

Compilation failed in require at C:\PROGRA~1\CSCOpx/lib/perl/install/InstallUtility.pm line 27.

BEGIN failed--compilation aborted at C:\PROGRA~1\CSCOpx/lib/perl/install/InstallUtility.pm line 27.

Compilation failed in require at C:\PROGRA~1\CSCOpx\campus\bin\utmEnable.pl line 11.

BEGIN failed--compilation aborted at C:\PROGRA~1\CSCOpx\campus\bin\utmEnable.pl line 11.

----- end of UHIC ununregistration

And found an internal bug CSCsx17409 that indicates an issue with the upgrade.  There is no work around mentioned.

Do you have a valid backup?

Thanks.

[e.gayosso]

Thanks for the feedback;

No the server has never belonged to a Windows domain

Not any software installed, the CW LMS is the only software on the server.

We apply the test to uninstall AntiVirus and restore processes, but remains the same.

And I commented that if I have a valid backup, but NOT charging that way.

I appreciate your advice ...

Greetings.