Description
CSCtj50884 covers a leak that will result in the Tomcat web service running out of memory. This will disrupt web services which includes the CCMAdmin and CCMUser web sites, Extension Mobility operations, and even IPCC agent logins.
Getting the fix
There are fixes available for every major version of CUCM impacted by this defect.
From cisco.com
6.1(5)SU3
7.1(5)SU4 this image will be posted soon to cisco.com
8.0(3a)SU2
An SU for 8.5(1) is planned but no release date is available.
From TAC via engineering special
Since all CUCM ES patches are cumulative if you are running an ES later than the ones listed below you have the fix.
6.1.5.13039-1
7.1.5.33043-1
8.0.3.22026-1
8.5.1.12008-3
Applying the fix
After upgrading CUCM to a version with the fix the last step is to download the new RTMT client from the upgraded server and completely reinstall RTMT on every PC that connects to CUCM. Failure to install the new RTMT client will expose the server to more failures even after applying the upgrade.
Symptoms
Tomcat memory leaks will show up over time and will range from slow response to web service through intermittent failures for some web services and eventually no web services will work.
You can recover service immediately by restarting Tomcat from the CLI with the command utils service restart Cisco Tomcat
In the Cisco Tomcat catalina.out log file you will see errors such as this:
Dumping heap to /usr/local/thirdparty/jakarta-tomcat/logs/java_pid5555.hprof ...
Heap dump file created [598225933 bytes in 69.829 secs]
Jan 01, 2011 12:14:55 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
SEVERE: Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$CharsetSE.encode(StringCoding.java:334)
at java.lang.StringCoding.encode(StringCoding.java:378)
at java.lang.String.getBytes(String.java:812)
at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
at java.io.File.lastModified(File.java:795)
This particular defect is not the only way that Tomcat can run out of heap memory and throw this error. The .hprof file is generated by the system any time the heap space is exhaused and its contents are the only way to prove conclusively the cause of the fault. The details for CSCtj50884 in Bug Toolkit contains instructions for decoding the .hprof file if you would like to get 100% confirmation.
