05-03-2011 02:02 PM - edited 03-12-2019 09:37 AM
CSCtj50884 covers a leak that will result in the Tomcat web service running out of memory. This will disrupt web services which includes the CCMAdmin and CCMUser web sites, Extension Mobility operations, and even IPCC agent logins.
There are fixes available for every major version of CUCM impacted by this defect.
7.1(5)SU4 this image will be posted soon to cisco.com
An SU for 8.5(1) is planned but no release date is available.
Since all CUCM ES patches are cumulative if you are running an ES later than the ones listed below you have the fix.
6.1.5.13039-1
7.1.5.33043-1
8.0.3.22026-1
8.5.1.12008-3
After upgrading CUCM to a version with the fix the last step is to download the new RTMT client from the upgraded server and completely reinstall RTMT on every PC that connects to CUCM. Failure to install the new RTMT client will expose the server to more failures even after applying the upgrade.
Tomcat memory leaks will show up over time and will range from slow response to web service through intermittent failures for some web services and eventually no web services will work.
You can recover service immediately by restarting Tomcat from the CLI with the command utils service restart Cisco Tomcat
In the Cisco Tomcat catalina.out log file you will see errors such as this:
Dumping heap to /usr/local/thirdparty/jakarta-tomcat/logs/java_pid5555.hprof ...
Heap dump file created [598225933 bytes in 69.829 secs]
Jan 01, 2011 12:14:55 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
SEVERE: Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$CharsetSE.encode(StringCoding.java:334)
at java.lang.StringCoding.encode(StringCoding.java:378)
at java.lang.String.getBytes(String.java:812)
at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
at java.io.File.lastModified(File.java:795)
This particular defect is not the only way that Tomcat can run out of heap memory and throw this error. The .hprof file is generated by the system any time the heap space is exhaused and its contents are the only way to prove conclusively the cause of the fault. The details for CSCtj50884 in Bug Toolkit contains instructions for decoding the .hprof file if you would like to get 100% confirmation.
Hi,
I suspect the V8.6.2.21900-5 to be also affected by this bug, and I hope there is a solution on the road.
After a cluster reboot on anew implemented BE 6000, because of some TFTP file uploads, the WEb SErvices (Tomcat, Axl,...) did not coming up anymore on the Pub and on the sub....Strange...,and did show "
Commanded Out of Service" and "Starting" (Tomcat)
None of my troubleshooting trials did help. Now I think about an Upgrade with the same Version on the second partition to force a reinstallation of the web services.
admin:utils service list
Requesting service status, please wait...
System SSH [STARTED]
Cluster Manager [STARTED]
Service Manager is running
Getting list of all services
>> Return code = 0
A Cisco DB[STARTED]
A Cisco DB Replicator[STARTED]
Cisco AMC Service[STARTED]
Cisco AXL Web Service[STOPPED] Commanded Out of Service
Cisco Audit Event Service[STARTED]
Cisco Bulk Provisioning Service[STARTED]
Cisco CAR DB[STARTED]
Cisco CAR Scheduler[STARTED]
Cisco CAR Web Service[STOPPED] Commanded Out of Service
Cisco CDP[STARTED]
Cisco CDP Agent[STARTING]
Cisco CDR Agent[STARTED]
Cisco CDR Repository Manager[STARTED]
Cisco CTIManager[STARTED]
Cisco CTL Provider[STARTED]
Cisco CallManager[STARTED]
Cisco CallManager Admin[STOPPED] Commanded Out of Service
Cisco CallManager Cisco IP Phone Services[STOPPED] Commanded Out of Service
Cisco CallManager Personal Directory[STOPPED] Commanded Out of Service
Cisco CallManager SNMP Service[STARTED]
Cisco CallManager Serviceability[STOPPED] Commanded Out of Service
Cisco CallManager Serviceability RTMT[STOPPED] Commanded Out of Service
Cisco Certificate Authority Proxy Function[STARTED]
Cisco Certificate Change Notification[STARTED]
Cisco Certificate Expiry Monitor[STARTED]
Cisco Change Credential Application[STOPPED] Commanded Out of Service
Cisco DHCP Monitor Service[STARTED]
Cisco DRF Local[STARTED]
Cisco DRF Master[STARTED]
Cisco Database Layer Monitor[STARTED]
Cisco Dialed Number Analyzer[STOPPED] Commanded Out of Service
Cisco Dialed Number Analyzer Server[STARTED]
Cisco DirSync[STARTED]
Cisco Extended Functions[STARTED]
Cisco Extension Mobility[STOPPED] Commanded Out of Service
Cisco Extension Mobility Application[STOPPED] Commanded Out of Service
Cisco IP Manager Assistant[STOPPED] Commanded Out of Service
Cisco IP Voice Media Streaming App[STARTED]
Cisco License Manager[STARTED]
Cisco Log Partition Monitoring Tool[STARTED]
Cisco RIS Data Collector[STARTED]
Cisco RTMT Reporter Servlet[STOPPED] Commanded Out of Service
Cisco SOAP - CDRonDemand Service[STARTED]
Cisco SOAP - CallRecord Service[STARTED]
Cisco Serviceability Reporter[STARTED]
Cisco Syslog Agent[STARTING]
Cisco TAPS Service[STARTED]
Cisco Tftp[STARTED]
Cisco Tomcat[STARTING]
Cisco Tomcat Stats Servlet[STOPPED] Commanded Out of Service
Cisco Trace Collection Service[STARTED]
Cisco Trace Collection Servlet[STOPPED] Commanded Out of Service
Cisco Trust Verification Service[STARTED]
Cisco UXL Web Service[STOPPED] Commanded Out of Service
Cisco Unified Mobile Voice Access Service[STOPPED] Commanded Out of Service
Cisco User Data Services[STOPPED] Commanded Out of Service
Cisco WebDialer Web Service[STOPPED] Commanded Out of Service
Host Resources Agent[STARTED]
MIB2 Agent[STARTED]
Platform SOAP Services[STOPPED] Commanded Out of Service
SNMP Master Agent[STARTED]
SOAP - Diagnostic Portal Database Service[STARTED]
SOAP -Log Collection APIs[STARTED]
SOAP -Performance Monitoring APIs[STARTED]
SOAP -Real-Time Service APIs[STARTED]
System Application Agent[STARTED]
Cisco Messaging Interface[STOPPED] Service Not Activated
Primary Node =true
Any quick help idea ?
None of simple starts or restarts did help. I've also have a look into the tomcat cert to check the device name.
Jacky
We are having very slow response from our CUCM web interface. (ver 8.6.2.21900-5) A restart of the Tomcat service seems to help, but a more permanant fix is needed.
Did you ever figure out what was causing it, or find a patch?
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: