Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
11758
Views
9
Helpful
9
Replies

RTMT message on low virtual memory

Ronit Malhotra
Level 1
Level 1

‎10-11-2013 02:42 AM - edited ‎03-16-2019 07:51 PM

Recieving following RTMT alerts on presence server CM 8

Available virtual memory below configured threshold . Configured low threshold is 25 % tomcat (1065 MB) uses most of the memory.

Any suggestion?

2 people had this problem
Labels:
0 Helpful
9 Replies 9

m.batts
Level 4
Level 4

‎10-11-2013 02:51 AM

What exact version of 8 are you running?

0 Helpful

Ronit Malhotra
Level 1
Level 1
In response to m.batts

‎10-11-2013 03:46 AM

Its 8.1

0 Helpful

Manish Gogna
Cisco Employee
Cisco Employee
In response to Ronit Malhotra

‎10-11-2013 04:37 AM

Hi Dheeraj,

Are the alerts being generated continuously or at some specific times of the day? This will help isolate if its happening due to some other process / activity on the server or not.

Have you tried restarting Tomcat or the CUPS server yet? Does the issue return after a restart ?

Can you check the output of the following command to see if there is any core dump on the server

utils core active list

Also , check for any issues by running 'utils diagnose test'

Please share the hardware specs of the server.

Manish

0 Helpful

Ronit Malhotra
Level 1
Level 1
In response to Manish Gogna

‎10-11-2013 05:33 AM

on Utils Diagnose test

I got following

test - disk_space          : Passed (available: 37124 MB, used: 4874 MB)

skip - disk_files          : This module must be run directly and off hours

test - service_manager     : Passed

test - tomcat              : Passed

test - tomcat_deadlocks    : Passed

test - tomcat_keystore     : Passed

test - tomcat_connectors   : Passed

test - tomcat_threads      : Passed

test - tomcat_memory       : Passed

test - tomcat_sessions     : Failed - The following web applications have an unusually large number of active sessions: ==>query .  Please collect all of the Tomcat logs for root cause analysis: file get activelog tomcat/logs/*

test - validate_network    : Passed

test - raid                : Passed

test - system_info         : Passed (Collected system information in diagnostic log)

test - ntp_reachability    : Passed

test - ntp_clock_drift     : Passed

test - ntp_stratum         : Passed

skip - sdl_fragmentation   : This module must be run directly and off hours

skip - sdi_fragmentation   : This module must be run directly and off hours

test - ipv6_networking     : Passed

Diagnostics Completed

admin:utils core active list

      Size         Date            Core File Name

=================================================================

  73536 KB   2013-07-25 22:39:39   core.25987.11.ma.1374809924

  71152 KB   2013-07-25 22:40:02   core.24570.11.tc.1374809924

  78776 KB   2013-03-29 22:35:43   core.27693.11.cm.1364614496

I have not restarted tomcat service yet, do you think it should be restarted?

0 Helpful

Manish Gogna
Cisco Employee
Cisco Employee
In response to Ronit Malhotra

‎10-11-2013 05:38 AM

Dheeraj,

Does the start of the issue coincide with the core dumps generation date ( 25th July ). if yes, the core dumpbacktrace needs to be investigated. If the issue is recent then please restart Tomcat to start with and if the issue returns there needs to be further investigation by collecting Tomcat logs as suggested in the utils diagnose test output.

HTH

Manish

0 Helpful

Vishal Parekh
Level 1
Level 1
In response to Manish Gogna

‎04-24-2017 01:58 AM

Hi Manish, 

I am facing the same kind of error on my Call manager (Version running 10.5.2.12901-1)

Below are the error which I am receiving from RTMT & also find the below output of "utils core active list" & "utils diagnose test"

 

RTMT Error:

1. UsedDiskSpace : 92 MessageString : Common Disk utilization hits LWM!

2.   Available virtual memory below configured threshold . Configured low threshold is 30 % tomcat#1 (1858 MB) uses most of the memory. 

Output from PUB CLI

admin:show status

Host Name : cucm01
Date : Mon Apr 24, 2017 12:55:06
Time Zone : Gulf Standard Time (Asia/Dubai)
Locale : en_US.UTF-8
Product Ver : 10.5.2.12901-1
Unified OS Version : 6.0.0.0-2

Uptime:
12:55:07 up 10 days, 21:22, 1 user, load average: 0.26, 0.42, 0.57

CPU Idle: 83.33% System: 07.29% User: 07.81%
IOWAIT: 01.04% IRQ: 00.00% Soft: 00.52%

Memory Total: 3925704K
Free: 113892K
Used: 3811812K
Cached: 387972K
Shared: 0K
Buffers: 51780K

Total Free Used
Disk/active 14643120K 1798244K 12696112K (88%)
Disk/inactive 14643056K 1896528K 12597764K (87%)
Disk/logging 50989088K 4112436K 44286476K (92%)

admin:utils core active list
No core files found


admin:utils diagnose test

Log file: platform/log/diag4.log

Starting diagnostic test(s)
===========================
test - disk_space : Passed (available: 1757 MB, used: 12399 MB)
skip - disk_files : This module must be run directly and off hours
test - service_manager : Passed
test - tomcat : Passed
test - tomcat_deadlocks : Passed
test - tomcat_keystore : Passed
test - tomcat_connectors : Passed
test - tomcat_threads : Passed
test - tomcat_memory : Passed
test - tomcat_sessions : Passed
skip - tomcat_heapdump : This module must be run directly and off hours
test - validate_network : Passed
test - raid : Passed
test - system_info : Passed (Collected system information in diagnostic log)
test - ntp_reachability : Passed
test - ntp_clock_drift : Passed
test - ntp_stratum : Passed
skip - sdl_fragmentation : This module must be run directly and off hours
skip - sdi_fragmentation : This module must be run directly and off hours

Diagnostics Completed


The final output will be in Log file: platform/log/diag4.log


Please use 'file view activelog platform/log/diag4.log' command to see the output

What should i do further to fix this issue ?

Regards, 

Vishal

0 Helpful

islam.kamal
Level 10
Level 10

‎10-11-2013 05:42 AM

Hi

You might be hitting a bug that is caused due to high no of http requests to the Tomcat by RTMT.Goto RTMT -> Performoance Monitoring -> Tomcat web-application -> Active sessions -> ast.you will see the number of active sessions. Please find the number of sessions

You will gradually see that this value goes high due to memory leak. You can set the RTMT alert for this value as well.May be you hit  CSCtj50884. Might be useful in resolving this issue.Use also the below command :-

utils service restart Cisco Tomcat

I suspect that this is being caused by the common partition being full. Please attach the output of the following commands:

show status

show hardware

show version active

show process using-most memory

Thank you

please rate all useful information

Ronit Malhotra
Level 1
Level 1
In response to islam.kamal

‎10-11-2013 06:24 AM

I have following as active session

  • _root
  • ast
  • axl
  • CallRecordService
  • ccmadmin
  • ccmservice
  • CCMTrustCerts
  • ccmuser
  • CDRonDemandService
  • changecredential
  • cmplatform
  • controlcenterservice
  • cucreports
  • cupadmin
  • cupuser
  • dpservice
  • drf
  • emapp
  • EPASSoap
  • grt
  • Help
  • log4jinit
  • logcollectionservice
  • manager
  • perfmonservice
  • platform-services
  • plugins
  • realtimeservice
  • reporter-servlet
  • SNMPService
  • tomcatstats
  • tracecollection
  • TroubleshooterService

Show hardware

admin:show hardware

HW Platform       : 7825H4

Processors        : 1

Type              : Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz

CPU Speed         : 3000

Memory            : 2048 MBytes

Object ID         : 1.3.6.1.4.1.9.1.583

OS Version        : UCOS 4.0.0.0-44

Serial Number     : MX29300086

RAID Version      :

1.86

BIOS Information  :

07/11/2009

RAID Details      :

Smart Array E200 in Slot 4

   Bus Interface: PCI

   Slot: 4

   Serial Number: PA6C90R9SXQ0U9

   Cache Serial Number: P9A3A0C9SXQ16W

   RAID 6 (ADG) Status: Disabled

   Controller Status: OK

   Chassis Slot:

   Hardware Revision: Rev A

   Firmware Version: 1.86

   Rebuild Priority: Medium

   Expand Priority: Medium

   Surface Scan Delay: 15 secs

   Post Prompt Timeout: 0 secs

   Cache Board Present: True

   Cache Status: OK

   Accelerator Ratio: 50% Read / 50% Write

   Drive Write Cache: Disabled

   Total Cache Size: 128 MB

   Battery Pack Count: 1

   Battery Status: OK

   SATA NCQ Supported: False

   Array: A

      Interface Type: SATA

      Unused Space: 0 MB

      Status: OK

      Logical Drive: 1

         Size: 232.9 GB

         Fault Tolerance: RAID 1

         Heads: 255

         Sectors Per Track: 32

         Cylinders: 59844

         Stripe Size: 128 KB

         Status: OK

         Array Accelerator: Enabled

         Unique Identifier: 600508B1001052395358513055390001

         Disk Name: /dev/cciss/c0d0

         Mount Points: /partB 42.1 GB, / 42.1 GB, /grub 258 MB, /common 146.5 GB

         Logical Drive Label: AF881CCEPA6C90R9SXQ0U96A8B

         Mirror Group 0:

            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 250 GB, OK)

         Mirror Group 1:

            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 250 GB, OK)

      physicaldrive 1I:1:1

         Port: 1I

         Box: 1

         Bay: 1

         Status: OK

         Drive Type: Data Drive

         Interface Type: SATA

         Size: 250 GB

         Firmware Revision: HPG8

         Serial Number: 9SF15C9Z

         Model: ATA     GB0250EAFJF

         SATA NCQ Capable: False

         PHY Count: 1

         PHY Transfer Rate: 1.5GBPS

      physicaldrive 1I:1:2

         Port: 1I

         Box: 1

         Bay: 2

         Status: OK

         Drive Type: Data Drive

         Interface Type: SATA

         Size: 250 GB

         Firmware Revision: HPG7

         Serial Number: Z2AEQMSD

         Model: ATA     VB0250EAVER

         SATA NCQ Capable: False

         PHY Count: 1

         PHY Transfer Rate: 1.5GBPS

admin:show version active

Active Master Version: 8.6.1.10000-34

Active Version Installed Software Options:

No Installed Software Options Found.

admin:show process using-most memory

MEM(K) PID  ARGS

108504 31987 /usr/local/cup/bin/syncAgent /usr/local/cup/bin/syncAgentJLauncherConfig.xml

138392 23355 /usr/local/cm/bin/cmoninit -w

191192 20690 /usr/local/cm/bin/RisDC

237128 321 /usr/local/pe/bin/pe

350416 759 /home/tomcat/tomcat /home/tomcat/tomcatJlauncherConfig.xml -Djava.library.path=/usr/local/sip/libexec:/usr/local/sip/modules:/usr/local/pe/lib:/usr/local/lib:/usr/local/thirdparty/java/j2sdk/jre/lib/i386:/usr/local/thirdparty/java/j2sdk/jre/lib/i386/server:/usr/lib/pgsql:/usr/lib:/usr/local/cm/lib::/usr/local/platform/lib:/usr/local/pe/tt/TimesTen/tt60/lib -XX:ErrorFile=/usr/local/thirdparty/jakarta-tomcat/logs/diagnostic-info.jvm-crash..tomcat.txt -XX:OnOutOfMemoryError=/home/tomcat/tomcat_diagnostics.sh -XX:OnError=/home/tomcat/tomcat_diagnostics.sh -Djavax.net.ssl.trustStore=/usr/local/platform/.security/tomcat/trust-certs/tomcat-trust.keystore -Djavax.net.ssl.trustStorePassword=DxDDqXEA1LrW8aTq -Djavax.net.ssl.trustStoreType=PKCS12 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/thirdparty/jakarta-tomcat/conf/logging.properties -Xmx640m -Xms256m -XX:MaxPermSize=256m -Djava.endorsed.dirs=/usr/local/thirdparty/jakarta-tomcat/endorsed -classpath :/usr/local/thirdparty/java/bcprov-jdk15-138.jar:/usr/local/platform/jar/xalan.jar:/usr/local/platform/jar/certMgmt.jar:/usr/local/platform/jar/certMonitor.jar:/usr/local/platform/jar/Iproduct.jar:/usr/local/platform/jar/Ihardware.jar:/usr/local/platform/jar/CiscoIPSec.jar:/usr/local/platform/jar/ciscoCmd.jar:/usr/local/platform/jar/ciscoCommon.jar:/usr/local/platform/jar/platform-api.jar:/usr/local/platform/jar/maverick-all.jar:/common/download:/usr/local/platform/application_locale/platform-api:/usr/local/platform/application_locale/cmplatform:/usr/local/cm/jar/commons-logging.jar:/usr/local/platform/jar/xstream-1.1.2.jar:/usr/local/cm/jar/log4j-1.2.8.jar:/usr/local/cm/application_locale/cmservices:/usr/local/cm/application_locale/car:/usr/local/cm/application_locale/ccmadmin:/usr/local/cm/application_locale/cupadmin:/usr/local/cm/application_locale/cupuser:/usr/local/cm/application_locale:/usr/local/thirdparty/jakarta-tomcat/bin/bootstrap.jar -Djava.security.manager -Djava.security.policy==/usr/local/thirdparty/jakarta-tomcat/conf/catalina.policy -Dcatalina.base=/usr/local/thirdparty/jakarta-tomcat -Dcatalina.home=/usr/local/thirdparty/jakarta-tomcat -Djava.io.tmpdir=/usr/local/thirdparty/jakarta-tomcat/temp start

0 Helpful

islam.kamal
Level 10
Level 10
In response to Ronit Malhotra

‎10-11-2013 03:51 PM

Hi

Did you try the command  "utils service restart Cisco Tomcat"?. If yes , the problem is still running.

Thank you

please rate all useful information

0 Helpful
Customers Also Viewed These Support Documents