CUCM performance is very slow

[Maximus2]

Hi

We have a 2 node CUCM cluster running on version 12_5_1_10000_22 on the EVE-NG environment in our lab. There are hardly any devices configured on the servers but the CUCM performance is very very slow.IT takes upto 5-6 minutes to print the DB replication runtimestate output. Basically, any command u run on the CLI it usually takes a long time than what i have seen in the past. Attached is the output of some commands..any idea why the server could be performing dis way

admin:show status

Date : Thu Jun 10, 2021 00:39:34
Time Zone : Pacific Daylight Time (America/Los_Angeles)
Locale : en_US.UTF-8
Product Ver : 12.5.1.10000-22
Unified OS Version : 7.0.0.0-4

Uptime:
00:39:37 up 63 days, 20:49, 1 user, load average: 12.82, 14.64, 14.48

CPU Idle: 00.00% System: 12.07% User: 29.03%
IOWAIT: 00.86% IRQ: 00.00% Soft: 06.61%

Memory Total: 7960232K
Free: 263472K
Used: 3688796K
Cached: 3225916K
Shared: 664892K
Buffers: 4007964K

Total Free Used
Disk/active 17921776K 4820104K 12901880K (73%)
Disk/inactive 17921776K 16943272K 45080K (1%)
Disk/logging 62681844K 31645888K 27828840K (47%)

************************************************************

admin:show process using-most cpu
PasreOption method called
PCPU PID CPU NICE STATE CPUTIME ARGS
%CPU PID CPU NI S TIME COMMAND
33.0 27584 - 10 S 00:00:34 java -DConsoleRows=24 -DConsoleColumns=80 -DCommonFileSystem="disk_full=false,inode_full=false,no_write=false,internal_error=false" -DJvmStartTime=1623308500 -noverify sdMain name=admin priv=4 master
28.0 31080 - 10 S 00:00:00 sudo /usr/local/platform/cli_scripts/listProcesses.sh -cpu 5
27.9 27328 - 10 S 00:00:36 java -DConsoleRows=24 -DConsoleColumns=80 -DCommonFileSystem="disk_full=false,inode_full=false,no_write=false,internal_error=false" -DJvmStartTime=1623308471 -noverify sdMain name=admin priv=4 master
15.6 26443 - 0 S 9-23:44:18 /home/tomcat/tomcat -user tomcat -home /usr/local/thirdparty/java/j2sdk -pidfile /usr/local/thirdparty/jakarta-tomcat/conf/tomcat.pid -procname /home/tomcat/tomcat -outfile /usr/local/thirdparty/jakarta-tomcat/logs/catalina.out -errfile &1 -Djava.library.path=/usr/local/cm/lib:/usr/local/lib:/usr/local/thirdparty/java/j2sdk/jre/lib/i386:/usr/local/thirdparty/java/j2sdk/jre/lib/i386/server:/usr/lib/pgsql:/usr/lib:/usr/local/cm/lib::/usr/local/platform/lib -Djavax.net.ssl.trustStore=/usr/local/platform/.security/tomcat/trust-certs/tomcat-trust.keystore -Djavax.net.ssl.trustStorePassword=EncqLiRj8r8nYtHy -XX:ErrorFile=/usr/local/thirdparty/jakarta-tomcat/logs/diagnostic-info.jvm-crash.%p.tomcat.txt -Dsun.zip.disableMemoryMapping=true -XX:OnOutOfMemoryError=/home/tomcat/tomcat_diagnostics.sh -XX:OnError=/home/tomcat/tomcat_diagnostics.sh -Djdk.tls.ephemeralDHKeySize=2048 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/thirdparty/jakarta-tomcat/conf/logging.properties -Xmx1824m -Xms256m -XX:MaxPermSize=448m -Djava.endorsed.dirs=/usr/local/thirdparty/jakarta-tomcat/endorsed -cp :/common/download:/usr/local/platform/application_locale/platform-api:/usr/local/platform/application_locale/cmplatform:/usr/local/platform/application_locale:/usr/local/cm/application_locale/cmservices:/usr/local/cm/application_locale/car:/usr/local/cm/application_locale/ccmadmin:/usr/local/cm/application_locale/ucmuser:/usr/local/cm/application_locale:/etc/opt/cisco/elm/server/resource:/usr/local/thirdparty/jakarta-tomcat/bin/bootstrap.jar::/usr/local/thirdparty/jakarta-tomcat/bin/tomcat-juli.jar -Djava.security.policy==/usr/local/thirdparty/jakarta-tomcat/conf/catalina.policy -Dcatalina.base=/usr/local/thirdparty/jakarta-tomcat -Dcatalina.home=/usr/local/thirdparty/jakarta-tomcat -Djava.io.tmpdir=/usr/local/thirdparty/jakarta-tomcat/temp org.apache.catalina.startup.Bootstrap start
Press <enter> for 1 line, <space> for one page, or <q> to quit
()_(********************************************

admin:show process load
top - 00:06:46 up 63 days, 20:17, 2 users, load average: 9.95, 13.91, 13.97
Tasks: 270 total, 3 running, 267 sleeping, 0 stopped, 0 zombie
%Cpu(s): 41.4 us, 27.3 sy, 6.1 ni, 14.1 id, 6.1 wa, 0.0 hi, 5.1 si, 0.0 s
KiB Mem : 7960232 total, 192960 free, 3770224 used, 3997048 buff/cache
KiB Swap: 4095996 total, 2911300 free, 1184696 used. 3153296 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26443 tomcat 20 0 2708940 1.7g 10900 S 38.1 22.4 14384:55 tomcat
32044 ctftp 20 0 607200 23260 6692 S 38.1 0.3 9796:55 ctftp
30426 informix 10 -10 352540 219076 218384 R 33.3 2.8 6561:38 cmoninit
2200 admin 30 10 155672 2492 1748 R 23.8 0.0 0:00.16 top
29184 ccmserv+ 20 0 319868 38412 17292 S 23.8 0.5 10203:24 RisDC
3920 root -62 0 0 0 0 S 19.0 0.0 8354:00 kipvmsMixe
31808 ccmserv+ 20 0 567336 59244 8988 S 19.0 0.7 3141:19 CCMDirSync
2226 root 20 0 113452 1708 1304 R 14.3 0.0 0:00.03 hssi_api.s
29898 root 20 0 119576 18380 7936 S 14.3 0.2 909:20.87 CiscoSyslo
29988 certbase 20 0 97948 5192 5072 S 14.3 0.1 898:06.75 capf
admin:

********************************
admin:show process using-most memory
PasreOption method called
MEM(K) PID ARGS
128980 30054 /usr/local/cm/bin/PnPLauncher /usr/local/cm/conf/PnPcfg.xml
153804 29699 /usr/local/cm/bin/carschlr /usr/local/cm/conf/car/carschlrCfg.xml
212312 12303 /usr/sbin/rsyslogd -n
219088 30426 /usr/local/cm/bin/cmoninit -w
1785400 26443 /home/tomcat/tomcat -user tomcat -home /usr/local/thirdparty/java/j2sdk -pidfile /usr/local/thirdparty/jakarta-tomcat/conf/tomcat.pid -procname /home/tomcat/tomcat -outfile /usr/local/thirdparty/jakarta-tomcat/logs/catalina.out -errfile &1 -Djava.library.path=/usr/local/cm/lib:/usr/local/lib:/usr/local/thirdparty/java/j2sdk/jre/lib/i386:/usr/local/thirdparty/java/j2sdk/jre/lib/i386/server:/usr/lib/pgsql:/usr/lib:/usr/local/cm/lib::/usr/local/platform/lib -Djavax.net.ssl.trustStore=/usr/local/platform/.security/tomcat/trust-certs/tomcat-trust.keystore -Djavax.net.ssl.trustStorePassword=EncqLiRj8r8nYtHy -XX:ErrorFile=/usr/local/thirdparty/jakarta-tomcat/logs/diagnostic-info.jvm-crash.%p.tomcat.txt -Dsun.zip.disableMemoryMapping=true -XX:OnOutOfMemoryError=/home/tomcat/tomcat_diagnostics.sh -XX:OnError=/home/tomcat/tomcat_diagnostics.sh -Djdk.tls.ephemeralDHKeySize=2048 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/thirdparty/jakarta-tomcat/conf/logging.properties -Xmx1824m -Xms256m -XX:MaxPermSize=448m -Djava.endorsed.dirs=/usr/local/thirdparty/jakarta-tomcat/endorsed -cp :/common/download:/usr/local/platform/application_locale/platform-api:/usr/local/platform/application_locale/cmplatform:/usr/local/platform/application_locale:/usr/local/cm/application_locale/cmservices:/usr/local/cm/application_locale/car:/usr/local/cm/application_locale/ccmadmin:/usr/local/cm/application_locale/ucmuser:/usr/local/cm/application_locale:/etc/opt/cisco/elm/server/resource:/usr/local/thirdparty/jakarta-tomcat/bin/bootstrap.jar::/usr/local/thirdparty/jakarta-tomcat/bin/tomcat-juli.jar -Djava.security.policy==/usr/local/thirdparty/jakarta-tomcat/conf/catalina.policy -Dcatalina.base=/usr/local/thirdparty/jakarta-tomcat -Dcatalina.home=/usr/local/thirdparty/jakarta-tomcat -Djava.io.tmpdir=/usr/local/thirdparty/jakarta-tomcat/temp org.apache.catalina.startup.Bootstrap start

********************************************************

[Nithin Eluvathingal · Accepted solution]

EVE-NG is a simulation tool. don't expect the performance which you get on the real UCS server running vmware. For better performance you need to run CUCM on compatible hardware.

[Maximus2]

Yes, i understand that we need to look at a better setup instead of EVE-NG

[Nithin Eluvathingal]

When preparing my CCIE I was using a Dell server to run the CUCM, it took half an  hour for all services to come up with just two end point.I used to go  for a coffee turning ON  my dell servers.  but with a UCS server I have no such issues. 

No where in the CUCM compatibility guides its mentioned that it support EVE-NG. so dont expect any support on this issue. only option is to use a compatibility hardware  or as I did go for a coffee turning ON the server

[Maximus2]

Yeah i have never faced such issues on UCS deployments and this is my first time working on a EVE-NG environment. Anyways , i will simply have to look to setup a UCS infra then.

Thanks

[Stephanie Knoop]

You seem to indicate that something has changed recently. If you've been on the same environment and the server has been up for some time, you could have a memory leak. Over the years I've seen many such instances call manager. If rebooting the system clears it for a while then there may be a bug.

https://bst.cloudapps.cisco.com/bugsearch/search?kw=Memory%20leak%20cucm&pf=prdNm&rls=12.5&sb=anfr

[Maximus2]

Nothing has changed in the recent past. Also, a reboot does not fix the issue.