cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

771
Views
0
Helpful
6
Replies
Maximus2
Beginner

CUCM performance is very slow

Hi

We have a 2 node CUCM cluster running on version 12_5_1_10000_22 on the EVE-NG environment in our lab. There are hardly any devices configured on the servers but the CUCM performance is very very slow.IT takes upto 5-6 minutes to print the DB replication runtimestate output. Basically, any command u run on the CLI it usually takes a long time than what i have seen in the past. Attached is the output of some commands..any idea why the server could be performing dis way

 

 

admin:show status

Date : Thu Jun 10, 2021 00:39:34
Time Zone : Pacific Daylight Time (America/Los_Angeles)
Locale : en_US.UTF-8
Product Ver : 12.5.1.10000-22
Unified OS Version : 7.0.0.0-4

Uptime:
00:39:37 up 63 days, 20:49, 1 user, load average: 12.82, 14.64, 14.48

CPU Idle: 00.00% System: 12.07% User: 29.03%
IOWAIT: 00.86% IRQ: 00.00% Soft: 06.61%

Memory Total: 7960232K
Free: 263472K
Used: 3688796K
Cached: 3225916K
Shared: 664892K
Buffers: 4007964K

Total Free Used
Disk/active 17921776K 4820104K 12901880K (73%)
Disk/inactive 17921776K 16943272K 45080K (1%)
Disk/logging 62681844K 31645888K 27828840K (47%)

************************************************************

admin:show process using-most cpu
PasreOption method called
PCPU PID CPU NICE STATE CPUTIME ARGS
%CPU PID CPU NI S TIME COMMAND
33.0 27584 - 10 S 00:00:34 java -DConsoleRows=24 -DConsoleColumns=80 -DCommonFileSystem="disk_full=false,inode_full=false,no_write=false,internal_error=false" -DJvmStartTime=1623308500 -noverify sdMain name=admin priv=4 master
28.0 31080 - 10 S 00:00:00 sudo /usr/local/platform/cli_scripts/listProcesses.sh -cpu 5
27.9 27328 - 10 S 00:00:36 java -DConsoleRows=24 -DConsoleColumns=80 -DCommonFileSystem="disk_full=false,inode_full=false,no_write=false,internal_error=false" -DJvmStartTime=1623308471 -noverify sdMain name=admin priv=4 master
15.6 26443 - 0 S 9-23:44:18 /home/tomcat/tomcat -user tomcat -home /usr/local/thirdparty/java/j2sdk -pidfile /usr/local/thirdparty/jakarta-tomcat/conf/tomcat.pid -procname /home/tomcat/tomcat -outfile /usr/local/thirdparty/jakarta-tomcat/logs/catalina.out -errfile &1 -Djava.library.path=/usr/local/cm/lib:/usr/local/lib:/usr/local/thirdparty/java/j2sdk/jre/lib/i386:/usr/local/thirdparty/java/j2sdk/jre/lib/i386/server:/usr/lib/pgsql:/usr/lib:/usr/local/cm/lib::/usr/local/platform/lib -Djavax.net.ssl.trustStore=/usr/local/platform/.security/tomcat/trust-certs/tomcat-trust.keystore -Djavax.net.ssl.trustStorePassword=EncqLiRj8r8nYtHy -XX:ErrorFile=/usr/local/thirdparty/jakarta-tomcat/logs/diagnostic-info.jvm-crash.%p.tomcat.txt -Dsun.zip.disableMemoryMapping=true -XX:OnOutOfMemoryError=/home/tomcat/tomcat_diagnostics.sh -XX:OnError=/home/tomcat/tomcat_diagnostics.sh -Djdk.tls.ephemeralDHKeySize=2048 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/thirdparty/jakarta-tomcat/conf/logging.properties -Xmx1824m -Xms256m -XX:MaxPermSize=448m -Djava.endorsed.dirs=/usr/local/thirdparty/jakarta-tomcat/endorsed -cp :/common/download:/usr/local/platform/application_locale/platform-api:/usr/local/platform/application_locale/cmplatform:/usr/local/platform/application_locale:/usr/local/cm/application_locale/cmservices:/usr/local/cm/application_locale/car:/usr/local/cm/application_locale/ccmadmin:/usr/local/cm/application_locale/ucmuser:/usr/local/cm/application_locale:/etc/opt/cisco/elm/server/resource:/usr/local/thirdparty/jakarta-tomcat/bin/bootstrap.jar::/usr/local/thirdparty/jakarta-tomcat/bin/tomcat-juli.jar -Djava.security.policy==/usr/local/thirdparty/jakarta-tomcat/conf/catalina.policy -Dcatalina.base=/usr/local/thirdparty/jakarta-tomcat -Dcatalina.home=/usr/local/thirdparty/jakarta-tomcat -Djava.io.tmpdir=/usr/local/thirdparty/jakarta-tomcat/temp org.apache.catalina.startup.Bootstrap start
Press <enter> for 1 line, <space> for one page, or <q> to quit
()_(********************************************

admin:show process load
top - 00:06:46 up 63 days, 20:17, 2 users, load average: 9.95, 13.91, 13.97
Tasks: 270 total, 3 running, 267 sleeping, 0 stopped, 0 zombie
%Cpu(s): 41.4 us, 27.3 sy, 6.1 ni, 14.1 id, 6.1 wa, 0.0 hi, 5.1 si, 0.0 s
KiB Mem : 7960232 total, 192960 free, 3770224 used, 3997048 buff/cache
KiB Swap: 4095996 total, 2911300 free, 1184696 used. 3153296 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26443 tomcat 20 0 2708940 1.7g 10900 S 38.1 22.4 14384:55 tomcat
32044 ctftp 20 0 607200 23260 6692 S 38.1 0.3 9796:55 ctftp
30426 informix 10 -10 352540 219076 218384 R 33.3 2.8 6561:38 cmoninit
2200 admin 30 10 155672 2492 1748 R 23.8 0.0 0:00.16 top
29184 ccmserv+ 20 0 319868 38412 17292 S 23.8 0.5 10203:24 RisDC
3920 root -62 0 0 0 0 S 19.0 0.0 8354:00 kipvmsMixe
31808 ccmserv+ 20 0 567336 59244 8988 S 19.0 0.7 3141:19 CCMDirSync
2226 root 20 0 113452 1708 1304 R 14.3 0.0 0:00.03 hssi_api.s
29898 root 20 0 119576 18380 7936 S 14.3 0.2 909:20.87 CiscoSyslo
29988 certbase 20 0 97948 5192 5072 S 14.3 0.1 898:06.75 capf
admin:

********************************
admin:show process using-most memory
PasreOption method called
MEM(K) PID ARGS
128980 30054 /usr/local/cm/bin/PnPLauncher /usr/local/cm/conf/PnPcfg.xml
153804 29699 /usr/local/cm/bin/carschlr /usr/local/cm/conf/car/carschlrCfg.xml
212312 12303 /usr/sbin/rsyslogd -n
219088 30426 /usr/local/cm/bin/cmoninit -w
1785400 26443 /home/tomcat/tomcat -user tomcat -home /usr/local/thirdparty/java/j2sdk -pidfile /usr/local/thirdparty/jakarta-tomcat/conf/tomcat.pid -procname /home/tomcat/tomcat -outfile /usr/local/thirdparty/jakarta-tomcat/logs/catalina.out -errfile &1 -Djava.library.path=/usr/local/cm/lib:/usr/local/lib:/usr/local/thirdparty/java/j2sdk/jre/lib/i386:/usr/local/thirdparty/java/j2sdk/jre/lib/i386/server:/usr/lib/pgsql:/usr/lib:/usr/local/cm/lib::/usr/local/platform/lib -Djavax.net.ssl.trustStore=/usr/local/platform/.security/tomcat/trust-certs/tomcat-trust.keystore -Djavax.net.ssl.trustStorePassword=EncqLiRj8r8nYtHy -XX:ErrorFile=/usr/local/thirdparty/jakarta-tomcat/logs/diagnostic-info.jvm-crash.%p.tomcat.txt -Dsun.zip.disableMemoryMapping=true -XX:OnOutOfMemoryError=/home/tomcat/tomcat_diagnostics.sh -XX:OnError=/home/tomcat/tomcat_diagnostics.sh -Djdk.tls.ephemeralDHKeySize=2048 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/thirdparty/jakarta-tomcat/conf/logging.properties -Xmx1824m -Xms256m -XX:MaxPermSize=448m -Djava.endorsed.dirs=/usr/local/thirdparty/jakarta-tomcat/endorsed -cp :/common/download:/usr/local/platform/application_locale/platform-api:/usr/local/platform/application_locale/cmplatform:/usr/local/platform/application_locale:/usr/local/cm/application_locale/cmservices:/usr/local/cm/application_locale/car:/usr/local/cm/application_locale/ccmadmin:/usr/local/cm/application_locale/ucmuser:/usr/local/cm/application_locale:/etc/opt/cisco/elm/server/resource:/usr/local/thirdparty/jakarta-tomcat/bin/bootstrap.jar::/usr/local/thirdparty/jakarta-tomcat/bin/tomcat-juli.jar -Djava.security.policy==/usr/local/thirdparty/jakarta-tomcat/conf/catalina.policy -Dcatalina.base=/usr/local/thirdparty/jakarta-tomcat -Dcatalina.home=/usr/local/thirdparty/jakarta-tomcat -Djava.io.tmpdir=/usr/local/thirdparty/jakarta-tomcat/temp org.apache.catalina.startup.Bootstrap start

********************************************************

1 ACCEPTED SOLUTION

Accepted Solutions
Nithin Eluvathingal
VIP Advisor

EVE-NG is a simulation tool. don't expect the performance which you get on the real UCS server running vmware. For better performance you need to run CUCM on compatible hardware.

 

 

 

 



Response Signature


View solution in original post

6 REPLIES 6
Nithin Eluvathingal
VIP Advisor

EVE-NG is a simulation tool. don't expect the performance which you get on the real UCS server running vmware. For better performance you need to run CUCM on compatible hardware.

 

 

 

 



Response Signature


View solution in original post

Yes, i understand that we need to look at a better setup instead of EVE-NG

When preparing my CCIE I was using a Dell server to run the CUCM, it took half an  hour for all services to come up with just two end point.I used to go  for a coffee turning ON  my dell servers.  but with a UCS server I have no such issues. 

 

No where in the CUCM compatibility guides its mentioned that it support EVE-NG. so dont expect any support on this issue. only option is to use a compatibility hardware  or as I did go for a coffee turning ON the server

 

 



Response Signature


Yeah i have never faced such issues on UCS deployments and this is my first time working on a EVE-NG environment. Anyways , i will simply have to look to setup a UCS infra then.

 

Thanks

You seem to indicate that something has changed recently. If you've been on the same environment and the server has been up for some time, you could have a memory leak. Over the years I've seen many such instances call manager. If rebooting the system clears it for a while then there may be a bug.

 

https://bst.cloudapps.cisco.com/bugsearch/search?kw=Memory%20leak%20cucm&pf=prdNm&rls=12.5&sb=anfr

Nothing has changed in the recent past. Also, a reboot does not fix the issue. 

Create
Recognize Your Peers
Content for Community-Ad