UCM 8.5.1
Unity Connection 8.5.1
Here's the background. I have two CUC servers in a cluster. About a week ago we started noticing extremely slow voicemail response (upwards of 30 seconds). During this time the web page and SSH response from the servers was extremely slow. To the point I could not manage them and had to hard boot the primary CUC server. This fixed the problem for 5 days then the same problem came back. Except this time I could pull up the web page and SSH to the backup CUC server fine. I still had problems accessing the primary CUC server and again hard booted it. This time it did not come back online so I'm running on the backup now as primary. We're still exhibiting the same delay except it is not on all calls this time. Sometimes when I access voicemail it picks up right away and works fine. Other times I see the long delay. Could I be hitting ports from the UCM to the CUC that are down because of the primary server being powered off? I've been working with TAC and they suspected the IMAP process that was showing high CPU utilization but we turned that off and the issue is still here.
Here's some output from the backup CUC server that is online:
admin:show status
Host Name : CUC-02
Date : Wed Jan 15, 2014 07:24:45
Time Zone : Eastern Standard Time (America/New_York)
Locale : en_US.UTF-8
Product Ver : 8.5.1.10000-26
OS Ver : 4.0.0.0-44
License MAC : E41F137781F3
Uptime:
07:24:47 up 153 days, 13 min, 1 user, load average: 0.58, 0.39, 0.24
CPU Idle: 97.50% System: 00.50% User: 01.50%
IOWAIT: 00.50% IRQ: 00.00% Soft: 00.00% Intr/sec: 1310.10
Memory Total: 2073260K
Free: 25332K
Used: 2047928K
Cached: 556944K
Shared: 0K
Buffers: 10768K
Total Free Used
Disk/active 43264144K 34715168K 8109440K (19%)
Disk/inactive 43264176K 35239096K 7585544K (18%)
Disk/logging 150538756K 101784728K 41107092K (29%)
admin:show process load most-used cpu
top - 07:24:58 up 153 days, 14 min, 1 user, load average: 0.49, 0.38, 0.23
Tasks: 193 total, 1 running, 179 sleeping, 0 stopped, 13 zombie
Cpu(s): 3.4% us, 1.9% sy, 0.2% ni, 94.1% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 2073260k total, 2051256k used, 22004k free, 11044k buffers
Swap: 2048248k total, 1025052k used, 1023196k free, 558492k cached
PID USER PR NI RES SHR S %CPU %MEM TIME COMMAND
4898 ccmservi 16 0 24m 12m S 2.0 1.2 38,45 RisDC
12582 administ 25 10 1148 804 R 2.0 0.1 0:00 top
14191 root 16 0 9.8m 8468 S 2.0 0.5 27:43 CiscoSyslogSubA
1 root 16 0 704 644 S 0.0 0.0 0:11 init
2 root RT 0 0 0 S 0.0 0.0 0:15 migration/0
3 root 34 19 0 0 S 0.0 0.0 0:00 ksoftirqd/0
4 root RT 0 0 0 S 0.0 0.0 0:19 migration/1
5 root 34 19 0 0 S 0.0 0.0 0:00 ksoftirqd/1
6 root 5 -10 0 0 S 0.0 0.0 0:00 events/0
7 root 5 -10 0 0 S 0.0 0.0 0:00 events/1
admin:show process load most-used memory
top - 07:25:14 up 153 days, 14 min, 1 user, load average: 0.38, 0.36, 0.23
Tasks: 193 total, 2 running, 178 sleeping, 0 stopped, 13 zombie
Cpu(s): 3.4% us, 1.9% sy, 0.2% ni, 94.1% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 2073260k total, 2047776k used, 25484k free, 10256k buffers
Swap: 2048248k total, 1025036k used, 1023212k free, 555436k cached
PID USER PR NI RES SHR S %CPU %MEM TIME COMMAND
29989 tomcat 17 0 477m 14m S 35.3 23.6 229,09 tomcat
26999 informix 6 -10 157m 157m S 0.0 7.8 630:35 unityoninit
4876 drf 16 0 102m 7860 S 0.0 5.1 0:52 CiscoDRFLocal
4931 drf 16 0 98m 10m S 0.0 4.9 0:59 CiscoDRFMaster
23831 informix 16 0 88m 87m S 0.0 4.4 21,53 cmoninit
27107 cumixer -61 0 85m 20m S 2.0 4.2 26:22 CuMixer
763 cusysage 17 0 65m 11m S 0.0 3.2 28:41 CuSysAgent
25117 cumta 18 0 58m 13m S 0.0 2.9 1:02 CuMta
5037 cusrm 15 0 55m 11m S 0.0 2.8 80:35 CuSrm
27151 cusmtpsv 19 0 54m 13m S 0.0 2.7 0:16 CuSmtpSvr
admin:show cuc dbserver disk page
Dbspaces
========
Dbspace Dbspace Size Used Free Percent
Number Name MB MB MB Free
------- ---------------------- ---- ---- ---- -------
1 rootdbs 391 101 289 74
2 cuc_er_sbspace 2750 371 2379 86
3 cuc_er_dbspace 250 5 245 98
4 dir16 700 174 526 75
5 phys 600 600 0 0
6 rpt 3500 520 2980 85
7 temp 1000 0 1000 99
8 mbx16 1000 30 970 97
9 temp2 1000 0 1000 99
10 dyn 1500 696 804 53
11 mbx 250 25 225 89
12 sadmin 100 3 97 96
13 dir 4500 682 3818 84
14 log 640 640 0 0
15 ciscounity_sbspacetemp 960 65 895 93
16 ciscounity_sbspace 20 1 18 92
Chunks
======
Size Free
Chunk Offset MB MB Path
----- ------ ---- ---- ---------------------------------------------------
1 0 391 289 /var/opt/cisco/connection/db/root_dbspace
2 0 2750 2379 /var/opt/cisco/connection/db/cuc_er_sbspace
3 0 250 245 /var/opt/cisco/connection/db/cuc_er_dbspace
4 0 700 526 /var/opt/cisco/connection/db/dir16_dbs
5 0 600 0 /var/opt/cisco/connection/db/phys_dbs
6 0 3500 2980 /var/opt/cisco/connection/db/rpt_dbs
7 0 1000 1000 /var/opt/cisco/connection/db/temp_dbs
8 0 1000 970 /usr/local/cm/db/informix/data/mbx16_dbs
9 0 1000 1000 /var/opt/cisco/connection/db/temp2_dbs
10 0 1500 804 /var/opt/cisco/connection/db/dyn_dbs
11 0 250 225 /usr/local/cm/db/informix/data/mbx_dbs
12 0 100 97 /var/opt/cisco/connection/db/sadmin_dbs
13 0 4500 3818 /var/opt/cisco/connection/db/dir_dbs
14 0 640 0 /var/opt/cisco/connection/db/log_dbs
15 0 960 895 /var/opt/cisco/connection/db/ciscounity_sbspacetemp
16 0 20 18 /var/opt/cisco/connection/db/ciscounity_sbspace
admin:utils core active list
Size Date Core File Name
=================================================================
138520 KB 2012-06-02 15:19:19 core.672.6.CuMta.1338664731
191488 KB 2012-06-02 15:19:23 core.715.6.CuNotifier.1338664732
339048 KB 2012-05-19 07:29:02 core.13398.11.cimserver.1337426941
admin:
