CUCM 11 Server Down

[cristian.munoz]

Hi All

I have the problem follow:

In the Cluster, a CUCM Subscriber has been boot a couple of times. In the RTMT only show the alert detail Server Down Ocurred. How to know more details for this issue?.

TIA
Cristian

[Hari Prasad]

Hi Cristian,

Please check the status of CCM SUB node from ESXi and check if you see any error for this node on ESXi

In additon to this, if server is reachable, you can ssh to the server and check the status with below command

show status

utils service list

utils diagnose test

If affected server in on single UCS box where no other node configured , you can verify the health of UCS through CIMC to know if any specific issue with UCS box.

Thanks

Hari    

[Rajan]

Hi Cristian,

You need to check the status of all activated services on that subscriber. Also you could pull the below logs for that server from RTMT to check for any other alarms:

Event viewer - application logs
Event viewer - system logs

HTH
Rajan
Pls rate all useful posts by clicking the star below

[cristian.munoz]

---

Thank you Rajan

In the Event viewer - system logs , the cause of the problem is shown:

Aug 27 16:39:27, CUCMSUB01-11, Error, Cisco CallManager, : 98: CUCMSUB01-11 : Aug 27 2019 19:39:27.606 UTC : %UC_CALLMANAGER-3-ICTCallThrottlingStart: %[DeviceName=ICT_8][IPAddress=10.113.227.11][DeviceType=125][Description=ICT to CUCM 8.6][ClusterID=StandAloneCluster][NodeID=CUCMSUB01-11]: Unified CM stops handling calls for the indicated H.323 device due to heavy traffic or a route loop over the H.323 trunk, 3796

Have a nice day.

Cristian

[Hari Prasad]

Hi Cristinan,

The logs you have pasted looks to be related to Trunk ICT_8.

It does not looks relevant to SUB Down.

Please share logs here to check.

[cristian.munoz]

Hi Hari

Here the syslog

[tgaur]

Hi,

can you copy paste the alert you are referring to, also provide the O/P of following :

show status

file view install system-history.log

Regards

Tarun

[cristian.munoz]

Dear Tarun

Here the info:

admin:show status

Host Name : CUCMSUB01-11
Date : Sun Sep 1, 2019 12:52:12
Time Zone : XXX Summer Time
Locale : en_US.UTF-8
Product Ver : 11.5.1.13901-3
Unified OS Version : 6.0.0.0-2

Uptime:
12:52:14 up 4 days, 23:02, 1 user, load average: 0.96, 0.29, 0.20

CPU Idle: 41.41% System: 14.14% User: 44.44%
IOWAIT: 00.00% IRQ: 00.00% Soft: 00.00%

Memory Total: 5994160K
Free: 155992K
Used: 5838168K
Cached: 2612308K
Shared: 305760K
Buffers: 204588K

Total Free Used
Disk/active 14154228K 984068K 13025032K (93%)
Disk/inactive 14154228K 1304988K 12704112K (91%)
Disk/logging 49573612K 22613376K 24435308K (52%)

admin:
admin:file view install system-history.log

=======================================
Product Name - Cisco Unified Communications Manager
Product Version - 11.5.1.13901-3
Kernel Image - 2.6.32-573.18.1.el6.x86_64
=======================================
09/07/2018 06:59:50 | root: Install 11.5.1.10000-6 Start
09/07/2018 02:42:03 | root: Boot 11.5.1.10000-6 Start
09/07/2018 14:59:00 | root: Install 11.5.1.10000-6 Success
09/07/2018 14:59:02 | root: Boot 11.5.1.10000-6 Start
09/12/2018 10:47:40 | root: Cisco Option Install cm-locale-es_CO-11.5.1.6000-1.cop Start
09/12/2018 10:48:14 | root: Cisco Option Install cm-locale-es_CO-11.5.1.6000-1.cop Success
09/12/2018 13:51:42 | root: Restart 11.5.1.10000-6 Start
09/12/2018 13:52:05 | root: Boot 11.5.1.10000-6 Start
10/08/2018 11:49:13 | root: Cisco Option Install cmterm-69xx-SCCP.9-4-1-3SR3.k3.cop Start
10/08/2018 11:49:17 | root: Cisco Option Install cmterm-69xx-SCCP.9-4-1-3SR3.k3.cop Success
10/08/2018 16:06:06 | root: Cisco Option Install cmterm-69xx-SCCP.9-4-1-3SR2.k3.cop Start
10/08/2018 16:06:10 | root: Cisco Option Install cmterm-69xx-SCCP.9-4-1-3SR2.k3.cop Success
03/06/2019 21:52:14 | root: Restart 11.5.1.10000-6 Start
03/06/2019 21:52:41 | root: Boot 11.5.1.10000-6 Start
03/17/2019 17:25:22 | root: Upgrade 11.5.1.13901-3 Start

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 43) :
03/17/2019 18:10:10 | root: Upgrade 11.5.1.13901-3 Success
03/17/2019 19:57:08 | root: Switch Version 11.5.1.10000-6 to 11.5.1.13901-3 Start
03/17/2019 19:59:25 | root: Switch Version 11.5.1.10000-6 to 11.5.1.13901-3 Success
03/17/2019 19:59:25 | root: Product Version 11.5.1.13901-3
03/17/2019 19:59:25 | root: Kernel Image 2.6.32-573.18.1.el6.x86_64
03/17/2019 19:59:27 | root: Restart 11.5.1.13901-3 Start
03/17/2019 19:59:27 | root: Restart 11.5.1.10000-6 Start
03/17/2019 20:01:28 | root: Boot 11.5.1.13901-3 Start
04/12/2019 17:01:26 | root: Boot 11.5.1.13901-3 Start
05/06/2019 21:44:03 | root: Boot 11.5.1.13901-3 Start
07/18/2019 08:11:39 | root: Boot 11.5.1.13901-3 Start
07/18/2019 08:19:28 | root: Restart 11.5.1.13901-3 Start
07/18/2019 08:20:01 | root: Boot 11.5.1.13901-3 Start
07/19/2019 14:06:47 | root: Boot 11.5.1.13901-3 Start
07/26/2019 09:53:52 | root: Boot 11.5.1.13901-3 Start
07/26/2019 16:50:13 | root: Boot 11.5.1.13901-3 Start
07/29/2019 04:01:01 | root: Shutdown 11.5.1.13901-3 Start
07/29/2019 04:06:20 | root: Boot 11.5.1.13901-3 Start
08/06/2019 11:35:49 | root: Boot 11.5.1.13901-3 Start
08/06/2019 16:36:15 | root: Boot 11.5.1.13901-3 Start

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 43) :
08/20/2019 17:59:59 | root: Boot 11.5.1.13901-3 Start
08/23/2019 10:53:34 | root: Boot 11.5.1.13901-3 Start
08/27/2019 13:49:44 | root: Boot 11.5.1.13901-3 Start

end of the file reached

[cristian.munoz]

Here more..

[tgaur]

Hi,

08/06/2019 11:35:49 | root: Boot 11.5.1.13901-3 Start
08/06/2019 16:36:15 | root: Boot 11.5.1.13901-3 Start

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 43) :
08/20/2019 17:59:59 | root: Boot 11.5.1.13901-3 Start
08/23/2019 10:53:34 | root: Boot 11.5.1.13901-3 Start
08/27/2019 13:49:44 | root: Boot 11.5.1.13901-3 Start

this BOOT means the system going down or non-graceful shutdown from ESXi, please check if we have an error is ESXi/server where these VM's are running.

Open a TAC/vmware case if required for detail analysis on this.

Regards

Tarun