RTMT Alert Name:CriticalServiceDown Detail: Service status is DOWN

[henrique.maia]

Hello,
I'm having a problem with CUCM 7.0.2 that some services changes to DOWN without any reason or intervention. I have 1 Publisher and 1 Subscriber.

Below are the log messages:

Oct 7 17:17:55, PRODCUCM, Critical, Cisco AMC Service, : 26534: Oct 07 20:17:55.471 UTC : %CCM_RTMT-RTMT-2-RTMT-ERROR-ALERT: RTMT Alert Name:CriticalServiceDown Detail: Service status is DOWN. Cisco Messaging Interface. The alert is generated on Thu Oct 07 17:17:55 BRT 2010 on node 10.21.5.21. Cluster ID: Node  ID:PRODCUCM, 76

Oct 7 17:18:26, GAVPRDCMG, Critical, Cisco AMC Service, : 26535: Oct 07 20:18:26.51 UTC : %CCM_RTMT-RTMT-2-RTMT-ERROR-ALERT: RTMT Alert Name:CriticalServiceDown Detail: Service status is DOWN. Cisco Messaging Interface. The alert is generated on Thu Oct 07 17:18:26 BRT 2010 on node 10.21.5.22. Cluster ID: Node ID:PRODCUCM, 77

Oct 7 17:18:26, PRODCUCM, Critical, Cisco AMC Service, : 26536: Oct 07 20:18:26.53 UTC : %CCM_RTMT-RTMT-2-RTMT-ERROR-ALERT: RTMT Alert Name:CriticalServiceDown Detail: Service status is DOWN. Cisco Messaging Interface. The alert is generated on Thu Oct 07 17:18:26 BRT 2010 on node 10.21.5.21. Cluster ID: Node  ID:PRODCUCM, 78

Oct 7 17:19:06, PRODCUCM, Critical, Cisco AMC Service, : 26537: Oct 07 20:19:06.286 UTC : %CCM_RTMT-RTMT-2-RTMT-ERROR-ALERT: RTMT Alert Name:CriticalServiceDown Detail: Service status is DOWN. Cisco Messaging Interface. The alert is generated on Thu Oct 07 17:19:06 BRT 2010 on node 10.21.5.22. Cluster ID: Node  ID:PRODCUCM, 79

Oct 7 17:19:06, PRODCUCM, Critical, Cisco AMC Service, : 26538: Oct 07 20:19:06.288 UTC : %CCM_RTMT-RTMT-2-RTMT-ERROR-ALERT: RTMT Alert Name:CriticalServiceDown Detail: Service status is DOWN. Cisco Messaging Interface. The alert is generated on Thu Oct 07 17:19:06 BRT 2010 on node 10.21.5.21. Cluster ID: Node  ID:PRODCUCM, 80

From CUCM (cli):

admin:utils service list

Requesting service status, please wait...

System SSH [STARTED]

Cluster Manager [STARTED]

Service Manager is running

Getting list of all services

>> Return code = 0

A Cisco DB[STARTED]

A Cisco DB Replicator[STARTED]

Cisco AMC Service[STARTED]

Cisco AXL Web Service[STOPPED]  Commanded Out of Service

Cisco Bulk Provisioning Service[STARTED]

Cisco CAR Scheduler[STARTED]

Cisco CAR Web Service[STOPPED]  Commanded Out of Service

Cisco CDP[STARTED]

Cisco CDP Agent[STARTED]

Cisco CDR Agent[STARTED]

Cisco CDR Repository Manager[STARTED]

Cisco CTIManager[STARTED]

Cisco CTL Provider[STARTED]

Cisco CallManager[STARTED]

Cisco CallManager Admin[STOPPED]  Commanded Out of Service

Cisco CallManager Attendant Console Server[STARTED]

Cisco CallManager Cisco IP Phone Services[STOPPED]  Commanded Out of Service

Cisco CallManager Personal Directory[STOPPED]  Commanded Out of Service

Cisco CallManager SNMP Service[STARTED]

Cisco CallManager Serviceability[STOPPED]  Commanded Out of Service

Cisco CallManager Serviceability RTMT[STOPPED]  Commanded Out of Service

Cisco Certificate Authority Proxy Function[STARTED]

Cisco Certificate Expiry Monitor[STARTED]

Cisco DHCP Monitor Service[STARTED]

Cisco DRF Local[STARTED]

Cisco DRF Master[STARTED]

Cisco Database Layer Monitor[STARTED]

Cisco Dialed Number Analyzer[STOPPED]  Commanded Out of Service

Cisco DirSync[STARTED]

Cisco Extended Functions[STARTED]

Cisco Extension Mobility[STOPPED]  Commanded Out of Service

Cisco Extension Mobility Application[STOPPED]  Commanded Out of Service

Cisco IP Manager Assistant[STOPPED]  Commanded Out of Service

Cisco IP Voice Media Streaming App[STARTED]

Cisco License Manager[STARTED]

Cisco Log Partition Monitoring Tool[STARTED]

Cisco Messaging Interface[STOPPED]  Component is not running

Cisco RIS Data Collector[STARTED]

Cisco RTMT Reporter Servlet[STOPPED]  Commanded Out of Service

Cisco SOAP - CDRonDemand Service[STOPPED]  Commanded Out of Service

Cisco Serviceability Reporter[STARTED]

Cisco Syslog Agent[STARTED]

Cisco TAPS Service[STARTED]

Cisco Tftp[STARTED]

Cisco Tomcat[STOPPED]  Component is not running

Cisco Tomcat Stats Servlet[STOPPED]  Commanded Out of Service

Cisco Trace Collection Service[STARTED]

Cisco Trace Collection Servlet[STOPPED]  Commanded Out of Service

Cisco UXL Web Service[STOPPED]  Commanded Out of Service

Cisco Unified Mobile Voice Access Service[STOPPED]  Commanded Out of Service

Cisco WebDialer Web Service[STOPPED]  Commanded Out of Service

Host Resources Agent[STARTED]

MIB2 Agent[STARTED]

Native Agent Adapter[STARTED]

SNMP Master Agent[STARTED]

SOAP -Log Collection APIs[STOPPED]  Commanded Out of Service

SOAP -Performance Monitoring APIs[STOPPED]  Commanded Out of Service

SOAP -Real-Time Service APIs[STOPPED]  Commanded Out of Service

System Application Agent[STARTED]

Anyone have any idea?

Best Regards

[David Hailey]

The logs are all for the Cisco Messaging Interface which is very rarely used.  You most likely have activated the service and it is shutting down.  You should deactivate this service.

For the CLI, "commanded out of service" means the service is deactivated and hence, cannot start.

Hailey

Please rate helpful posts!

[henrique.maia]

Thank you for your reply David.

If I deactivate the Cisco Messaging service the log messages will stop.  ok

I have 1 Publisher and 1 Subscriber, when the services goes DOWN my IP Phones stop working and don't register in the Subscriber because the Publisher is still ACTIVE.

Do you have any other troubleshooting that I can use to know why the services are going DOWN?

[David Hailey]

The CMI service has nothing to do with phone registration, so that's not causing you any issue.

You have a number of critical services that are not running on your servers.  The CCM service is running which would allow phones to register.  However, the other services are largely administrative - but critical.  The first thing that comes to mind is that you are running a 7.0 version of code which is largely asking for problems with bugs.  So, once you get this squared away - I'd upgrade to 71.(3b)SU2. For now, a large number of the services you've mentioned are dependent upon Cisco Tomcat which is stopped.  You can only start this service from the CLI.

The first thing I'd do is to reboot the Publisher, then the Subscriber.  You should then check to see that Tomcat and the other services that are stopped now start after reboot.

If not, you can attempt to start services manually (starting with Tomcat) via CLI:

utils service start (Use the name as displayed in the utils service list output)

Hailey

Please rate helpful posts!

[henrique.maia]

Hi,

Yestesday night we started all the services but after 10 minutes all the same services changed to DOWN again!!!

If I upgrade the CUCM to version 7.1 will the licences still work?

Regards

[David Hailey]

Yes, a jump from 7.0 to 7.1 is fine without changing licenses.  You are going to have to do the upgrade via CLI though.  I have a blog on that here:

http://www.netcraftsmen.net/resources/blogs/cucm-7x-application-upgrade---updates-to-the-cli-method.html?blogger=David+Hailey

This actually sounds like a potential memory issue.  What servers are you running on?  How much memory do they have?

Before you upgrade to 7.1, I would focus in on finding out if any bugs exist in your current 7.0 version related to Tomcat or web services not working properly.  If not, upgrading to 7.1 may not do you much good.

Is this your production system?

Hailey