CUCM service operational service is down

[mtaneja712]

Hi All,

In my organisation suddenly all of the ip phones goes to the registering stage for few minutes and comes up.

During the outage time i was able to ping the server but i was not able to get the concsole and RTMT of server.

After outage when i checked the alert on RTMT i got following logs for Publisher server

"
 Service operational status is DOWN. Cisco AXL Web Service;Cisco CAR Web Service;Cisco CallManager Admin;Cisco CallManager Cisco IP Phone Services;Cisco CallManager Personal Directory;Cisco CallManager Serviceability;Cisco CallManager Serviceability RTMT;Cisco Change Credential Application;Cisco Dialed Number Analyzer;Cisco Extension Mobility;Cisco Extension Mobility Application;Cisco RTMT Reporter Servlet;Cisco Tomcat Stats Servlet;Cisco Trace Collection Servlet;Cisco Unified Mobile Voice Access Service;Cisco User Data Services;Cisco WebDialer Web Service;Platform SOAP Services. The alert is generated on Fri Mar 13 13:16:41 IST 2015 on node 10.31.220.1."

Could someone please help me troubleshot what goes wrong and how to stop this in future

Thanks,

Mohit

[Terry Cheema]

Please mention version of your CUCM and platform.

Are the phones and the server on the same site? If not, first of all confirm network connectivity was not broken.

Secondly go to the CLI of your primary server (or if not sure, on all servers) and run:

utils core active list

post the output of the command.

Also check from the RTMT - your event viewer log and check if any useful information shows up there.

-Terry

Please rate helpful posts

[Aditya Gupta]

Hi Mohit

There can be various reasons to the problem you are facing. As Terry mentioned above Core can be one of the suspects ,  there can be others as well such  high cpu/memory/any process/application  internal or external , eating away the resources.or a network flap, I/O issues etc. For such kind of problem it is very important to identify :

1. Which cucm server/node faced the problem and what is exact version (this is important to identify any defect)

2. Date/Time Stamp of the issue

3. When you say all the phones got unregistered, were they related to single site ( Terry mentioned above) or were they all registered to single CUCM Node ?

4. Event Viewer Logs from same the date/time stamp and what error are they throwing.

5. Was any action taken to resolve the issue ?

6. What kind of server are you using (MCS or UCS)? 

It's best suggested you can take all this information and open a TAC SR and they would definitely help you identify the root cause.

Regards

Aditya

[mtaneja712]

Hi Aditya ,

Here are the answer to your question

1) Publisher faced the problem and Version 8.6.2.20000-2

2)  Fri Mar 13 13:16:41 IST 2015

3)The publisher and IP phones are on the same site (territory). Few phones are registered on publisher and few are on subscriber. The alert are for publisher only.

4) Log timestamp is same when the incident happend

5) We haven't taken any action. The phones comes up after 2-3 mins. Now when i checked at RTMT i can see that only 1 sccp phone  is registered on Publisher , rest are on subscriber

6) Its a virtual installation on VMware ESXi

Please help me to troubleshoot it furthur.

Thanks,

Mohit

[Aditya Gupta]

Hi Mohit

Please attach the event viewer logs from the time when the issue happened.

Regards

Aditya Gupta

[Terry Cheema · Accepted solution]

Mohit,

Read my post, I asked for an output - that was to check if there is any service crash on your server, specially call manager service, that would dump a core file generally if it happens.

Secondly go through the RTMT event viewer logs to get an idea and also check from RTMT alert central, the services will show a date and time stamp for any alerts that were last triggered to get you an idea of which alerts were triggered.

You can also go to CLI:

 file view activelog syslog/CiscoSyslog*

file view activelog syslog/AlternateSyslog*

to view from CLI or use get command to download the logs.

But I would suggest to open a TAC case, TAC can review all the logs including perfmon etc. to find out whats causing the issue.

-Terry

Please rate all helpful posts.