Service operational status is DOWN. Cisco SOAP - CallRecord Service.

[dimensyssrl]

Hello.

I've successfully upgraded CallManager from version 6.1.5 (hw on mcs) to 9.1.2 (virtual on UCS).

All seems ok.

My only doubt is about an error that I receive on RTMT:

Service operational status is DOWN. Cisco SOAP - CallRecord Service.

This error cycle continuosly, with another one related:

the following SyslogSeverityMatchFound events generated:  SeverityMatch : Critical MatchedEvent : Jan 18 11:12:48 CCM01 local7 2 : 1059: CCM01.local: Jan 18 2014 10:12:48.430 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco SOAP - CallRecord Service][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CCM01]: Service terminated. AppID : Cisco Syslog Agent

Anyone can help me?

Thanks

Daniele

[dimensyssrl]

Another information: the error occurs every 3 or 4 hours, than the service restart...

I can't understand why this happen.

Any suggestion will be appreciated.

Daniele

[dimensyssrl]

Please, anyone can't help how to troubleshoot this problem?

[Nadeem Ahmed]

Hello

Can you please check the system having core and check system status with below command

utils dbreplication runtimestate (run on the pub of the cluster)
Utils diagnose test
Utils core active list

SOAP-Real-Time services APIs allow you to collected real time information for devices and CTI
applicaitons,though its an optional servies and it has not impact when it goes down, if the servies
goes down its normal event that could have been due to API's not recieved and to recover them it restarted.

Br,
Nadeem 

Please rate all useful post.

[dimensyssrl]

Hello Nadeem.

Thanks for your reply.

I attach here commands output:

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

DB CLI Status: No other dbreplication CLI is running...

Cluster Replication State: BROADCAST SYNC Completed on 2 servers at: 2014-01-14-11-20

     Last Sync Result: SYNC COMPLETED  603 tables sync'ed out of 603

     Sync Errors: NO ERRORS

DB Version: ccm9_1_2_11900_12

Repltimeout set to: 300s

PROCESS option set to: 1

Cluster Detailed View from CCM01 (3 Servers):

                                           PING                          CDR Server             REPL.          DBver&           REPL.          REPLICATION SETUP

SERVER-NAME          IP ADDRESS            (msec)          RPC?          (ID) & STATUS           QUEUE          TABLES           LOOP?          (RTMT) & details

-----------          ------------          ------          ----          --------------          -----          -------          -----          -----------------

CCM01          192.168.57.7          0.061          Yes          (2)  Connected           0            match          Yes            (2) PUB Setup Completed

CCM02          192.168.57.11          0.380          Yes          (3)  Connected           0            match          Yes            (2) Setup Completed

CCM03          192.168.57.26          0.185          Yes          (5)  Connected           0            match          Yes            (2) Setup Completed

admin:utils diagnose test

Log file: platform/log/diag1.log

Starting diagnostic test(s)

===========================

test - disk_space          : Passed (available: 2652 MB, used: 11503 MB)

skip - disk_files          : This module must be run directly and off hours

test - service_manager     : Passed                     

test - tomcat              : Passed                     

test - tomcat_deadlocks    : Passed                     

test - tomcat_keystore     : Passed                     

test - tomcat_connectors   : Passed                     

test - tomcat_threads      : Passed                     

test - tomcat_memory       : Passed                     

test - tomcat_sessions     : Passed                     

skip - tomcat_heapdump     : This module must be run directly and off hours

test - validate_network    : Passed                     

test - raid                : Passed                     

test - system_info         : Passed (Collected system information in diagnostic log)

test - ntp_reachability    : Passed              

test - ntp_clock_drift     : Passed              

test - ntp_stratum         : Passed              

skip - sdl_fragmentation   : This module must be run directly and off hours

skip - sdi_fragmentation   : This module must be run directly and off hours

test - ipv6_networking     : Passed              

Diagnostics Completed

admin:utils core active list

No core files found

admin:

I know that SOAP service can fail and restart itself, but it happens too frequently...

Thanks

Daniele

[Nadeem Ahmed]

Did you try to restart the services. can try that and check whether this having frequent after this

You can s restart  for the following page: Cisco Unified
Serviceability> Tools> Control Center - Network Services> SOAP Services>
SOAP -Performance Monitoring APIs.

Br,
Nadeem 

Please rate all useful post.

[dimensyssrl]

I've tried to restart service yet.

Service die and restart about every two hours...

I can't understand why this happens

[dimensyssrl]

Any other suggestions?

[Carlo Poggiarelli]

Hi Daniele.
Try to put to false run on flag value of call detail record service parameter and restart Cisco Soap Call record network service.

Let me know

Regards

Carlo

Sent from Cisco Technical Support iPhone App

[Trent McKee]

We were receiving the SOAP alerts on one server in our cluster of 4 Subs. Restarting the SOAP services that were already restarting did nothing, but restarting the Tomcat service did the trick without having to regenerate the Tomcat certificate.

[System Administration · Accepted solution]

Hi Daniele,

have you been able to resolve this?

We're facing similar errors after upgrading from 9.1.2.11900-12 to 9.1.2.12900-11:

ServiceName=Cisco SOAP - CallRecord Service and ServiceName=Cisco SOAP - CDRonDemand Service are crashing periodically every 5 minutes.

Thank you,

Helge

[tom golan]

Hi Guys

Did you manage to solve this ?

Thanks,

Tom.

[System Administration]

Resolution Summary:

Regenerated Tomcat certificate resolved the issue.

Cheers, Helge

[tom golan]

Hi Helge,

Thanks for the response.

I tried that and unfortunately the service is still crashing randomly. 

Any other thoughts ? i'm running CUCM 10.5.2.

Tom.

[dianareyes]

Hi All,

I am coming across the same issue and wondering if there is a known solution for this or if any of you manage to solve your issues, if so any suggestions?

So far, I have performed the following;

utils dbreplication runtimestate (run on the pub of the cluster)
Utils diagnose test
Utils core active list

I did not find anything out of the norm, everything passed, no core dumps found, and dbreplication status is "(2) Setup Completed".  I've reviewed the Tomcat certificates and all are up to date, none expired.  

Any insight or suggestions are greatly appreciated.

Thanks in Advance!

Diana