CUCM 8.6.2a SU2 Cisco CAR Scheduler stopped abruptly

[Erick Bergquist]

Anyone seen this before on 8.6.2a SU2 (22900-9)

The Cisco CAR Scheduler stopped abruptly due to memory issue. I am not finding any bugs on this in the bug toolkit or release notes for SU3.

There is bug CSCtn99418 which is for the CARIDSPerfmon error but that bug is fixed in earlier 8.6.2 version.

There are no core dumps for this.

%UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco CAR Scheduler][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CM1]: Service terminated. AppID : Cisco Syslog Agent ClusterID : NodeID : CM1 TimeStamp : Thu Jan 09 14:13:24 CST 2014][AppID=Cisco AMC Service][ClusterID=][NodeID=CM1]:

%[FailureDetail=CAR Scheduler restarted, update ALL 2 job(s) with status: '-1','-3' to status: '-2'.][FailureCause=Update unfinished (InProgress / Scheduled) jobs from tbl_event_log.][JobName=2 VARIOUS JOBS][AppID=Cisco CAR Scheduler][ClusterID=][NodeID=CM1]: CAR scheduled job failed

%[ServiceName=Cisco CAR Scheduler][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CM1]: Service terminated.

%[AlertName=CARSchedulerJobFailed][AlertDetail= FailureDetail : JVM status: [Max = 455.12MB, Total = 368.62MB, Free = 2.19MB (0.59%), Used = 366.43MB (99.41%)]. FailureCause : OutOfMemoryError while running the job [CARIDSPerfmon]. JobName : CARIDSPerfmon AppID : Cisco CAR Scheduler ClusterID : NodeID : CM1 TimeStamp : Thu Jan 09 14:13:12 CST 2014. The alarm is generated on Thu Jan 09 14:13:12 CST 2014.][AppID=Cisco AMC Service][ClusterID=][NodeID=CM1]:

%[FailureDetail=JVM status: [Max = 455.12MB, Total = 368.62MB, Free = 2.19MB (0.59%), Used = 366.43MB (99.41%)].][FailureCause=OutOfMemoryError while running the job [CARIDSPerfmon].][JobName=CARIDSPerfmon][AppID=Cisco CAR Scheduler][ClusterID=][NodeID=CM1]: Critical CAR scheduled job failed.

[Manish Gogna]

Hi,

The OutOfMemory alert points to the following

https://tools.cisco.com/bugsearch/bug/CSCta71863/?reffering_site=dumpcr

Symptom:

High memory usage on CAR Scheduler (carschlr) during auto-purging of partitions.

From "top" command:

PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND

9182 ccmservi  17   0    6 142:11.79 14.8  459m 300m 7116 S carschlr

Possibly,  the "OutOfMemoryError" message can be caught in the CAR scheduler  traces. And/or Total JVM memory usage is close to Max JVM memory  allocated for CAR scheduler. For example:

... JVM status: [Max = 246.56MB, Total = 244.88MB, Free = 3.87MB (1.58%), Used = 241.01MB (98.42%)]

...OutOfMemoryError seen somewhere during CAR Loader Run - Round #2648

Conditions:

When  MAX_CAR_DATABASE_AGE setting is set lower than 34 (31+today+2) days,  and loader is actively running with relatively light traffic. This  setting is controlled via the CAR admin System -> Database ->  Configure Auto purge page. Max Age of Call Detail Records below 34 days  can lead to this condition.

Workaround:

Simply raise  the  MAX_CAR_DATABASE_AGE setting (configurable in CAR's automatic-purge  page), to either its default of 60 days or a value higher than 34.

Further Problem Description:

When  traffic is low, it does not hurt to keep the MAX age at the default of  60 days.  General speaking, raise the MIN/MAX age setttings when call  traffic is low, and lower the MIN/MAX age settings when traffic is high.

The bug is still not fixed , can you try applying the workaround if the setting falls under 34 days.

HTH

Manish

[Erick Bergquist]

The setting is at default (60 days) and still failing. The CAR Scheudler fails like this daily.

[Manish Gogna]

In that case i would suggest engaging TAC if this is a Production system, unless someone else wants to add to this post.

Manish