Showing results for 
Search instead for 
Did you mean: 

Troubleshooting Core Dumps


    Core dumps occur when a Linux process experiences a fault.  This results in an outage of the affected process or service.  The process or service must restart to recover.  During these incidents, the server may remain up, but certain services may experience a brief outage.  This document covers troubleshooting core dumps and backtraces that may occur on Communications Manager(CM or CUCM), Unity Connection(UC), Cisco Emergency Responder(CER), Cisco Unified Presence Server(CUPS), Cisco Unified Contact Center Express (UCCX or IPCC Express), or any product based on Cisco's Voice Operating System (VOS) appliance model.



    Identifying Core Dump Events

    Two of the most common ways in which to identify the occurrence of a core dump in CallManager are the following:


    • CoreDumpFileFound RTMT alert messages found in Alert Central



             Within RTMT Alert Central, more detail on the specific application that generated the core can be found by right-clicking

             on the alert selection.  An example of the core dump alert details information can be found below:


    • Application Event log alert messages indicating a core dump has occurred:

    May 15 05:32:09 ccm-pub local7 2 : 0: May 15 09:32:08.865 UTC :

    %CCM_LPM-LPMTCT-2-CoreDumpFileFound: The new core dump file(s) have been found in the

    system. TotalCoresFound:1 CoreDetails:The following lists up to 6 cores dumped by

    corresponding applications. Core1:Cisco CallManager (core.10499.6.ccm.1273915815) App

    ID:Cisco Log Partition Monitoring Tool Cluster ID: Node ID: ccm-pub

    May 15 05:32:15 ccm-pub local7 2 : 138: May 15 09:32:15.231 UTC :
    %CCM_RTMT-RTMT-2-RTMT-ERROR-ALERT: RTMT Alert Name:CoreDumpFileFound Detail:
    CoreDumpFileFound TotalCoresFound : 1 CoreDetails : The following lists up to 6 cores
    dumped by corresponding applications. Core1 : Cisco CallManager
    (core.10499.6.ccm.1273915815) AppID : Cisco Log Partition Monitoring Tool ClusterID :
    NodeID : ccm-pub . The alarm is generated on Sat May 15 05:32:08 EDT 2010. App
    ID:Cisco AMC Service Cluster ID: Node ID:ccm-pub


    Listing Core Dump Files

    On the CallManager server in question, a list of core dumps can be obtained by issuing the following command:


    utils core list               (CallManager version 5.x, 6.x)

    utils core active list     (CallManager version 7.x and later)


    An example of 'utils core list' is provided below, where we observe the CCM service as the core dump generator:



    Output of the command 'utils core active list' is similar to the example depicted above, with the exception of the inclusion of the "active" parameter.  This parameter was added in later CallManager releases to allow core file listing from the CM Inactive partition (previous CM version on the system, if an upgrade has taken place) without the need to perform a version switch and reboot. Instead of supplying "active" as the command line parameter, inactive partition core file listing is performed via 'utils core inactive list'.


    An example of 'utils core active list' is provided below:



    Performing Core Analysis

    Once the core dump instance has been identified via the list command, the next step is to obtain the core file backtrace for review.  This function is provided by the following command:


    utils core analyze <CoreFileName>                    (CallManager version 5,x, 6.x)

    utils core active analyze <CoreFileName>          (CallManager version 7.x and later)


    An example of the 'utils core anayze' command is provided below, where we are supplying a ccm service core file that was generated on 11/30/2009 at 11:11:50:



    Like the 'utils core active list' command, one can also perform core file analysis on the inactive partition via the 'utils core inactive analyze <CoreFileName>' command.  This feature is available in CallManager 7.x and later, and a screenshot of the 'utils core active analyze' command is provided below:



    In both examples, a warning is provided stating that this procedure will take a considerable amount of I/O and may impact system performance.  During the analysis process, the raw core file is parsed and interpreted into a backtrace output that can be used to identify the cause of the core dump.


    The analysis process normally takes a minute or less to complete on average.  The warning about impact to system performance is a suggestion to run this command during a non-peak time period to avoid a potential resource issue.


    Understanding the Backtrace of a Core File

    The chief component of the core analysis process is retrieving the backtrace for review.  Once the analysis command has been executed, a section titled "Backtrace" will be displayed on the command line, similar to the screenshot below:



    The core backtrace output is composed of several process calls, denoted by #0, #1, #2, etc.  These lines indicate process calls stored in memory at the time of the service fault.  In many cases, these backtrace signatures are a unique fingerprint that can identify a particular known or new defect in CallManager.

    Example 1: File Size Limit Exceeded
    Core was generated by `/usr/local/cm/bin/ccm'.
    Program terminated with signal 25, File size limit exceeded.

    #0  0x0067a211 in __write_nocancel () from /lib/tls/
    #1  0x00616d0f in _IO_new_file_write () from /lib/tls/
    #2  0x00615c6e in new_do_write () from /lib/tls/
    #3  0x00615c06 in _IO_new_do_write () from /lib/tls/
    #4  0x006164ba in _IO_new_file_sync () from /lib/tls/
    #5  0x0060abbb in fflush () from /lib/tls/
    #6  0x08271749 in dBProcs::addDevice (this=0xee6b1a0,
        deviceName=0xbc59c3e0 "SEP0022905BC978", deviceProtocol=0,
        deviceType=Device7941G) at dBProcs.cpp:7388


    In this example the process was attempting to write to a file.  The write attempt generted an exception and generated a core file.  The cause, "Program terminated with signal 25, File size limit exceeded." is a direct match to

    CSCsu94937    Multiple services core dumping with signal 25, File size limit exceeded.


    Example 2: Core when memory leak reaches maximum process memory size

    Memory leak in CCM process, resulting in intentional abort.


    #0  0x00a157a2 in _dl_sysinfo_int80 () from /lib/

    #1  0x01276825 in raise () from /lib/tls/

    #2  0x01278289 in abort () from /lib/tls/

    #3  0x0050d58b in __gnu_cxx::__verbose_terminate_handler () from /usr/local/cm/lib/

    #4  0x0050b2a1 in __cxxabiv1::__terminate () from /usr/local/cm/lib/

    #5  0x0050b2d6 in std::terminate () from /usr/local/cm/lib/

    #6  0x0050b41f in __cxa_throw () from /usr/local/cm/lib/

    #7  0x0050b86c in operator new () from /usr/local/cm/lib/

    #8  0x0a06bb2d in SdlProcessBase::operator new (size=102700) at SdlProcessBase.cpp:105

    #9  0x0a0014e2 in H245SessionManager::create (parentId={mSdlProcessName = 0x0, mSdlNodeId = 4, mSdlAppId     = 100, mSdlProcessNumber = 150, mSdlProcessInstance = 2629}, vH245TerminalType=H245_Gateway,         vH245TransportConnectionMode=H245Client, vH245IpAddress=404699044, vH245IpPort=40076, vTCPTos=96,         vPassThruMSD=false, vTCSTimeout=10, vFastStartInd=0, vFsAudioOutgoingLCN=0, vFsAudioIncomingLCN=0,     pktCaptureContext=0xbffab74d "", allowTCPKeepAlivesForH323=true) at             ProcessH245SessionManager.cpp:221

    #10 0x08a5629c in H245Interface::start_Transition (this=0xbff99008, s=@0x5c70990) at /vob/ccm/Common        /Include/Sdl/SdlProcessBase.hpp:123

    #11 0x08a99354 in H245Interface::fireSignal (this=0xbff99008, sdlSignal=@0x5c70990) at /vob/ccm/Common        /Include/Sdl/SdlProcessBase.hpp:175

    #12 0x0a06c904 in SdlProcessBase::inputSignal (this=0xbff99008, rSignal=0x5c70990,         traceType=SdlSystemLog::SignalRouterThread, highPriority=0, normalPriority=0, lowPriority=0,         veryLowPriority=0, lazyPriority=0, dbUpdatePriority=0) at SdlProcessBase.cpp:397

    #13 0x0a0746ce in SdlRouter::callProcess (this=0xe225ac0, _sdlSignal=0x5c70990, _deleteSignal=@0x36b8d07,     _traceType=SdlSystemLog::SignalRouterThread, _hp=0, _np=0, _lp=0, _vlp=0, _lzp=0, _dbp=0) at         SdlRouter.cpp:371

    #14 0x0a0740f3 in SdlRouter::scheduler (sdlRouter=0xe225ac0) at SdlRouter.cpp:281

    #15 0x05514bd7 in ACE_OS_Thread_Adapter::invoke (this=0xfe57a30) at OS_Thread_Adapter.cpp:94

    #16 0x054d5087 in ace_thread_adapter (args=0x0) at Base_Thread_Adapter.cpp:137

    #17 0x00db73cc in start_thread () from /lib/tls/

    #18 0x0131a96e in clone () from /lib/tls/


    In this example, the CCM process cores due to a memory leak and subsequent resource exhaustion.  Backtraces that include calls to "operator new" are typically a result of memory leak.  The process has requested the maximum amount of memory allowed by the operating system so a core is forced.  It is not possible to identify the specific memory leak from the core only to state it is result of memory leak. Other methods must be used to identify the source of the leak. Frequently this is possible by parsing SDL traces to identify objects that are "Started" or "Created" and not subsequently "Stopped".  From traces the above core was eventually diagnosed back to:

    CSCte50152    Memory Leak in CCM due to Transient SIP Connections.


    Example 3: Core Stack Corruption

    Memory corruption results in corrupted stack with "??" characters in place of function calls.


    #0  0x4e52500a in ?? ()
    #1  0xaffb3070 in ?? ()
    #2  0xaffb9084 in ?? ()
    #3  0x030dc678 in ?? ()
    #4  0x00000000 in ?? ()


    In this example, a memory corruption incident had ocurred that resulted in the stack being overwritten.  In place of function calls, we observe "??" characters in its place.  Unfortunately, a search against this backtrace alone will not correlate to a known defect.  It is recommended that the corresponding service log (e.g. ccm traces, tomcat logs) and the complete core file be retrieved from the affected system for TAC review.


    Cisco Bug Toolkit Search

    Once a backtrace has been retrieved for the core dump event, the next step is to search the Bug Toolkit for potential known defects.  The following defect will be used for this example:


    CSCta39769    UnicastBridgeControl Causes CUCM to Crash


    #0  0x097a0850 in UnicastBridgeControl::removeConfResources  (this=0x6a69f698) at
    #1   0x097ab5ae in UnicastBridgeControl::star_StationClose  (this=0x6a69f698, s=@0x6a981938)
    at  ProcessUnicastBridgeControl.cpp:2193
    #2  0x097bff64 in  UnicastBridgeControl::fireSignal (this=0x6a69f698,
    sdlSignal=@0x6a981938)  at /vob/ccm/Common/Include/Sdl/SdlProcessBase.hpp:174
    #3  0x09e4ae58  in SdlProcessBase::inputSignal (this=0x6a69f698, rSignal=0x6a981938,
    traceType=SdlSystemLog::SignalRouterThread,  highPriority=0, normalPriority=0,
    lowPriority=0, veryLowPriority=0,  lazyPriority=0, dbUpdatePriority=0) at
    #4   0x09e52c1a in SdlRouter::callProcess (this=0xde9bcc8,  _sdlSignal=0x6a981938,
    _deleteSignal=@0x324bd97,  _traceType=SdlSystemLog::SignalRouterThread, _hp=0, _np=0,
    _lp=0,  _vlp=0, _lzp=0, _dbp=0) at SdlRouter.cpp:372
    #5  0x09e5263f in  SdlRouter::scheduler (sdlRouter=0xde9bcc8) at SdlRouter.cpp:282
    #6   0x00a00ef3 in ACE_OS_Thread_Adapter::invoke (this=0x10b70b90) at
    #7   0x009c1abf in ace_thread_adapter (args=0x0) at  Base_Thread_Adapter.cpp:137
    #8  0x003bf371 in start_thread () from  /lib/tls/
    #9  0x01339ffe in clone () from  /lib/tls/


    The following line will be used to perform initial searching in the Bug Toolkit:


    #2  0x097bff64 in  UnicastBridgeControl::fireSignal (this=0x6a69f698,
    sdlSignal=@0x6a981938)   at /vob/ccm/Common/Include/Sdl/SdlProcessBase.hpp:174




    In the search example screenshot, unique memory location identifiers have been removed from the search statement to ensure that matches are found.  It may also be necessary to refine the search criteria to a specific CUCM version if no matches are presented after the search attempt, as shown below:



    Software version 7.0 was selected in the modified search above to narrow down to a specific subset of defects applicable to CUCM.  With the search re-submitted for defects related to version 7.0, the following results are displayed:



    Troubleshooting Intentional Aborts

    Core dumps that include the "IntentionalAbort" statement indicate a  system resource issue that was responsible for the service fault.  The  following ccm service core dump backtrace example will be used to  demonstrate steps involved in troubleshooting intentional aborts:


    #0  0x001627a2 in _dl_sysinfo_int80 () from /lib/

    #1  0x00d64815 in raise () from /lib/tls/

    #2  0x00d66279 in abort () from /lib/tls/

    #3  0x084c4e7a in preabort () at ProcessCMProcMon.cpp:101

    #4  0x084c4e92 in IntentionalAbort (reason=0xa9fdbdc "CallManager's timers appear

    incorrect. This may be due to CPU or blocked function. Attempting to restart

    CallManager.") at ProcessCMProcMon.cpp:106

    #5  0x084c66c3 in CMProcMon::verifySdlTimerServices () at ProcessCMProcMon.cpp:843

    #6  0x084c7035 in CMProcMon::callManagerMonitorThread (cmProcMon=0xec122d0) at


    #7  0x0107e5fb in ACE_OS_Thread_Adapter::invoke (this=0xf3ef3b8) at


    #8  0x01040cbf in ace_thread_adapter (args=0x0) at Base_Thread_Adapter.cpp:137

    #9  0x002dc3cc in start_thread () from /lib/tls/

    #10 0x00e061ae in clone () from /lib/tls/



    The RIS Data Collector Perfmonlog information should be retrieved from  the CUCM node that experienced the core dump via RTMT for review, for  the timestamp of the core dump alert.  Using Windows Performance log viewer, the process CPU utilization counters are reviewed first, as shown below:



    In the screenshot above, it is observed that CPU utilization appears stable prior to the core dump incident.  CPU utilization dips during the crash as resources are released.  In troubleshooting a potential CPU utilization issue, the concern would be a trend in CPU increase leading up to the core dump incident.


    The next component to examine in the Perfmon data is percentage VM used by the system.  In the current example, it is observed that this counter is particularly high for the time period leading up to the core dump incident:



    Next, VMSize specific to all processes are examined to determine what caused the gradual increase in memory utilization on the system.  In this example, it was found that the VMSize counter for the CCM process is relatively high and sloping upwards.  This indicates that CCM had cored due to a memory leak:


    Useful Information for Creating TAC Service Requests

    When opening a new TAC service request to troubleshoot a core dump incident, the following information is useful to provide to TAC to expedite the process:


    • Full CallManager version in use (e.g.
    • Date/Time of the core dump incident    
      • Application Event log is useful to provide for this information
      • Provide output of 'utils core list' command for absolute timestamps and core file names
    • Offending service that generated the core dump (e.g. CCM, CEF, Tomcat)
    • Core file backtrace output
    • Core file
    • Service logs for offending process    
      • e.g. If a CCM core dump, provide Cisco CallManager traces for time period of incident
    • RIS Data Collector Perfmonlog    
      • See Troubleshooting Intentional Aborts section
    Cisco Employee

    Brilliant Doc Matthew..


    Very helpful, Matt!

    Maybe link CSCtf61575, for those who wonder why analysis of cimserver and cimlistener core dumps doesn't work as expected?



    I've a UCCX server generating Core dumps.

    I was following this wonderful document.  but got stuck as none of the above conditions seems helping me out. I tried various other keywords to find out any relevant bug but was no good.

    UCCX System version:

    Following is my core dump file, let me know any pointers you may think of.

    admin:utils core active analyze core.11200.6.slapd.1314126556

    This command may use a considerable amount of I/O

    and running it may impact system performance.

    It is highly recommended that this command be

    run off-hours.

    Continue (y/n)?y

    (no debugging symbols found)

    Using host libthread_db library "/lib/tls/".

    Core was generated by `/opt/cisco/uccx/desktop/bin/slapd -x CadSplkStd -d 256 -f /opt/cisco/uccx/deskt'.

    Program terminated with signal 6, Aborted.

    Reading symbols from /lib/csa/ debugging symbols found)...done.

    Loaded symbols for /lib/csa/

    Reading symbols from /opt/cisco/uccx/desktop/lib/

    Loaded symbols for /opt/cisco/uccx/desktop/lib/

    Reading symbols from /opt/cisco/uccx/desktop/lib/

    Loaded symbols for /opt/cisco/uccx/desktop/lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /opt/cisco/uccx/desktop/lib/

    Loaded symbols for /opt/cisco/uccx/desktop/lib/

    Reading symbols from /opt/cisco/uccx/desktop/lib/

    Loaded symbols for /opt/cisco/uccx/desktop/lib/

    Reading symbols from /usr/lib/

    Loaded symbols for /usr/lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /lib/tls/

    Loaded symbols for /lib/tls/

    [1mPress <enter> for 1 line, <space> for one page, or <q> to quit [0m

    [KReading symbols from /lib/tls/

    Loaded symbols for /lib/tls/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /lib/tls/

    Loaded symbols for /lib/tls/

    Reading symbols from /usr/lib/

    Loaded symbols for /usr/lib/

    Reading symbols from /usr/lib/

    Loaded symbols for /usr/lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /usr/lib/

    Loaded symbols for /usr/lib/

    Reading symbols from /usr/lib/

    Loaded symbols for /usr/lib/

    Reading symbols from /lib/tls/

    Loaded symbols for /lib/tls/

    Reading symbols from /usr/lib/

    [1mPress <enter> for 1 line, <space> for one page, or <q> to quit [0m

    [KLoaded symbols for /usr/lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /opt/cisco/uccx/desktop/lib/

    Loaded symbols for /opt/cisco/uccx/desktop/lib/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /usr/lib/sasl2/

    Loaded symbols for /usr/lib/sasl2/

    Reading symbols from /lib/

    Loaded symbols for /lib/

    Reading symbols from /usr/lib/sasl2/

    Loaded symbols for /usr/lib/sasl2/

    Reading symbols from /usr/lib/sasl2/

    Loaded symbols for /usr/lib/sasl2/

    Reading symbols from /usr/lib/sasl2/

    Loaded symbols for /usr/lib/sasl2/

    #0  0x00c367a2 in _dl_sysinfo_int80 ()

       from /lib/




    #0  0x00c367a2 in _dl_sysinfo_int80 () from /lib/

    [1mPress <enter> for 1 line, <space> for one page, or <q> to quit [0m

    [K#1  0x00668825 in raise () from /lib/tls/

    #2  0x0066a289 in abort () from /lib/tls/

    #3  0x00661da1 in __assert_fail () from /lib/tls/

    #4  0x080b0875 in slap_send_ldap_result ()

    #5  0x080e1d27 in bdb_bind ()

    #6  0x0808f70e in fe_op_bind ()

    #7  0x0808f0a6 in do_bind ()

    #8  0x080cf4c0 in connection_operation ()

    #9  0x080cf928 in connection_read_thread ()

    #10 0x08124fed in ldap_int_thread_pool_wrapper ()

    #11 0x00b2d3cc in start_thread () from /lib/tls/

    #12 0x0070c96e in clone () from /lib/tls/


    info registers


    eax            0x0          0

    ecx            0x2bd1          11217

    edx            0x6          6

    ebx            0x2bc0          11200

    esp            0xb2c03e28          0xb2c03e28

    ebp            0xb2c03e3c          0xb2c03e3c

    esi            0x2bd1          11217

    edi            0x76bff4          7782388

    [1mPress <enter> for 1 line, <space> for one page, or <q> to quit [0m

    [Keip            0xc367a2          0xc367a2

    eflags         0x200206          2097670

    cs             0x73          115

    ss             0x7b          123

    ds             0x7b          123

    es             0x7b          123

    fs             0x0          0

    gs             0x33          51


    info threads


       6 process 11200  0x00c367a2 in _dl_sysinfo_int80 () from /lib/

      5 process 11202  0x00c367a2 in _dl_sysinfo_int80 () from /lib/

      4 process 11203  0x00c367a2 in _dl_sysinfo_int80 () from /lib/

      3 process 11204  0x00c367a2 in _dl_sysinfo_int80 () from /lib/

      2 process 11216  0x00c367a2 in _dl_sysinfo_int80 () from /lib/

    * 1 process 11217  0x00c367a2 in _dl_sysinfo_int80 () from /lib/


    Cisco Employee

    Looks like you have a crash in slapd.  Unfortunately we, Cisco, haven't done a great job documenting this.  However, it looks like slapd cores are common with:

    CSCtq13277    Two instances of LDAP Monitor service running during backup

    Did this happen during a backup?  The timestamp of your file in epoch is: 1314126556.  For further investigation I highly recommend a TAC case.


    I found other cases attached to CSCtq13277 and the associated backtraces. Those backtraces match yours.  That is very likely your issue. I updated the Release-note of the bug to include the backtrace. You should see an update in Bug Toolkit in 48 hours.  There are patches and upgrades available that resolve CSCtq13277. Please work with the UCCX TAC team to obtain the fixes.

    Very helpful Matthew, thank you!

    CSCtq13277 two instances of LDAP monitor service running during the backup CSCtn93085 UCCX 2 alarms generated in syslog during normal backups CSCtj93842 UCCX DRS mechanism to prevent cor
    ruption LDAP CAD needs. vcp550 vce Dumps