11-08-2014 08:16 PM - edited 03-14-2019 02:05 PM
I am getting a new alert CoreDumpFileFound on UCCX RTMT.
If any one facing same issue please share
The running Cisco Unified CCX version is 10.5.1.11001-49
At Tue Jun 02 09:43:32 AST 2015 on node 172.XX.XX.XX;
the following CoreDumpFileFound events generated: TotalCoresFound : 1
CoreDetails : The following lists up to 6 cores dumped by corresponding applications.
Core1 : Cisco Desktop LDAP Monitor Service (core.19621.11.LdapMonSvr.1433227379)
AppID : Cisco Log Partition Monitoring Tool ClusterID : NodeID : UCCX1
TimeStamp : Tue Jun 02 09:43:01 AST 2015
06-02-2015 01:20 AM
I got same alerts on previous uCCX version also (8.X)
06-02-2015 01:21 AM
Here I updated my old post with latest alert
08-10-2016 12:58 AM
Hi,
Sorry, my bad.
Passed all to Cisco TAC and they confirmed this Bug:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuu20569/?reffering_site=dumpcr
and implemented same workaround and all stabilised.
Needs root access to do.
Neil
06-04-2015 12:46 AM
Following are my core dump file, let me know any points you may think of.
admin:utils core active list
Size Date Core File Name
=================================================================
38068 KB 2015-06-02 09:42:59 core.19621.11.LdapMonSvr.1433227379
38068 KB 2015-03-17 14:45:17 core.4453.11.LdapMonSvr.1426592717
38068 KB 2015-04-14 16:46:14 core.31823.11.LdapMonSvr.1429019174
38316 KB 2015-04-28 10:13:14 core.11363.11.LdapMonSvr.1430205193
52552 KB 2014-12-03 06:21:02 core.13320.6.FCVoIPMonSvr.1417576862
37040 KB 2015-03-25 00:19:14 core.16844.11.LdapMonSvr.1427231954
70028 KB 2014-12-03 07:38:28 core.13559.6.FCVoIPMonSvr.1417581508
38068 KB 2015-02-24 22:19:10 core.2382.11.LdapMonSvr.1424805550
38068 KB 2015-03-10 17:46:29 core.32509.11.LdapMonSvr.1425998789
39476 KB 2015-03-31 14:14:48 core.25141.11.LdapMonSvr.1427800488
38068 KB 2015-02-10 20:18:16 core.18079.11.LdapMonSvr.1423588696
67712 KB 2014-12-03 07:23:39 core.20282.6.FCVoIPMonSvr.1417580619
38068 KB 2015-05-19 14:45:14 core.1422.11.LdapMonSvr.1432035913
39468 KB 2015-02-04 00:18:33 core.4269.11.LdapMonSvr.1422998312
39428 KB 2015-05-12 08:43:58 core.18253.11.LdapMonSvr.1431409438
38212 KB 2015-04-07 15:45:29 core.12913.11.LdapMonSvr.1428410729
38068 KB 2015-02-17 14:45:34 core.15014.11.LdapMonSvr.1424173534
38068 KB 2015-05-05 06:41:54 core.30786.11.LdapMonSvr.1430797314
38068 KB 2015-05-26 13:44:40 core.30898.11.LdapMonSvr.1432637080
admin:utils core active analyze core.19621.11.LdapMonSvr.1433227379
This command may use a considerable amount of I/O
and running it may impact system performance.
It is highly recommended that this command be
run off-hours.
Continue (y/n)?y
Reading symbols from /opt/cisco/uccx/desktop/bin/LdapMonSvr...(no debugging symbols found)...done.
[New Thread 1515]
[New Thread 19621]
[New Thread 19622]
[New Thread 19623]
[New Thread 19624]
[New Thread 19629]
[New Thread 19631]
[New Thread 19632]
[New Thread 19633]
[New Thread 19638]
[New Thread 19639]
[New Thread 19641]
[New Thread 19643]
[New Thread 19645]
[New Thread 19649]
[New Thread 19651]
[New Thread 19654]
[New Thread 19655]
[New Thread 21816]
[New Thread 31190]
[New Thread 31188]
[New Thread 21771]
[New Thread 19659]
[New Thread 19646]
[New Thread 19644]
[New Thread 19642]
[New Thread 19640]
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /opt/cisco/uccx/desktop/lib/libsasl2.so.3...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libsasl2.so.3
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/local/platform/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/local/platform/lib/libz.so.1
Reading symbols from /opt/cisco/uccx/desktop/lib/libxerces-c.so.28...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libxerces-c.so.28
Reading symbols from /opt/cisco/uccx/desktop/lib/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libcrypto.so.1.0.0
Reading symbols from /opt/cisco/uccx/desktop/lib/libCadSplkStd.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libCadSplkStd.so
Reading symbols from /opt/cisco/uccx/desktop/lib/libSplkSecurity.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libSplkSecurity.so
Reading symbols from /opt/cisco/uccx/desktop/lib/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libssl.so.1.0.0
Reading symbols from /opt/cisco/uccx/desktop/lib/libSplkSocket.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libSplkSocket.so
Reading symbols from /opt/cisco/uccx/desktop/lib/libLDAPClientDLL.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libLDAPClientDLL.so
Reading symbols from /opt/cisco/uccx/desktop/lib/libCVDClient.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libCVDClient.so
Reading symbols from /opt/cisco/uccx/desktop/lib/libSplkStd.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libSplkStd.so
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libelf.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libelf.so.1
Reading symbols from /opt/cisco/uccx/desktop/lib/libAlarmAppenderLib.so...(no debugging symbols found)...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libAlarmAppenderLib.so
Reading symbols from /usr/local/platform/lib/libstlport.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/platform/lib/libstlport.so
Reading symbols from /usr/local/platform/lib/libACE.so...done.
Loaded symbols for /usr/local/platform/lib/libACE.so
Reading symbols from /opt/cisco/uccx/desktop/lib/libAlarmAppenderWrapper.so...done.
Loaded symbols for /opt/cisco/uccx/desktop/lib/libAlarmAppenderWrapper.so
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `/opt/cisco/uccx/desktop/bin/LdapMonSvr'.
Program terminated with signal 11, Segmentation fault.
#0 0x00b622e6 in std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::~basic_string() () from /usr/lib/libstdc++.so.6
====================================
backtrace - CUCM
===================================
#0 0x00b622e6 in std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::~basic_string() () from /usr/lib/libstdc++.so.6
#1 0x05bcf37b in splk_std::LogFunctionCall::~LogFunctionCall() () from /opt/cisco/uccx/desktop/lib/libSplkStd.so
#2 0x00fc460b in ldap_client::LDAPConnectionCheckerThread::ldapMonitor() () from /opt/cisco/uccx/desktop/lib/libLDAPClientDLL.so
#3 0x00fc1d34 in ldap_client::LDAPConnectionCheckerThread::run() () from /opt/cisco/uccx/desktop/lib/libLDAPClientDLL.so
#4 0x05c64cf1 in splk_std_internal::ThreadPimpl::thread_func(void*) () from /opt/cisco/uccx/desktop/lib/libSplkStd.so
#5 0x005edb39 in start_thread () from /lib/libpthread.so.0
#6 0x0180dd6e in clone () from /lib/libc.so.6
====================================
info registers
====================================
eax 0xfffffff4 -12
ecx 0xf7500030 -145752016
edx 0xffffffff -1
ebx 0xb9fff4 12189684
esp 0xf56e8d40 0xf56e8d40
ebp 0xf56e8d68 0xf56e8d68
esi 0x2 2
edi 0x0 0
eip 0xb622e6 0xb622e6 <std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::~basic_string()+54>
eflags 0x210286 [ PF SF IF RF ID ]
cs 0x23 35
ss 0x2b 43
ds 0x2b 43
es 0x2b 43
fs 0x0 0
gs 0x63 99
====================================
info threads
====================================
27 Thread 0xf65f8b70 (LWP 19640) 0x00603430 in __kernel_vsyscall ()
26 Thread 0xf63f6b70 (LWP 19642) 0x00603430 in __kernel_vsyscall ()
25 Thread 0xf61f4b70 (LWP 19644) 0x00603430 in __kernel_vsyscall ()
24 Thread 0xf5ff2b70 (LWP 19646) 0x00603430 in __kernel_vsyscall ()
23 Thread 0xf5aedb70 (LWP 19659) 0x00603430 in __kernel_vsyscall ()
22 Thread 0xf6dfeb70 (LWP 21771) 0x00603430 in __kernel_vsyscall ()
21 Thread 0xf58ebb70 (LWP 31188) 0x00603430 in __kernel_vsyscall ()
20 Thread 0xf57eab70 (LWP 31190) 0x00603430 in __kernel_vsyscall ()
19 Thread 0xf59ecb70 (LWP 21816) 0x00603430 in __kernel_vsyscall ()
18 Thread 0xf5beeb70 (LWP 19655) 0x00603430 in __kernel_vsyscall ()
17 Thread 0xf5cefb70 (LWP 19654) 0x00603430 in __kernel_vsyscall ()
16 Thread 0xf5df0b70 (LWP 19651) 0x00603430 in __kernel_vsyscall ()
15 Thread 0xf5ef1b70 (LWP 19649) 0x00603430 in __kernel_vsyscall ()
14 Thread 0xf60f3b70 (LWP 19645) 0x00603430 in __kernel_vsyscall ()
13 Thread 0xf62f5b70 (LWP 19643) 0x00603430 in __kernel_vsyscall ()
12 Thread 0xf64f7b70 (LWP 19641) 0x00603430 in __kernel_vsyscall ()
11 Thread 0xf68f9b70 (LWP 19639) 0x00603430 in __kernel_vsyscall ()
10 Thread 0xf69fab70 (LWP 19638) 0x00603430 in __kernel_vsyscall ()
9 Thread 0xf6afbb70 (LWP 19633) 0x00603430 in __kernel_vsyscall ()
8 Thread 0xf6bfcb70 (LWP 19632) 0x00603430 in __kernel_vsyscall ()
7 Thread 0xf6cfdb70 (LWP 19631) 0x00603430 in __kernel_vsyscall ()
6 Thread 0xf6effb70 (LWP 19629) 0x00603430 in __kernel_vsyscall ()
5 Thread 0xf73feb70 (LWP 19624) 0x00603430 in __kernel_vsyscall ()
4 Thread 0xf74ffb70 (LWP 19623) 0x00603430 in __kernel_vsyscall ()
3 Thread 0xf7779b70 (LWP 19622) 0x00603430 in __kernel_vsyscall ()
2 Thread 0xf777e6e0 (LWP 19621) 0x00603430 in __kernel_vsyscall ()
* 1 Thread 0xf56e9b70 (LWP 1515) 0x00b622e6 in std::basic_string<wchar_t, std::
char_traits<wchar_t>, std::allocator<wchar_t> >::~basic_string() () from /usr/lib/libstdc++.so.6
06-28-2016 08:20 AM
Hello,
Have same version and same issue:
admin:utils core active list
Size Date Core File Name
=================================================================
68848 KB 2016-02-18 07:28:13 core.12023.11.LRMServer.1455780492
38088 KB 2016-04-28 22:20:07 core.26346.11.LdapMonSvr.1461878406
38068 KB 2016-06-13 03:10:12 core.1149.11.LdapMonSvr.1465783812
38360 KB 2016-05-29 09:00:38 core.8822.11.LdapMonSvr.1464508837
38088 KB 2015-12-17 23:27:15 core.14456.11.LdapMonSvr.1450394835
48364 KB 2015-06-21 04:24:18 core.5342.11.FCVoIPMonSvr.1434857058
38088 KB 2016-01-14 20:16:53 core.7478.11.LdapMonSvr.1452802613
38092 KB 2015-06-26 04:05:37 core.13472.11.LdapMonSvr.1435287937
59916 KB 2016-05-07 00:50:31 core.588.11.LRMServer.1462578631
73164 KB 2015-12-31 20:13:54 core.30526.6.FCVoIPMonSvr.1451592827
Did you get to the bottom of it?
Rgrds,
neil
06-28-2016 08:39 AM
Hi Neil,
I believe you need to share these files with TAC and see if there is any memory leak or bug in the current version.You can share active analyse output with cisco
admin:utils core active analyze core.19621.11.LdapMonSvr.1433227379>>>>>>>>>>>>>>
I do see a bug for UCCX older version (9.X) with same sysmptoms.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCub91461/?referring_site=bugquickviewredir
06-28-2016 08:46 AM
I found 4 bugs and seems main one is not fixed
Looks like cisco jobbie
Thanks Deepak.
06-28-2016 09:34 PM
Hi Neil,
Still we are getting same issue.And our running version is 10.5.1.11001-49.
I think this issue will resolve by an upgrade to 10.6.
06-29-2016 12:50 AM
Thanks fenilantony
Am looking at asking Cisco TAC so can be sure - will update once I get any useful
Neil
08-08-2016 02:17 AM
Hi Neil,
Any update on this?
09-27-2017 08:01 AM
In July 2016 I upgraded UCCX to 10.6.. I have always been getting alerts on the LDAP stopping at night when the DRS runs.
Now I see I am getting this core dump issues and the Document manger is in Partial on the UCCX SUB..
At Wed Sep 27 07:53:36 PDT 2017 on node , the following CoreDumpFileFound events generated:
TotalCoresFound : 1
CoreDetails : The following lists up to 6 cores dumped by corresponding applications.
Core1 : Cisco Desktop VoIP Monitor Service (core.4822.6.FCVoIPMonSvr.1506524004)
AppID : Cisco Log Partition Monitoring Tool ClusterID :
NodeID : uccx-sub
TimeStamp : Wed Sep 27 07:53:29 PDT 2017
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide