07-19-2017 07:56 AM - edited 03-17-2019 10:49 AM
Hi,
We have two servers Pub and Sub
After replication problem, execute reset replication cluster, but setup not completed
DB and Replication Services: ALL RUNNING
DB CLI Status: No other dbreplication CLI is running...
Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2017-07-19-17-55
Last Sync Result: SYNC COMPLETED 603 tables sync'ed out of 603
Sync Errors: 4 Errors or failures were found
Use this command to see the output- file view activelog /cm/trace/dbl/2017_07_19_17_53_02_dbl_repl_cdr_Broadcast.logDB Version: ccm9_1_2_11900_12
Repltimeout set to: 300s
PROCESS option set to: 1Cluster Detailed View from callmanager1 (2 Servers):
PING CDR Server REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? (ID) & STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- -------------- ----- ------- ----- -----------------
callmanager1 10.0.3.11 0.051 Yes (2) Connected 0 match No (2) PUB Setup Completed
callmanager2 10.0.4.11 4.861 Yes (4) Connected 0 match N/A (4) Setup Completed
We have 4 sync errors, for example one of this:
Jul 18 2017 15:44:01 ------ Table scan for
ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic start --------Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1 0 0 1 0
The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:abcdd7a0-4c70-46f6-9e58-d56bf1d409ff
------------------------------------------------------------------
WARNING: replicate is not in sync
Ok, i understand, problem in replicationdynamic table.
Now show this table on Pub, execute run sql select * from replicationdynamic
admin:run sql select * from replicationdynamic
pkid fkprocessnode datetimestamp
==================================== ==================================== =============
abcdd7a0-4c70-46f6-9e58-d56bf1d409ff b5efd7a0-4c70-46f6-9e58-d56bf1d409ff 1497420013
Same command on Sub
admin:run sql select * from replicationdynamic
pkid fkprocessnode datetimestamp
==================================== ==================================== =============
abcdd7a0-4c70-46f6-9e58-d56bf1d409ff b5efd7a0-4c70-46f6-9e58-d56bf1d409ff 1497420013
This value the same on Pub and Sub, why cucm not sync?
Plz help, i dont have a choice to call TAC
07-19-2017 08:07 AM
Hi,
Try below steps the way they are listed and let me know how it goes.
1) Stopped the replication on all the nodes by running the command "utils dbreplication stop" (1st on sub's and then on pub)
(2) Ran the command "utils dbreplication dropadmindb" on all the nodes (1st the pub and then the sub's)
(3) Ran the command "utils dbreplication reset all" on the publisher.
(Rate if it helps)
JB
07-20-2017 12:30 AM
Thanks for answer, i try it before, this not help.
Sync log after cluster reset, same error in the table, maybe we can delete this row in sql request?
It just one row in credentialdynamic table.
Jul 20 2017 10:35:53 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0
The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:35:59 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end
Jul 20 2017 10:35:31 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic start --------Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1 0 0 1 0
The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:abcdd7a0-4c70-46f6-9e58-d56bf1d409ff
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:35:36 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic end ---------
07-20-2017 01:16 AM
Hi
Lets start from basics
Can you attach the below output in .txt file to post, all from PUB
1) show network cluster
2) utils diagnose test
3) utils dbreplication status
4) show tech network hosts
JB
07-20-2017 01:31 AM
Thank you for taking time
I attach, what you need.
Now i compared credentialdynamic table with the field where sync show error
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
On Pub
eb4f68a2-744a-401b-a41e-5515d68c5733 0 0 0 1497420200
66892066-fa9c-4789-816e-9fa6d15adb32 1497420200 1497420145
On Sub
eb4f68a2-744a-401b-a41e-5515d68c5733 0 0 0 1500537611 66892066-fa9c-4789-816e-9fa6d15adb32 1500537610 1500537611
In a field timelastaccessed datetimestamp lastsuccessfullogintime value not the same, how you see above.
This pkid connected to Application user "axluser"
How i can change this value in the table?
07-20-2017 01:42 AM
Hi
I think a cluster reboot has already been done, is that correct?
can you attach the output of .txt file for below.
file view activelog /cm/trace/dbl/2017_07_20_10_33_40_dbl_repl_cdr_Broadcast.log
JB
07-20-2017 02:04 AM
07-20-2017 02:58 AM
Hi
Most of the things look good with your replication other then below
Jul 20 2017 10:24:24 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic start --------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1 0 0 1 0
The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:abcdd7a0-4c70-46f6-9e58-d56bf1d409ff
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:24:29 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic end ---------
Jul 20 2017 10:24:45 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 8
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 8 0
The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:24:51 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------
Jul 20 2017 10:25:09 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_561_callforwardhistorydynamic start --------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 627 0 0 0 2
g_callmanager2_ccm9_1_2_11900_12 627 0 0 2 0
The repair operation completed. Validating the repaired rows ...
Validation completed successfully.
Jul 20 2017 10:25:11 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_561_callforwardhistorydynamic end ---------
Jul 20 2017 10:25:11 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_560_callforwarddynamic start --------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 183 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 183 0 0 1 0
The repair operation completed. Validating the repaired rows ...
Validation completed successfully.
Jul 20 2017 10:25:13 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_560_callforwarddynamic end ---------
admin:utils dbreplication runtimestate
DB and Replication Services: ALL RUNNING
DB CLI Status: No other dbreplication CLI is running...
Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2017-07-20-10-36
Last Sync Result: SYNC COMPLETED 603 tables sync'ed out of 603
Sync Errors: 4 Errors or failures were found
Use this command to see the output- file view activelog /cm/trace/dbl/2017_07_20_10_33_40_dbl_repl_cdr_Broadcast.log
DB Version: ccm9_1_2_11900_12
Repltimeout set to: 300s
PROCESS option set to: 1
Cluster Detailed View from callmanager1 (2 Servers):
PING CDR Server REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? (ID) & STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- -------------- ----- ------- ----- -----------------
callmanager1 10.0.3.11 0.040 Yes (2) Connected 0 match No (2) PUB Setup Completed
callmanager2 10.0.4.11 5.874 Yes (4) Connected 0 match N/A (4) Setup Completed
"REPL. LOOP?" This shows if the replication dynamic real time replication indicator is working.
Can you try "utils dbreplication rebuild all" once
JB
07-20-2017 03:12 AM
Thanks, yeah, i know this error, cucm not repair table credentialdynamic, because
On Pub
eb4f68a2-744a-401b-a41e-5515d68c5733 0 0 0 1497420200
66892066-fa9c-4789-816e-9fa6d15adb32 1497420200 1497420145
On Sub
eb4f68a2-744a-401b-a41e-5515d68c5733 0 0 0 1500537611 66892066-fa9c-4789-816e-9fa6d15adb32 1500537610 1500537611
In a field timelastaccessed datetimestamp lastsuccessfullogintime value not the same, how you see above.
I think this problem connected with another, because i do not connect to Pub with login "axluser" (login/pass error), but on Sub connected is good.
What i can do to write normal value in this table
How i know, command "utils dbreplication rebuild all", just replace next command:
utils dbreplication stop
utils dbreplication dropadmindb or dropadmindbforce
utils dbreplication reset
I use it today.
07-20-2017 03:39 AM
Hi,
I think you have already tried
utils dbreplication repairtable credentialdynamic all on PUB correct?
JB
07-20-2017 03:48 AM
Yeah, i try it too, same result
Maybe you have idea how fix this situation
To determine if replication is suspect, look for the following:
(1) Number of rows in a table do not match on all nodes.
(2) Non-zero values occur in any of the other output columns for a tableFirst, a cdr list of the replication status of the servers
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_callmanager1_ccm9_1_2_11900_12 2 Active Local 0
g_callmanager2_ccm9_1_2_11900_12 4 Active Connected 0 Jul 20 10:17:22
Jul 20 2017 14:45:29 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 1g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0
The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 14:45:35 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------command failed -- WARNING: replicate is not in sync (178)
Running a cdr check after repair to insure table is in sync
Jul 20 2017 14:45:36 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 0
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0WARNING: replicate is not in sync
Jul 20 2017 14:45:37 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------command failed -- WARNING: replicate is not in sync (178)
end of the file reached
07-20-2017 04:01 AM
Hi,
I have seems some similar issues reposted like this getting fixed with repair commands, but this one seems to have stuck.
Can you provide below
utils create report database
and informix from Both server for last 6 hours.
JB
07-20-2017 04:42 AM
07-20-2017 05:22 AM
Can you also collect informix logs from RTMT
http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200787-How-to-Collect-Traces-for-CUCM-9-x-10-x.html
JB
07-20-2017 05:35 AM
Hi,
Try below and let me know if it helps
1. Run -- utils dbreplication repairtable certificate all
2. After the previous step completes run -- utils dbreplication repairtable certificateservicecertificatemap all
3. After the previous step completes run -- utils dbreplication repairtable certificatehashmap all
4. After the previous step completes run -- utils dbreplication repairtable certificatetrustrolemap all
5. After the previous step completes run -- utils dbreplication repairtable certificateprocessnodemap all
6. After the previous step completes run -- utils dbreplication runtimestate
7. Send the output of the runtimestate
JB
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide