Nodes losing sync on CUCM 10.5.2 cluster

[UCguy42]

Greetings,

      It seems I have an issue with my cluster that no matter what I do, all nodes but one (a subscriber) loses sync. All of them give me the following:

Cluster Replication State: BROADCAST SYNC Completed on 4 servers at: 2016-02-23-02-26
Last Sync Result: 6 tables repaired. SYNC COMPLETED on 681 tables out of 681
Sync Status: All Tables are in sync.
Use CLI to see detail: 'file view activelog cm/trace/dbl/20160223_010133_dbl_repl_output_Broadcast.log'

DB Version: ccm10_5_2_12901_1
Repltimeout set to: 300s
PROCESS option set to: 1

Cluster Detailed View from CUCM-PUB (5 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
CUCM-PUB X.X.X.32 0.019 Y/Y/Y 0 (g_2) (3) Out Of Sync
CUCM-SUB01 X.X.X.33 0.116 Y/Y/Y 0 (g_3) (3) Out Of Sync
CUCM-SUB03 X.X.X.32 82.854 Y/Y/Y 0 (g_4) (2) Setup Completed
CUCM-SUB02 X.X.X.35 0.211 Y/Y/Y 0 (g_7) (3) Out Of Sync
CUCM-SUB04 X.X.X.33 62.751 Y/Y/Y 0 (g_6) (3) Out Of Sync

After a "utils dbreplitcation reset all" they all sync up successfully and return all 2s, and then again go back to the state above. When I try to run "utils dbreplication repair all"  nothing happens and I get the following:

chmod: changing permissions of `/var/log/active/cm/trace/dbl/sdi/replication_scripts_output.log': Operation not permitted

Replication Repair is now running in the background.
Use command 'utils dbreplication runtimestate' to check its progress

Any help would be greatly appreciated. 

[Deepak Rawat]

Try running utils dbreplication rebuild all command from the CUCM Publisher. The command will run a combination of the following CLI commands on specified node(s) and is pretty successful in resolving the replication issues:

utils dbreplication stop
utils dbreplication dropadmindb or dropadmindbforce
utils dbreplication reset

Note: After running the above command leave the system for a considerable amount of time such as 30 to 40 minutes and do not run any commands from the CLI during that time (specially Replication based commands)

Regards

Deepak

[UCguy42]

Nothing, no luck after running that. Any other suggestions? 

[UCguy42]

Wound up redoing the whole setup from the ground up in regards to the VM environment since the network admin really didn't give me any details on his port configuration except that it was trunked to allow the tagged vlans I detailed. So after that I ran a "utils dbreplication dropadmindb", and a "utils dbreplication rebuild all" and they all came back with perfect 2s.