Troubleshooting Database Replication of Cisco Unity Connection

CscTsWebDocs · ‎10-06-2014

1. Overview

This document introduces how to troubleshoot Database Replication problems which could cause the following problems in the Cisco Unity Connection (CUC) cluster:

Setting changes are not reflected correctly
Voice Messages are not distributed to the user's mailbox

The CUC cluster consists of the Publisher and Subscriber. The Databases within the cluster will be synchronized by the following two types of Database Replications:

Enterprise Replication (ER)
Unity Connection Replication (UCR)

2. Details of Two Types of Database Replications

2-1. Enterprise Replication (ER)

It is the same replication as in the Cisco Unified Communications Manager (CUCM) 6.x and later.

Reference URL:
CUCM Database Replication Troubleshooting on the Linux Appliance Model

Without the proper ER running, UCR does not work correctly. So the ER state must be normal at all times.

Specifically, the Replication Status that can be verified by Real Time Monitoring Tool (RTMT) must be 2 for both the Publisher and Subscriber, as shown in the above link.
The Replication Status of RTMT can also be confirmed by
show perf query class "Number of Replicates Created and State of Replication".

----- Command execution example -----

[[{"fid":"1036606","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"01_1","field_file_image_title_text[und][0][value]":"01_1","field_media_description[und][0][value]":"01_1"},"type":"media","attributes":{}}]]

2-2. Unity Connection Replication (UCR)

With ER running normally (the status is 2), the databases are synchronized between the Publisher and the Subscriber of CUC by UCR.
Use the show cuc cluster status command to see if UCR is running normally.

If the status of the replications shown below is Connected, it is running normally.

[[{"fid":"1036611","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"show_cuc","field_file_image_title_text[und][0][value]":"show_cuc","field_media_description[und][0][value]":"show_cuc"},"type":"media","attributes":{}}]]

3. Recovering Database Replication
3-1. If the ER Status is "2" but the UCR status is "Dropped"
You may be able to recover the Database by executing utils cuc cluster overwritedb on the Subscriber to overwrite the Database from the Publisher to the Subscriber.

Note)
Overwriting is completed in a little over five minutes in the laboratory environment where the databases are smaller, but it may take about one hour, depending on the environment.
The Subscriber restarts automatically after executing this command.

There will be no effect to the Publisher, but it is recommended to perform this operation in maintenance window or off hours.

----- Command execution example -----

[[{"fid":"1036616","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"02","field_file_image_title_text[und][0][value]":"02","field_media_description[und][0][value]":"02"},"type":"media","attributes":{}}]]

3-2. If the ER Status is "2" but the UCR Status is "Disconnected" or UCR is not recovered by 3-1

It is highly possible that the Subscriber's information on thePublisher Database may be corrupt. In many cases, it requires a reinstallation of the Subscriber.

We may be able to recover your data without reinstalling the Subscriber, depending on our investigation of the case. Please contact Cisco TAC as needed.

In addition, please reinstall the Subscriber as follows:

Shutdown the Subscriber.
Log into Cisco Unity Connection Administration. Go to [System Setting] > [Cluster] and delete the Subscriber information.
Restart the Publisher.
Log into Cisco Unity Connection Administration. Go to [System Setting] > [Cluster] and add the Subscriber again.
Reinstall the Subscriber.

3-3. If the ER Status is other than "2"
Recovery may be possible with the utils dbreplication clusterreset command.

(*) Please perform this task in maintenance window or off hours.

utils dbreplication clusterreset command procedure
(1) Execute the utils dbreplication stop command on the Subscriber. Upon completion, proceed to the next step.

----- Command execution example -----

[[{"fid":"1036621","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"03","field_file_image_title_text[und][0][value]":"03","field_media_description[und][0][value]":"03"},"type":"media","attributes":{}}]]

(2) Execute the utils dbreplication stop command on the Publisher. Upon completion, proceed to the next step.

(*) The command execution example is the same as in (1).

(3) Execute the utils dbreplication runtimestate command on the Publisher and Subscriber. Verify that "RPC?" is “YES”.

----- Command execution example -----

[[{"fid":"1036626","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"04","field_file_image_title_text[und][0][value]":"04","field_media_description[und][0][value]":"04"},"type":"media","attributes":{}}]]

(*) Same procedure for the Subscriber.

(4) Execute the utils dbreplication dropadmindb command on the Publisher. Upon completion, proceed to the next step.

----- Command execution example -----

[[{"fid":"1036631","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"05","field_file_image_title_text[und][0][value]":"05","field_media_description[und][0][value]":"05"},"type":"media","attributes":{}}]]

(5) Execute the utils dbreplication dropadmindb command on the Subscriber. Upon completion, proceed to the next step.

(*) The command execution example is the same as in (4).

(6) Execute the utils dbreplication reset all command on the Publisher.

----- Command execution example -----

[[{"fid":"1036636","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"06","field_file_image_title_text[und][0][value]":"06","field_media_description[und][0][value]":"06"},"type":"media","attributes":{}}]]

(7) Execute the utils dbreplication runtimestate command on the Publisher and Subscriber periodically. When the output is as follows, the process is complete.

----- Command execution example -----

[[{"fid":"1036641","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"07","field_file_image_title_text[und][0][value]":"07","field_media_description[und][0][value]":"07"},"type":"media","attributes":{}}]]

(8) If the ER Status is not "2" after about one hour following execution, collect the following logs and contact Cisco TAC.

utils dbreplication runtimestate
file get activelog cm/trace/dbl/*.log
file get activelog cm/trace/dbl/sdi/dbmon*.txt
file get activelog cm/log/informix/dbl_repl*.log
file get activelog cm/log/informix/ccm*.log

4. Useful Links

Related Information

Original Document: https://supportforums.cisco.com/ja/document/12180166
Author: Shuhei Mentani
Posted on April 21, 2014