Introduction
Some of the output from the CUCM CLI command utils dbreplicaiton runtimestate is fairly clear while some is not. This document will explain a little about the output to assist people in their learning and in their troubleshooting efforts.
The output from CUCM version 10.5.2
NOTE: THESE COMMANDS SHOULD BE RUN FROM THE PUBLISHER
The explanation
1: This lets you know the last action performed and the time of the action. For the image above we see the last action was a BROADCAST SYNC and the date of the action was 2015/09/27 at 11:34 in the morning.
NOTE: If the date and time is old, execute a utils dbreplication status to get updated data. Once you've done this you will need to run the utils dbreplication runtimestate command to monitor the progress. Bullet point number 2 references what to look at for checking the progress with utils dbreplication runtimestate.
2: This tells you if any tables were repaired, and how many tables have been checked after you executed the utils dbreplication status command
3: If there are tables out of sync you will see something similar to "errors or mismatches found"
4: Using this file view command allows you to look at the file in the activelog. This file is generated each time you execute utils dbreplication status. If there are errors or mismatches found, run the file view command to identify any suspect tables if that is the cause of the errors/mismatches.
5: This is the database version. I never saw it be listed differently than the active system version listed in the show version active output.
6: This is the replication timeout that is discussed here:
7: This is the ping time between the servers. If this is above 80 ms then the network is not in compliance the SRND.
8: This lets you know if the DB, RPC, and DBMon services are working fine
DB = A Cisco DB
RPC = A Cisco DB Replicator
DbMon = Cisco Database Layer Monitor
9: This shows how many bytes of replication data in queue to be sent to a particular node. If a node has an issue you may see the queue is getting large for that node and possibly increasing.
10: This shows the node id. g_# with the number being the node id. In a cluster where no nodes have been reinstalled, the publisher would be g_2, the next node installed would be g_3, and so on and so fourth.
11: This shows the RTMT states for database replication. There are 5 states.
State 0: Replication setup is in progress.
Possible causes: CLIs reset, rebuild, auto-recovery State 1: This means the number of replicates is not correct. Possible causes: Mismatched number of replicates on the pub and / or sub. NOTE: This is an outdated state and is no longer around. State 2: Replication Set-up completed successfully. State 3: Real-Time replication not occurring on the replication dynamic table. Possible causes: Intra-cluster communication issues will cause all servers to go to
3 (even if only one is affected). If you see an active disconnect as the statues
then replication is down, or a drop as the status this means the database itself is down.
Utils diagnose test and utils network connectivity will help to identify if there is a
problem with the cluster manager, a potential communication problem on certain ports, or
a firewalls as these could cause this state to occur. State 4: Replication set-up failure. Possible causes: Network connection issues, firewalls, intra-cluster communication
failure, bad timing of a reboot/stop/reset, rebuild or forcedatasynccli. You should
gather the ccm.log (utils create report database) and review it.
Some things you may see in runtimestate that are not listed above
Based on the version of CUCM in use you may see the following:
i. "RPC" only instead of DB/RPC/DBMon
ii. "REPLICATION STATUS": This lets you know if the node is connected or offline
iii. "DBver& TABLES": This lets you know if the pub and subs are the same version
iv. "REPL. LOOP?" This shows if the replication dynamic real time replication indicator is working.
SERVER-NAME IP ADDRESS (msec) RPC? STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- ----------- ----- ------- ----- -----------------
PUB X.X.X.80 0.173 Yes Connected 0 match Yes (2) PUB Setup Completed
tftp1 X.X.X.81 0.259 Yes Connected 0 match Yes (2) Setup Completed
tftp2 X.X.X.82 0.203 Yes Connected 0 match Yes (2) Setup Completed
sub1 X.X.X.83 0.267 Yes Connected 0 match Yes (2) Setup Completed
sub2 X.X.X.84 0.358 Yes Connected 0 match Yes (2) Setup Completed
sub3 X.X.X.85 0.247 Yes Connected 0 match Yes (2) Setup Completed
sub4 X.X.X.86 0.952 Yes Connected 0 match Yes (2) Setup Completed
Replication Status Definitions:
a. Connected
i. Queue: 0 or varying numbers
ii. Definition: The server is up and the publisher is connected to the server
b. Connecting
i. Queue: Blank
ii. Definition: the connection is being established
c. Dropped
i. Queue: Continuously rising / accumulating
ii. Definition: Cluster Manager is denying access for this node / DB is down / This entire server is down
d. Disconnect
i. Queue: Continuously rising / accumulating
ii. Definition: Replication is down on the target server
