cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Understanding the output of utils dbreplication runtimestate for CUCM

14286
Views
90
Helpful
7
Comments

Introduction

 

Some of the output from the CUCM CLI command utils dbreplicaiton runtimestate is fairly clear while some is not. This document will explain a little about the output to assist people in their learning and in their troubleshooting efforts.

 

The output from CUCM version 10.5.2

 

NOTE: THESE COMMANDS SHOULD BE RUN FROM THE PUBLISHER

 

 

 

The explanation

 

1:  This lets you know the last action performed and the time of the action. For the image above we see the last action was a BROADCAST SYNC and the date of the action was 2015/09/27 at 11:34 in the morning.

 

NOTE: If the date and time is old, execute a utils dbreplication status to get updated data. Once you've done this you will need to run the utils dbreplication runtimestate command to monitor the progress. Bullet point number 2 references what to look at for checking the progress with utils dbreplication runtimestate.

2:  This tells you if any tables were repaired, and how many tables have been checked after you executed the utils dbreplication status command

3: If there are tables out of sync you will see something similar to "errors or mismatches found"

4: Using this file view command allows you to look at the file in the activelog. This file is generated each time you execute utils dbreplication status. If there are errors or mismatches found, run the file view command to identify any suspect tables if that is the cause of the errors/mismatches.

5: This is the database version. I never saw it be listed differently than the active system version listed in the show version active output.

6: This is the replication timeout that is discussed here:

https://supportforums.cisco.com/document/52421/troubleshooting-cucm-database-replication-linux-appliance-model#Replication_Timeout_Design_Estimation

 

7: This is the ping time between the servers. If this is above 80 ms then the network is not in compliance the SRND.

8: This lets you know if the DB, RPC, and DBMon services are working fine

    DB = A Cisco DB
    RPC = A Cisco DB Replicator
    DbMon = Cisco Database Layer Monitor
    
9: This shows how many bytes of replication data in queue to be sent to a particular node. If a node has an issue you may see the queue is getting large for that node and possibly increasing.

10: This shows the node id. g_# with the number being the node id. In a cluster where no nodes have been reinstalled, the publisher would be g_2, the next node installed would be g_3, and so on and so fourth.

11: This shows the RTMT states for database replication. There are 5 states.

 

State 0: Replication setup is in progress. 
Possible causes: CLIs reset, rebuild, auto-recovery State 1: This means the number of replicates is not correct. Possible causes: Mismatched number of replicates on the pub and / or sub. NOTE: This is an outdated state and is no longer around. State 2: Replication Set-up completed successfully. State 3: Real-Time replication not occurring on the replication dynamic table. Possible causes: Intra-cluster communication issues will cause all servers to go to
3 (even if only one is affected). If you see an active disconnect as the statues
then replication is down, or a drop as the status this means the database itself is down.
Utils diagnose test and utils network connectivity will help to identify if there is a
problem with the cluster manager, a potential communication problem on certain ports, or
a firewalls as these could cause this state to occur. State 4: Replication set-up failure. Possible causes: Network connection issues, firewalls, intra-cluster communication
failure, bad timing of a reboot/stop/reset, rebuild or forcedatasynccli. You should
gather the ccm.log (utils create report database) and review it.

Some things you may see in runtimestate that are not listed above


Based on the version of CUCM in use you may see the following:        

i. "RPC" only instead of DB/RPC/DBMon
ii. "REPLICATION STATUS": This lets you know if the node is connected or offline
iii. "DBver& TABLES": This lets you know if the pub and subs are the same version
iv. "REPL. LOOP?" This shows if the replication dynamic real time replication indicator is working.

 

                                                    PING                 REPLICATION   REPL.       DBver&   REPL.     REPLICATION SETUP

SERVER-NAME IP ADDRESS  (msec)   RPC?   STATUS             QUEUE    TABLES   LOOP?   (RTMT) & details

-----------             ------------           ------     ----         -----------              -----           -------       -----         -----------------

PUB                   X.X.X.80     0.173     Yes       Connected         0               match       Yes         (2) PUB Setup Completed

tftp1                   X.X.X.81     0.259     Yes       Connected         0               match       Yes         (2) Setup Completed

tftp2                   X.X.X.82     0.203     Yes       Connected         0               match       Yes         (2) Setup Completed

sub1                   X.X.X.83     0.267     Yes       Connected        0               match       Yes         (2) Setup Completed

sub2                   X.X.X.84     0.358     Yes       Connected        0               match       Yes         (2) Setup Completed

sub3                   X.X.X.85     0.247     Yes       Connected        0               match       Yes         (2) Setup Completed

sub4                   X.X.X.86     0.952     Yes       Connected        0               match       Yes         (2) Setup Completed

 

 

Replication Status Definitions:
        a. Connected
            i. Queue: 0 or varying numbers
            ii. Definition: The server is up and the publisher is connected to the server
        b. Connecting
            i. Queue: Blank
            ii. Definition: the connection is being established
        c. Dropped
            i. Queue: Continuously rising / accumulating
            ii. Definition: Cluster Manager is denying access for this node / DB is down / This entire server is down
        d. Disconnect
            i. Queue: Continuously rising / accumulating
            ii. Definition: Replication is down on the target server

Comments
Advisor

Good piece of work.[+5]

regds,

aman

Cisco Employee

Thank you, Aman.

Hi,

Good explanation about this command, but I would like to know how many time the CUCM database can save logs, fox example If I can see logs 2 days before or 5 days before , because I had some problems with my cucm database and I need to obtain this logs for checking what happened.

I expect someone can help me.

Cisco Employee

You could probably pull the following and see if you find anything. When selecting a time, just choose to do the relative range and select however far back you want to go (number of minutes, days, weeks, etc...). You may get what you are looking for, or you might not.

    Cisco Database CLI Output
    Cisco Database Installation Service
    Cisco Database Layer Monitor
    Cisco Database Library Trace
    Cisco Database Notification Service
    Cisco Database Replicator Server

    Cisco Informix Database Service
    Event Viewer-Application Log
    Event Viewer-System Log

You can also take a look in the ccm.log files on the different servers via the CLI:

"file search activelog cm/log/informix/ccm.log error"

"file search activelog cm/log/informix/ccm.log fail"

 

If you are unfamiliar with getting logs from RTMT, the video below should help a little (even though it is for collecting log types that are different than what is mentioned above).

 

https://supportforums.cisco.com/t5/collaboration-voice-and-video/rtmt/ba-p/3102764

Beginner

This is likely the best summary of dbreplication I've found yet. Thanks for taking the time to put it together.

Cisco Employee

Thank you, Mark.

Beginner

Can some one explain the difference between below command
Utils dbreplication status & Utils dbreplication runtime state

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards