Troubleshooting CUCM Replication Issues

[Ahmadalkayyali]

First, what is replication actually doing?

Think of a CUCM cluster as an office with one manager and several assistants.

The Publisher is the manager. It's the only one allowed to write into the database — every new phone, DN, user, partition, route pattern you configure goes into the Publisher's copy first. The Subscribers are the assistants. They keep their own copy of that database so they can answer calls fast, without asking the manager every single time.

Replication is the memo system between them. When the manager writes something down, a memo goes out and every assistant updates their notebook.

When replication breaks, nobody gets the memo. The scary part is that nothing looks broken at first. Phones keep registering. Calls keep going through. Then a week later someone asks, "Why is the new hire's phone not working on the second site?" — and you find out the Subscriber's notebook has been stuck on last Tuesday.

That's why replication problems are sneaky. They're quiet, and then they're expensive.

The one command you'll type a thousand times

Everything starts here. SSH into the Publisher (always the Publisher — the Subscribers only know their own side of the story):

Note on the examples: The CLI outputs below are dummy examples built to look similar to real CUCM output, but they are not copied from a production system. Hostnames, IPs, timestamps and counts are invented. Use them to learn the shape of the output, not as a reference for exact values.

admin:utils dbreplication runtimestate

Server Time: Tue Jul 14 09:12:44 CDT 2026

Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2026-07-13-22-40
    Last Sync Result: SYNC COMPLETED on 674 tables out of 674
    Sync Errors: NO ERRORS

DB Version: ccm14_0_1_11900_5
Repltimeout set to: 300s
PROCESS option set to: 1

Cluster Detailed View from PUB (3 Servers):

                                PING      DB/RPC/   REPL.   Replication    REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec)    DbMon?    QUEUE   Group ID       (RTMT) & Details
-----------     ----------      ------    -------   -----   -----------    ------------------
cucm-pub        10.10.10.10     0.028     Y/Y/Y      0      (g_2)          (2) Setup Completed
cucm-sub1       10.10.10.11     0.412     Y/Y/Y      0      (g_3)          (2) Setup Completed
cucm-sub2       10.20.30.12     38.221    Y/N/N     254     (g_4)          (4) Setup Failed

Read that table right to left, honestly. The last column is the verdict, the rest is the evidence.

cucm-sub2 is your problem child. It pings fine, so the box is alive. But DB/RPC/DbMon shows Y/N/N — the database is reachable but the RPC channel and the DB Monitor aren't answering. And 254 items are sitting in the replication queue with nowhere to go. That queue number is the memo pile stacking up on the desk of an assistant who left the building.

What the number in parentheses means

This is the RTMT replication state, and it's the whole ballgame:

State What it means What you do

0	Replication hasn't been set up, or it's still initializing	If you just rebooted, wait. If it's been hours, it's stuck.
1	Replicates created, but the counts don't match yet	Wait — this is a normal in-between state during setup.
2	Good. Setup completed, data is flowing	Nothing. Go get coffee.
3	Replication is up, but tables are mismatched	Data drift. Repair it.
4	Setup failed. Replication never came up	Something underneath is broken. Fix that first, then reset.

A quick way to remember it: 2 is good, 3 is dirty, 4 is dead. Everything else is "give it a minute."

---

The mistake almost everyone makes

Here's the trap. You see state 4, you panic, and you immediately fire off:

admin:utils dbreplication reset all

...and then you wait an hour, and it fails again. Why? Because replication is a symptom, not a disease.

Replication runs on top of three things: the network, the clock, and the cluster's trust relationship. If any of those three is broken, replication cannot come up, and resetting it just makes it fail again more slowly.

So before you touch a single dbreplication command, run the boring checks.

---

Step 1: Is the cluster still on speaking terms?

admin:show network cluster

10.10.10.10  cucm-pub.lab.local   cucm-pub   Publisher  authenticated
10.10.10.11  cucm-sub1.lab.local  cucm-sub1  Subscriber authenticated using TCP since Mon Jul 13 22:41:03 2026
10.20.30.12  cucm-sub2.lab.local  cucm-sub2  Subscriber not authenticated

There it is: not authenticated. That's the cluster equivalent of the assistant losing their badge. The node is on the network, but the cluster doesn't trust it, so no data will ever flow to it.

A common cause is the cluster security password not matching across nodes. Someone changed it on the Publisher and never updated the Subscriber, or the Subscriber was built with a typo in it. CUCM nodes use this password to authenticate cluster communication, so if it doesn't match, the node is simply not trusted — and no amount of replication resetting will change that.

The fix is set password user security, but the order matters: change it on the Publisher first, then on the other nodes, with a restart after the change on each. Yes, a reboot. Yes, at night.

Step 2: Do they agree on what time it is?

This one gets underestimated constantly. Replication is timestamp-driven. If the Subscriber's clock has drifted, it will genuinely reject updates as "old news."

admin:utils ntp status

ntpd (pid 5241) is running...

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.10.10.10     .LOCL.           1 u   42   64  377    0.421   -0.038   0.052

synchronised to NTP server (10.10.10.10) at stratum 2
   time correct to within 24 ms

That's a healthy one — note the * next to the remote and the word synchronised. What you don't want to see is unsynchronised, a stratum of 16, or a reach value of 0. Any of those and you stop troubleshooting replication and go fix NTP, because nothing downstream will work until you do.

Rule of thumb: Subscribers sync to the Publisher. The Publisher syncs to an external NTP source. If the Publisher's NTP is bad, the whole cluster's clock is bad, and you'll chase replication ghosts all day.

Step 3: Can they actually reach each other?

Ping is not enough. Replication uses specific ports, and a helpful firewall admin somewhere may have "cleaned up" the rules between your data centres.

admin:utils network connectivity

This command can take up to 3 minutes to complete.
Continue (y/n)? y

Running test, please wait ...

Network connectivity test to 10.10.10.11 : Successful
Network connectivity test to 10.20.30.12 : Failed
        Port 1515 (DB replication)  : Unreachable
        Port 8500 (Cluster Manager) : Unreachable

Please check firewall and routing between nodes.

That output tells you everything. The node is reachable but the replication ports aren't. This is a network ticket, not a CUCM ticket — go talk to the firewall team.

The ports worth knowing by heart:

Port Who uses it

1500 / 1501 (TCP)	Database connection (Informix)
1515 (TCP)	Database replication between nodes during installation
1516 (TCP)	Database replication during upgrade
2552 (TCP)	Subscribers receiving CUCM database change notifications
8500 (TCP/UDP)	Cluster Manager (the heartbeat between nodes)
22 (TCP)	SSH — you already know this one

The 1516 and 2552 entries are the ones people forget. A firewall rule that allows replication to build but blocks change notification will give you a cluster that looks healthy and still doesn't propagate config changes — which is exactly the silent failure this whole article is about.

Also watch your latency. Cisco design guidance commonly uses 80 ms RTT as the upper limit for clustering over the WAN, and notes that latency at or above that level can significantly lengthen database replication time. But treat 80 ms as a ceiling, not a target: even well under that limit, jitter and packet loss can still make replication unstable. That 38.221 ms on cucm-sub2 in our first output isn't fatal on its own — but a link that intermittently spikes is the worst kind of problem, because it heals itself right before you start looking at it.

Step 4: Let CUCM check its own homework

CUCM has a built-in self-diagnostic that catches most of the above in one shot:

admin:utils diagnose test

Log file: platform/log/diag1.log

Starting diagnostic test(s)
===========================
test - disk_space          : Passed (available: 41 GB, used: 39 GB)
skip - disk_files          : This module must be run directly and off hours
test - service_manager     : Passed
test - tomcat              : Passed
test - tomcat_deadlocks    : Passed
test - tomcat_keystore     : Passed
test - tomcat_connectors   : Passed
test - tomcat_threads      : Passed
test - tomcat_memory       : Passed
test - cluster_manager     : Failed - Cluster Manager is unable to reach 10.20.30.12
test - dbreplication       : Failed - Database replication is broken on 1 node(s)
test - network             : Passed
test - ntp_reachability    : Passed
test - ntp_clock_drift     : Passed
test - ntp_stratum         : Passed

Diagnostics Completed

Notice how honest this is: cluster_manager failed first, and dbreplication failed because of it. Cause on top, symptom underneath. That ordering is a gift — read it top to bottom and fix the first red line, not the last one.

OK, the foundation is solid. Now fix replication.

Only once network, NTP and authentication are all green do you get to touch the replication commands. Which one you use depends entirely on that RTMT number.

If you're in state 3 — mismatched tables

Replication is running, it just delivered some memos wrong. The data has drifted. You don't need to tear anything down — you need a repair.

First, find out how bad it is:

admin:utils dbreplication status

This command can take significant time to complete.
Please monitor progress with:  utils dbreplication runtimestate

Output will be in file: /var/log/active/cm/trace/dbl/sdi/ReplicationStatus.2026_07_14_09_31_02.out

Then read the report:

admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2026_07_14_09_31_02.out

TABLE            PUB ROWS   SUB1 ROWS   SUB2 ROWS   STATUS
--------------   --------   ---------   ---------   ---------------
device               2841        2841        2779   MISMATCH (62)
numplan              1904        1904        1904   MATCH
enduser               612         612         598   MISMATCH (14)
callingsearchspace     88          88          88   MATCH
devicepool             24          24          24   MATCH

Replication status check completed.
Mismatched tables found: 2

Now repair. This is safe to run in production, and it pulls the Subscriber's tables back in line with the Publisher's:

admin:utils dbreplication repair all

This command can take several minutes to complete.
Please wait...

Repairing replicated tables...

--------------------------------------------------
Replication Repair Summary
--------------------------------------------------
Node          Tables Checked   Repaired   Errors
cucm-sub1     674              0          0
cucm-sub2     674              2          0
--------------------------------------------------

The Replication Repair is complete.
Please run 'utils dbreplication runtimestate' to verify.

Then go back and confirm you're at state 2. Repair is the gentle option — always try it before you reach for the sledgehammer.

If you're in state 4 or a stuck 0 — full reset

Replication never came up at all, so there's nothing to repair. You have to rebuild the memo system from scratch.

You can reset a single node, which is what you want if only one Subscriber is broken:

admin:utils dbreplication reset cucm-sub2

Or, if the whole cluster is a mess:

admin:utils dbreplication reset all

Before you press enter, know what you're signing up for:

• Run it only from the Publisher.
• It can take 30 minutes to several hours, depending on cluster size and database size.
• It is disruptive-ish. Calls generally survive, but configuration changes during the reset are a bad idea.
• Do it in a maintenance window. Every engineer who has ignored this advice has a story about it.

Here's what a healthy reset looks like:

admin:utils dbreplication reset all

This command will try to start Replication reset and will return in 1-2 minutes.
Background repair of replication will continue after that for 1 hour.
Please watch RTMT replication state. It should go from 0 to 2.
When all subs have an RTMT Replicate State of 2, replication is complete.
If Sub replication state becomes 4, there is a problem.

OK to proceed? (y/n) y

Requested Replication Reset is in progress.
Please watch 'utils dbreplication runtimestate' for progress.

And then you watch. Poll runtimestate every ten minutes. You are looking for every node to walk 0 → 1 → 2 and stay there.

admin:utils dbreplication runtimestate

Cluster Replication State: Replication reset in progress on 3 servers

SERVER-NAME     IP ADDRESS      PING     DB/RPC/   REPL.   REPLICATION SETUP
                                (msec)   DbMon?    QUEUE   (RTMT) & Details
-----------     ----------      ------   -------   -----   ------------------
cucm-pub        10.10.10.10     0.031    Y/Y/Y      0      (2) Setup Completed
cucm-sub1       10.10.10.11     0.408    Y/Y/Y      0      (2) Setup Completed
cucm-sub2       10.20.30.12     0.996    Y/Y/Y      12     (1) Replicates created

cucm-sub2 is at (1) with a draining queue. That's not a problem — that's progress. Leave it alone and check back in fifteen minutes.

The last resort

If a reset genuinely will not take, the deeper option is to tear down the replication admin database (syscdr) across the cluster and rebuild it from nothing.

Follow the documented order exactly. This is not a place to improvise:

Step 1 — on the PUBLISHER:
admin:utils dbreplication stop all

Step 2 — on each SUBSCRIBER, one at a time:
admin:utils dbreplication dropadmindb

Step 3 — then on the PUBLISHER:
admin:utils dbreplication dropadmindb

Step 4 — back on the PUBLISHER:
admin:utils dbreplication reset all

Three things to burn into your memory here:

• stop all runs on the Publisher, and it stops replication cluster-wide. You don't walk around stopping it node by node.
• dropadmindb runs on the Subscribers first, one at a time, and the Publisher last. Drop the Publisher's admin database first and you've removed the reference point the Subscribers are supposed to be torn down against.
• reset all runs on the Publisher, and only after every node has had its admin database dropped.

This is deep-water territory — full maintenance window, recent backup, nobody making config changes. If you're here and it still won't come up, that's a TAC case, and that's fine. That's what TAC is for.

A field checklist you can actually use

Print this. Tape it to something.

1. SSH to the Publisher. Not a Subscriber. The Publisher.
2. utils dbreplication runtimestate — get your RTMT number.
3. State 2 everywhere? You're done. The problem is something else.
4. show network cluster — is every node authenticated?
5. utils ntp status — is every node synchronised?
6. utils network connectivity — are 1500/1501/1515/8500 open, and is latency under 80 ms?
7. Fix whatever failed in 4–6. Do not skip this.
8. State 3? → utils dbreplication status, then utils dbreplication repair all.
9. State 4 or stuck at 0? → utils dbreplication reset <node> or reset all, in a maintenance window.
10. Wait. Watch runtimestate. Look for 0 → 1 → 2.
11. Still broken? Collect show tech dbstateinfo and open a TAC case.

The habit that saves you

The best replication troubleshooting is the kind you never have to do.

Run utils dbreplication runtimestate after every change window — after a reboot, after an upgrade, after a certificate regeneration, after anything that touched the network. It takes fifteen seconds. It costs you nothing.

Because the difference between "replication broke last night" and "replication broke sometime in the last three weeks and I don't know what changed" is the difference between a ten-minute repair and a very long Saturday.

2 is good. Check for the 2.

Works Cited

Cisco. “Troubleshoot CUCM Database Replication Issues.” Cisco Support, Cisco Systems.
Cisco. Command Line Interface Reference Guide for Cisco Unified Communications Solutions, Release 12.5(1)SU2. Cisco Systems.
Cisco. System Configuration Guide for Cisco Unified Communications Manager, Release 14 and SUs. Cisco Systems.
Cisco. Upgrade and Migration Guide for Cisco Unified Communications Manager and the IM and Presence Service, Release 12.5(1). Cisco Systems.