Call Manager 7.1.3 DB Replication issues after changing IPs

[jasunf]

In our lab we tried to change the IPs of all our call center servers.  We have 3 call managers we executed the change based on a document found here:

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/install/7_1_2/ipchange/ipchg712.html#wp41907

After following the article we started having DB replication issues.  We performed the following troubleshooting steps below:

1. "show network cluster" (see if IP addr/hostnames are what you expect)

2. "show tech network hosts".  (see if IP addr/hostnames are what you expect)

3. Also, perform a "utils diagnose module validate_network".  You should always validate the network before doing a dbreplication reset

All 3 show the call managers having the proper IP addresses and node names.

I ran another set of steps I found in another community post where the issue was reoslve:

http://vannis.tk/?p=316

I run a utils dbreplication runtimestate on the pub and get this:

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: Replication status command started at: 2012-01-16-13-52
     Replication status command COMPLETED 1 tables checked out of 427
     Processing Table: typedberrors with 992 records
     No Errors or Mismatches found.

     Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2012_01_16_13_52_28.out' to see the details

DB Version: ccm7_1_3_31900_1
Number of replicated tables: 427

Cluster Detailed View from PUB (3 Servers):

                                PING            REPLICATION     REPL.   DBver&  REPL.   REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec)  RPC?    STATUS          QUEUE   TABLES  LOOP?   (RTMT) & details
-----------     ------------    ------  ----    -----------     -----   ------- -----   -----------------
CMPHQ01 172.16.64.X    0.052   Yes     Connected       0       match   N/A     (3) PUB Setup Completed
CMHQS01 172.16.64.X    0.300   Yes     Off-Line        N/A             N/A     () Not Setup
CMHQS04 172.16.64.X    0.247   Yes     Off-Line        N/A             N/A     () Not Setup

Its showing off line for the subs so there appears to be something missing some where in my changes.

Here is the resulting output from the publisher dbrepliation status command:

No Errors or Mismatches found.
Replication status is good on all available servers.

utils dbreplication status output

To determine if replication is suspect, look for the following:
        (1) Number of rows in a table do not match on all nodes.
        (2) Non-zero values occur in any of the other output columns for a table

Processing ccmdbtemplate_cmphq01_ccm7_1_3_31900_1_1_95_typedberrors with 992 rows group 1
Sync target server is not defined for the replicate ccmdbtemplate_cmphq01_ccm7_1_3_31900_1_1_95_typedberrors
command failed -- participants required for operation specified  (17)

I'm going over all my changes again but it appears I'm missing something.  In the RTMT tool also it shows AMC server down on the subs when in fact the service is running.  The network is good as I can reach each subscriber via ssh to admin the box. 

Any suggestions on what I may have missed would be well received!!

[jasunf]

Subsequently I have found most all the services on the subs won't start (call manager, ris, db layer monitor, and other major services).

The system logs from RTMT show this:

Jan 15 05:01:41

CMHQS01

Critical

Cisco Database Layer Monitor

: SDIDBConfigData:Read failed. ProccessNodeIdException = Failed to connect to datasource: [Informix][Informix ODBC Driver][Informix]Attempt to connect to database server (cmhqs01_ccm7_1_3_31900_1) failed.

[james.ferris]

I always run into issues when changing the IP or hostname on CUCM.  For some reason the host file dosn't always reconize the change.

Here is a good link on DB replication from the support forums:

https://supportforums.cisco.com/docs/DOC-13672

You can try to repair the replicaton using this command;

      utils dbreplication repair

it may need to be run a few times.

If that dosn't work you can try this one:

      utils dbreplication forcedatasyncsub 

This forces a sub to basiclly drop the database and rebuild it from the Pub so be careful.

Somtimes i've had to rebuild the subscribers from scratch. 

The last time I had this issue TAC had to get root access to the box and manually change the host file.

I hope this helps. 

[jasunf]

Thanks James I'll give that a shot.  At this point in the day I reverted all virtuals back to their latest snapshot and am going to follow the steps again thinking maybe I botched up a step or something. 

If I get the same results I'll follow your path.

Thanks a bunch!

[Leonardo Santana]

Hello take a look on this link!

https://supportforums.cisco.com/docs/DOC-13672

Regards!