RTMT not working after changing hostname on subscriber (unity co

latintrpt · ‎04-15-2012

Hello all,

Yesterday I moved my subscriber for Cisco Unity Connection to another data center. I had it defined by hostname so I changed DNS to reflect the new IP.

Also changed the IP, name, and gateway under the OS administration.

The cluster was rebooted and replication is working just fine.

My only problem is that RTMT is reporting that the subscriber node "AMC service is down". I have verified that the service is up on both the publisher and subscriber. In RTMT the only node I'm seeing is the publisher.

Anyone have an idea as to what I'm doing wrong or missing?

I'm running version 7.1.3ES50.33039-50 on both nodes

Thanks

ronpatel · ‎04-15-2012

Hi

Is there a possibility to do following procedure

This might be due to enterprise replication being broken between the publisher and subscriber.

on sub:

utils dbreplication stop


on pub:

utils dbreplication stop

utils dbreplication reset all


This should rebuild the enterprise replication (OS level informix replication).


Confirm RTMT state is 2 on both pub and sub with:

utils dbreplication runtimestate

utils dbreplication status


Then make sure the UC application replication is good with:

show cuc cluster status

Regards

Ronak Patel

Rate helpful posts.

Regards Ronak Patel Rate all helpful post by clicking stars below the answer.

latintrpt · ‎04-15-2012

I think I have a replication issue

Publisher:

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: Replication status command started at: 2012-04-15-21-59
There no active nodes in the replication network
DB Version: ccm7_1_3_33039_1
Number of replicated tables: 427

Cluster Detailed View from PUB (2 Servers):

                                PING            REPLICATION     REPL.   DBver& REPL.   REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec) RPC?    STATUS          QUEUE   TABLES LOOP?   (RTMT) & details
-----------     ------------    ------ ----    -----------     -----   ------- -----   -----------------
rudua-unity001 10.74.13.229    0.042   Yes     Off-Line        N/A     match   Yes     (2) PUB Setup Completed
rupua-unity001 10.74.213.229   0.226   Yes     Off-Line        N/A             N/A     () Not Setup

admin:show cuc cluster status

Server Name     Member ID Server State Internal State Reason
-------------- --------- ------------ -------------- ------
rudua-unity001 0          Primary       Pri Active      Normal
rupua-unity001 1          Secondary     Sec Active      Normal

SERVER                 ID STATE    STATUS     QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_rudua_unity001 100 Active   Local           0
g_ciscounity_rupua_unity001 101 Active   Connected       0 Apr 15 05:19:13

Subscriber:

admin:utils dbreplication runtimestate

Cisco Database Layer Monitor is not running

admin:show cuc cluster status

Server Name     Member ID Server State Internal State Reason
-------------- --------- ------------ -------------- ------
rudua-unity001 0          Primary       Pri Active      Normal
rupua-unity001 1          Secondary     Sec Active      Normal

SERVER                 ID STATE    STATUS     QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_rudua_unity001 100 Active   Connected       0 Apr 15 05:19:13
g_ciscounity_rupua_unity001 101 Active   Local           0

Any idea on how to fix this and do I need to this on a down time?

Thank You

anchoudh · ‎04-15-2012

Hi,

You can use the command for forceing the replication on to subscriber

utils dbreplication forcedatasyncsub

Reference:

http://www.cisco.com/en/US/partner/docs/voice_ip_comm/cucm/cli_ref/7_1_2/cli_ref_712.html#wp69932

utils dbreplication forcedatasyncsub

This command forces a subscriber server to have its data restored from data on the publisher server.

Use this command only after you have run the utils dbreplication repair command several times, but the utils dbreplication status command still shows non-dynamic tables that are not in sync

Note Do not run this command if only dynamic tables are out of sync; dynamic tables can be out of sync during normal system operation.

You can only run this command from the publisher server. Use the all parameter to force sync on all subscriber servers in the cluster. If only one subscriber server is out of sync, use the hostname parameter.

After you run this command, you must restart the restored subscriber servers.

This command can take a significant amount of time to execute and can affect the system-wide IOWAIT.

Hope it helps.

Anand

Pls rate helpful posts !!

anchoudh · ‎04-15-2012

Hi,

Did you reinstall the RTMT after you made changes on subscriber?

Please add the hostname entries in the RTMT client box and select the new ip address once again to login.

Hope it helps.

Anand

Pls rate helpful posts !!

latintrpt · ‎04-15-2012

Whenever I start the Cisco Database Layer Monitor on the subscriber, it starts and then it stops.

Should I still do the above with forcing replication on the subscriber?

Thanks

anchoudh · ‎04-15-2012

Hi,

Could you please check the servicability page to verify if the services are running and stable?

Thanks,

Anand

latintrpt · ‎04-15-2012

All services are started except for: Cisco Database Layer Monitor, Cisco RIS Data Collector, and Cisco Log Partition Monitoring Tool. I tried starting these services. They do start up but then stop running.

This is only an issue on the subscriber. Publisher services are all fine.

ronpatel · ‎04-15-2012

Hi

Please do above procedure which is mentioned in my previous post. It will help you.

regards

Ronak Patel

Rate helpful posts.

Regards Ronak Patel Rate all helpful post by clicking stars below the answer.

latintrpt · ‎04-15-2012

Ron, I have tried the above you listed, but i get the following:

Publisher:

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: Replication status command started at: 2012-04-15-21-59
There no active nodes in the replication network
DB Version: ccm7_1_3_33039_1
Number of replicated tables: 427

Cluster Detailed View from PUB (2 Servers):

                                PING            REPLICATION     REPL.   DBver& REPL.   REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec) RPC?    STATUS          QUEUE   TABLES LOOP?   (RTMT) & details
-----------     ------------    ------ ----    -----------     -----   ------- -----   -----------------
rudua-unity001 10.74.13.229    0.042   Yes     Off-Line        N/A     match   Yes     (2) PUB Setup Completed
rupua-unity001 10.74.213.229   0.226   Yes     Off-Line        N/A             N/A     () Not Setup

admin:show cuc cluster status

Server Name     Member ID Server State Internal State Reason
-------------- --------- ------------ -------------- ------
rudua-unity001 0          Primary       Pri Active      Normal
rupua-unity001 1          Secondary     Sec Active      Normal

SERVER                 ID STATE    STATUS     QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_rudua_unity001 100 Active   Local           0
g_ciscounity_rupua_unity001 101 Active   Connected       0 Apr 15 05:19:13

Subscriber:

admin:utils dbreplication runtimestate

Cisco Database Layer Monitor is not running

admin:show cuc cluster status

Server Name     Member ID Server State Internal State Reason
-------------- --------- ------------ -------------- ------
rudua-unity001 0          Primary       Pri Active      Normal
rupua-unity001 1          Secondary     Sec Active      Normal

SERVER                 ID STATE    STATUS     QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_rudua_unity001 100 Active   Connected       0 Apr 15 05:19:13
g_ciscounity_rupua_unity001 101 Active   Local           0

latintrpt · ‎04-15-2012

Any reason why the Cisco Database Layer Monitor keeps turning off on the subscriber? I would think this would cause issues if I try to rebuild the database on the subscriber.

ronpatel · ‎04-15-2012

Hi

"utils cuc cluster overwritedb" in Sub will copies data from the publisher to the subscriber. See if this command helps, If not I would suggest you to raise the Cisco TAC case.

Regards

Ronak Patel

Regards Ronak Patel Rate all helpful post by clicking stars below the answer.

Gajanan Pande · ‎04-15-2012

Latin,

Do you see any error while booting the Sub ? Anything mentioning about using recovery CD to fix any issue as such ?

Due to malfunction or file corruption, Services will have similar issues & using recovery disk while booting fixes it.

GP.

latintrpt · ‎04-15-2012

Date: Apr 15 05:20:02

Machine Name: rupua-unity001

Severity: Critical

Process: Cisco Database Layer Monitor

Message: : SDIDBConfigData:Read failed. ProccessNodeIdException = Could not locate process node id

I saw this in RTMT

Gajanan Pande · ‎04-15-2012

No, I meant, did you see any message on server console ( not RTMT ) asking to use " Recovery Disk " while booting the server ?

GP.

RTMT not working after changing hostname on subscriber (unity connection)

utils dbreplication forcedatasyncsub

utils dbreplication forcedatasyncsub