Cisco UC 9.1 Database replication issue & repair

[mbroberson1]

We currently are running UC 9.1 with 1 pub and 3 subscribers in our cluster. We are currently unable to make any changes to the system (user phones for example) as our database replication status for the pub and subscribers all show a status of 3 and the db replication needs to be repaired. Will running a "utils dbreplication repair all" from the publisher take the system down and cause an outage? Just curious if anyone has ran into a similar issue and had to run a repaid all.

Regards

[Terry Cheema]

No running the database commands doesn't take the system down and doesn't will even impact call processing.

Sometimes even after running the repair command databases replication is not good, then you will be need to stop on all nodes (from pub) and reset all (again from pub).

Depending on your cluster size (and location) it may take some time for everything to return to normal.

If I do a complete stop and reset the replication, from memory it took anything between 20-40 mins for a 3 node cluster and around 1-1.5 hour for a 9 node cluster.

-Terry

[mbroberson1]

Terry,

Thank you and I really appreciate the information. We have around 250 ip phones total with our one publisher and 3 sub's. I'm hoping the fix won't take too long.

Thanks,

Brandon

[Terry Cheema]

Hi Brandon,

Not a problem. Yes it should fix in around 30 mins or so roughly as long as servers are not located geographically too far away.

Once you have done monitor from RTMT and from CLI by running the command:

utils dbreplication runtimestate

and confirm you see replication status 2 on each server.

And please do rate any helpful posts :-)

-Terry

[Francisco Granados]

Quick note -

Since you are saying this is happening on all subscribers/pub, make sure that you check your NTP status "utils ntp status". I've run into a few cases where the dbreplication fails because of NTP not being synced. You don't want to wait hours just to find out you had to restart  NTP and then repair the db again!

Thanks,

Frank

[mbroberson1]

Hi Frank,

Thanks for the post. Quick question for you on the NTP setup. In our environment our subscribers point to the publisher for time, and publisher to an NTP server. Does this seem like the correct NTP setup?

Thanks

[Francisco Granados]

Correct. The subs get time with the Pub, the Pub gets time from a NTP source. Cisco recommends to use either a Linux or IOS device as the NTP server for your Pub (Not windows). 

The most frustrating experience I had was having db issues because NTP stopped working on the NTP source (6500 switch) and the Pub did not show any errors. The database was down because of this. Once I fixed the NTP on the 6500 switch I was able to repair the db and everything was back to normal.

Thanks,

Frank