DB Replication failure of entire cluster after moving 2 subscribers to DR location

Suresh Hudda · ‎03-07-2015

Dear Supportforums's Members,

We have a customer who is running on Cisco Callmanager 6.1 and this product is end of support so cant take help from TAC :-( Could someone can help me here to resolve the dbreplication issue. Below are CUCM nodes in cluster and there is not any voice gateway and not other cisco voice application.

CUCM Pub
TFTP Server
Sub1
Sub2
Sub3
Sub4

Background: Few days back customer has moved Sub3 & Sub4 to DR location and changed the IP addresses for these two subscribers. After that dbreplication got messed up. In RTMT, for pub it showing 0 and for other DC nodes it is showing 3 & sometimes 4 and for moved subscribers (DR) it showing nothing.

Observation and activities which have been done till now:

-> Not much latency between DC and DR location, RTT is 55 msec to 60 msec between DC and DR nodes.
-> IP addresses of subscribers have been changed as per Cisco doc.
-> There is no error when executing "utils diagnose test"
-> Given reboot to Entire cluster but no luck
-> Tried to dbreplication repair but no luck.
-> Tried to dbreplication reset all but no luck.
-> When executing "utils network connectivity" from all subscribers then it saying that cluster may be in bad state, below is the output.

admin:utils network connectivity
This command can take up to 3 minutes to complete.
Continue (y/n)?y
Running test, please wait ...
....................................
Network connectivity test timed out.
Cluster Manager may be in a bad state.
Use the command 'utils service restart' to restart Cluster Manager

-> When executing "utils firewall list" from publisher then is showing entries for DC subscribers as below, but not showing for DR subscribers. So. I am suspecting firewall issue as well. Could you please confirm it. I have attached entire output of "utils firewall list", pls find in attachment.

ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:WIP_dc_cucm_pub_ccm6_1_2_1000_13
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1500
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1501
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1501
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1502
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1502
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1503
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1503
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1504
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1504
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1505
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1505
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1515
ACCEPT     udp -- WIP-DC-CUCM-SUB01   anywhere           udp dpt:1515
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:8001
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:2555
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:2556
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1099
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:1090
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:2551
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:2552
ACCEPT     tcp -- WIP-DC-CUCM-SUB01   anywhere           tcp dpt:4040

Suresh

Terry Cheema · ‎03-08-2015

1) First of all - confirm there is no network connectivity issue, ping both ways from the Pub and the affected subs - use - utils network ping and post the output.

2) Once its established network connectivity is not an issue run the below from both the Pub and the impacted Subs:

Run the utils dbreplication runtimestate and post the output.

3) Run the below command from both the Pub and impacted Subs and post the output:
show network cluster

4) Lastly if the entries are not showing up in the Pub firewall for the Subs 3 & 4 even after we establish network connectivity is fine as in Step 1, the try the below and disable the firewall on Pub server:

utils firewall ipv4 diable

Check if after disabling the firewall, the entries show up and dbreplication return to normal.

-Terry

Aman Soi · ‎03-08-2015

Hi Suresh,

In addition to tips provided by Terry,you can check the ports

http://www.cisco.com/c/dam/en/us/td/docs/voice_ip_comm/cucm/port/6_1/61plrev1.pdf

after the same are verified in firewall , u can run the command utils service restart cluster manager from SUB3 and SUB4 servers.

regds,

aman

Suresh Hudda · ‎03-08-2015

Thank you Aman (+5), will update you after doing this.

Suresh

Wilson Samuel · ‎03-08-2015

Hi Suresh,

Hopefully your issue has been resolved by now. If not please read on.

For some reason I think it may be due to the Expired Certificates on the CCM Servers esp the self signed .pem certificates.

Try regenerating them if they are expired.

HTH

Suresh Hudda · ‎03-08-2015

Thanks a lot Terry (+5), really appreciate it :-) I will follow these steps and let u know.

Suresh

Terry Cheema · ‎03-08-2015

No worries - I am assuming there is no firewall between your DR site and the Pub DC site. But if there is a firewall first of all confirm its not blocking any traffic.

If not then follow the steps above and lets see what are the results.

-Terry

Suresh Hudda · ‎03-08-2015

Hi Terry,

There are two ASA firewalls between DC and DR nodes. Tomorrow, I will take pcap and will check it and confirm.

Suresh

j.huizinga · ‎03-08-2015

How much bandwidth is between the sites?

Think you need 2MB

j.huizinga · ‎03-08-2015