Recover Publisher from Subscriber

[austinwilly]

Hey All,

I need some direction/Advise. A cluster that I've been brought in to help with is in a really tricky situation. After a 2 drive failure in a RAID 5 array their PUB and 2 SUBs are just BOOM. No hope of recovery and drives have been replaced to get the hardware back online ASAP. They have 1 SUB in an offsite location that is currently handling the load. 

Problems:

• Working SUBs DRS is un-responsive so I cannot create a backup.
• All Backups are totally corrupted as apparently the SFTP server they were on flatlined 6+ Months ago and nobody noticed
• Outage is Not an option (Life Safety issue) so we either fix or totally replace and Lead time on Hardware is an issue so we need to at least get a pub up as a stop gap.

I attempted the procedure outlined here: https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/116946-technote-product-00.html#anc20

But when I reached the place where I was supposed to select the Sub to recover from there was no section in the recovery wizard that even gave me the option. Even the one button recovery was missing. 

To make matters worse when I brought the pub up all of the tables seemed to disappear from the sub as if it had replicated the blank tables over from the pub to the sub. I did stop Replication on the sub before i brought the new pub up and I confirmed that the phones at the site never went down. I have since shut the new pub down and all the tables re appeared on the sub (As did my soul which had left my body in fear up until this point) 

Does anyone have any thoughts of what I'm doing wrong that the option to recover the pubs database from the sub is Non-Existent. does anyone have suggestions.

and no the certs on the Sub are not the reason the DRS isn't accessible. Also I attempted to access the DRS system via the command line and it is non responsive. 

Version: Call manager 11.5.1.18900-97

[Roger Kallberg]

With the impact in mind I would think that you should seek the hands-on help from TAC on this as it's not very likely something that you could get the level of help needed from the community.

[austinwilly]

I would love to speak to TAC on this issue.

1. EOS EOL hardware
2. EOS EOL software
3. Someone let SMART lapse

TAC wont help, been down that road already. I'm prepared to Rebuild this cluster from scratch if necessary. This post is just a last ditch attempt to see if anyone has been in this situation and has any suggestions. 

[Jonathan Schulenberg · Accepted solution]

You have to get DRS working on the sub for this restore procedure to work; you have to take a backup successfully.

Have you restarted the DRS service on that sub?

PS- The Cisco account team can override entitlement if they want to and open a case anyway. Usually this only happens if the customer agrees to reinstate support (ie cut a PO) while working the case. And TAC can help on a best-effort basis with EoL versions if they want to (ie the account team advocates on the customer’s behalf). It’s an easy argument if this is life safety. We’re not talking CCM 4.1 here; this procedure is the same on 11.5 and 14.

[austinwilly]

Turns out The DRF Master service on the Sub was stopped. It was in a "Commanded Shutdown" state. I restarted the DRF master service on the sub (and restarted the DRF Local service for good measure) and repeated the procedure and was able to recover the Pubs database from the Sub.

Now to get the rest of the subs online, but that should be easy now the pub is back.

Thanks for pointing me in the right direction.