CUCM-Pub rebuild and restore data from Sub

[Ali Zaghir]

Our CUCM-Pub has failed and no good DRS backup. During the following recovery plan based on this guide
https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/116946-technote-product-00.html

Although I did everything correctly according to the guide above but during the cucm-pub backup I noticed that the ccmdb server is not cucm-sub but cucm-pub and therefore when restoring i didn't have the option to select subscriber as to restore from. It was only the first part till selecting the pub only. I continued the restore process and now I ended up with having only 8 devices on cucm-sub where it was too much more. I verified using run sql select count(*) from device. I think the publisher wiped the sub db in this case. I shut down the pub and now the sub is still online. Althoug the sub's db says only 8 but the phones are all still working. I'am afraid if I reboot the new values in db will replace. How to recover at this point ?

My CUCM version is 9.1.2

Appreciate your support.

[Roger Kallberg]

I’m going to guess based on your outline that you did not even made sure that you had a working DRS backup of your subscriber before you embarked on the process of restoring your publisher? If so your data is lost and you’ll need to recreate it manually or with bulk import.

[Ali Zaghir]

Unfortunately it didn't come to my mind and wasn't part of the procedures. Right now the pub is in shutdown state and the sub is online and all the phones are working properly although when I issue the command run sql select count(*) from device it outputs 8 only while 200+ are registered. I guess if I reboot the cucm will read from the sql which is almost empty isn't it?

Also, on subscriber it's not showing any phones or enduser on gui.

However, we have an old DRS backup from 2019. I'm thinking of turning on the pub in isolation of subscriber and see if the db restores most of the config at least. The reason I'm fearful of doing it without isolating them is that the DRS backup might be with an old security password that is undocumented and i'm afraid the pub not able to decrypt it. What do you think the best thing to do in this situation before going to manual method of recreating all?

[Roger Kallberg]

Doing it in isolation would be the recommended cause of action in your situation.

On the procedure not including the step. You would need to think a little for yourself. You cannot expect that someone else can create a document that contains all aspects that would cover all the details that pertains to your specific situation.

[Ali Zaghir]

Thanks for the update. Of course I agree not everything can be included in
the document and I'm not criticizing the document but on the contrary it's
well demonstrated. I just wanted to say that I didn't read it and during
the stressful situation I simply missed that part.

Do you have any idea why the pub didn't take the ccmdb from sub and instead
copied his to it? Although i followed all the steps precisely.

[Roger Kallberg]

Not really. A guess would be that you did something not totally accurate. It’s an easy thing to do and not read the whole information that comes up during the process.

[Roger Kallberg]

I took the time to look through the document as it was a few years ago since I last read it and it does actually include a step to create a backup before you do the steps to start the restore of the database from a subscriber node. Based on that you said that you did not do that,and it was not included in the document, I’m going to venture out on a speculation and guess that you might not have followed the step to stop replication of the database on the subscriber node(s) before you installed the new publisher or that you did something wrong in the step where you selected the subscriber to restore from. Either way I’m sorry to say that it is quite evident that you did not actually follow the full outline of the document, as if you would you would not have lost your data.

[Ali Zaghir]

You might be referring to " Cisco strongly recommends that you keep a full Disaster Recovery Framework (DRF) backup of the entire cluster.". If so, it points to backing up the entire cluster before such disaster occur. 

To make sure we're talking about the same document it's the below one which i referred to in the beginning of my post:

 https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/116946-technote-product-00.html

I followed the same steps and db replication was stopped on subscriber, but I'm doubting that it could be the one thing that I didn't do which is installing same cop files on publisher. I selected the pub to be restored but the weird thing is that there was no option of the db to restore from as in the figure of the support document but i thought it might be because of different version.

I'm not the guy who is responsible for this system but I'm just working on it after the failure of publisher to help an IT guy in there. The customer is in a remote area and I didn't try to restore it in isolate state yet.

One last question if you don't mind. Can I have a full backup from subscriber at the moment? Knowing that all phones configurations are working properly on subscriber although the db contains the new wiped data from pub, verified by run sql commands, which i think is due to the fact that it's not restarted yet.

[Roger Kallberg]

 

I’m referring to this step.

[Ali Zaghir]

That's meant for the Publisher not the subscriber to restore the non-DB components. Also, explained in the note:

"Note: Although you can use the subscriber DB for the restore, a backup is still required in order to restore the non-DB components."

[Roger Kallberg]

From what I understand it would as well have included the database in the state it was before you proceeded with the restore process. So you would have had a backup of your subscriber database if you’d had followed the instructions in the document.

[Ali Zaghir]

Yes I did follow the instructions in the document and did a backup for the publisher, but the dbreplectation was stopped in the subscriber as per the document and as I did actually. So, I'm not sure why it didn't take the ccmdb from it but copied its empty db to sub. The only thing I feel worried about that I could avoid it is that I shouldn't restore the pub since I didn't see the option to copy db from sub as per the document. For some reason, the restore wizard on pub didn't show that option of copying db from subscriber and I'm not sure if it's related to my version 9.1.2 or related to that one cop file is missing in the newly installed publisher.

Also, there is a note in the document saying that:

"Note: It is critical that the publisher node has the CCMDB component registered."

And I did see the server is "CCM-Pub" for the component "ccmdb" where I think it should have been "CCM-Sub".

However, in my current situation now. Can I have a full backup from subscriber at the moment? Knowing that all phones configurations are working properly on subscriber although the db contains the new wiped data from pub, verified by run sql commands, which i think is due to the fact that it's not restarted yet.

[Roger Kallberg · Accepted solution]

There is as I can tell no point in taking a backup at this moment. You’ll need to come to terms about that your data is lost. You’re option is to recreate it by suggestions in previous replies. I’m sorry to be this blunt.

[Ali Zaghir]

Ok. I will try the old backup on pub in isolation state and if data is good i will restore it to subscriber. Otherwise will go on it from scratch which is very tough but inevitable. Thank you for your time.