CUCM 7.1.2 DRS Issue

[crush5454]

Hello all,

I am working on an issue where attempting to add a backup device to DRS in CUCM 7.1.2 fails with an error message indicating "Status: Local Agent is not responding. This may be due to Master or Local Agent being down."

Initially the CUCM had all certificates expired. I have regenerated all certificates and downloaded the ipsec.pem file from the certificate store of the publisher. I have uploaded it with the certificate name ipsec-trust and root certificate of ipsec.pem. I have used this same file on all of the subscribers, and checked that the serial number matches the ipsec.pem file of the publisher. However this issue is persisting.

I have reviewed the following Cisco document and followed these steps from the first problem:

http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-communications-manager-version-71/111796-cucm-drf.html#topic1

I have the following questions:

1) Can you please clarify this step? Is this just the drop down ipsec-trust from the drop down menu? I see nothing or no options for entering a "caption."

1. Upload the downloaded ipsec.pem file with the caption ipsec-trust.

2) Is it OK to use the downloaded ipsec.pem file from the publisher on each of the subs or is necessary to download the ipsec.pem from each sub and then upload that to each sub as the ipsec-trust certificate name?

3) If I have done the steps correctly and the serial numbers match on all servers what is the next step? Anything else that can be checked?

[Brandon Pierce]

Have you verified DRF is actually running?  I would try to restart that before messing with certificates.  So all you are trying to do at this point is just add a backup device then? 

[crush5454]

Hi Brandon,

Yes, I've confirmed DRF Master and Local are running on the Pub, and Subs. I've tried resetting the services on all of the servers, and I've even been as far as rebooting the entire cluster and confirming the services are running after the reboot.

You are correct at this point all I'm trying to do is add a backup device and the system won't let me. There is about a 15 second delay after I select backup device from the drop down menu and finally the system gives me the error message about the local agent issue.

[GRANT GATHAGAN]

Any further progress on this ?

Since the documentation you linked to states that the goal is to store the Publisher's info on all Subs, that would indicate that you're doing it correctly.

It's a pity the original documentation isn't clearer. Its description would lead you to believe that you should download the sub's ipsec cert and install it as the ipsec-trust cert for that sub.

I've just discovered that I have the identical problem.
The ipsec and ipsec-trust certs on my Pub and Sub expired around August of last year.

My additional wrinkle:
I also back up my Unity Connection server with the same method.
As it's not part of the UCM cluster, it has its own ipsec and ipsec-trust certs.
Unlike my Pub and Sub, it's still backing up successfully, even though its ipsec and ipsec-trust certs also expired last year.
Since the Unity server is backing up, I'm disinclined to fix what isn't broken.

Before I try this same process, are there any disruptions of the phone system caused by either the regeneration process or re-starting the DRF services?
I'd hate to mess up something else when trying to fix this issue.

[crush5454]

Hi Grant,

Yes, I have further info to report on this issue and the resolution. Before I go any further please know that messing with the certificates will cause your phones to reboot so make sure you do this in a maintenance window.

My particular issue was not covered by the documentation at all. I used the RTMT tool to pull the DRF Master and Local agent logs and I saw the following snippet:

INFO [main] - drfUtils:ReadXMLObject: File: /usr/local/platform/conf/drfConfiguration.xml

INFO [main] - drfUtils:ReadXMLObject: Unable to read XML file: /usr/local/platform/conf/drfConfiguration.xml

INFO [main] - drfUtils:ReadXMLObject: The Message:  : Content is not allowed in prolog.

So what this basically means is that the file at the following path: /usr/local/platform/conf/drfConfiguration.xml is corrupt. In my case the file was completely blank. The only way to manipulate this file is via the root account which only TAC is supposed to get. I opened a TAC case and provided them my logs and we went through the procedure required to create and activate the root account. In this particular version another bug existed that prevented the root account from working. I had to do a minor version upgrade to the latest version of 7.1 to get around the TAC root bug.

The root account bug is CSCti05853 incase anyone hits this issue. This bug is fixed by upgrading to the latest version of 7.1.

Since the file drfConfiguration.xml was blank we used vi to manually rebuild the file with the appropriate text. Once we did this we restarted the DRF Master and Local agents and the CUCM backup completed successfully.

If I can answer any other questions regarding this issue please let me know.

Thanks,

Jonathan

[GRANT GATHAGAN]

Thanks for the update, Jonathan

I'll be sure and schedule this for off-hours.

I don't have RTMT set up on my system, so I'd better do that and check for that corruption issue beforehand.

[Gordon Ross]

I thought DRS used SSH to communicate across servers. In which case, fiddling with the IPSEC certs isn't necessary.

GTG

[crush5454]

Hi Gordon,

The link above from Cisco indicates that one should fiddle with the ipsec.pem file and ipsec-trust files when the local agent is not responding issue presents. Unfortunately I haven't been able to find any info regarding how to deal with this if it's not a certificate issue.

[Brandon Pierce]

You said you already tried this guys method that solved the same problem?

https://supportforums.cisco.com/discussion/11971041/cucm-91-backup-issue-local-agent-not-responding

Try that if you haven't done it in that exact order.  Running out of ideas.

[crush5454]

Yes, I've tried these steps and I'm still having the problem. I did this exactly.

1) Regen ipsec.pem on the Pub and all Subs (it was expired because the system is more than 5 years old.)

2) Download ipsec.pem from the Pub and save to local computer.

3) Delete ipsec-trust.pem from the Pub and all Subs.

4) Upload the saved local ipsec.pem file to the Pub with the title ipsec-trust.

5) Upload the saved local ipsec.pem file to each sub with the title ipsec-trust.

What isn't clear in the instructions is if it is OK to use the local saved ipsec.pem that has been regenerated on the pub on all subs as the uploaded ipsec-trust title. Or does this process have to be done on each sub individually?

Last but not least is this fixed in the latest version of CUCM 7? Would an upgrade take care of this issue?