Dealing with Expired Certs in Mixed-Mode

[William Bell]

I have done a fair amount of research on this topic and while I have deployed mixed-mode clusters, I haven't had a situation quite like the one I need to contend with in the next couple of weeks. I am trying to be as prepared as possible and am looking for feedback on some procedures I am drafting.

The situation:

• (4) node cluster (clustering over the WAN)
• TFTP is enabled on all nodes (that is going to change as a result of our assessment findings)
• Cluster is running in mixed-mode
• Most certificates on the Publisher node are expired
  • tomcat cert
  • ipsec cert
  • host-name ipsec-trust cert
  • call manager cert (callmanager.pem)
  • CAPF cert
  • CAPF trust cert
• One of the subscriber nodes is in the same boat as the Publisher node (they were deployed at the same time and were the first nodes in this cluster)
• The other two nodes (in a DR datacenter) have valid certificates  (until 2016) except for the publisher node server cert (which has expired)
• The publisher node and the subscriber node that has the expired certs were also installed without DNS being enabled (no domain and no DNS resolvers specified - therefore, I expect that DNS client was not enabled during install)
• It is worth noting the following:
  • Customer enabled mixed-mode because one of the security folks got hot and heavy on encryption. However, they limited the scope to phones only. So, IP Phone to IP Phone == authenticated/encrypted. They have a Unity Connection system with secure ports and that is it. Gateways: no encryption. CCX, etc. == no encryption
  • During discovery we also found that LSC distribution is fubar. Only a percentage of the phones are using LSC. Likely due to a flaw in the provisioning process. That will be addressed later.
• The version they are running is 6.1(3)  (base, no service releases)

The goal: Get the present solution into a VMware environment running CUCM 9.1(2). Planning on doing the Jump Upgrade procedure (interim hop to 6.1.4).

We found out about the certificate issues during our discovery phase. We have built in time to remediate the certificate issue.

The plan (well, thus far). I am still pulling together my notes and trying to come up with a way to test an implementation plan off line so that I can avoid bricking the phones (they are spread all over north america).

Here is the 10,000 foot view of the plan (obviously, the actually plan will be more detailed):

• Use BAT to disable phone security and uninstall LSC
  • Security Profile mod
  • Certificate Ops
• Reset phones
• DRS Back up
• Download/backup current certs
• Configure DNS
  • set DNS domain name
  • set DNS resolver (primary and secondary)
• Pub node:
  • regenerate tomcat cert
  • restart tomcat service
  • regenerate ipsec.pem
  • regenerate callmanager.pem
  • regenerate capf.pem
• Sub node (repeat above)
• ?should we update the Subs not affected by the cert issue?
• Run the CTL client and update CTL
• Reboot servers
  • Pub then Subs
  • Phones will reset as a result of this process

The customer has said that they are actually fine with the idea of going back to square one and start over with provisioning a secured (mixed-mode) cluster after the 9.1 upgrade. That would be great except that if I uninstall LSCs, change phones to non-secure, and use CTL client to change back to standard-mode, I still have the CTLs left on the phones. No way to bulk delete them in UCM. I am considering using something like UnfiedFX to help me get back to square 1. Right now, I consider this a plan B. Unless feedback to this thread and other research suggests a different tact.

Thanks in advance for any assist.

-Bill

[Karthik Sivaram]

Hi William,

You have a quite a few requirements here. Just to clear things up, there are two type of certificates, first is called "certificate trust", and the other is called "Certificate".  For the trust certificates such as Callmanager_trust you can just click on the certificate, make sure that it is expired, and then delete it. this has no impact on the phones. The other type of certificate is called "Certs", you will need to regenerate those certificates, This will regenerate the certificate and also recreates the new "CAPF-trust" or "CallManager-trust" certificates with new date/time ranges.

Doing the above will not impact the phones are the services, however after regenerating the certificates, you will need to restart all the services related to this certificate, for example if you regenerate the tftp certificate, you will need to restart the tftp service on all the servers in the cluster. Same for the Callmanager and the Tomcat.

Please note that whenever you regenerate the Call manager certificate, you will need to run the CTL client with the same Token you used when the server was changed to mixed mode.

In General the below is the procedure to regenerate the certificate

- log into the "Cisco Unified OS Administration" page of the publisher

- choose Security>Certificate Management

- click the link for the expiring certificate

- click "Regenerate"

- restart the service that uses the certificate

That will regenerate the certificate on the publisher. Within the next

10-15 minutes, the updated certificate will  be propagated

to the subscribers.

For more details you could refer to :

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/cucos/6_1_1/cucos/iptpch6.html#wp1040760

http://www.cisco.com/en/US/prod/collateral/voicesw/direct_upgrade_procedure_for_cisco_unified_communications_manager_releases_6.1_2.pdf

Hope this Helps!

Regards,

Karthik Sivaram

[William Bell]

Karthik,

Thanks for the input. Does the flow I laid out in the original post jive? I have looked through the Cisco documentation. I understand the content provided by Cisco. I just think it is light on details. I have also read some great docs posted to the forum. I am trying to cobble together a cohesive process from different resources.

-Bill

[Wilson Gonzalez]

Another thing to consider and make sure is that you have the same original token that was used to enable security on the cluster. Otherwise this is a bit more painful and will need Cisco TAC to help you. They have to log in with root access and disable security on the cluster.

Also you need to make sure all phones are changed to non-secure profiles and you will have to delete the CTL cert from each phone in which you will require a bulk tool that could do this.

Wilson