CUCM 11 - Tomcat / Tomcat-trust Certificates not synchronizing

[RL5901]

I have a subscriber node that had certificates that were expiring this month. I regenerated them per instructions...

Regenerate Tomcat:

Upon regeneration, the Tomcat certificate automatically uploads itself to tomcat-trust.

OS Admin > Security > Certificate Management > Find > Click tomcat certificate > Regenerate

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200199-CUCM-Certificate-Regeneration-Renewal-Pr.html#anc9

The Tomcat certificate was renewed on the subscriber node but did not synchronize to the other nodes (including the publisher). 

The Cisco Tomcat service was restarted on the publisher first via CLI, then the subscriber node second, then on the other subscribers. 

Still, the Tomcat certificate still did not synchronize to the other nodes (including the publisher). Because of the Tomcat certificate mismatch, I can't get to that subscriber node in Control Center from the publisher in the WebGUI. 

I restarted the Callmanager service on the publisher, then the subscriber. No synch of the certificate. 

Any ideas why the certificate on the subcriber is not synchronizing across the cluster? All documentation claims this is automatic. Can I do the download/upload of the certificate to fix this problem? What are the steps I need to take? 

[b.winter · Accepted solution]

Hi,

as you correctly stated, you can just download the tomcat-cert of the subscriber and upload it as "tomcat-trust" to every other node in the cluster and then restart the Tomcat Service.

Also check, if you have to delete the old tomcat-cert from the tomcat-trust store.

[RL5901]

I haven't deleted the old tomcat-trust certs of this node from the other nodes in the cluster. I only regenerated the tomcat certificate on the subscriber node and restarted the Cisco Tomcat service (via CLI) on all nodes (publisher first, then this subscriber, then the other subscribers). This was what was in the documentation and even in a video from a Cisco TAC engineer. I checked replication and I see only "2" for all nodes in the cluster. So it seems that the documentation on this task is not correct or there is some bug that prevents my version of CUCM (11.0) from behaving correctly. 

If I delete the old tomcat-trust certs and restart the Cisco Tomcat service from all of the other nodes will this correctly perform the certificate synch across the cluster? None of the documentation (Cisco or other) shows that I need to do any of that. 

I can perform the workaround @b.winter but do I download the certificate from tomcat.pem file or the tomcat-trust file to upload it across the cluster? 

[b.winter]

If I delete the old tomcat-trust certs and restart the Cisco Tomcat service from all of the other nodes will this correctly perform the certificate synch across the cluster? None of the documentation (Cisco or other) shows that I need to do any of that

You can try that first, yes.

I can perform the workaround @b.winter but do I download the certificate from tomcat.pem file or the tomcat-trust file to upload it across the cluster?

You need to download the certificate from the "tomcat" store from the subscriber, and upload it into "tomcat-trust" store on all the other nodes in the cluster.

I always check the certificate replication on all nodes in the cluster, if I do certificate changes. And if anything is wrong / missing / old certificates still in there, I correct it manually. That's a normal doing.

The stores without the "-trust" (like tomcat, CallManager, TVS, ...) always contain the certificate of the node itself.

The stores with the "-trust" (like tomcat-trust, CallManager-trust, TVS-trust, ...) contain certificates of other servers (other CUCM- or IMP-nodes, root-certificates, ...), that this node should trust.

[b.winter]

Please remember to accept a post as solution, if your problem could be solved

[Roger Kallberg]

If you where to manually distribute the certificate to your other nodes in the cluster I would recommend you to get the pem file from the top level, ie not from the trust store.

About why it is not distributed across your other nodes, I’m not sure if it would work that way when you have individual tomcat certificates for each node. I’m not 100% sure, but from what I know I think that the distribution of certificates only works if your using a multi SAN certificate and that’s created on the publisher.

One other possible option could be that you would upload the certificates from the other nodes on your publisher in the tomcat trust store as from what I know the functionality for distribution of certificates is on the publisher.

[RL5901 · Accepted solution]

My best guess is that CUCM 11.0 does not follow the documentation I see online. Tomcat-trust certificates (stores) do not distribute automatically across the cluster.

What I did to fix this...

1. Signed in to the Cisco Unified OS Administration of all nodes in the cluster. 
2. Selected the tomcat.pem certificate ("tomcat" certificate) on the subscriber node that needed to be distributed.
3. Downloaded the .PEM File and saved it to my PC
4. On the publisher, in Cisco Unified OS Administration,
   1. selected Upload Certificate/Certificate chain (the Upload Certificate/Certificate Chain window will open)
      1. You will see a message "Warning: Uploading a cluster-wide certificate will distribute it to all servers in this cluster"
         1. This is not going to happen, don't get your hopes up, just continue with the steps
   2. Under Upload Certificate/Certificate Chain section of the window, 
      1. For the Certificate Purpose, choose "tomcat-trust" from the drop-down menu
      2. For the Description(friendly name), type "Trust Certificate"
      3. For the Upload File, found the tomcat.pem file that downloaded and selected "Open"
         1. I observed the "tomcat.pem" file name next to the "Choose File" button at this point
      4. Selected Upload
   3. In the Status box, two messages appeared...
      1. Success: Certificate Uploaded and,
      2. Restart Cisco Tomcat Service using the CLI "utils service restart Cisco Tomcat".
5. Opened my favorite SSH app (SecureCRT, but this could be PuTTY for you.)
   1. Signed in to the publisher via CLI
   2. Ran the command "utils service restart Cisco Tomcat"
   3. Waited for a very long time for the WebGUI to become available again (my gawd, it took forever... or 10 minutes)
6. Checked the publisher via the WebGUI for the new tomcat-trust certificate issued by the subscriber node and it was there with a new expiration date matching the tomcat certificate on the subscriber node.
7. Repeated these actions for the remaining subscribers in the node (except for the issuing subscriber, of course).  

And that solved the problem. 

[Roger Kallberg]

A comment on this part.

1. You will see a message "Warning: Uploading a cluster-wide certificate will distribute it to all servers in this cluster"
   1. This is not going to happen, don't get your hopes up, just continue with the steps

If you where to have a cluster wide certificate it would be distributed.

A cluster wide certificate is in this case a multi SAN certificate where you have one tomcat or call manager certificate, that is created on the publisher, that holds the names of all your nodes in the cluster within one certificate.

The certificates that you’re trying to get distributed are not cluster wide. That’s why they don’t get distributed.