cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Cisco UC Certificates Renewal Guide

1197
Views
30
Helpful
4
Comments

As this is something that is quite often asked in the community I thought I had a go at creating my first document. It's about how to generate or renew certificates in a Cisco UC environment.

Overview

Unified Communication system uses self-signed and third-party-signed certificates. Certificates are used between devices in the system to securely authenticate devices, encrypt data, and hash the data to ensure its integrity from source to destination. When a system trusts a certificate, this means that there is a preinstalled certificate on your system which states it is fully confident that it shares information with the correct destination. Otherwise, it terminates the communication between these points. In order to trust a certificate, trust must already be established with a third-party certificate authority (CA). The devices must know that they can trust both the CA and intermediate certificates before they can trust the server certificate presented by the exchange of messages called the secure sockets layer (SSL) handshake.

Types of Certificates

NameDescriptionMulti-Server supportRelated service
tomcatThis self-signed root certificate is generated during installation for the HTTPS node.YesTomcat and TFTP
ipsecThis self-signed root is generated during isntallation for IPsec connection with MGCP and H.323 gateways.NoCisco Disaster recovery system (DRS) Local and Cisco DRF Master
CallManagerThis self-signed root is installed automatically when you install Cisco Unified Communication Manager. this certificate provide node identification, including the node name and the global unique identifier (GUID).YesCallManager, CAPF and CTI
CAPFThe system copies this root certificate to your node or to all nodes in the cluster after you complete the Cisco client configuration.NoCallManager and CAPF
TVSThis is a self-signed root certificate.NoTrust Verification Service (TVS)
ITLRecoveryThis certificate is used when devices lose their trusted statusNoTrust Verification Service (TVS)
cup-xmpp

Presented to a Cisco Jabber client, Third-Party XMPP client, or a CAXL based application when the XMPP session is being created.

Yes

Cisco XCP Connection Manager, Cisco XCP Web Connection Manager, Cisco XCP Directory service and Cisco XCP Router service

cup-xmpp-s2sPresented for XMPP interdomain federation when connecting to externally federated XMPP systemsYesCisco XCP XMPP Federation Connection Manager
cupUsed for internal server communication.No

Cisco SIP Proxy and Cisco Presence Engine

AUTHZUsed for Keys for OAuth Refresh LoginsNo

 

 

CUCM/CUC/CUPS/UCCX Versions 10.X and Later - Certificate Regeneration

In general the same applies to UC systems version 8.x and 9.x, besides that these do not support multi SAN certificates, so each server needs it's own CA signed certificate.

 

Warning:

DO NOT regenerate CallManager and TVS certificates at the same time.  This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster.

 

Note:

* When regenerating certificates, it is advised that this action be performed after hours due to the needs of restarting services and rebooting all phones in the cluster.

** Monitor phone registration through RTMT tool.

*** Check Relevant Certificates Procedure as per UC nodes for CUC/CUPS/UCCX

 

To regenerate expiring or expired certificates please follow the procedures below.

Common certificates


Tomcat Certificate

Note: This certificate will only need to be regenerated on the publisher since it is pushed to all the nodes.

Specific note for UCCX:
Snag_1416d85.png

  1. Log into Cisco Unified OS Administration Page of Publisher
    • Security
    • Certificate Management
    • Find
  2. Click on tomcat.pem Certificate.
  3. Once open click “Generate CSR”.

Snag_1541eec.png

  1. Make sure the below is selected
    • Choose Certificate Purpose either “Tomcat” OR “CallManager” depending on what certificate you're about to renew.
    • Choose Distribution as “Multi-Server SAN”.
    • Click on Generate and wait until you get successful generation prompt.
    • After Successful generation close the window.

Snag_1578836.png

  • Download CSR File and send it to the team that handles signing of certificates.
  • After receiving the “.CER” file from this team, upload that certificate to the OS admin Publisher GUI. Make sure to choose correct certificate purpose as per the CSR file, if its for call manager then choose “CallManager” and if its for Tomcat then choose “tomcat”. If not already present in the system the signing CA root certificates needs to be uploaded to the trust store for each of these, “callmanager-trust” for CallManager or “tomcat-trust” for Tomcat.

Snag_15e04fb.png

  • When all Nodes have received the signed tomcat certificate, restart the Cisco Tomcat service on all the nodes beginning with the publisher, followed by the TFTP and then subscribers one by one.

Snag_15f8939.png

Make sure services are not restarted on all nodes simultainiously as that will disrupt services. Recommended to do the restart of services with a gap of minimum 15 minutes between nodes. Disregard the part with disable and re-enable SSO. This is not required.

  1. To restart tomcat you will need to open a CLI to each node and enter the following command, "utils service restart Cisco Tomcat".

If SSO is used the below needs to be done after Tomcat certificate has been renewed.

  1. From System > SAML Single Sign-On, download new metadata file for system and send appropriate team to update trust in IdP.
  2. After trust in IdP has been updated do a SSO test on all nodes from System > SAML Single Sign-On on the CUCM publisher.

 

CallManager Certificate

Warning:

DO NOT regenerate CallManager and TVS certificates at the same time.  This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster. A reboot of ALL phone will be required after the regeneration of CallManager certificate.

Note: This certificate will only need to be regenerated on the publisher since it is pushed to all the nodes.

 

  1. Open publisher GUI to Cisco Unified OS Administration >  Security > Certificate Management
  2. Click “Find” showing all the certificates.
    • Click on the “CallManager.pem” Certificate.
  3. Generate CallManager CSR file by following same steps as in Tomcat Certificates Generation.
  4. After all nodes have received the signed CallManager certificate, the following services needs to be restarted in the following order:

 

Make sure services are not restarted on all nodes simultaneously as that will disrupt services. Recommended to do the restart of services with a gap of minimum 15 minutes between nodes.

  1. Log into Publisher’s Cisco Unified Serviceability
    • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco CallManager Service where running.
    • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco CTIManager Service only where running.
    • Cisco Unified Serviceability > Tools > Control Center - Network Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Trust Verification Service (TVS).
    • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Tftp Service only where running.
  2. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  3. Phones will now upload the new ITL while they reboot.

 

IPSEC Certificate

Note: IPSEC certificates will affect the DRF master and local service witch deals with backup and restore.

 

  1. Open a GUI for each node in the cluster starting with the publisher, then each subscriber/TFTP and navigate to Cisco Unified OS Administration >  Security > Certificate Management.
  2. On all of the GUI pages beginning with the publisher Click “Find” showing all the certificates.
    • Click on the “IPSEC.pem” Certificate.
    • Once open Click “Regenerate” Wait until you see “Success” then close pop-up or go back to “Find/List”.
  3. Continue with subsequent Subscribers, follow the procedure in step 2 and complete on all subscribers in your cluster.
  4. After all Nodes have regenerated the IPSEC certificate, The following services will need to be restarted in the following order:
    1. Log into Publisher’s Cisco Unified Serviceability.
      • Cisco Unified Serviceability > Tools > Control Center - Network Services.
        • On the Publisher only, restart Cisco DRF Master Service.
        • Once the service restart completes, restart Cisco DRF Local Service on the Publisher.
        • Continuing with the subscribers, restart Cisco DRF Local Service.

 

ITLRecovery Certificate

Note:

The ITLRecovery Certificate is used when devices lose their trusted status. The certificate appears in both the ITL and CTL (when CTL provider is active).
If devices lose their trust status, you can use the command
utils itl reset localkey for non-secure clusters and the command utils ctl reset localkey for mix-mode clusters. Read the Security guide for your Call Manager version to become familiar with how the ITLRecovery certificate is used and the process required to recover trusted status.
If the cluster has been upgraded to a version that supports a key length of 2048 and the clusters server certificates have been regenerated to 2048 and the ITLRecovery has not been regenerated and is currently 1024 key length, the ITL recovery command will fail and the ITLRecovery method will not be able to be used.

 

  1. Navigate to each server in your cluster (in separate tabs of your web browser) begin with the publisher, then each subscriber.  Navigate to Cisco Unified OS Administration > Security > Certificate Management > Find.
    • Select the ITLRecovery pem Certificate.
    • Once open select Regenerate and wait until you see the Success pop-up then close pop-up or go back and select Find/List.
  2. Continue with subsequent Subscribers; follow the same procedure in step 2 and complete on all subscribers in your cluster.
  3. After all Nodes have regenerated the ITLRecovery certificate, these services will need to be restarted in the order as follows:
    • Navigate to Cisco Unified Serviceability > Tools > Control Center - Network Services.
    • On the publisher select Restart on Cisco Trust Verification Service.
    • If you are in Mixed Mode – Update the CTL before you proceed Token - Tokenless.
    •  Once the service restart completes, continue with the subscribers and restart the Cisco Trust Verification Service.
  4. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  5. Phones will now upload the new ITL while they reboot.

 

CUCM specific certificate

 

CAPF Certificate

Note: Ensure the cluster is not in Secure or Mixed Mode. 

Check if the cluster is in secure cluster mode.

  1. Navigate to the Cisco Unified CM Administration > System > Enterprise Parameters.
    • Section “Security Parameters” Check Cluster Security Mode and ensure is is set to “0” for Cluster Secure Mode. If the value if 0 it is not in a secure cluster mode. If it is anything else it is in a secure mode and you will need to use a CTL client to update the CTL file. The link to the CTL phone security guide can be found at this link. https://supportforums.Cisco.com/docs/DOC-18834
  2. Open a GUI for each server in the cluster starting with the publisher, then each subscriber/TFTP in sequence and navigate to Cisco Unified OS Administration >  Security > Certificate Management
  3. On all of the GUI pages beginning with the publisher Click “Find” showing all the certificates.
    • Click on the “CAPF.pem” Certificate.
    • Once open Click “Regenerate” Wait until you see “Success” then close pop-up or go back to “Find/List”.
  4. Remove the outdated CAPF certificate from the trust store on each node after regeneration of new.
  5. Continue with subsequent Subscribers, follow the same procedure in step 3-4 on all subscribers in the cluster.
  6. After all nodes have regenerated the CAPF certificate, the following services needs to be restarted in the following order:
    1. Log into Publisher’s Cisco Unified Serviceability.
      • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
        • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Certificate Authority Proxy Function Service ONLY where running. (Even if not running on any node, continue with instructions).
      • Cisco Unified Serviceability > Tools > Control Center - Network Services.
        • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Trust Verification Service (TVS).
      • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
        • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Tftp Service only where running.
  7. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  8. Phones will now upload the new ITL while they reboot.

 

TVS Certificate

Warning:

DO NOT regenerate CallManager and TVS certificates at the same time.  This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster. A reboot of ALL phone will be required after the regeneration of TVS certificate.

 

  1. Open a GUI for each node in the cluster starting with the publisher, then each subscriber/TFTP in sequence and navigate to Cisco Unified OS Administration >  Security > Certificate Management.
  2. On all of the GUI pages beginning with the publisher Click “Find” showing all the certificates.
    • Click on the “TVS.pem” Certificate.
    • Once open Click “Regenerate” Wait until you see “Success” then close pop-up or go back to “Find/List”.
  3. Continue with subsequent Subscribers; following the same procedure in step 2 and complete on all subscribers in the cluster.
  4. After all nodes have regenerated the TVS certificate, The following services needs to be restarted in the following order:
    1. Log into Publisher’s Cisco Unified Serviceability.
      • Cisco Unified Serviceability > Tools > Control Center - Network Services.
      • On the Publisher only, restart Cisco Trust Verification Service.
  • Once the service restart completes, continuing with the TFTP followed by subscribers one by one  and restart Cisco Trust Verification Service.
  1. Beginning with the Publisher then continuing with the TFTP followed by subscribers one by one, restart Cisco Tftp Service only where running.
  2. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  3. Phones will now upload the new ITL while they reboot.

 

CUPS specific certificate

 

CUP-XMPP Certificate

Note:

Presented to a Cisco Jabber client, Third-Party XMPP client, or a CAXL based application when the XMPP session is being created.

The associated trust-store is used to verify connections made by Cisco XCP Directory service in performing LDAP search operations for third-party XMPP clients.

The associated trust-store is used by the Cisco XCP Router service when establishing secure connections between IM and Presence Service servers if the Routing Communication Type is set to Router-to-Router.

This certificate will only need to be regenerated on the publisher since this certificate is pushed to all the nodes.

 

  1. If you click on Generate and save, this will remove the current certificate if the server is rebooted or services restarted.
  2. Log into Cisco Unified IM and Presence OS Administration Page of Publisher.
    • Security -> Certificate Management
    • Find Certificate List Where the Certificate contains cup-xmpp.
  3. Click on cup-xmpp certificate and you can view the old certificate.
  4. Next click “Generate CSR”.

Snag_16f2726.png

Make sure the below is selected.

  • Choose Certificate Purpose “cup-xmpp”.
  • Choose distribution as “Multi-Server SAN”.
  • Applicable for cup-xmpp: In Subject Alternate Names fill in the applicable names for your setup.

For example altNames:

          usdecups02.domain.com (dNSName)

          domain.com (dNSName)

          usdecups01.domain.com (dNSName)

 

  • Remove or click the minus side for the other Auto-populated names to remove them from the SAN.
  • The Key Length will be 2048 and the Hash Algorith will be SHA256.

Snag_17304e5.png

  1. Click on Generate and wait until you get successful generation prompt.
  2. After successful generation close the window.
  3. Download CSR File and send it to the team that handles signing of certificates.
  4. After receiving the “.cer” file from this team, upload that certificate to the OS admin Publisher GUI. Make sure to choose correct certificate purpose as per the CSR file.
    • Go to Security -> Certificate Management.
    • Press Upload Certificate/Certificate chain.
    • Select Certificate Purpose: cup-xmpp
    • Choose the .cer file and press Upload. 
    • After successful upload close the window.

Redo step 3 from first section to view the new certificate and verify the expiration has changed to the new date.

 

After all nodes have received the signed cup-xmpp certificate, the following services needs to be restarted in the following order:

Make sure services are not restarted on all nodes simultaneously as that will disrupt services. Recommended to do the restart of services with a gap of minimum 15 minutes between nodes.

  1. Login into CLI of the Cisco IM and Presenece (CUPS servers) and restart Cisco Intercluster Sync Agent.
    • Run the following command on the Publisher "utils service restart Cisco Intercluster Sync Agent".
    • After the restart finishes on the Publisher, then run the same command on the Subscriber.
  2. Restart Cisco XCP Router services on all nodes. This can be done from either CLI or Serviceability web page.
    • From CLI run the following command on the Publisher "utils service restart Cisco XCP Router".
    • After the restart finishes on the Publisher, then run the same command on the Subscriber.
  3. To verify the new certificate you can check Inter-Clustering on Cisco Unified IM and Presence Administration page from a different cluster to test the connection to the cluster where the certificate has been renewed.
    • Go to Presence -> Inter-cluster.
    • Select the Cluster that you renewed the certificate and verify the connectivity status.

 

Troubleshoot Certificate Errors

If you encounter an error when you attempt to access Unified Communications Manager services from an IM and Presence Service node or IM and Presence Service functionality from a Unified Communications Manager node, the source of the issue is the tomcat-trust certificate. The error message Connection to the Server cannot be established (unable to connect to Remote Node) appears on the following Serviceability interface windows:

  • Service Activation
  • Control Center - Feature Services
  • Control Center - Network Services

Use this procedure to help you resolve the certificate error. Start with the first step and proceed, if necessary. Sometime, you may only have to complete the first step to resolve the error; in other cases, you have to complete all the steps.

 

  1. From Cisco Unified OS Administration, verify that the required tomcat-trust certificates are present: Security > Certificate Management.
    • If the required certificates are not present, wait 30 minutes before checking again.
  2. Choose a certificate to view its information. Verify that the content matches with the corresponding certificate on the remote node.
  3. From the CLI, restart the Cisco Intercluster Sync Agent service with command "utils service restart Cisco Intercluster Sync Agent".
  4. After the Cisco Intercluster Sync Agent service restarts, restart the Cisco Tomcat service with command "utils service restart Cisco Tomcat".
  5. Wait 30 minutes. If the previous steps do not address the certificate error and a tomcat-trust certificate is present, delete the certificate. After you delete the certificate, you must manually exchange it by downloading the Tomcat and Tomcat-ECDSA certificate for each node and uploading it to its peers as a tomcat-trust certificate.
  6. After the certificate exchange is complete, restart Cisco Tomcat on each affected server with command "utils service restart Cisco Tomcat".
Comments
VIP Mentor

Hi Roger,

Great document. Thank you for putting this together. It's so much easier for us all to have a single go to reference for cert related information like this.

 

Collaborator

Thanks Ayodeji,

Appreciate your comment.

VIP Rising star

Hello Roger,

 

That’s a great post, thank you for collecting all this info in one piece!

 

George

Collaborator

Your welcome George. Glad you found it useful.

CreatePlease to create content
Content for Community-Ad

Cisco COVID-19 Survey