Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Create a new article
50230
Views
101
Helpful
31
Comments

Cisco UC Certificates Renewal Guide

Roger Kallberg
VIP
VIP

‎04-30-2020 06:45 AM - edited ‎02-07-2024 04:14 AM

Overview

Unified Communication system uses self-signed and third-party-signed certificates. Certificates are used between devices in the system to securely authenticate devices, encrypt data, and hash the data to ensure its integrity from source to destination. When a system trusts a certificate, this means that there is a preinstalled certificate on your system which states it is fully confident that it shares information with the correct destination. Otherwise, it terminates the communication between these points. In order to trust a certificate, trust must already be established with a third-party certificate authority (CA). The devices must know that they can trust both the CA and intermediate certificates before they can trust the server certificate presented by the exchange of messages called the secure sockets layer (SSL) handshake.

Types of Certificates

Name Description Multi-Server support Related service

tomcat
tomcat-ECDSA

 This self-signed root certificate is generated during installation for the HTTPS node. Yes Tomcat and TFTP
ipsec This self-signed root is generated during isntallation for IPsec connection with MGCP and H.323 gateways. No Cisco Disaster recovery system (DRS) Local and Cisco DRF Master

CallManager
CallManager-ECDSA

 This self-signed root is installed automatically when you install Cisco Unified Communication Manager. this certificate provide node identification, including the node name and the global unique identifier (GUID). Yes CallManager, CAPF and CTI
CAPF The system copies this root certificate to your node or to all nodes in the cluster after you complete the Cisco client configuration. No CallManager and CAPF
TVS This is a self-signed root certificate. No Trust Verification Service (TVS)
ITLRecovery This certificate is used when devices lose their trusted status No Trust Verification Service (TVS)
cup-xmpp

Presented to a Cisco Jabber client, Third-Party XMPP client, or a CAXL based application when the XMPP session is being created.

Yes

Cisco XCP Connection Manager, Cisco XCP Web Connection Manager, Cisco XCP Directory service and Cisco XCP Router service
cup-xmpp-s2s Presented for XMPP interdomain federation when connecting to externally federated XMPP systems No Cisco XCP XMPP Federation Connection Manager
cup Used for internal server communication. No

Cisco SIP Proxy and Cisco Presence Engine
AUTHZ Used for Keys for OAuth Refresh Logins No

 

 

CUCM/CUC/CUPS/UCCX Versions 10.X and Later - Certificate Regeneration

In general the same applies to UC systems version 8.x and 9.x, besides that these do not support multi SAN certificates, so each server needs it's own CA signed certificate.

 

Warning:

DO NOT regenerate CallManager and TVS certificates at the same time.  This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster.

 

Note:

* When regenerating certificates, it is advised that this action be performed after hours due to the needs of restarting services and rebooting all phones in the cluster.

** Monitor phone registration through RTMT tool.

*** Check Relevant Certificates Procedure as per UC nodes for CUC/CUPS/UCCX

 

To regenerate expiring or expired certificates please follow the procedures below.

Common certificates


Tomcat Certificate

Note: This certificate will only need to be regenerated on the publisher since it is pushed to all the nodes.

Specific note for UCCX:
Snag_1416d85.png

  1. Log into Cisco Unified OS Administration Page of Publisher
    • Security
    • Certificate Management
    • Find
  2. Click on “tomcat.pem” Certificate.
  3. Once open click “Generate CSR”.

Snag_1541eec.png

  1. Make sure the below is selected
    • Choose Certificate Purpose either “Tomcat” OR “CallManager” depending on what certificate you're about to renew.
    • Choose Distribution as “Multi-Server SAN”.
    • Click on Generate and wait until you get successful generation prompt.
    • After Successful generation close the window.

Snag_1578836.png

  • Download CSR File and send it to the team that handles signing of certificates.
  • After receiving the “.CER” file from this team, upload that certificate to the OS admin Publisher GUI. Make sure to choose correct certificate purpose as per the CSR file, if its for call manager then choose “CallManager” and if its for Tomcat then choose “tomcat”. If not already present in the system the signing CA root certificates needs to be uploaded to the trust store for each of these, “callmanager-trust” for CallManager or “tomcat-trust” for Tomcat.

Snag_15e04fb.png

  • When all Nodes have received the signed tomcat certificate, restart the Cisco Tomcat service on all the nodes beginning with the publisher, followed by the TFTP and then subscribers one by one.

Snag_15f8939.png

Make sure services are not restarted on all nodes simultainiously as that will disrupt services. Recommended to do the restart of services with a gap of minimum 15 minutes between nodes. Disregard the part with disable and re-enable SSO. This is not required.

  1. To restart tomcat you will need to open a CLI to each node and enter the following command, "utils service restart Cisco Tomcat".

If SSO is used the below needs to be done after Tomcat certificate has been renewed.

  1. From System > SAML Single Sign-On, download new metadata file for system and send appropriate team to update trust in IdP.
  2. After trust in IdP has been updated do a SSO test on all nodes from System > SAML Single Sign-On on the CUCM publisher.

 

CallManager Certificate

Warning:

DO NOT regenerate CallManager and TVS certificates at the same time.  This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster. A reboot of ALL phone will be required after the regeneration of CallManager certificate.

Note: This certificate will only need to be regenerated on the publisher since it is pushed to all the nodes.

 

  1. Open publisher GUI to Cisco Unified OS Administration >  Security > Certificate Management
  2. Click “Find” showing all the certificates.
    • Click on the “CallManager.pem” Certificate.
  3. Generate CallManager CSR file by following same steps as in Tomcat Certificates Generation.
  4. After all nodes have received the signed CallManager certificate, the following services needs to be restarted in the following order:

 

Make sure services are not restarted on all nodes simultaneously as that will disrupt services. Recommended to do the restart of services with a gap of minimum 15 minutes between nodes.

  1. Log into Publisher’s Cisco Unified Serviceability
    • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco CallManager Service where running.
    • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco CTIManager Service only where running.
    • Cisco Unified Serviceability > Tools > Control Center - Network Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Trust Verification Service (TVS).
    • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
      • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Tftp Service only where running.
  2. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  3. Phones will now upload the new ITL while they reboot.

 

IPSEC Certificate

Note: IPSEC certificates will affect the DRF master and local service witch deals with backup and restore.

 

  1. Open a GUI for each node in the cluster starting with the publisher, then each subscriber/TFTP and navigate to Cisco Unified OS Administration >  Security > Certificate Management.
  2. On all of the GUI pages beginning with the publisher Click “Find” showing all the certificates.
    • Click on the “IPSEC.pem” Certificate.
    • Once open Click “Regenerate” Wait until you see “Success” then close pop-up or go back to “Find/List”.
  3. Continue with subsequent Subscribers, follow the procedure in step 2 and complete on all subscribers in your cluster.
  4. After all Nodes have regenerated the IPSEC certificate, The following services will need to be restarted in the following order:
    1. Log into Publisher’s Cisco Unified Serviceability.
      • Cisco Unified Serviceability > Tools > Control Center - Network Services.
        • On the Publisher only, restart Cisco DRF Master Service.
        • Once the service restart completes, restart Cisco DRF Local Service on the Publisher.
        • Continuing with the subscribers, restart Cisco DRF Local Service.

 

ITLRecovery Certificate

Note:

The ITLRecovery Certificate is used when devices lose their trusted status. The certificate appears in both the ITL and CTL (when CTL provider is active).
If devices lose their trust status, you can use the command utils itl reset localkey for non-secure clusters and the command utils ctl reset localkey for mix-mode clusters. Read the Security guide for your Call Manager version to become familiar with how the ITLRecovery certificate is used and the process required to recover trusted status.
If the cluster has been upgraded to a version that supports a key length of 2048 and the clusters server certificates have been regenerated to 2048 and the ITLRecovery has not been regenerated and is currently 1024 key length, the ITL recovery command will fail and the ITLRecovery method will not be able to be used.

 

  1. Navigate to each server in your cluster (in separate tabs of your web browser) begin with the publisher, then each subscriber.  Navigate to Cisco Unified OS Administration > Security > Certificate Management > Find.
    • Select the ITLRecovery pem Certificate.
    • Once open select Regenerate and wait until you see the Success pop-up then close pop-up or go back and select Find/List.
  2. Continue with subsequent Subscribers; follow the same procedure in step 2 and complete on all subscribers in your cluster.
  3. After all Nodes have regenerated the ITLRecovery certificate, these services will need to be restarted in the order as follows:
    • Navigate to Cisco Unified Serviceability > Tools > Control Center - Network Services.
    • On the publisher select Restart on Cisco Trust Verification Service.
    • If you are in Mixed Mode – Update the CTL before you proceed Token - Tokenless.
    •  Once the service restart completes, continue with the subscribers and restart the Cisco Trust Verification Service.
  4. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  5. Phones will now upload the new ITL while they reboot.

 

CUCM specific certificate

 

CAPF Certificate

Note: Ensure the cluster is not in Secure or Mixed Mode. 

Check if the cluster is in secure cluster mode.

  1. Navigate to the Cisco Unified CM Administration > System > Enterprise Parameters.
    • Section “Security Parameters” Check Cluster Security Mode and ensure is is set to “0” for Cluster Secure Mode. If the value if 0 it is not in a secure cluster mode. If it is anything else it is in a secure mode and you will need to use a CTL client to update the CTL file. The link to the CTL phone security guide can be found at this link. https://supportforums.Cisco.com/docs/DOC-18834
  2. Open a GUI for each server in the cluster starting with the publisher, then each subscriber/TFTP in sequence and navigate to Cisco Unified OS Administration >  Security > Certificate Management
  3. On all of the GUI pages beginning with the publisher Click “Find” showing all the certificates.
    • Click on the “CAPF.pem” Certificate.
    • Once open Click “Regenerate” Wait until you see “Success” then close pop-up or go back to “Find/List”.
  4. Remove the outdated CAPF certificate from the trust store on each node after regeneration of new.
  5. Continue with subsequent Subscribers, follow the same procedure in step 3-4 on all subscribers in the cluster.
  6. After all nodes have regenerated the CAPF certificate, the following services needs to be restarted in the following order:
    1. Log into Publisher’s Cisco Unified Serviceability.
      • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
        • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Certificate Authority Proxy Function Service ONLY where running. (Even if not running on any node, continue with instructions).
      • Cisco Unified Serviceability > Tools > Control Center - Network Services.
        • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Trust Verification Service (TVS).
      • Cisco Unified Serviceability > Tools > Control Center - Feature Services.
        • Beginning with the Publisher then continue with the TFTP, followed by subscribers one by one, restart Cisco Tftp Service only where running.
  7. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  8. Phones will now upload the new ITL while they reboot.

 

TVS Certificate

Warning:

DO NOT regenerate CallManager and TVS certificates at the same time.  This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster. A reboot of ALL phone will be required after the regeneration of TVS certificate.

 

  1. Open a GUI for each node in the cluster starting with the publisher, then each subscriber/TFTP in sequence and navigate to Cisco Unified OS Administration >  Security > Certificate Management.
  2. On all of the GUI pages beginning with the publisher Click “Find” showing all the certificates.
    • Click on the “TVS.pem” Certificate.
    • Once open Click “Regenerate” Wait until you see “Success” then close pop-up or go back to “Find/List”.
  3. Continue with subsequent Subscribers; following the same procedure in step 2 and complete on all subscribers in the cluster.
  4. After all nodes have regenerated the TVS certificate, The following services needs to be restarted in the following order:
    1. Log into Publisher’s Cisco Unified Serviceability.
      • Cisco Unified Serviceability > Tools > Control Center - Network Services.
      • On the Publisher only, restart Cisco Trust Verification Service.
  • Once the service restart completes, continuing with the TFTP followed by subscribers one by one  and restart Cisco Trust Verification Service.
  1. Beginning with the Publisher then continuing with the TFTP followed by subscribers one by one, restart Cisco Tftp Service only where running.
  2. Reboot all Phones.
    • Cisco Unified CM Administration > System > Enterprise Parameters.
    • Select Reset then you will see a pop-up with the statement You are about to reset all devices in the system. This action cannot be undone. Continue?,select OK and then select Reset.
  3. Phones will now upload the new ITL while they reboot.

 

CUPS specific certificate

 

CUP-XMPP Certificate

Note:

Presented to a Cisco Jabber client, Third-Party XMPP client, or a CAXL based application when the XMPP session is being created.

The associated trust-store is used to verify connections made by Cisco XCP Directory service in performing LDAP search operations for third-party XMPP clients.

The associated trust-store is used by the Cisco XCP Router service when establishing secure connections between IM and Presence Service servers if the Routing Communication Type is set to Router-to-Router.

This certificate will only need to be regenerated on the publisher since this certificate is pushed to all the nodes.

 

  1. If you click on Generate and save, this will remove the current certificate if the server is rebooted or services restarted.
  2. Log into Cisco Unified IM and Presence OS Administration Page of Publisher.
    • Security -> Certificate Management
    • Find Certificate List Where the Certificate contains cup-xmpp.
  3. Click on “cup-xmpp” certificate and you can view the old certificate.
  4. Next click “Generate CSR”.

Snag_16f2726.png

Make sure the below is selected.

  • Choose Certificate Purpose “cup-xmpp”.
  • Choose distribution as “Multi-Server SAN”.
  • Applicable for cup-xmpp: In Subject Alternate Names fill in the applicable names for your setup.

For example altNames:

          usdecups02.domain.com (dNSName)

          domain.com (dNSName)

          usdecups01.domain.com (dNSName)

 

  • Remove or click the minus side for the other Auto-populated names to remove them from the SAN.
  • The Key Length will be 2048 and the Hash Algorith will be SHA256.

Snag_17304e5.png

  1. Click on Generate and wait until you get successful generation prompt.
  2. After successful generation close the window.
  3. Download CSR File and send it to the team that handles signing of certificates.
  4. After receiving the “.cer” file from this team, upload that certificate to the OS admin Publisher GUI. Make sure to choose correct certificate purpose as per the CSR file.
    • Go to Security -> Certificate Management.
    • Press Upload Certificate/Certificate chain.
    • Select Certificate Purpose: cup-xmpp
    • Choose the .cer file and press Upload. 
    • After successful upload close the window.

Redo step 3 from first section to view the new certificate and verify the expiration has changed to the new date.

 

After all nodes have received the signed cup-xmpp certificate, the following services needs to be restarted in the following order:

Make sure services are not restarted on all nodes simultaneously as that will disrupt services. Recommended to do the restart of services with a gap of minimum 15 minutes between nodes.

  1. Login into CLI of the Cisco IM and Presenece (CUPS servers) and restart Cisco Intercluster Sync Agent.
    • Run the following command on the Publisher "utils service restart Cisco Intercluster Sync Agent".
    • After the restart finishes on the Publisher, then run the same command on the Subscriber.
  2. Restart Cisco XCP Router services on all nodes. This can be done from either CLI or Serviceability web page.
    • From CLI run the following command on the Publisher "utils service restart Cisco XCP Router".
    • After the restart finishes on the Publisher, then run the same command on the Subscriber.
  3. To verify the new certificate you can check Inter-Clustering on Cisco Unified IM and Presence Administration page from a different cluster to test the connection to the cluster where the certificate has been renewed.
    • Go to Presence -> Inter-cluster.
    • Select the Cluster that you renewed the certificate and verify the connectivity status.

 

Troubleshoot Certificate Errors

If you encounter an error when you attempt to access Unified Communications Manager services from an IM and Presence Service node or IM and Presence Service functionality from a Unified Communications Manager node, the source of the issue is the tomcat-trust certificate. The error message Connection to the Server cannot be established (unable to connect to Remote Node) appears on the following Serviceability interface windows:

  • Service Activation
  • Control Center - Feature Services
  • Control Center - Network Services

Use this procedure to help you resolve the certificate error. Start with the first step and proceed, if necessary. Sometime, you may only have to complete the first step to resolve the error; in other cases, you have to complete all the steps.

 

  1. From Cisco Unified OS Administration, verify that the required tomcat-trust certificates are present: Security > Certificate Management.
    • If the required certificates are not present, wait 30 minutes before checking again.
  2. Choose a certificate to view its information. Verify that the content matches with the corresponding certificate on the remote node.
  3. From the CLI, restart the Cisco Intercluster Sync Agent service with command "utils service restart Cisco Intercluster Sync Agent".
  4. After the Cisco Intercluster Sync Agent service restarts, restart the Cisco Tomcat service with command "utils service restart Cisco Tomcat".
  5. Wait 30 minutes. If the previous steps do not address the certificate error and a tomcat-trust certificate is present, delete the certificate. After you delete the certificate, you must manually exchange it by downloading the Tomcat and Tomcat-ECDSA certificate for each node and uploading it to its peers as a tomcat-trust certificate.
  6. After the certificate exchange is complete, restart Cisco Tomcat on each affected server with command "utils service restart Cisco Tomcat".
Comments
Ayodeji Okanlawon
VIP Alumni
VIP Alumni
‎05-11-2020 12:07 AM

Hi Roger,

Great document. Thank you for putting this together. It's so much easier for us all to have a single go to reference for cert related information like this.

 

Roger Kallberg
VIP
VIP
‎05-11-2020 06:45 AM

Thanks Ayodeji,

Appreciate your comment.

George Sotiropoulos
Cisco Employee
Cisco Employee
‎05-11-2020 11:19 AM

Hello Roger,

 

That’s a great post, thank you for collecting all this info in one piece!

 

George

Roger Kallberg
VIP
VIP
‎05-11-2020 12:04 PM

Your welcome George. Glad you found it useful.

Sunny Khandale
Spotlight
Spotlight
‎07-09-2020 07:00 AM

Dear Roger, Appreciate the details you covered in this post. It will be really helpful for all. Thanks. :)

Roger Kallberg
VIP
VIP
‎07-09-2020 07:29 AM

Your welcome Sunny. Glad it was of use to you.

Ritesh Desai
Spotlight
Spotlight
‎01-02-2021 09:21 PM

@Roger Kallberg 

Very fantastic Post... There are many post covering to certs regen but in this post which I came by was about creating blank ITL files and being pushed to Cisco IP Phones. What's your view on this? Although I haven't tried experimenting on it. I skipped this step and followed the TAC tech notes post while performing the CUCM cluster cert regeneration.

 

https://community.cisco.com/t5/collaboration-voice-and-video/regenerating-call-manager-tvs-ipsec-and-other-certificates/ta-p/3700503

Roger Kallberg
VIP
VIP
‎01-03-2021 12:37 AM

@Ritesh Desai 

That is a precautionary step that you could use to make sure that all phones have a blank ITL before you regenerate TVS and Callmanager certificates. If you follow the correct process for certificate registration this should not be required, but there is always a risk that a few phones would not have updated the ITL and then they would not be able to get configuration changes as they would not trust the TVS. The recommendation is to if possible have a time gap between regeneration of these two types of certificates, the longer the better, to allow time for phones to update this information. Best practices would be to spread them out by a week or so.

derek.andrew
Level 1
Level 1
‎02-03-2021 08:44 AM

This is absolutely fantastic. Please keep writing more.

 

With respect to this warning:

 

DO NOT regenerate CallManager and TVS certificates at the same time. This will cause an unrecoverable mismatch to the installed ITL on the phone to the newly generated ITL in CUCM causing the need to remove the ITL from ALL phones in the cluster. A reboot of ALL phone will be required after the regeneration of TVS certificate.

 

Is there any way to see if the current phones have any certificate problems and need to be reset prior to regenerating these certificates?

 

 

 

Roger Kallberg
VIP
VIP
‎02-06-2021 04:45 AM

Thank you for your kind words @derek.andrew 

There are some 3:rd party tools that can be used to test/check if the phones have an issue with ITL. Please do a search for “itl file cisco phone” to find out more.

You can also test if a device has ITL issues by doing some configuration change that’s visible or easy to verify, like turn on/off the web server on the device and then test if you can reach the web page of the device.

mduling
Level 1
Level 1
‎09-05-2021 06:55 AM

Thank you for your work. I'm sure it should be obvious, but one clarification in the above instructions about meaning of "one by one" if I may. Do you mean one by one according to host or by service?

 

By host like this #1

 

On pub restart:
Cisco CallManager
Cisco CTIManager
Cisco Trust Verification
Cisco Tftp

 

Then repeat process for subs one by one with 15 minutes between nodes? If so, how long should I wait between restarts of each service on the same node?

 

Or according to service this #2

 

On pub restart:
Cisco CallManager wait 15

 

On sub1 restart:
Cisco CallManager wait 15

 

Etc. Repeating for the same service on each host with 15 minutes between service restart before restarting same service on next host until all services on all hosts have been restarted?

 

Many thanks

Roger Kallberg
VIP
VIP
‎09-05-2021 11:02 AM

The note reference restart of services per node, with 15 minutes gap between the nodes.

Jahangir Tahirov
Level 1
Level 1
‎11-04-2021 03:45 AM

@Roger Kallberg , thanks this is a great post. 

 

I have one question. Do we have the option to create CSR using CLI (or GUI) for changing CSR location parameters ( O= , S= , L= , C=) on UCCX?

Roger Kallberg
VIP
VIP
‎11-04-2021 05:22 AM

@Jahangir Tahirov It is possible to change this, however that would not be done during the CSR creation as there is no option AFAIK available for this at that part of the webUI.

image.png

What you can do is change this from the CLI by this command "set web-security orgunit orgname locality state [country] [alternatehostname]"

image.png

You can check what the settings for this is current set to by this command "show web-security"

image.png

Jahangir Tahirov
Level 1
Level 1
‎11-06-2021 06:05 AM

@Roger Kallberg thanks for the detailed information.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents