Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Create a new article
11065
Views
36
Helpful
7
Comments

Renew self-signed ipsec.pem capf.pem callmanager.pem tvs.pem and tomcat.pem one after the other

pkinane
Cisco Employee
Cisco Employee

‎10-06-2017 09:33 AM - edited ‎06-19-2019 06:05 AM

Introduction

 

Sometimes administrators want to regenerate all of their certificates. The documentation that currently exists only tells you how to do this one certificate at a time. The purpose of this document is to explain how all of the certificates can be regenerated one after the other.

 

Preflight Checklist

 

Confirm the rollback parameter is set to false

 

The command below can assist with identifying the current status of the parameter.

 

run sql select paramname,paramvalue from processconfig where paramname='RollBackToPreGrayback'

 

The command output should look like this before you move forward:

 

admin:run sql select paramname,paramvalue from processconfig where paramname='RollBackToPreGrayback'
paramname             paramvalue
===================== ==========
RollBackToPreGrayback F

 

Confirm you have more than one server in the CallManager Group

 

If you have a single CUCM server, then this document is not for you. You can read more here:

        Communications Manager Security By Default and ITL Operation and Troubleshooting

 

If you have multiple nodes in the cluster, make sure you have more than one CallManager server in each of your CallManager Groups used by the phones.

 

Determine if you are using LSCs on the phones

 

If you are using 802.1x, Phone VPN using certificate authentication, or secure phone profiles then you will need to upload the new CAPF to the RADIUS/ASA appliance after you update the CAPF certificate and resign the LSCs on the phones.

 

NOTE: If you have LSCs on your phones, do not delete the old capf-trust certificates until after you've updated the LSCs on the phones. To update the LSCs on the phones you can use the Install/Upgrade option mentioned in the Configure LSC on Cisco IP Phone with CUCM document.

 

Confirm whether or not you are in mixed mode

 

admin:run sql select paramname,paramvalue from processconfig where paramname='ClusterSecurityMode'
paramname           paramvalue
=================== ==========
ClusterSecurityMode 0
 
If you see a 0 for the paramvalue column, like what is shown above, then you are not in mixed mode; however, if you see a 1 then you are in mixed mode and you must be mindful of the CTLFile. For those not in mixed mode, you can ignore the notes about updating the CTLFile throughout this document.

 

Confirm there are no CTL or ITL issues

 

If you have CTL or ITL issues before modifying your certificates you will want to know. Administrators who don't perform this check will find the CTL/ITL issues after the cert change and come to the conclusion that the CTL/ITL issues are a result of the certificate change; however, they can't be sure as they didn't perform the check prior to modifying the certificates.

 

When I check for ITL issues I use this tool:

        UnifiedFX ITL Scanner

 

        This video explains how to install/use the tool.

                https://www.youtube.com/watch?v=E_jBp3O7VzA

 

NOTE: There are other tools out there, such as the Bulk CTL / ITL Eraser so do some research to see what is available and best suits your environment.

 

If I am not able to use the ITL Scanner I simply pick phones at random and check their status messages for CTL/ITL/Trust List failures.

 

Verify the health of database replication

 

This document explains the output of utils dbreplication runtimestate.

        Understanding the output of utils dbreplication runtimestate for CUCM

 

Confirm backups are working

 

This document explains some things to do if your backups are failing.

        Before opening a TAC case: Info to gather/think about when troubleshooting DRS in CUCM/CER

 

Review the output of show ctl

 

Enter the command show ctl on the CLI of the publisher and TFTP servers. This should be done on the publisher and all servers in the cluster that are running the TFTP service. Confirm there aren't any failures or errors. If there are any errors or failures, please resolve them before moving forward.

 

It may be good to document the output of this command from each TFTP server in case someone wants to review it later.

 

The output is expected to look like the example below if your cluster is not in mixed mode and never has been:

 

 

admin:show ctl
Length of CTL file: 0
CTL File not found. Please run CTLClient plugin or run the CLI - utils ctl.. to generate the CTL file.
Error parsing the CTL File.

 

Review the output of show itl

 

Enter the command show itl on the CLI of the TFTP servers. This should be done on all servers in the cluster that are running the TFTP service. Confirm there aren't any failures or errors. If there are any errors or failures, please resolve them before moving forward.

 

It may be good to document the output of this command from each TFTP server in case someone wants to review it later.

 

Good to know information

 

What if I don't read this whole document or I go out of order

 

You run the risk of causing a CTL/ITL issue on the phones. People that followed this process step by step never reported a CTL/ITL issue at the time of me writing this statement.

 

What impact comes with renewing the different certificates?

 

NOTE: The phones will need to be reset at different times in this process; however, they will reset a lot while removing old certificates from the system due to https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut58407

 

Once you upload a new cert you need to reset the associated services. Below is a list of information about the impact of restarting the services:


1. For the Tomcat certificate you need to restart the Tomcat service. Restarting Tomcat is very quick; however, it will impact anything that uses HTTPS (mostly login attempts):


            1. Extension Mobility
            2. Secure AD integration
            3. The Admin and end user GUI
            4. Etc…

 

NOTE: You will also need to perform the exchange of CUCM metadata with the IdP if you are using SSO.


2. The IPsec certificate is primarily used for DRS, Secure SIP profiles, CTL Client, and secure connections to gateways. If the IPsec certificate is renewed, reset DRF Master and DRF Local on the publisher AND DRF local on the subscribers. Restarting these will restart the DRS feature (DRS is used for backups and restoring for backup).
    
3. The CallManager certificate deals primarily with ITL which ITL files are introduced in CUCM Version 8. A new CallManager cert requires the restart of the CallManager, CTIManager, TFTP, and Trust Verification Service (TVS) services. Restarting CallManager may drop calls (however, I have restarted CCM while on a call with a customer using a phone on the cluster where I restarted the service and our call stayed connected while features were not available due to call preservation) and may cause phones to failover to their secondary node. Restarting TVS and TFTP may cause devices to reset; however, you need to reboot the phones after restarting the service so they can get a new ITL file anyway.
    
4. The CAPF certificate is associated with the LSC certificates on phones. Once the CAPF Certificate is renewed you will need to restart the CAPF, TFTP, CallManager, and TVS services followed by a reboot of the phones. NOTE: Restarting the CAPF, TFTP or TVS services may cause devices to reset. Also after restarting the CAPF service if you are using 802.1x, Phone VPN, or Phone Proxy then you will need to upload the new CAPF to the RADIUS/ASA appliance.
    
5. The TVS certificate deals primarily with ITL files for the phones. If you renew the TVS file you need to restart the TVS service, restart the TFTP service, and reset the phones so they have download the new ITL file.

 

How do I restart the different services?

 

You can restart many from the CLI; however, others you need to reset from the GUI.


1. Tomcat: go to the whichever node as the new certificate and execute utils service restart Cisco Tomcat
    
2. DRF Master: go to the publisher and execute utils service restart Cisco DRF Master or Cisco Unified serviceability GUI -> Tools -> Control Center - Network Services -> Select Cisco DRF Master -> click Restart
    
3. DRF Local: go to the whichever node as the new certificate and execute utils service restart Cisco DRF Local or Cisco Unified serviceability GUI -> Tools -> Control Center - Network Services -> Select Cisco DRF Local -> click Restart
    
4. CallManager: go to the whichever node as the new certificate and execute Cisco Unified serviceability GUI -> Tools -> Control Center - Feature Services -> Select Cisco CallManager -> Click Restart
    
5. TVS service: go to the whichever node as the new certificate and execute Cisco Unified serviceability GUI -> Tools -> Control Center - Network Services -> Select Cisco Trust Verification Service -> click Restart
    
6. TFTP service: go to the whichever node as the new certificate and execute Cisco Unified serviceability GUI -> Tools -> Control Center - Feature Services -> Select Cisco TFTP -> Click Restart
    
7. CAPF service: go to the whichever node as the new certificate and Cisco Unified serviceability GUI -> Tools -> Control Center - Feature Services -> Select Cisco Certificate Authority Proxy Function -> Click Restart
    
8. CTIManager service: go to the whichever node as the new certificate and execute Cisco Unified serviceability GUI -> Tools -> Control Center - Feature Services -> Select Cisco CTIManager -> Click Restart

 

The process to regenerate the certificates

 

It is recommended these steps are performed during a maintenance window.

 

1) Regenerate the IPSEC.PEM on ccm1, ccm2, ccm3, etc…

 

NOTE: You can regenerate the cert from the cli of each node with the command below:

set cert regen ipsec


               a. Restart DRF Master and DRF local at the Publisher
               b. Restart DRF local on all Subscribers

 

2) Regenerate CAPF.PEM on ccm1, ccm2, ccm3, etc…

 

NOTE: You can regenerate the cert from the cli of each node with the command below:

set cert regen CAPF

 

3) Regenerate CallManager.PEM on ccm1, ccm2, ccm3, etc…

 

NOTE: You can regenerate the cert from the cli of each node with the command below:

set cert regen CallManager


a. If you are in mixed mode, update the CTLFile then confirm the update using show ctl on the publisher and all the TFTP servers.

 

NOTE: You can use the USB hardware tokens or the tokenless method to update the CTL file as described in this document:

 

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/118893-technote-cucm-00.html

 

b. Restart CAPF service on Publisher


c. Restart TFTP service on all nodes running the service


d. Restart the CallManager service on all nodes running the service


e. Restart the CTIManager service on all nodes running the service


f. Restart TVS service on all nodes


g. Confirm the ITL update with the command show itl (run this on all the TFTP servers)


h. Reboot the phones cluster-wide. You can Reboot the phones by going to System > Enterprise Parameters and click Reset, or in a more controlled manor by resetting device pools.


4) Wait for all phones to register again, then check for CTL/ITL issues on the phones

 

     Spot check the status messages on the phone using web access

     or use the ITL Scanner software

 

5) Regenerate TVS.PEM on ccm1, ccm2, ccm3, etc…

 

NOTE: You can regenerate the cert from the cli of each node with the command below:

set cert regen TVS


a. Restart TVS service on all CUCM nodes

 

b. Restart TFTP service on all nodes running the service

 

c. Reboot the phones cluster-wide. You can Reboot the phones by going to System > Enterprise Parameters and click Reset, or in a more controlled manor by resetting device pools.

 

 

6) Wait for all phones to register again, then check for ITL issues on the phones

 

     Spot check the status messages on the phone using web access

     or use the ITL Scanner software

 

7) Regenerate tomcat.pem on ccm1, ccm2, ccm3, etc…

 

NOTE: It typically takes around 10 to 15 minutes for the web page to work after restarting tomcat.

 

NOTE: If you are using SSO, you will also need to perform the exchange of CUCM metadata with the IdP.

 

NOTE: You can regenerate the cert from the cli of each node with the command below:

set cert regen tomcat

 

a. Restart Tomcat service on all CUCM nodes where you regenerated the tomcat certificate

 

8) Stop the Cisco Certificate Change Notification service on all nodes

 

9) Remove old trust certificates from all nodes that have them

 

10) Wait 30 minutes then start the Cisco Certificate Change Notification service on all nodes

Comments
Hard_Boiled_Greg
Level 1
Level 1
‎10-27-2017 01:51 PM

So the main question I have and nobody within Cisco has been able to give me a good answer is if I really need to roll Security By Default back by setting the "Prepare Cluster for Rollback to pre 8.0" parameter to True before doing any regeneration of CallManager, TVS and CAPF certificates on a cluster that not in Mixed Mode. This forum post, which is great, doesn't address use of the parameter. I did an engagement with Cisco Advanced Services last fall where we regenerated certs on a number of non-secure clusters, and they recommended disabling the Security by Default feature. I have a CUCM 10.5 cluster with seven nodes and Mixed Mode is set to 0.

pkinane
Cisco Employee
Cisco Employee
‎10-30-2017 01:59 PM

@Hard_Boiled_Greg

 

I am just going to copy the majority of my reply in our messages to each other and post it here for other folks to see as well.

 

"If I am running without Mixed Mode enabled and need to regenerate my CallManager.pem, TVS.pem and CAPF. pem certificates, do I really need to disable Security by Default?"

The short answer is no.

 

Disabling SBD by changing the rollback parameter simply modifies the ITL (you can do a show itl on the TFTP server before and after modifying the parameter if you want to see what I mean -- focus mainly on the TVS certs in the ITL). The reason people disable SBD is so they don't cause ITL issues with the phones; however, you won't face ITL issues if you regenerate the certificates in the proper order, and if you do the proper checks/service restarts along the way.

 

Even if you are facing ITL issues with the phones, you aren't in that bad of shape these days. Here is why:

1: https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/117598-technote-itl-00.html

2: TAC can go into root, access what is called the tftp_backup directory, find old ITL files, and restore the old ITLs.

Hard_Boiled_Greg
Level 1
Level 1
‎11-02-2017 12:28 PM

Thanks pkinane. I wish I would have had your reply before last weekend. :) It probably would have saved me a lot of trouble. I did an engagement with Cisco Advanced Services last fall. They recommended disabling SBD before doing any of the certificate work we were doing. It was the same scenario as I had this past weekend. We had just replaced the Tomcat certs with new externally signed certs, we had to use SET WEB-SECURITY to update the web security parameters to match our new CA, and then we had to regenerate all self-signed certs (because Cisco has told me that if modifying the web security all self-signed certs need to be regenerated). In the case of this past weekend, I did disabled SBD because of the recommendations I had from Advanced Services. It turned out to be a mistake. I had 400 Cisco 8800 series phones that would not register after turning off Security By Default. It turns out there is a bug in the 7800 and 8800 series 10.x firmware that causes this issue. I couldn't even get to the web pages for the phones once they became unregistered, so my UnifiedFX PhoneView couldn't help me. So, of the 9,000 phones on my cluster I had 400 of the 8800 series phones that were locked out. We had to find and manually clear the security settings on them. I would have been better off just following your process (which, BTW, is similar to the process another TAC engineer provided to me when I opened a case after locking my phones out) and not messing with SBD.

Lesson learned. I appreciate the reply.

pkinane
Cisco Employee
Cisco Employee
‎11-02-2017 01:58 PM

@Hard_Boiled_Greg

 

Thank you for the update. Did you scan the cluster for ITL issues before starting the change? This is critical because many people have a few phones with ITL issues and they don't know it. So they move forward with the change (without checking for ITL issues) then when the change is done they see the ITL issues surface and attribute the issue to the change.

 

I see the bug mentioned in your case is this one: CSCuz38655

 

I don't believe you hit this bug as the bug requires an old CTL file to be on the system. The problem with this bug is the CTL doesn't update automatically (you need to run a command to update the CTLFile). Unlike the ITL that updated when it needs to or whenever you restart TFTP. So when the phone downloads the CTL it will accept the CTL, then the phone requests the ITL, but the ITL is signed by updated certs that are not in the CTL (so the phone will reject the ITL file). The phone will then show no ITL installed. I've helped a few customers with this type of issue; however, I didn't know there is a bug out there for it (which I don't think there should be as this is working as designed). I filed an enhancement a while back to have the CTL file update like the ITL file does so that people wouldn't face this issue. This is the enhancement I filed: CSCux52914

The big requirement to hit CSCuz38655, that you seem to not meet, is that you be in mixed mode (if not in mixed mode, then no CTL).

I will likely update CSCuz38655 because this happens on more than just 78XX and 88XX.

 

For more detail that is not available on the external page of the bug is the description of the bug. This is only internally visible; however, I don't see any sensitive data in the description so here is a snippet:

If a customer is in mixed-mode and updates all of their CallManager certificates at the same (either by regenerating the cert on
each node simultaneously or uploading a CA-signed multi-server cert), the phone will go from having the old ITL to updating to the new ITL,
and then seconds later having no ITL (listed as "Not Installed" on the device).



Rafael Ochoa
Level 1
Level 1
‎05-22-2018 01:06 PM

I followed Patrick's document over the weekend and had no issues reported. Thanks again Patrick

Ryan Kotulak
Cisco Employee
Cisco Employee
‎06-24-2018 07:52 AM

All,

 

Wanted to share an updated process for those of you who are using Mixed-Mode and CTL. I've worked with other CUCM TAC Engineers as well as Technical Leaders to work the kinks out in this process. I've used this process with customers and known and confirmed working, below are the instructions with some other externalized links that are helpful for future reading:

 

Update CTL first "“utils ctl update CTLFile” as seen here:  https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/118893-technote-cucm-00.html#anc6

Next, you’ll want to Regenerating the certificates as shown here:

 

  1. Regenerate the IPSEC.PEM on ccm1, ccm2, ccm3, etc…
    a. Restart DRF Master and DRF local at the Publisher
    b. Restart DRF local on all Subscribers
  2. Regenerate CAPF.PEM on ccm1, ccm2, ccm3, etc…
  3. Regenerate CallManager.PEM on ccm1, ccm2, ccm3, etc…
    a. Restart CAPF service on Publisher
    b. Restart TFTP service on all nodes running the service
    c. Restart the CallManager service on all nodes running the service
    d. Restart the CTIManager service on all nodes running the service
    e. Restart TVS service on all nodes
    f. Reboot the phones cluster-wide - You can Reboot the phones by going to System > Enterprise Parameters and click Reset. Wait for all phones to register again.
  4. Regenerate TVS.PEM on ccm1, ccm2, ccm3, etc…
    a. Restart TVS service on all CUCM nodes
    b. Restart TFTP service on all nodes running the service
    c. Reboot the phones cluster-wide - You can Reboot the phones by going to System > Enterprise Parameters and click Reset. Wait for all phones to register again.
  5. Then use CTL Recovery by issuing "utils ctl reset localkey" then update CTLFile using “utils ctl update CTLFile”.
  6. Stop the Cisco Certificate Change Notification service on all nodes
  7. Remove old trust certificates from all nodes that have them. Wait 30 minutes then start the Cisco Certificate Change Notification service on all nodes.
Derek Liu #65178
Level 1
Level 1
‎05-31-2020 11:39 PM

Nice Doc.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents