CUCM Certificates Renewal procedures for mixed mode cluster with USB Token

edchandr · ‎11-07-2019

1. Reference URLs:

Certificate type and function:

http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200199-CUCM-Certificate-Regeneration-Renewal-Pr.html

How to renew a certificate:

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/117299-problemsolution-product-00-html.html

2. Type of Certificates:

	CallManager	Unity Connection	CTL/ITL Impact
Tomcat Certificates: Cisco Tomcat Certificate is leveraged by https web services to access callmanager/CUC webpages (Administration, OS Admin and user GUI). It does not have any functional impact if the certificate expires, only web access to the above services will be affected that users will be prompted about expired SSL certificates.	X	X	-
IPSec Certificates: IP Security certificates are vital for DRF services DRF and also used to provide secure services between nodes in the cluster. DRS backup and restore can fail if IPSec certificates are not synchronized (or when expired) between nodes. It has no impact in a single node CUC cluster or when DRF is only set to back up the primary Publisher node. It is also used for SRTP (Secure RTP) encryption towards voice gateways.	X	X	-
Call Manager Certificate: CallManager certificate is a self-signed root certificate which is generated by the system. It provides server identification including server name and GUID and is used for integration with multiple applications and devices such as secure DSP farm, CUPS, Unity Connection etc. Expiration of this certificate will result in loss of trust and secure communication between Call Manager, applications and devices.	X	-	X
CAPF Certificate: CUCM CAPF certificate is leveraged for secure endpoint communication and phone authentication (802.1x). It enables creation of certificate trusted list (CTL) and LSC certificates, which is used by the endpoints for authentication and encryption. Expiry of this certificate may lead to encryption not working and LSC not issued to the phones.	X	-	X
TVS Certificate: The Trusted Verification Service (TVS) is the main component of Security by Default and is activated by default. It authenticates certificates on behalf of the phone. TVS certificate and a few key certificates are bundled in a new ITL file and gets downloaded by the phone. If the TVS certificate expires, the IP phones will not be able to trust any other certificates other than the ones they already have.	X	-	X

Certificates need to be renewed when they are: expiring/expired, hostname/ip address change, node add/deletion, and certain DRF restore scenarios.
Phone uses either CTL or TVS Trust List to validate phone config file hash (based of CallManager certificate)

3. Regenerate certificates that have no CTL/ITL impact [no phone impact] – on all the nodes:

Certificate Type

Certificate Regeneration Steps

Remark

Tomcat

1. SSH to each node with expired certificate

set cert regen tomcat

utils service restart Cisco Tomcat

2. Renewing 3^rd party CA signed Tomcat certificate [separate steps not covered in this doc]

Webpage won’t be available for up to 30 minutes

IPSec

1. SSH to each node with expired certificate

set cert regen ipsec

2. Restart Cisco DRF Master (on Publisher)

utils service restart Cisco DRF Master

3. Restart Cisco DRF Local (on all nodes)

utils service restart Cisco DRF Local

Only Pub’s IPSec is used by DRF.

Other nodes’ IPSec regeneration is to suppress expiration alert in RTMT.

4. Regenerate certificates that have CTL/ITL impact [medium phone impact] – for “usb token”-based mixed-mode cluster:

<Do not use for Soft-Token [non-usb] mixed-mode cluster. Step adjustments will be needed for non-usb and non mixed mode clusters>

Prerequisite:

Verify number of registered devices via RTMT
Verify a valid recent DRF backup is available
Make note of current CTL/ITL content (on the phone > Settings > Security > Trust List) – see sample below:

Phone Type	CTL Signature	ITL Signature
	CTL is signed by both USB e-tokens and contains CallManager and TFTP certificates – shown below by red arrow. CAPF Server id – signed by usb eToken is shown in the red box below.	ITL is signed TVS certificate that is derived from CallManager certificate on the same node with TVS service – shown below by red arrow.
78xx/88xx
79xx with ITL Support
79xx Legacy [exclude 79x1/79x5/797x]		*No ITL support*

CTL signatures are of different values when comparing between 78xx/88xx vs 79xx legacy phones.

WARNING:

recommended not to carry out Week #0 [CTL update] and Week #1 [TVS update] back to back on same day. This to ensure phones can still trust either CTL or ITL at any given time.
if CallManager and/or CAPF certificates are changed, CTL client update is required to restore config update and/or encryption capability respectively.

Week #0 CallManager+CAPF certificates regeneration and CTL Client update – follow top down:

Certificate Type	Certificate Regeneration Steps	Remark
CallManager	SSH to each node with expired CallManager’s certificate set cert regen CallManager	a. Phones *will soft reset* and “Trust List Updated” in phone status message b. Phones will register back to its original ccm node c. Phones that support ITL will have an *update to its ITL signature. CTL signature stay the same. d. Phones that do not support ITL will reject* any config changes (File Auth Fail in phone status message)
CAPF	SSH to the Publisher with expired CAPF’s certificate set cert regen CAPF	a. Phones *will soft reset* and “Trust List Updated” in phone status message b. Phones will register back to its original ccm node c. Phones that support ITL will have an *update to its ITL signature*. CTL signature stay the same.
Update CTL Client	1. Run CTL client application on a Windows PC 2. Select “CTL Update” 3. Click Next and input the proper usb token password [after 15 incorrect pw entry, usb token will be self delete and unusable] 4. Click “Finish” at the screen where it shows all the CCM and TFTP nodes: 5. Restart TFTP services (on all nodes) 6. Restart CallManager Services (on all nodes) 7. Restart CTI Manager Services if secure CTI is used (on all nodes) 8. *Reset the phones* in groups (either via device pool or bulk tool) – to ensure that all phones reset and download new updated CTL Trust List. Especially needed for legacy 7940 and 7960 phones that do not support ITL. 9. Verify new CTL is updated the phone – *”CAPF server id” will be changed* from prior value.	At step “6. Restart CallManager Services (on all nodes)” – the following will happen: a. some phones might switch to SRST mode and might take up to 5 minutes before they switch back to CallManager b. Newer phones 78xx and 88xx will auto soft reset and download *update to their CTL Trust List* c. Older phones 79XX will require a phone reset (step 8) to initiate download of updated CTL Trust List As CTL is updated with new CallManager and CAPF certificates, phones will start accepting configuration changes delivered via its xml config file.

Week #1 TVS certificate regeneration – follow top down:

Certificate Type

Certificate Regeneration Steps

Remark

TVS

1. Verify that CTL Trust List has been updated properly and phones configuration can be changed with no issue

2. SSH to each node with expired TVS’s certificate

set cert regen TVS

3. Verify new ITL updated on phones

a. Phones that register to this node will soft reset and “Trust List Updated” in phone status message

b. Phones will register back to its original ccm node

c. Phones that support ITL will have an update to its ITL signature. CTL signature stay the same.

5. Step to delete expired certificates to silent “expired” certs alert in RTMT

Stop “Cisco Certificate Change Notification” service on all nodes
Remove old trust certificates from all nodes that have them
Wait 30 minutes then start “Cisco Certificate Change Notification” service on all nodes

6. Setup RTMT to send alert 90 days before next certificate expiration

/cmplatform > Security > Certificate Monitor
Change the “Notification Start Time” to 90 days [max is 90]
Click “Save”

7. Other Observations

Scenarios	No CTL/ITL	CTL phones	CTL + ITL phones
Phone Models	793X, IP Communicator	7940, 7960, Jabber	7941, 7945, 7961, 7965, 797X, 78XX, 88XX
CallManager, CAPF, TVS has impact on phones?	No	Yes	Yes
Impact of callmanager certificate regeneration (same node with TFTP server) without CTL update? TVS and CAPF stay the same	No	Yes – subsequent config update is rejected	No – TVS will be use to validate config update
Impact of callmanager certificate regeneration (and TFTP server on different node) without CTL update? TVS and CAPF stay the same	No	No – provided TFTP node did not have callmanager certificate changed	No
Impact of only CAPF certificate regeneration without CTL client update. CallManager and TVS are unchanged	No	No – phone can register and accept config update	No – phone can register and accept config update
Impact of callmanager + CAPF certificate regeneration with CTL update?	No	Phone reset and restored config update capability	No
Phone reset required after CTL Client update?	No	Yes	No – auto update/soft reset
Impact of TVS certificate regeneration and cluster is mixed mode with USB Token	No	No	No – provided CTL Trust List is valid.

Hope the above helps other in their journey to update the certificates.

JFerello · ‎11-08-2019

I would love to see this document for non-usb/mixed mode and non-usb/non mixed mode. But thanks for this document!