Regenerating Call Manager, TVS, IPSEC and Other certificates

[voipeee]

 Hi Experts,

I am writing this blog for the work around we need to follow while regenerating the Call Manager certificates especially when you have to regenerate TVS and Call manager certificates at the same time.

This action plan is suggested based on the observation that the Call Manager certificates

are expired on all nodes and cluster is in non-secure node.

Before Performing the regeneration activity do the following pre-requisites.

1. Take a backup of the whole cluster
2. Make sure Phones are having updated ITL.

You can verify the above by running show itl command on TFTP server and then matching the ITL hash with one present in the Phone.

If they are not matching then Phone reboot might be required.

There is no such order to be followed to regenerate the Certificates. You can regenerate any of the certificates first namely Call manager, TVS, CAPF, IPSEC. However, the only thing to make sure during the certificate regeneration is to avoid regenerating TVS and Call Manager Certificates together. Please read the document below to understand the process.

Steps for re-generating certificates.

1. Set pre8.0 rollback parameter in Enterprise Parameters  to True  and restart TFTP service on all TFTP nodes.

This will create an blank ITL file.

1. Reset all the Phones to make sure they get blank ITL . Ideally in step 1 only when you will change the Enterprise parameters , Phones will reset but resetting them once will make sure they get blank itl.

Ensure all phones register back so they have the empty ITL.

1. Regenerate TVS across all nodes

                Restart Trust Verification Service on all nodes(In some cases it restarts automatically)

Here is a catch "Please restart all Devices" either from enterprise parameters or select all devices and then restart. This is very helpful because when you restart Call Manager service phones will use TVS to register.

1. Regenerate Call Manager certificate on all nodes

Restart Call Manager and TFTP service on all nodes

"Please restart all Devices again"

1. Verify updated ITL on all TFTP nodes by using 'show ITL' via CLI ensuring the new certificates are listed (Call Manager & TVS)

1. Remove rollback parameter and ensure all phones register. (set to false)

                  (Phones will obtain a valid ITL)

-----------------------------------------------------------------

Please note if the step 1 and 2 didn't work as mentioned don't worry at all. We have one more option left. 

Please follow the below steps.

1 Regenerate the TVS certificate

2 Restart all the phones (This restart is very crucial, don't think we will regenerate both certificates then restart the devices only one time. It can cause a huge problem and you may have to reset all the devices manually).

3 Regenerate the call manager certificate

4 Restart the phones again

-----------------------------------------------------------------

Remaining certificates won’t have any impact  on ITL and you can regenerate them in any order. Perform the below steps to regenerate them.

1. Regenerate Tomcat across all nodes

Restart Tomcat service on all nodes

(Impacts corporate directory access and extension mobility)

1. Regenerate CAPF on all nodes

(No impact as CAPF service is not running. Recommended so alerts aren't giving via RTMT)

1. Regenerate IPSEC ( Applicable for CUCM/CUC/IMP...)

Note: CUCM/Instant Messaging and Presence (IM&P) before version10.X the DRF Master Agent runs on both CUCM Publisher and IM&P Publisher.  DRF Local service runs on the subscribers respectively.  Versions 10.X and higher, DRF Master Agent runs on the CUCM Publisher only and DRF Local service will be on CUCM Subscribers and IM&P Publisher and Subscribers.

Note: The Disaster Recovery System uses an Secure Socket Layer (SSL) based communication between the Master Agent and the Local Agent for authentication and encryption of data between the CUCM cluster nodes. DRS makes use of the IPSec certificates for its Public/Private Key encryption. Be aware that if you delete the IPSEC truststore (hostname.pem) file from the Certificate Management page, then DRS will not work as expected. If you delete the IPSEC-trust file manually, then you must ensure that you upload the IPSEC certificate to the IPSEC truststore. For more details, refer to the certificate management help page in the Cisco Unified Communications Manager Security Guides.

1. Navigate to each server in your cluster (in separate tabs of your web browser) begin with the publisher, followed by each subscriber.  Navigate to Cisco Unified OS Administration > Security > Certificate Management > Find
   
   • Select the IPSEC pem Certificate.
   • Once open select Regenerate and wait until you see the Success pop-up then close pop-up or go back and select Find/List
2. Continue with subsequent Subscribers; follow the same procedure in step 1 and complete on all subscribers in your cluster
3. After all Nodes have regenerated the IPSEC certificate then restart services.
   
   • Navigate to the Publisher's Cisco Unified Serviceability
     1. Cisco Unified Serviceability > Tools > Control Center - Network Services
     2. Select Restart on Cisco DRF Master service
     3. Once the service restart completes, select Restart on the Cisco DRF Local Service on the publisher then continue with the subscribers and select Restart on the Cisco DRF Local Service

The IPSEC.pem certificate in the publisher must be valid and must be present in all subscribers as IPSEC truststores. The subscribers IPSEC.pem certificate will not be present in the publisher as IPSEC truststore in a standard deployment. In order to verify the validity compare the serial numbers in the IPSEC.pem certificate  from the PUB with the IPSEC -trust in the SUBs. They MUST match.

What to do incase the Serial Number is not matching?

 1. Go to Publisher download ipsec.pem and rename it with ipsec-trust.

 2. Go back to Certificate Management Page on all the nodes(Publisher and all Subcribers) and find the existing ipsec-trust file and Delete it.

 3. Upload the file you downloaded(ipsec.pem renamed with ipsec-trust) in step 1. in Publisher and all subscribers.

 4. Restart the services in the following order:

    a. Restart DRS Master on Publisher

    b. Restart DRS Local on Publisher

    c. Restart DRS Local on all Subscribers.

(Impacts backups and ipsec tunnels)

There will be some certificates ending with –trust  and you won't get any option to regenerate them: you can go ahead and Delete them without any issues. 

If you delete -trust (expired certificates) normally we don’t need to stop any services but with 10.5 CUCM, we need to stop Certificate Change Notification service due to a known bug that deleted certificates keep coming back even after deleting them. Please start Certificate Change Notification service again after you delete the unnecessary certificates. 

Summary of the document:

1. Take backup of whole cluster.

2. Set pre8.0 rollback parameter in Enterprise Parameters  to True

3. Restart TFTP services on all the TFTP nodes.

4. Reset all the Phones.

5. Regenerate the certificates one by one in any order.

6. Don't Regenerate Call Manager and TVS Certificates without restarting all the devices between them. 

    Example: Let's assume you are regenerating Call manager certificate first and then TVS.

    So you will regenerate Call Manager Certificate first.

    restart all the devices.

    Then Regenerate TVS and restart all the devices again.

Please rate "Helpful", if this helps you.

[derek.andrew]

If you run the "show itl" command on TFTP server then how to you get the matching the ITL hash with one present in the Phone to compare it? Does this mean visiting every phone and going into settings... or is there any other way?

[jaheshkhan]

what about the third party CA certificate steps renewal? is it the same?

[Roger Kallberg]

It depends @jaheshkhan what you refer to with "third party CA certificate"? If it's certificates that are signed by a third party CA then the process is the same, for example Tomcat or CallManager certificates that are signed by a CA. However if it's for CA root/intermediate certificates that is in the trust store, to form what is know in PKI as the "circle of trust", the process would be either to upload them manually or that they will get updated by an upgrade of the system.

For additional information have a look at this document that I wrote last year. https://community.cisco.com/t5/collaboration-voice-and-video/cisco-uc-certificates-renewal-guide/ta-p/4077131

[jaheshkhan]

can we use wild card certificate for tomcat ?

im still little confused from your answer. while renewing the tomcat certificate that are signed by public CA there is no option of regenerate. regenerate is for self sign. for renewal, we again need to generate CSR and get the certificate signed by CA. then upload them manually to trust store.

Am I correct?

[Roger Kallberg]

This is correct, apart from that you don’t upload the certificate into the trust store, it is uploaded into the tomcat store and the system will automatically distribute it into the trust store on the local server (Publisher) and also to any subscribers. To renew a CA signed certificate you would need to create a CSR to initiate the signing process. Please note that this does not have any impact on system or service until the signed certificate is uploaded onto the system.

No AFAIK you can not use wild card certificate.

[jaheshkhan]

No CSR generated instead they renewed certificated based on old CSR request from certificate provider. 

so in this case we didnt generate any CSR request. but we have new certificate. its a wild card certificate for tomcat. but when we upload certificate tomcat-trust accepted but while uploading to tomcat store it get failed. may i know what could be the reason?

[Roger Kallberg]

Two possible options for the error as I see it.

1. The CSR user is an old one, meaning that the private key do not match with the one in use on CM.
2. Use of wild card certificates are not supported in CM. You can use multi SAN, but that's not the same as wild card.

[jaheshkhan]

if upload certicate to tomcat store of CUCM publisher will it distribute to IMP publisher and IMP subscriber too. do i need to separately upload IMP Publisher and it will also distribute to IMP subscribter?

[Roger Kallberg]

You only upload the Tomcat certificate on the CM publisher, given that you use a multi SAN certificate that is signed by a CA.

[jwilson]

You mention that the tomcat-trust gets auto updated in the CUCM Cluster once you renew the tomcat with a CA.  If you go and run the Cert Bulk process because you have multiple clusters and you use the features that the Cert Bullk process provides do you know if the tomcat-trust get updated in the other clusters or does one still need to delete the tomcat-trust certs if they are in the trust store because the Cert Bulk process was run in the past?

[Roger Kallberg]

The Bulk Certificate Management process is a standalone process that is manually run and the outcome of this is manually put on each CM AFAIK. So no the tomcat-trust would not get auto updated in either way on the account of this.