09-04-2018 08:47 AM - edited 01-01-2020 01:23 AM
Hi Experts,
I am writing this blog for the work around we need to follow while regenerating the Call Manager certificates especially when you have to regenerate TVS and Call manager certificates at the same time.
This action plan is suggested based on the observation that the Call Manager certificates
are expired on all nodes and cluster is in non-secure node.
Before Performing the regeneration activity do the following pre-requisites.
You can verify the above by running show itl command on TFTP server and then matching the ITL hash with one present in the Phone.
If they are not matching then Phone reboot might be required.
There is no such order to be followed to regenerate the Certificates. You can regenerate any of the certificates first namely Call manager, TVS, CAPF, IPSEC. However, the only thing to make sure during the certificate regeneration is to avoid regenerating TVS and Call Manager Certificates together. Please read the document below to understand the process.
Steps for re-generating certificates.
This will create an blank ITL file.
Ensure all phones register back so they have the empty ITL.
Restart Trust Verification Service on all nodes(In some cases it restarts automatically)
Here is a catch "Please restart all Devices" either from enterprise parameters or select all devices and then restart. This is very helpful because when you restart Call Manager service phones will use TVS to register.
Restart Call Manager and TFTP service on all nodes
"Please restart all Devices again"
(Phones will obtain a valid ITL)
-----------------------------------------------------------------
Please note if the step 1 and 2 didn't work as mentioned don't worry at all. We have one more option left.
Please follow the below steps.
1 Regenerate the TVS certificate
2 Restart all the phones (This restart is very crucial, don't think we will regenerate both certificates then restart the devices only one time. It can cause a huge problem and you may have to reset all the devices manually).
3 Regenerate the call manager certificate
4 Restart the phones again
-----------------------------------------------------------------
Remaining certificates won’t have any impact on ITL and you can regenerate them in any order. Perform the below steps to regenerate them.
Restart Tomcat service on all nodes
(Impacts corporate directory access and extension mobility)
(No impact as CAPF service is not running. Recommended so alerts aren't giving via RTMT)
Note: CUCM/Instant Messaging and Presence (IM&P) before version10.X the DRF Master Agent runs on both CUCM Publisher and IM&P Publisher. DRF Local service runs on the subscribers respectively. Versions 10.X and higher, DRF Master Agent runs on the CUCM Publisher only and DRF Local service will be on CUCM Subscribers and IM&P Publisher and Subscribers.
Note: The Disaster Recovery System uses an Secure Socket Layer (SSL) based communication between the Master Agent and the Local Agent for authentication and encryption of data between the CUCM cluster nodes. DRS makes use of the IPSec certificates for its Public/Private Key encryption. Be aware that if you delete the IPSEC truststore (hostname.pem) file from the Certificate Management page, then DRS will not work as expected. If you delete the IPSEC-trust file manually, then you must ensure that you upload the IPSEC certificate to the IPSEC truststore. For more details, refer to the certificate management help page in the Cisco Unified Communications Manager Security Guides.
The IPSEC.pem certificate in the publisher must be valid and must be present in all subscribers as IPSEC truststores. The subscribers IPSEC.pem certificate will not be present in the publisher as IPSEC truststore in a standard deployment. In order to verify the validity compare the serial numbers in the IPSEC.pem certificate from the PUB with the IPSEC -trust in the SUBs. They MUST match.
What to do incase the Serial Number is not matching?
1. Go to Publisher download ipsec.pem and rename it with ipsec-trust.
2. Go back to Certificate Management Page on all the nodes(Publisher and all Subcribers) and find the existing ipsec-trust file and Delete it.
3. Upload the file you downloaded(ipsec.pem renamed with ipsec-trust) in step 1. in Publisher and all subscribers.
4. Restart the services in the following order:
a. Restart DRS Master on Publisher
b. Restart DRS Local on Publisher
c. Restart DRS Local on all Subscribers.
(Impacts backups and ipsec tunnels)
There will be some certificates ending with –trust and you won't get any option to regenerate them: you can go ahead and Delete them without any issues.
If you delete -trust (expired certificates) normally we don’t need to stop any services but with 10.5 CUCM, we need to stop Certificate Change Notification service due to a known bug that deleted certificates keep coming back even after deleting them. Please start Certificate Change Notification service again after you delete the unnecessary certificates.
Summary of the document:
1. Take backup of whole cluster.
2. Set pre8.0 rollback parameter in Enterprise Parameters to True
3. Restart TFTP services on all the TFTP nodes.
4. Reset all the Phones.
5. Regenerate the certificates one by one in any order.
6. Don't Regenerate Call Manager and TVS Certificates without restarting all the devices between them.
Example: Let's assume you are regenerating Call manager certificate first and then TVS.
So you will regenerate Call Manager Certificate first.
restart all the devices.
Then Regenerate TVS and restart all the devices again.
Please rate "Helpful", if this helps you.
If you run the "show itl" command on TFTP server then how to you get the matching the ITL hash with one present in the Phone to compare it? Does this mean visiting every phone and going into settings... or is there any other way?
what about the third party CA certificate steps renewal? is it the same?
It depends @jaheshkhan what you refer to with "third party CA certificate"? If it's certificates that are signed by a third party CA then the process is the same, for example Tomcat or CallManager certificates that are signed by a CA. However if it's for CA root/intermediate certificates that is in the trust store, to form what is know in PKI as the "circle of trust", the process would be either to upload them manually or that they will get updated by an upgrade of the system.
For additional information have a look at this document that I wrote last year. https://community.cisco.com/t5/collaboration-voice-and-video/cisco-uc-certificates-renewal-guide/ta-p/4077131
can we use wild card certificate for tomcat ?
im still little confused from your answer. while renewing the tomcat certificate that are signed by public CA there is no option of regenerate. regenerate is for self sign. for renewal, we again need to generate CSR and get the certificate signed by CA. then upload them manually to trust store.
Am I correct?
This is correct, apart from that you don’t upload the certificate into the trust store, it is uploaded into the tomcat store and the system will automatically distribute it into the trust store on the local server (Publisher) and also to any subscribers. To renew a CA signed certificate you would need to create a CSR to initiate the signing process. Please note that this does not have any impact on system or service until the signed certificate is uploaded onto the system.
No AFAIK you can not use wild card certificate.
No CSR generated instead they renewed certificated based on old CSR request from certificate provider.
so in this case we didnt generate any CSR request. but we have new certificate. its a wild card certificate for tomcat. but when we upload certificate tomcat-trust accepted but while uploading to tomcat store it get failed. may i know what could be the reason?
Two possible options for the error as I see it.
if upload certicate to tomcat store of CUCM publisher will it distribute to IMP publisher and IMP subscriber too. do i need to separately upload IMP Publisher and it will also distribute to IMP subscribter?
You only upload the Tomcat certificate on the CM publisher, given that you use a multi SAN certificate that is signed by a CA.
You mention that the tomcat-trust gets auto updated in the CUCM Cluster once you renew the tomcat with a CA. If you go and run the Cert Bulk process because you have multiple clusters and you use the features that the Cert Bullk process provides do you know if the tomcat-trust get updated in the other clusters or does one still need to delete the tomcat-trust certs if they are in the trust store because the Cert Bulk process was run in the past?
The Bulk Certificate Management process is a standalone process that is manually run and the outcome of this is manually put on each CM AFAIK. So no the tomcat-trust would not get auto updated in either way on the account of this.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: