05-23-2018 07:18 AM - edited 03-17-2019 12:51 PM
Hello,
Last year I deployed a CUCM 11.5 cluster for a customer and used certificates signed by their internal certificate authority (Microsoft) for Tomcat and CallManager. These certificates will expire in approximately one month and so need to be replaced. I need to understand the process to do this safely without causing ITL issues for IP phones.
I have found a Cisco document that covers this process for self-signed certificates but it seems to have some gaps for my requirement.
The document has a section which discusses how to avoid these issues
This procedure provides a TFTP server with a valid/updated ITL file from a trusted TFTP server that is available.
Caution: Do NOT edit certificates on both TFTP servers at the same time. This gives the phones no TFTP server to trust and requires the local administrator to manually remove the ITL from all phones.
I guess this process should work for certs signed by a CA but for this cluster I have used a multi-SAN certificate rather than one per server. I am therefore not sure that this process would work as the multi-SAN cert would be copied to the secondary TFTP as soon as it is loaded on the primary server.
Has anyone done this successfully for this type of environment? - there are 3000+ phones connected to this custer so avoiding ITL issues is critical
Solved! Go to Solution.
05-24-2018 08:18 AM - edited 05-25-2018 11:39 PM
James,
There is one key piece you are missing and thats the TVS.pem cert. After you generate CUCM cert, a new CUCM+TFTP signature is generated. When the phone resets and attempts to download new file, the signature in its old ITL file will not match with the signature, it will then reach out to TVS store. The key here is not to generate both Calmanager.pem and TVS.pem at same time
++ Details from this +++
If the CallManager.pem file is regenerated, a new CCM+TFTP certificate will be generated
with a new private key. Additionally the ITL file will now be signed by this new CCM+TFTP
key.
1. The phone will attempt to download the new ITL file signed by the new CCM+TFTP from the TFTP server.
2. The phone only has the old ITL file at this point, and the new keys are not in the ITL file present on the phone.Since the phone could not find the new CCM+TFTP signature in the old ITL, it will attempt to contact the TVS service.
3. This part is extremely important. The TVS certificate from the old ITL file must still match. This is because the phone connects securely to the TVS store, hence it will use it's old cert and therefore the TVS cert must match the old cert on the phone. Ie The phone must be able to verify the TVS cert presented by TVS service for it to securely connect to it.
If both the CallManager.pem and TVS.pem are regenerated at the same exact time, then phones will not
be able to download any new files without deleting the ITL from the phone manually.
4. When the phone contacts TVS, the CM server running TVS will have the new CallManager.pem certificate in the OS Certificate Store.
5. The TVS server will return success and the phone will load the new ITL file into memory.
6. The phone will now attempt to download a configuration file, which has been signed by the new
CallManager.pem key.
7. Since the new ITL has been loaded, the newly signed config file will be successfully verified by the ITL in memory.
Key Points:
•
Never regenerate both the CallManager.pem and TVS.pem certificates at the same time.
•
If either TVS.pem or CallManager.pem is regenerated, TVS and TFTP should be restarted and phones
reset to get the new ITL files. Newer versions of CUCM will handle this phone reset automatically and warn the user at certificate regeneration time. ( This should happen with your version)
If more than one TVS server exists (more than one server in the CallManager Group), the additional
servers can authenticate the new CallManager.pem certificate.
05-24-2018 08:18 AM - edited 05-25-2018 11:39 PM
James,
There is one key piece you are missing and thats the TVS.pem cert. After you generate CUCM cert, a new CUCM+TFTP signature is generated. When the phone resets and attempts to download new file, the signature in its old ITL file will not match with the signature, it will then reach out to TVS store. The key here is not to generate both Calmanager.pem and TVS.pem at same time
++ Details from this +++
If the CallManager.pem file is regenerated, a new CCM+TFTP certificate will be generated
with a new private key. Additionally the ITL file will now be signed by this new CCM+TFTP
key.
1. The phone will attempt to download the new ITL file signed by the new CCM+TFTP from the TFTP server.
2. The phone only has the old ITL file at this point, and the new keys are not in the ITL file present on the phone.Since the phone could not find the new CCM+TFTP signature in the old ITL, it will attempt to contact the TVS service.
3. This part is extremely important. The TVS certificate from the old ITL file must still match. This is because the phone connects securely to the TVS store, hence it will use it's old cert and therefore the TVS cert must match the old cert on the phone. Ie The phone must be able to verify the TVS cert presented by TVS service for it to securely connect to it.
If both the CallManager.pem and TVS.pem are regenerated at the same exact time, then phones will not
be able to download any new files without deleting the ITL from the phone manually.
4. When the phone contacts TVS, the CM server running TVS will have the new CallManager.pem certificate in the OS Certificate Store.
5. The TVS server will return success and the phone will load the new ITL file into memory.
6. The phone will now attempt to download a configuration file, which has been signed by the new
CallManager.pem key.
7. Since the new ITL has been loaded, the newly signed config file will be successfully verified by the ITL in memory.
Key Points:
•
Never regenerate both the CallManager.pem and TVS.pem certificates at the same time.
•
If either TVS.pem or CallManager.pem is regenerated, TVS and TFTP should be restarted and phones
reset to get the new ITL files. Newer versions of CUCM will handle this phone reset automatically and warn the user at certificate regeneration time. ( This should happen with your version)
If more than one TVS server exists (more than one server in the CallManager Group), the additional
servers can authenticate the new CallManager.pem certificate.
05-25-2018 07:52 AM
Thank you for writing such a comprehensive response. It is most appreciated.
05-25-2018 07:53 AM - edited 05-25-2018 07:54 AM
duplicate removed
04-22-2020 06:38 PM
Did you get your answer if this process works the same way with Multi-san callmanager certs signed by the CA?
I know the process works with self-signed for sure and probably ca signed by I havent seen much documentation on multi-san process myself.
04-22-2020 10:18 PM
It would work the same in both scenarios. Only difference is that you only upload the CA signed multi SAN certificate on the Pub, it should take care of distribution of it to any other nodes in the cluster.
The key thing is to never renew TVS and CallManager certificates at the same time. Preferably there should be a good time window in-between, rule of thumb would be at least a couple of weeks or more. The longer the better to allow any seldom active devices to pickup the new ILT information.
04-23-2020 02:55 AM
@Roger Kallberg wrote:It would work the same in both scenarios. Only difference is that you only upload the CA signed multi SAN certificate on the Pub, it should take care of distribution of it to any other nodes in the cluster.
The key thing is to never renew TVS and CallManager certificates at the same time. Preferably there should be a good time window in-between, rule of thumb would be at least a couple of weeks or more. The longer the better to allow any seldom active devices to pickup the new ILT information.
In a cluster to cluster migration, since it is CA signed, would we simply upload the Issuing CA certs as CallManager-trust and phone-Sast trust to allow for a trusted relationship or do we still need to upload the actual multisan CallManager cert itself ?
04-23-2020 05:11 AM
@hellohova wrote:
In a cluster to cluster migration, since it is CA signed, would we simply upload the Issuing CA certs as CallManager-trust and phone-Sast trust to allow for a trusted relationship or do we still need to upload the actual multisan CallManager cert itself ?
That's a completely different question. I would recommend you to have a look at this document that explains the migration process in detail. https://community.cisco.com/t5/collaboration-voice-and-video/migrating-ip-phones-between-clusters-with-cucm-8-and-itl-files/ta-p/3108501
05-11-2020 06:27 PM
I'll answer my own question..yes it works only thing I needed to do was upload the ITLRecovery cert on the source as well.
05-12-2020 12:48 AM
@hellohova wrote:I'll answer my own question..yes it works only thing I needed to do was upload the ITLRecovery cert on the source as well.
Do you mean that you took the ITLRecovery cert from the destination and uploaded it to the source cluster? If so that's not the intended way to do it. Per the document that I refereed to earlier you should use the process of Bulk certificate export and import.
The process as outlined in the document is this:
"Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.
The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate Management page:
Where TFTP cert in the Bulk Certificate Management actually refers to CallManager certificate as this is what is used to sign the configuration files in TFTP.
05-12-2020 02:25 AM
Spot on Roger.
Certificate consolidation is what you need to do. ITL recovery is a different thing. Take some time to understand the process. I have done this a few times and not once have I touched ITL recovery file...
05-12-2020 05:27 AM - edited 05-12-2020 05:33 AM
@Ayodeji Okanlawon wrote:Spot on Roger.
Certificate consolidation is what you need to do. ITL recovery is a different thing. Take some time to understand the process. I have done this a few times and not once have I touched ITL recovery file...
I’m well aware this is considered cert consolidation but you can also do this the manual way which is the way I had needed to do it. And yes you do need to upload the ITLRecovery file of the Source cluster and upload it into the Destination cluster as Phone-Sast if you are going from version 10.x to 12.x. Here is the document that calls this out
05-12-2020 06:17 AM - edited 05-12-2020 06:18 AM
Nope you don't need ITL recovery file when migrating from 10.x to 12.x. Here is what the documentation says..
"Steps 3, 5, 6 will apply to scenarios of migrating phone from 8.x to 12.x"
05-13-2020 05:10 PM - edited 05-13-2020 05:12 PM
I don't know what to tell you as it only worked (from 10.5.X to 12.5) when I uploaded the source's ITLRecovery to the destination cluster. Doing the Bulk method takes care of this process.
From the document https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/admin/12_5_1/admin/cucm_b_administration-guide-1251/cucm_b_administration-guide-1251_chapter_010000.html
When the bulk certificate import is performed, the certificates are then uploaded to the remote cluster as follows:
CAPF certificate gets uploaded as a CallManager-trust
Tomcat certificate gets uploaded as a Tomcat-trust
CallManager certificate gets uploaded as a CallManager-trust
CallManager certificate gets uploaded as a Phone-SAST-trust
ITLRecovery certificate gets uploaded as a PhoneSast-trust and CallManager-trust
Bear in mind, this is with using signed Callmanager certs using a private CA, and not self signed certs.
I would upload the Private CA certs as the Callmanager-trust and Phone-sast-trust. We run a multi-SME-leaf cluster and add more leafs into the SME ILS network so doing Bulk export with self signed certs can be cumbersome.
05-13-2020 09:29 PM - edited 05-13-2020 11:03 PM
NB: This is a process to add a new node to a leaf cluster not migrate phones...
I think I understand your point now. Because the certs are CA signed except for the ITL recovery, you manually uploaded the root CA cert as tomcat-trust, callmanager-trust, phone-sass- trust and in addition you uploaded the self signed ITL recovery as a PhoneSast-trust and CallManager-trust?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide