Replacing certificates signed by an internal CA on CUCM 11.5

[James Hawkins]

Hello,

Last year I deployed a CUCM 11.5 cluster for a customer and used certificates signed by their internal certificate authority (Microsoft) for Tomcat and CallManager. These certificates will expire in approximately one month and so need to be replaced. I need to understand the process to do this safely without causing ITL issues for IP phones.

I have found a Cisco document that covers this process for self-signed certificates but it seems to have some gaps for my requirement.

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200199-CUCM-Certificate-Regeneration-Renewal-Pr.html#anc8

The document has a section which discusses how to avoid these issues 

Regenerate Certificates in Specific Order

This procedure provides a TFTP server with a valid/updated ITL file from a trusted TFTP server that is available.

1. Stop TFTP service on the Primary TFTP server.
2. Make changes on the Primary TFTP server's certificates (as needed).
3. Reset the phones (in order to get a new ITL file from the Secondary TFTP server) - dependent upon which certificates are regenerated, this might happen automatically.
4. Once phones have returned, start the Primary TFTP server's TFTP service.
5. Make certificate changes on the Secondary TFTP server.
6. Reset the phones (in order to get a new ITL file from the Primary TFTP server).

Caution: Do NOT edit certificates on both TFTP servers at the same time. This gives the phones no TFTP server to trust and requires the local administrator to manually remove the ITL from all phones.

 I guess this process should work for certs signed by a CA but for this cluster I have used a multi-SAN certificate rather than one per server. I am therefore not sure that this process would work as the multi-SAN cert would be copied to the secondary TFTP as soon as it is loaded on the primary server.

Has anyone done this successfully for this type of environment? - there are 3000+ phones connected to this custer so avoiding ITL issues is critical

[Ayodeji Okanlawon · Accepted solution]

James,

There is one key piece you are missing and thats the TVS.pem cert. After you generate CUCM cert, a new CUCM+TFTP signature is generated. When the phone resets and attempts to download new file, the signature in its old ITL file will not match with the signature, it will then reach out to TVS store. The key here is not to generate both Calmanager.pem and TVS.pem at same time

++ Details from this +++

If the CallManager.pem file is regenerated, a new CCM+TFTP certificate will be generated
with a new private key. Additionally the ITL file will now be signed by this new CCM+TFTP
key.

1. The phone will attempt to download the new ITL file signed by the new CCM+TFTP from the TFTP server.
2. The phone only has the old ITL file at this point, and the new keys are not in the ITL file present on the phone.Since the phone could not find the new CCM+TFTP signature in the old ITL, it will attempt to contact the TVS service.
3. This part is extremely important. The TVS certificate from the old ITL file must still match. This is because the phone connects securely to the TVS store, hence it will use it's old cert and therefore the TVS cert must match the old cert on the phone. Ie The phone must be able to verify the TVS cert presented by TVS service for it to securely connect to it.

If both the CallManager.pem and TVS.pem are regenerated at the same exact time, then phones will not
be able to download any new files without deleting the ITL from the phone manually.
4. When the phone contacts TVS, the CM server running TVS will have the new CallManager.pem certificate in the OS Certificate Store.
5. The TVS server will return success and the phone will load the new ITL file into memory.
6. The phone will now attempt to download a configuration file, which has been signed by the new
CallManager.pem key.
7. Since the new ITL has been loaded, the newly signed config file will be successfully verified by the ITL in memory.
Key Points:
•
Never regenerate both the CallManager.pem and TVS.pem certificates at the same time.
•
If either TVS.pem or CallManager.pem is regenerated, TVS and TFTP should be restarted and phones
reset to get the new ITL files. Newer versions of CUCM will handle this phone reset automatically and warn the user at certificate regeneration time. ( This should happen with your version)
If more than one TVS server exists (more than one server in the CallManager Group), the additional
servers can authenticate the new CallManager.pem certificate.

[James Hawkins]

Thank you for writing such a comprehensive response. It is most appreciated.

[James Hawkins]

duplicate removed

[hellohova]

Did you get your answer if this process works the same way with Multi-san callmanager certs signed by the CA?

I know the process works with self-signed for sure and probably ca signed by I havent seen much documentation on multi-san process myself.

[Roger Kallberg]

It would work the same in both scenarios. Only difference is that you only upload the CA signed multi SAN certificate on the Pub, it should take care of distribution of it to any other nodes in the cluster.

The key thing is to never renew TVS and CallManager certificates at the same time. Preferably there should be a good time window in-between, rule of thumb would be at least a couple of weeks or more. The longer the better to allow any seldom active devices to pickup the new ILT information.

[hellohova]

---

@Roger Kallberg wrote:

It would work the same in both scenarios. Only difference is that you only upload the CA signed multi SAN certificate on the Pub, it should take care of distribution of it to any other nodes in the cluster.

The key thing is to never renew TVS and CallManager certificates at the same time. Preferably there should be a good time window in-between, rule of thumb would be at least a couple of weeks or more. The longer the better to allow any seldom active devices to pickup the new ILT information.

---

In a cluster to cluster migration, since it is CA signed, would we simply upload the Issuing CA certs as CallManager-trust and phone-Sast trust to allow for a trusted relationship or do we still need to upload the actual multisan CallManager cert itself ?

[Roger Kallberg]

---

@hellohova wrote:

---

In a cluster to cluster migration, since it is CA signed, would we simply upload the Issuing CA certs as CallManager-trust and phone-Sast trust to allow for a trusted relationship or do we still need to upload the actual multisan CallManager cert itself ?

---

That's a completely different question. I would recommend you to have a look at this document that explains the migration process in detail. https://community.cisco.com/t5/collaboration-voice-and-video/migrating-ip-phones-between-clusters-with-cucm-8-and-itl-files/ta-p/3108501

[hellohova]

I'll answer my own question..yes it works only thing I needed to do was upload the ITLRecovery cert on the source as well. 

[Roger Kallberg]

---

@hellohova wrote:

I'll answer my own question..yes it works only thing I needed to do was upload the ITLRecovery cert on the source as well. 

---

Do you mean that you took the ITLRecovery cert from the destination and uploaded it to the source cluster? If so that's not the intended way to do it. Per the document that I refereed to earlier you should use the process of Bulk certificate export and import.

The process as outlined in the document is this:
"Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.

The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate Management page:

1. Export certificates from both destination cluster (TFTP only) and original cluster to a central SFTP server.
2. From original cluster, run Consolidate certificates (TFTP only) on the SFTP server using the Bulk Certificate Management interface.
3. On the original cluster use the Bulk Certificate function to import the TFTP certificates from the central SFTP server.
4. Restart TVS services on original cluster.
5. Use DHCP option 150, or some other method, to point the phones to the new destination cluster.
6. Phones will download the new destination cluster ITL file and attempt to verify it against their existing ITL file.
7. The cert will not be in the existing ITL file so the phone will ask the old TVS server to verify the signature of the new ITL file. The phone sends a TVS query to the old original cluster on TCP port 2445 to make this request.
8. If the certificate export/consolidate/import process worked correctly then TVS returns success, and the phone replaces the in memory ITL file with the newly downloaded ITL file from the destination cluster.
9. The phones can now download and verify the signed configuration files from the new cluster."

Where TFTP cert in the Bulk Certificate Management actually refers to CallManager certificate as this is what is used to sign the configuration files in TFTP.

[Ayodeji Okanlawon]

Spot on Roger. 

Certificate consolidation is what you need to do. ITL recovery is a different thing. Take some time to understand the process. I have done this a few times and not once have I touched ITL recovery file...

[hellohova]

---

@Ayodeji Okanlawon wrote:

Spot on Roger. 

Certificate consolidation is what you need to do. ITL recovery is a different thing. Take some time to understand the process. I have done this a few times and not once have I touched ITL recovery file...

---

I’m well aware this is considered cert consolidation but you can also do this the manual way which is the way I had needed to do it. And yes you do need to upload the ITLRecovery file of the Source cluster and upload it into the Destination cluster as Phone-Sast if you are going from version 10.x to 12.x. Here is the document that calls this out

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/213407-migrate-phones-between-secure-clusters.html

[Ayodeji Okanlawon]

Nope you don't need ITL recovery file when migrating from 10.x to 12.x. Here is what the documentation says..

---

"Steps 3, 5, 6 will apply to scenarios of migrating phone from 8.x to 12.x"

[hellohova]

I don't know what to tell you as it only worked (from 10.5.X to 12.5) when I uploaded the source's ITLRecovery to the destination cluster. Doing the Bulk method takes care of this process.

From the document https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/admin/12_5_1/admin/cucm_b_administration-guide-1251/cucm_b_administration-guide-1251_chapter_010000.html

When the bulk certificate import is performed, the certificates are then uploaded to the remote cluster as follows:

• CAPF certificate gets uploaded as a CallManager-trust

• Tomcat certificate gets uploaded as a Tomcat-trust

• CallManager certificate gets uploaded as a CallManager-trust

• CallManager certificate gets uploaded as a Phone-SAST-trust

• ITLRecovery certificate gets uploaded as a PhoneSast-trust and CallManager-trust

Bear in mind, this is with using signed Callmanager certs using a private CA, and not self signed certs.

I would upload the Private CA certs as the Callmanager-trust and Phone-sast-trust. We run a multi-SME-leaf cluster and add more leafs into the SME ILS network so doing Bulk export with self signed certs can be cumbersome. 

[Ayodeji Okanlawon]

NB: This is a process to add a new node to a leaf cluster not migrate phones...

 I think I understand your point now. Because the certs are CA signed except for the ITL recovery, you manually uploaded the root CA cert as tomcat-trust, callmanager-trust, phone-sass- trust and in addition you uploaded the self signed ITL recovery as a PhoneSast-trust and CallManager-trust?