Cluster wide TVS regeneration

[Gordon Ross]

All the TVS certificates in my cluster are due to expire at the same time in the next few days. I've been reading about the CUCM "Security By Design" and its certificates. The general idea appears to be: Either play with the CUCM certificate OR the TVS certificate, but not both. I'm OK with that.

As I have to regenerate every servers' TVS certificate, and a TVS certificate regeneration forces a cluster wide phone reboot, is there any trick to regenerating all TVS certificates and only rebooting the phones once, or do I have to suffer every phone rebooting multiple times?

I'm on CUCM 10.5(2)

Thanks,

GTG

[Jaime Valencia]

If you try to change all the TVS at once, and then reboot all phones, you'd lock yourself out.

If you have 3 servers in your CUCM group, they'll be used for TVS, if you re-generate the 1st node, it will fail to recognize it as valid, BUT, it will recognize as valid the 2nd node (which you have not re-generated), which in turn, will contain the new TVS from the 1st node, next time, it will also recognize the 1st node as valid.

If you do all 3 servers at once, and phones do not have a TVS server on which they trust, you will not be able to connect or authenticate to anything requiring TVS.

That's also the reason why you can only play with CUCM, OR TVS at any point in time, but not both, it would be the same, you'd lose the authentication to them, and phone would not connect.

[Gordon Ross]

But from my reading of the docs, one of the reasons for TVS is for when the CallManager cert changes. (The TVS validates the new CallManager cert).

As the CallManager & TFTP certs are used for signing, etc., then changing the TVS shouldn't affect that.

GTG

[Ayodeji Okanlawon]

TVS is a trust store. Since IP phones contain a limited amount of memory, we need a place to store certs. This is the TVS.
Any time the phone cannot verify a signature or certificate via the CTL or ITL files, it will ask
the TVS server for verification
TVS runs on all servers that have the cucm service activated. The CUCM group the phone belongs to determines its TVS store..

To understand why you must not regenerate TVS certs at once just as Jamie mentioned, you need to understand the phone boot up sequence

1.phone boots up,
2.receives vlan from cdp
3.receives dhcp (tftp server present)
4.Requests CTL and ITL file (ITLSEPxxxx.tlv)
5.Once ITL file is succesfully received, phone then request a signed config file.
6.Next phone must verify ITL..

7.If the phone already has ITL it must verify the downloaded ITL files match the signature in the ITL /CTL or TVS server it already has..

In step 7 above, if you regenerate all TVS certs at once and reboot the phones. Here is what happens..

Since your phones already have ITL file downloaded. These ITL files were signed by the old TVS certs before you regenerated them. Now the phone tries to verify the signature of the ITL files, but it cant because none of your TVS servers have the old signature. Hence you are locked out and your only solution will be to delete the ITL files.

NB: your phones may still register at this point but certain things will be broken. One of which is your directory services. This requires https connection and hence the phone needs to validate the cert been exchanged and since there is no TVS trust store available to do this, this connection fails. There are other features that could also be affected

[Gordon Ross]

OK. So as I have to regenerate all the TVS certs, how long should I wait between each regeneration? Some phones (e.g. 8851) don't do a reboot when reloading the ITL file and just do it in the background, so watching registrations won't work.

GTG

[Ayodeji Okanlawon]

No regenerate cucm certs, TVS certs one by one. So just do one server first in the cucm group of the phones and then let the phones reboot. Ensure they all reboot.. Then once they are back. Do the rest 

[Ali Amir]

Do communicate phones with all CUCM servers in the CUCM group for TVS cert or only with the TFTP servers?
I mean, how is the update TVS cert when there are a publisher (TFTP) and one subscriber?
 
- Is the TVS cert required however the cluster is unsecured?
- What could be happened when the TVS cert is expired in unsecured cluster?
- Could be any problem happened during TVS cert update?

[Manish Gogna]

TVS is a enabled by default in versions 8.x and higher. It is an important component of the ITL file and hence this certificate should not be in expired state whether the cluster is secure or non-secure. You can check the following link which provides all the details about TVS

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/security/8_0_1/secugd/sec-801-cm/secusbd.html#wp1091977

Manish

[pkinane]

+5

[pkinane · Accepted solution]

Do the phones communicate with all the CUCM servers in the CUCM group for TVS cert or only with the TFTP servers?

The phones will communicate with the TFTP server for files (CTL, ITL, Config, Ring list, etc.)

The TVS runs on all servers where the CallManager service is activated and the ITLFile should have the TVS certificate from all servers running the TVS service.

A phone knows of it's TVS servers by looking at it's call manager group. The phone will communicate with the TVS servers when it cannot validate the signer of a file (or the certificate of a server like when doing EM login) If none of the servers in the callmanager group can validate what the phones needs validated, the phone will try the TFTP server for validation (as a backup TVS server. i.e. If the TFTP server is running the CCM service and isn't in the phone's Callmanager group, the phone can still reach out on port 2445 for validation. This can allow phones to have more than 3 TVS servers).

How do we update the TVS cert when there are a publisher (TFTP) and one subscriber?

I believe a comment I made in a different section of the conversation here should help answer this question.

"You should be able to regenerate all the TVS certs at once. The reason is the config file, ITLFile, and CTLFile are not signed by the TVS cert. They are signed by the CCM cert (or SAST tokens for the CTL on older versions of CUCM) which is why the callmanager.pem certificate is also known as the TFTP certificate. The TVS only comes into play when the phone can't validate the signer of a file (or the certificate of a service) using the CTL or ITL (Hence the full name of TVS... Trust Verification Service).

When you regenerate all the TVS certs (or even just one) the ITL should be updated and pushed out to the phones, that ITL should be signed by the callmanager.pem certifcate, the callmanager.pem cert didn't change and it is known by the old ITL that will be replaced on the phone; therefore, the phone should be able to verify the signer of the new ITL just by looking in the old once currently on the phone, now the phone accepts the new ITL and has all of the new TVS entries. The cardinal rule is to not replace all the TVS and callmanager.pem certs at the same time. Newer versions of CUCM actually prevent you from being able to replace them at the same time on a single node.

This document should help with understanding it all:

https://supportforums.cisco.com/document/68701/communications-manager-security-default-and-itl-operation-and-troubleshooting"

 
- Is the TVS cert required however the cluster is unsecured?

Yes

- What might happen when the TVS cert is expired in unsecured cluster?

I've not tested this scenario; however, I would imagine it would cause issues when the phones need to use the TVS service to validate a certificate. Your best option is to not let it expire by replacing the certificate beforehand.

- Could problems occur during the TVS cert update?

If you renew only the TVS cert, there should be no problem. This is assuming you don't have an ITL issue already. You can use this tool to check:

http://www.unifiedfx.com/itl-scanner/

Please rate helpful content (i.e. videos, documents, comments). Also, please select the correct answer(s) if any comment(s) answer your question otherwise the questions remains on the support forums as unanswered.

[Ali Amir]

I couldn't understand why the TVS certs must be updated one by one in the cucm group. Because:

1- If we have only one CUCM in our instllation how could I regenerate the TVS certs?

2- If I regenerate the TVS certs for first and second server, the phone could verify the signature of the ITL files with the third server but if I regenerate the TVS certs for third server, the phone could not verify the signature of the ITL files because none of your TVS servers have the old signature.

[pkinane]

You should be able to regenerate all the TVS certs at once. The reason is the config file, ITLFile, and CTLFile are not signed by the TVS cert. They are signed by the CCM cert (or SAST tokens for the CTL on older versions of CUCM) which is why the callmanager.pem certificate is also known as the TFTP certificate. The TVS only comes into play when the phone can't validate the signer of a file (or the certificate of a service) using the CTL or ITL (Hence the full name of TVS... Trust Verification Service).

In short: If you renew only the TVS cert, there should be no problem. This is assuming you don't have an ITL issue already. You can use this tool to check:

http://www.unifiedfx.com/itl-scanner/

When you regenerate all the TVS certs (or even just one) the ITL should be updated and pushed out to the phones, that ITL should be signed by the callmanager.pem certifcate, the callmanager.pem cert didn't change and it is known by the old ITL that will be replaced on the phone; therefore, the phone should be able to verify the signer of the new ITL just by looking in the old once currently on the phone, now the phone accepts the new ITL and has all of the new TVS entries. The cardinal rule is to not replace all the TVS and callmanager.pem certs at the same time. Newer versions of CUCM actually prevent you from being able to replace them at the same time on a single node.

This document should help with understanding it all:

https://supportforums.cisco.com/document/68701/communications-manager-security-default-and-itl-operation-and-troubleshooting

Please rate helpful content (i.e. videos, documents, comments). Also, please select the correct answer(s) if any comment(s) answer your question otherwise the questions remains on the support forums as unanswered.