UCM 12 certificate expired re-generation

[test60]

hi Experts,

I have few question on re-generate expired certs for UCM 12 ( not in mixed mode )
Following is the expired certs :

- Call Manager
-CAPF
-TVS
- ITL Recovery
-ipsec
-Tomcat

* what is the impact if I let it expired ?
* Which order I need to do it first ? How many hours it take , we have 1 Pub and 5 Subs, 2 TFTP, 10,000 phones
* What tool to check and delete ITL remotely ?
* If I don't restart phone after "Call Manager" cert regenerate, will phone still registered?
* if I don't use Mixed mode, means no need to worry on TVS and CAPF ?

Thanks,
C

[Leonardo Santana]

Hi,

This table at the first link (Table 3. Certificate Names and Descriptions) shows the services related to each certificate of the CUCM:

Check this links i think all your questions will be answered:

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/admin/12_0_1/admin/cucm_b_administration-guide-1201/cucm_b_administration-guide-1201_chapter_01111.html#CUCM_TK_R8B9F2BB_00 

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200199-CUCM-Certificate-Regeneration-Renewal-Pr.html 

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/214231-certificate-regeneration-process-for-cis.html 

Regards

Leonardo Santana

[test60]

thanks, I read these docs already before i posted question.

But what i look for if someone had experienced before and able to answer  from practical perspective.

* what is the impact if I let it expired ?
* Which order I need to do it first ? How many hours it take , we have 1 Pub and 5 Subs, 2 TFTP, 10,000 phones
* What tool to check and delete ITL remotely ?
* If I don't restart phone after "Call Manager" cert regenerate, will phone still registered?
* if I don't use Mixed mode, means no need to worry on TVS and CAPF ?

[Leonardo Santana · Accepted solution]

Hi from the practical side:

* what is the impact if I let it expired ?

Service Impact by the Certificate Store

It is critical for good functionality of the system to have all certificates updated across the CUCM cluster. If your certificates are expired or invalid they might significantly affect normal functionality of the system. A list of potential issues you might have when any of the specific certificates is invalid or expired is shown here. The impact might differ dependent upon your system setup.

CallManager.pem

• Encrypted/authenticated phones do not register.
• TFTP not trusted (phones do not accept signed configuration files and/or ITL files).
• Phone services might be affected.
• Secure Session Initiation Protocol (SIP) trunks or media resources (Conference bridges, Media Termination Point (MTP), Xcoders, and so on) will not register or work.
• The AXL request fails.

Tomcat.pem

• Phones are not able to access HTTPs services hosted on the CUCM node, such as Corporate Directory.
• CUCM's web GUI issues, such as unable to access service pages from other nodes in the cluster.
• Extension Mobility or Extension Mobility Cross Cluster issues.

CAPF.pem

• Phones do not authenticate for Phone VPN, 802.1x, or Phone Proxy. 
• Cannot issue LSC certificates for the phones.
• Encrypted configuration files do not work.

IPSec.pem

• Disaster Recovery System (DRS)/Disaster Recovery Framework (DRF) might not function properly. 
• IPsec tunnels to Gateway (GW) to other CUCM clusters do not work.

TVS (Trust Verification Service)

• The phone cannot authenticate HTTPS service. The phone cannot authenticate configuration files (this can affect nearly everything on CUCM).

phone-vpn-trust

• The phone VPN will not work, because the VPN's HTTPS URL cannot be authenticated.

* Which order I need to do it first ? How many hours it take , we have 1 Pub and 5 Subs, 2 TFTP, 10,000 phones

First blanks out the ITL for the IP Phones:

The restart of the services are very fast, but it depends of the features that you have like EM and other stuff, but i think within 3 or 4 hours you can handle this maintenance window but again you have to know your environment, see all the features that you have ( like IM&P, UCCX, Extension Mobility). But do not wait them to expire to do this.

This feature "blanks" out your ITL on all servers, so the phones will trust any TFTP server. Phone services (for example, extension mobility) will NOT work when this parameter is set to True. However, users will be able to continue to make and receive basic phone calls.

Once this feature is set, all TFTP servers need to be restarted ( in order to supply the new ITL) and all phones need to be reset in order to force them to request the new "blank" ITL. Once the certificate changes are completed and all necessary services have been restarted, this feature can be set back to "False", TFTP service restarted, and the phone reset (so the phone can obtain the valid ITL file). Then all features will continue to work as they did previously.

This procedure provides a TFTP server with a valid/updated ITL file from a trusted TFTP server that is available.

1. Stop TFTP service on the Primary TFTP server.
2. Make changes on the Primary TFTP server's certificates (as needed).
3. Reset the phones (in order to get a new ITL file from the Secondary TFTP server) - dependent upon which certificates are regenerated, this might happen automatically.
4. Once phones have returned, start the Primary TFTP server's TFTP service.
5. Make certificate changes on the Secondary TFTP server.
6. Reset the phones (in order to get a new ITL file from the Primary TFTP server).

* What tool to check and delete ITL remotely ?

There are many applications at the market but you have to pay to use them:

https://www.uplinx.com/phonecontrol/erase-ctl-itl-files-from-cisco-phones/ 

https://www.unifiedfx.com/products/phoneview-itl-delete 

* If I don't restart phone after "Call Manager" cert regenerate, will phone still registered?

When you restart CUCM Service the IP Phone will be restarted automatically

* if I don't use Mixed mode, means no need to worry on TVS and CAPF ?

For CAPF you need to worry if you have encryption (mixed mode), but the TVS you need to worry because is responsible for the IP Phones to authenticate with Extension Mobility if you are using this feature.

My advice map all the variables of you environment if you have a valid cisco support contract open a case with TAC to map all these, but they will tell you basically the same thing that we are discussing here. But if you have any problem during the maintenance window your case will oppened and you can raise the severity of the case depending how is the level of your issue.

Regards

Leonardo Santana

[test60]

Hi Leonardo,

Thanks, much appreciated , this is very clear.

- what you mean by mapping out environment ?   You mean feature we used in UCM ?

- Also I have Jabber MRA (Expressway), IMP, UCCX, Unity, Media Sense, CER.

    * Is there anything break if I update each apps' certs  at different time?

    * Do I need to do all UC apps above at the same window?

-  I have "Tomcat" certs on UCM that is signed by CA Windows Server, any specific process when  i request CSR and apply?

- In case something went sideway , what I need to backup before hand for ITL /TVS , when TAC need to restore?

Thanks,

[Leonardo Santana · Accepted solution]

Hi,

- what you mean by mapping out environment ?   You mean feature we used in UCM ?

I mean the features that you have, like Extension Mobility, or like you mentioned MRA, UCCX, Media Sense and CER. You need to check if the certificates of CUCM will impact these applications.

- Also I have Jabber MRA (Expressway), IMP, UCCX, Unity, Media Sense, CER.

    * Is there anything break if I update each apps' certs  at different time? All the certificates will expire at the same data?

    * Do I need to do all UC apps above at the same window? Your apps will expire the certificates?

-  I have "Tomcat" certs on UCM that is signed by CA Windows Server, any specific process when  i request CSR and apply? This certificate will expire?

- In case something went sideway , what I need to backup before hand for ITL /TVS , when TAC need to restore?

I suggest that you have the backup of your collab enviroment updated

[test60]

hi Leonardo,

I suggest that you have the backup of your collab enviroment updated

----->

if I have UCM backup , do I still need to physically remove ITL when i need to roll back?

Rgds,

T

[Leonardo Santana]

Hi,

if I have UCM backup , do I still need to physically remove ITL when i need to roll back?

Are talking to do a restore on the cluster?

Regards

Leonardo Santana

[test60]

correct, let say i took UCM snapshot  and restore the snapshot.

after that , do i need to remove ITL for each phone to be registered ?

[Leonardo Santana · Accepted solution]

The official answer to this is that the only supported method of backup is to capture regular and complete DRS backup sets from the cluster using the built-in DRS components.

Snapshot is not supported

https://www.cisco.com/c/dam/en/us/td/docs/voice_ip_comm/uc_system/virtualization/virtualization-software-requirements.html#snapshots

For example if you server you need to do a restore, if is the same hostname, domain i think you dot not need to delete the ITL Files. But if you change your domain/hostname or IP  you need to delete the ITL.

Regards