cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
148937
Views
70
Helpful
57
Comments
Jason Burns
Level 1
Level 1

 

 

Purpose

CUCM 8 introduced the new Security By Default feature and the use of ITL (Initial Trust List) files. (More documentation here). With this new feature, care must be taken when moving phones between different CUCM clusters. Without following the proper steps it is possible to encounter a situation where thousands of phones must manually have their ITL files deleted.

 

Background

Phones supporting the new ITL file download this special file from their CUCM TFTP server. Once an ITL file is installed on a phone, all future configuration files and ITL file updates must be either:

 

  1. Signed by the CCM+TFTP Server certificate currently installed in the phone's CTL file (If cluster security with CTLs is enabled).
  2. Signed by the CCM+TFTP Server certificate installed in the phone's ITL file.
  3. Signed by a certificate that exists in one of the CUCM Server TVS certificate stores that are listed in the ITL file.

 

With this new security functionality in mind here are the three problems that we can encounter when moving a phone from one cluster to another cluster.

 

  1. The ITL file of the new cluster is not signed by the phone's existing ITL CCM+TFTP cert, so the phone will not accept the new ITL file or config files.
  2. The TVS servers listed in the phone's existing ITL may not be reachable when the phones are moved to the new cluster.
  3. Even if the TVS servers are reachable for certificate verification, the old cluster TVS servers may not have the new servers' certificates.

 

If these three problems are encountered one possible option is to delete the ITL file manually from all phones being moved between clusters. This is not a desirable solution sine it requires massive effort as the number of phones increases.

 

Symptoms

Any changes that phones receive through TFTP or HTTP of configuration files will not be honored.  Configuration options passed by configuration file partially include:

  • URLs including Authentication URL, Directories URL, Services URL; this includes internal/external directories configuration
  • some locale features
  • callmanager groups for primary and secondary registration

The phone very likely will register, it registers to configured TFTP server by default. The phone likely will not register if the new TFTP server is not running the CallManager service.

 

When a phone has an incorrect ITL file for the current TFTP server the phone console logs show a message similar to:

 

 

1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed </usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not in CTL)

1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED, </usr/ram/SEP00260BD749E9.cnf.xml>

 

Migration Options

With the three previous problems taken into account we can come up with a plan that allows the migration of phones seamlessly from one cluster to the next without requiring manual phone intervention.

 

Bulk Certificate Export

Note

The Bulk Certificate Export method will only work if both clusters are online with network connectivity while the phones are being migrated.

 

Another possible option if both the old and new clusters will be online at the same time is to use the Bulk Certificate migration method.

 

Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.

 

The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate page:

  1. Export certificates from new destination cluster (TFTP only) and original cluster to a central SFTP server.
  2. From original cluster, run Consolidate certificates (TFTP only) on the SFTP server using the Bulk Certificate interface.
  3. On the old origination cluster use the Bulk Certificate function to import the TFTP certificates from the central SFTP server.
  4. Restart TVS services on old origination cluster.
  5. Use DHCP option 150, or some other method, to point the phones to the new destination cluster.
  6. Phones will download the new destination cluster ITL file and attempt to verify it against their existing ITL file.
  7. The cert will not be in the existing ITL file so the phone will ask the old TVS server to verify the signature of the new ITL file. The phone sends a TVS query to the old origination cluster on TCP port 2445 to make this request.
  8. If the certificate export/consolidate/import process worked correctly then TVS returns success, and the phone replaces the in memory ITL file with the newly downloaded ITL file.
  9. The phones can now download and verify the signed configuration files from the new cluster.

 

 

Rollback Enterprise Parameter

Note

This method is only valid if completed before the phone migration is attempted. This cannot be used once phones are already in the "verify file failed" state.

Phones supporting TVS will potentially lose access to secure URL services such as the Corporate Directory before they are migrated to the new cluster after the "Prepare Cluster for Rollback to pre-8.0" parameter is set to True on the original cluster.  Once migrated to the new cluster, the phones will download their new ITLs and Secure URL operation should go back to normal.

 

Another option is to make use of the CUCM Enterprise Parameter "Prepare Cluster for Rollback to pre-8.0." Full documentation of the Rollback parameter can be found here. Once this parameter is set to True, the phones will download a special ITL file that contains empty TVS and TFTP certificate sections.

 

When a phone has an empty ITL file it will accept any unsigned configuration file (for migrations to pre CUCM 8.X clusters), and will also accept any new ITL file (for migrations to different CUCM 8.X clusters).

 

The empty ITL file can be verified by checking "Settings > Security > Trust List > ITL". Empty entries will appear where the old TVS and TFTP servers used to be.

 

The phones must have access to the old CUCM servers only as long as it takes them to download the new empty ITL files. Once the phone has an empty ITL file, the old servers can be decommissioned, powered down, thrown into a river, or rebuilt (depending on business requirements).

 

Using Hardware Security tokens (KEY-CCM-ADMIN-K9=)

If hardware security tokens (product number KEY-CCM-ADMIN-K9=) have been used to generate a CTL (Certificate Trust List) on both the old and new clusters, the phones will be able to freely migrate between clusters as long as at least one of the same hardware tokens was used on both the old and new clusters.

 

When a phone that has a CTL from the old cluster is moved to the new cluster, it will accept the new cluster's CTL since the new CTL contains a security token certificate in common with it's current CTL. Because the CTL also contains the certificate for the CCM+TFTP server, the new cluster's ITL will also be accepted by the phone so there will be no issues in moving the phone between clusters.

 

For phones that don't utilize security by default (ITLs) such as the 7960s and 7940s, you will need to re-run the CTL client on the original cluster first to add new TFTP entries for the TFTP servers of the new cluster prior to moving the phones to the new cluster.  This is due to those phone models not even reaching out for TFTP files for a server not in the CTL.

 

This security token method requires this additional hardware and must be configured on the old cluster. Normally security tokens are used to allow for encrypted signaling and media (SRTP) in a cluster and encrypted / authenticated config files. Also, once a cluster has had security enabled with security tokens, disabling security on that cluster requires manually removing the CTL from each phone on that cluster from the phone itself.

 

Manual ITL Delete

If some catastrophic failure happens and the TFTP key/certificate is no longer available from the old cluster (this is maintained in a DRF backup... TAKE A BACKUP), then the only available option to migrate a phone to a new cluster is to manually delete the ITL file from the phones.

 

This differs for each phone model.  The steps needed for the most common phone models are listed here, but the steps for other models can be found in the Phone Administration Guides.

 

79XX - Settings > Security > Trust List > ITL File > **# (to unlock the settings) > Erase

 

89XX/99XX - Settings > Administrator Settings > Reset Settings > Security Settings

Comments
pkinane
Cisco Employee
Cisco Employee

Craig,

Thank you for the clarification. Blocking ITL and CTL updates would not mitigate man in the middle attacks and they are not blocked for VPN phones. Having an ITL or CTL actually helps prevent man in the middle attacks.

When you have an ITL or CTL on the phone these will be checked before the phone accepts a signed config file. If the ITL or CTL isn't there, the phone won't have the ability to confirm the signer of the config file; therefore, the config file can come from anyone.

R/s,

Patrick.

pkinane
Cisco Employee
Cisco Employee

Hello Isidro,

I know this post is old, but I want to reply for others that might have thesaurus question.

This should not work because the phone will query the TVS server, the server will be new and when it presents the phone with a TVS certificate the phone won't be able to verify the TVS server's identity because it will only know about the old ones.

R/s,

Patrick

srinivascisco
Level 1
Level 1

Can you tell me if this is still unchanged to move between CUCM 6 migration to CUCM 11.5.

Have phones in 6 and will be moved to anther cluster running on 11.5, appreciate the procedure

Mohamed Gaffoor
Level 1
Level 1

if you are running version 6.1(5) then you can use PCD to upgrade directly to 11.5

This will be a Migration from the MCS7800 server to a Virtual Server

Note : BE3000 and BE5000 deployments are not supported via the PCD upgrade method

srinivascisco
Level 1
Level 1

Mine is PCD Eligible but my question is, we are choosing new IP Addressing schema along with new hosts. how do we safely migrate the phones when both the clusters are up and running. I want to avoid CTL/ITL file deletion

Adnan Kolakovic
Level 1
Level 1

You don't have to worry about the certificates as CUCM 6 doesn't use security by default. 

Phones will register automatically to the new cluster when you change your DHCP Option 150. 

Look at the table under "Run a Migration Task" heading in the following document:

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/pcdadmin/10_5_1/CUCM_BK_U35347D2_00_pcd-administration-guide-1051/CUCM_BK_U35347D2_00_ucmap-administration-guide-1051_chapter_010.html

If you do need to roll back, you can use the enterprise parameter Prepare for rollback to pre 8.0.

Adnan

srinivascisco
Level 1
Level 1

I have BE6K destination, i see a note on PCD which says "Cisco Business Edition 6000 and Cisco Business Edition 7000 servers are preinstalled with Cisco UC Virtualization Hypervisor. If you plan to use Cisco Prime Collaboration Deployment with application VMs on these servers, you must substitute a higher virtualization software feature level." What does this mean?

Adnan Kolakovic
Level 1
Level 1

PCD is supported only when using certain types of VMware ESXi licenses. The ESXi license that BE6K comes with (Cisco UC Virtualization Hypervisor) is not supported.

You need something like Foundation, Standard, Enterprise or Enterprise Plus ESXi license. Otherwise the APIs used by PCD are not available.

Adnan

srinivascisco
Level 1
Level 1

My BE6K came with PCD templates pre-deployed along with other Prime collaborations tools, I have installed it already, at what stage it might fail?

Adnan Kolakovic
Level 1
Level 1

You need to check what VMware ESXi license is currently installed on your UCS host (you need to log in using vSphere client - Google for specific instructions).

If it's not one of the supported license levels, it will fail when trying to connect to the host from PCD (if i recall correctly).

Adnan

silversnc
Level 1
Level 1
The cert consolidation process seems easy enough but I had a question on fall back if needed. Does the cert consolidation allow for fall back to old cluster? or would I need to consolidate old cluster certs onto new cluster? I am working on consolidating 2 - 8.6 clusters with 3+ subs each to a single 10.5 cluster with 5+ subs and trying to figure out a valid fall back plan for cutovers, if the customer has any issues with the migration.
rhallinan7387
Level 1
Level 1

I would suggest adding to the article that when migrating to a cluster in non-secure mode which is CUCM 12.0 or more then the ITL Recovery certificate also needs to be imported to the source cluster as a CallManager-trust and Phone-SAST-trust in order for the phones to migrate successfully.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: