cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Migrating IP Phones Between Clusters with CUCM 8 and ITL Files

126767
Views
65
Helpful
57
Comments

 

 

Purpose

CUCM 8 introduced the new Security By Default feature and the use of ITL (Initial Trust List) files. (More documentation here). With this new feature, care must be taken when moving phones between different CUCM clusters. Without following the proper steps it is possible to encounter a situation where thousands of phones must manually have their ITL files deleted.

 

Background

Phones supporting the new ITL file download this special file from their CUCM TFTP server. Once an ITL file is installed on a phone, all future configuration files and ITL file updates must be either:

 

  1. Signed by the CCM+TFTP Server certificate currently installed in the phone's CTL file (If cluster security with CTLs is enabled).
  2. Signed by the CCM+TFTP Server certificate installed in the phone's ITL file.
  3. Signed by a certificate that exists in one of the CUCM Server TVS certificate stores that are listed in the ITL file.

 

With this new security functionality in mind here are the three problems that we can encounter when moving a phone from one cluster to another cluster.

 

  1. The ITL file of the new cluster is not signed by the phone's existing ITL CCM+TFTP cert, so the phone will not accept the new ITL file or config files.
  2. The TVS servers listed in the phone's existing ITL may not be reachable when the phones are moved to the new cluster.
  3. Even if the TVS servers are reachable for certificate verification, the old cluster TVS servers may not have the new servers' certificates.

 

If these three problems are encountered one possible option is to delete the ITL file manually from all phones being moved between clusters. This is not a desirable solution sine it requires massive effort as the number of phones increases.

 

Symptoms

Any changes that phones receive through TFTP or HTTP of configuration files will not be honored.  Configuration options passed by configuration file partially include:

  • URLs including Authentication URL, Directories URL, Services URL; this includes internal/external directories configuration
  • some locale features
  • callmanager groups for primary and secondary registration

The phone very likely will register, it registers to configured TFTP server by default. The phone likely will not register if the new TFTP server is not running the CallManager service.

 

When a phone has an incorrect ITL file for the current TFTP server the phone console logs show a message similar to:

 

 

1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed </usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not in CTL)

1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED, </usr/ram/SEP00260BD749E9.cnf.xml>

 

Migration Options

With the three previous problems taken into account we can come up with a plan that allows the migration of phones seamlessly from one cluster to the next without requiring manual phone intervention.

 

Bulk Certificate Export

Note

The Bulk Certificate Export method will only work if both clusters are online with network connectivity while the phones are being migrated.

 

Another possible option if both the old and new clusters will be online at the same time is to use the Bulk Certificate migration method.

 

Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.

 

The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate page:

  1. Export certificates from new destination cluster (TFTP only) and original cluster to a central SFTP server.
  2. From original cluster, run Consolidate certificates (TFTP only) on the SFTP server using the Bulk Certificate interface.
  3. On the old origination cluster use the Bulk Certificate function to import the TFTP certificates from the central SFTP server.
  4. Restart TVS services on old origination cluster.
  5. Use DHCP option 150, or some other method, to point the phones to the new destination cluster.
  6. Phones will download the new destination cluster ITL file and attempt to verify it against their existing ITL file.
  7. The cert will not be in the existing ITL file so the phone will ask the old TVS server to verify the signature of the new ITL file. The phone sends a TVS query to the old origination cluster on TCP port 2445 to make this request.
  8. If the certificate export/consolidate/import process worked correctly then TVS returns success, and the phone replaces the in memory ITL file with the newly downloaded ITL file.
  9. The phones can now download and verify the signed configuration files from the new cluster.

 

 

Rollback Enterprise Parameter

Note

This method is only valid if completed before the phone migration is attempted. This cannot be used once phones are already in the "verify file failed" state.

Phones supporting TVS will potentially lose access to secure URL services such as the Corporate Directory before they are migrated to the new cluster after the "Prepare Cluster for Rollback to pre-8.0" parameter is set to True on the original cluster.  Once migrated to the new cluster, the phones will download their new ITLs and Secure URL operation should go back to normal.

 

Another option is to make use of the CUCM Enterprise Parameter "Prepare Cluster for Rollback to pre-8.0." Full documentation of the Rollback parameter can be found here. Once this parameter is set to True, the phones will download a special ITL file that contains empty TVS and TFTP certificate sections.

 

When a phone has an empty ITL file it will accept any unsigned configuration file (for migrations to pre CUCM 8.X clusters), and will also accept any new ITL file (for migrations to different CUCM 8.X clusters).

 

The empty ITL file can be verified by checking "Settings > Security > Trust List > ITL". Empty entries will appear where the old TVS and TFTP servers used to be.

 

The phones must have access to the old CUCM servers only as long as it takes them to download the new empty ITL files. Once the phone has an empty ITL file, the old servers can be decommissioned, powered down, thrown into a river, or rebuilt (depending on business requirements).

 

Using Hardware Security tokens (KEY-CCM-ADMIN-K9=)

If hardware security tokens (product number KEY-CCM-ADMIN-K9=) have been used to generate a CTL (Certificate Trust List) on both the old and new clusters, the phones will be able to freely migrate between clusters as long as at least one of the same hardware tokens was used on both the old and new clusters.

 

When a phone that has a CTL from the old cluster is moved to the new cluster, it will accept the new cluster's CTL since the new CTL contains a security token certificate in common with it's current CTL. Because the CTL also contains the certificate for the CCM+TFTP server, the new cluster's ITL will also be accepted by the phone so there will be no issues in moving the phone between clusters.

 

For phones that don't utilize security by default (ITLs) such as the 7960s and 7940s, you will need to re-run the CTL client on the original cluster first to add new TFTP entries for the TFTP servers of the new cluster prior to moving the phones to the new cluster.  This is due to those phone models not even reaching out for TFTP files for a server not in the CTL.

 

This security token method requires this additional hardware and must be configured on the old cluster. Normally security tokens are used to allow for encrypted signaling and media (SRTP) in a cluster and encrypted / authenticated config files. Also, once a cluster has had security enabled with security tokens, disabling security on that cluster requires manually removing the CTL from each phone on that cluster from the phone itself.

 

Manual ITL Delete

If some catastrophic failure happens and the TFTP key/certificate is no longer available from the old cluster (this is maintained in a DRF backup... TAKE A BACKUP), then the only available option to migrate a phone to a new cluster is to manually delete the ITL file from the phones.

 

This differs for each phone model.  The steps needed for the most common phone models are listed here, but the steps for other models can be found in the Phone Administration Guides.

 

79XX - Settings > Security > Trust List > ITL File > **# (to unlock the settings) > Erase

 

89XX/99XX - Settings > Administrator Settings > Reset Settings > Security Settings

Comments
Beginner

if i migrate the servers from physical to UCS same IPs same version 8.5.1 am i going to face this issue !!

Beginner

You wont face the issue  .I tested the same in a Lab environment and during this time I  registered an  IP Communicator and that was working fine without any issues .My 7900 Series Phones gave me the problem .

 If you are facing the problem u can delete bulk ITL files for Phones by using 3rd Party Application bulk ctl eraser.

http://www.voipintegration.com/software/bulk-ctl-eraser/

 

Thanks,

Harish

 

Beginner

Hi, I am building a new be6k single CUCM (pub, tftp) using backups from the original server to move from v9.1 to v10.5 onto new hardware. I have the new server built, database restored and now upgraded to 10.5 and I have changed the IP Address.

I will install this with network connectivity to the old 9.1 server ready to migrate devices. I plan to use the Bulk Certificate Export method once both servers are on line. While I am happy the local phones will migrate ok, can anyone confirm if the VPN phones will be able to migrate to the new server this way?

We have a lot of VPN phones out on various locations which we don't want to have to bring back into register locally. As the new server was restored from the old server database all the VPN certificates look to have been carried across. 

Have anyone done this before with VPN phones?

Beginner

Hi Steven,

how did you fix your issue ? i am in same situation. I m migrating from ccm ver 8.5 to ccm ver 10.5.

I exported TFTP certificates from new & old. After successful export from both cluster I can find only one file name " CCMPUB_tftp.pkcs12 " because both clusters TFTP certificate has same name hence one overwrite to the other. Do i need to change the file names ? I can't find consolidate option under Bulk Certificate Export option ?

Would appreciate if you can help

Cisco Employee

If both server has same hostname pkcs12 will be overwritten.

Consolidate option will come when platform detects multiple cluster certificates at the SFTP cert store.

HTH,

Vasanth

Beginner

We had upgrade project from ver 8.5 to ver 10.5 and also migrate from MCS to UCS.

We built UCS with 8.5 than migrate database from MCS ver 8.5 to UCS ver 8.5 and than upgrade UCS 8.5 to 10.5. Once we successfully upgraded to 10.5 than we change call manager ip address. Hostnames are same.

So we have currently two clusters with different IP & version in production. We disconnect one phone from ver 8.5 and connected to new CCM ver 10.5 it didn't registered.

After manually delete trust certificate from phone it registered immediately.

To overcome this manual delete trust certificate we are trying Bulk Certificate Export procedure.

Is there any other recommendation for my scenario ?

Cisco Employee

This looks like a limitation with Bulk certificate tool in CUCM. One should have the ability to do, consolidation of certificate in such scenario. This looks like a feature request.

In your case the Rollback parameter can come handy.

 

Thanks,

Vasanth

Beginner

You are right Vasank this should be consider.

I tried with Rollback and it worked with no issues.

Regards,

 

Will the bulk certificate export method work when migrating to a new cluster with same ip addressing?

The ITL check will fail, not sure what will happen when querying the old TVS server, which in turn will be the new one but with the TFTP certificates of the old cluster consolidated. Will that return success?

Thanks for the good post!

Isidro

Enthusiast

I'm just going though this process to move from 8.5 to 11 and have hit an issues with the bulk certificate method. When I try to import the consolidated certificate into my original 8.5 cluster I get a very generic 'null' message back as the below screen shot.

I currently have a case open with TAC, who said they replicated the issue in the lab and suggested that installing ciscocm.version3-keys.cop into 8.5 would fix the issue as my 8.5 version does not understand RSAv3 which v11 would do. However this has not fixed the issues for me.

Has anyone come across this before?

Thanks
Matty 

Cisco Employee

I tried this in the lab and experienced the same error as you (even after installing the cop file and doing a cluster reboot). I made sure the traces were set to debug and looked for a good reason. I didn't see much in the traces though. It seems the 8.5 doesn't like something about the consolidated certs because I was able to upload the callmanager certs to the CUCM Version 8.5 individually as callmanager-trust after downloading them from the 11 cluster. The goal of exporting, consolidating, and importing the certs is to get the calmanager cert of the new cluster on to the old cluster. If the callmanager cert (also known as TFTP cert) is downloaded from version 11 and is uploaded as callmanager-trust and phone-sassy-trust to version 8.5, the phone should be able to authenticate the ITL when migrating to version 11 from version 8.

I think you should give this workaround a try if you've not resolved the error you are seeing.

Enthusiast

Thanks for the reply. What's odd is when I build a vanilla 8.5 in my lab I can import the consolidated certificate sucessfully!

In the above work around, can you confirm it's the "Callmanager" self-signed certificate where the description and common name is the hostname that I need to export from my v11?

Many thanks

Matty

Cisco Employee

CallManger.pem will be the cert of interest as it is the cert used to sign the ITL and the configuration file.

Enthusiast

Sorry for the delay in my reply. I have been on other project since the above date.

I have tried this export import method and found that I still face the same error.

What is interesting now is it appears to trust the TFTP server of the new cluster but does not actually register.

E.g. phone on v8 TFTP pointing to v8. Change DHCP option 150 to v11 server.

The phone will actually upgrade it's phone load based on the device defaults in v11 but then registers with v8 again!

Some very odd behaviour!

Matty

Cisco Employee

What do you see in the status messages of the phone? Did you look in the console logs for the phone as well?