Purpose
CUCM 8 introduced the new Security By Default feature and the use of ITL (Initial Trust List) files. (More documentation here). With this new feature, care must be taken when moving phones between different CUCM clusters. Without following the proper steps it is possible to encounter a situation where thousands of phones must manually have their ITL files deleted.
Background
Phones supporting the new ITL file download this special file from their CUCM TFTP server. Once an ITL file is installed on a phone, all future configuration files and ITL file updates must be either:
- Signed by the CCM+TFTP Server certificate currently installed in the phone's CTL file (If cluster security with CTLs is enabled).
- Signed by the CCM+TFTP Server certificate installed in the phone's ITL file.
- Signed by a certificate that exists in one of the CUCM Server TVS certificate stores that are listed in the ITL file.
With this new security functionality in mind here are the three problems that we can encounter when moving a phone from one cluster to another cluster.
- The ITL file of the new cluster is not signed by the phone's existing ITL CCM+TFTP cert, so the phone will not accept the new ITL file or config files.
- The TVS servers listed in the phone's existing ITL may not be reachable when the phones are moved to the new cluster.
- Even if the TVS servers are reachable for certificate verification, the old cluster TVS servers may not have the new servers' certificates.
If these three problems are encountered one possible option is to delete the ITL file manually from all phones being moved between clusters. This is not a desirable solution sine it requires massive effort as the number of phones increases.
Symptoms
Any changes that phones receive through TFTP or HTTP of configuration files will not be honored. Configuration options passed by configuration file partially include:
- URLs including Authentication URL, Directories URL, Services URL; this includes internal/external directories configuration
- some locale features
- callmanager groups for primary and secondary registration
The phone very likely will register, it registers to configured TFTP server by default. The phone likely will not register if the new TFTP server is not running the CallManager service.
When a phone has an incorrect ITL file for the current TFTP server the phone console logs show a message similar to:
1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed </usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not in CTL)
1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED, </usr/ram/SEP00260BD749E9.cnf.xml>
Migration Options
With the three previous problems taken into account we can come up with a plan that allows the migration of phones seamlessly from one cluster to the next without requiring manual phone intervention.
Bulk Certificate Export
Note
The Bulk Certificate Export method will only work if both clusters are online with network connectivity while the phones are being migrated. |
Another possible option if both the old and new clusters will be online at the same time is to use the Bulk Certificate migration method.
Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.
The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate page:
- Export certificates from new destination cluster (TFTP only) and original cluster to a central SFTP server.
- From original cluster, run Consolidate certificates (TFTP only) on the SFTP server using the Bulk Certificate interface.
- On the old origination cluster use the Bulk Certificate function to import the TFTP certificates from the central SFTP server.
- Restart TVS services on old origination cluster.
- Use DHCP option 150, or some other method, to point the phones to the new destination cluster.
- Phones will download the new destination cluster ITL file and attempt to verify it against their existing ITL file.
- The cert will not be in the existing ITL file so the phone will ask the old TVS server to verify the signature of the new ITL file. The phone sends a TVS query to the old origination cluster on TCP port 2445 to make this request.
- If the certificate export/consolidate/import process worked correctly then TVS returns success, and the phone replaces the in memory ITL file with the newly downloaded ITL file.
- The phones can now download and verify the signed configuration files from the new cluster.
Rollback Enterprise Parameter
Note
This method is only valid if completed before the phone migration is attempted. This cannot be used once phones are already in the "verify file failed" state. Phones supporting TVS will potentially lose access to secure URL services such as the Corporate Directory before they are migrated to the new cluster after the "Prepare Cluster for Rollback to pre-8.0" parameter is set to True on the original cluster. Once migrated to the new cluster, the phones will download their new ITLs and Secure URL operation should go back to normal. |
Another option is to make use of the CUCM Enterprise Parameter "Prepare Cluster for Rollback to pre-8.0." Full documentation of the Rollback parameter can be found here. Once this parameter is set to True, the phones will download a special ITL file that contains empty TVS and TFTP certificate sections.
When a phone has an empty ITL file it will accept any unsigned configuration file (for migrations to pre CUCM 8.X clusters), and will also accept any new ITL file (for migrations to different CUCM 8.X clusters).
The empty ITL file can be verified by checking "Settings > Security > Trust List > ITL". Empty entries will appear where the old TVS and TFTP servers used to be.
The phones must have access to the old CUCM servers only as long as it takes them to download the new empty ITL files. Once the phone has an empty ITL file, the old servers can be decommissioned, powered down, thrown into a river, or rebuilt (depending on business requirements).
Using Hardware Security tokens (KEY-CCM-ADMIN-K9=)
If hardware security tokens (product number KEY-CCM-ADMIN-K9=) have been used to generate a CTL (Certificate Trust List) on both the old and new clusters, the phones will be able to freely migrate between clusters as long as at least one of the same hardware tokens was used on both the old and new clusters.
When a phone that has a CTL from the old cluster is moved to the new cluster, it will accept the new cluster's CTL since the new CTL contains a security token certificate in common with it's current CTL. Because the CTL also contains the certificate for the CCM+TFTP server, the new cluster's ITL will also be accepted by the phone so there will be no issues in moving the phone between clusters.
For phones that don't utilize security by default (ITLs) such as the 7960s and 7940s, you will need to re-run the CTL client on the original cluster first to add new TFTP entries for the TFTP servers of the new cluster prior to moving the phones to the new cluster. This is due to those phone models not even reaching out for TFTP files for a server not in the CTL.
This security token method requires this additional hardware and must be configured on the old cluster. Normally security tokens are used to allow for encrypted signaling and media (SRTP) in a cluster and encrypted / authenticated config files. Also, once a cluster has had security enabled with security tokens, disabling security on that cluster requires manually removing the CTL from each phone on that cluster from the phone itself.
Manual ITL Delete
If some catastrophic failure happens and the TFTP key/certificate is no longer available from the old cluster (this is maintained in a DRF backup... TAKE A BACKUP), then the only available option to migrate a phone to a new cluster is to manually delete the ITL file from the phones.
This differs for each phone model. The steps needed for the most common phone models are listed here, but the steps for other models can be found in the Phone Administration Guides.
79XX - Settings > Security > Trust List > ITL File > **# (to unlock the settings) > Erase
89XX/99XX - Settings > Administrator Settings > Reset Settings > Security Settings
