cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
150797
Views
70
Helpful
57
Comments
Jason Burns
Level 1
Level 1

 

 

Purpose

CUCM 8 introduced the new Security By Default feature and the use of ITL (Initial Trust List) files. (More documentation here). With this new feature, care must be taken when moving phones between different CUCM clusters. Without following the proper steps it is possible to encounter a situation where thousands of phones must manually have their ITL files deleted.

 

Background

Phones supporting the new ITL file download this special file from their CUCM TFTP server. Once an ITL file is installed on a phone, all future configuration files and ITL file updates must be either:

 

  1. Signed by the CCM+TFTP Server certificate currently installed in the phone's CTL file (If cluster security with CTLs is enabled).
  2. Signed by the CCM+TFTP Server certificate installed in the phone's ITL file.
  3. Signed by a certificate that exists in one of the CUCM Server TVS certificate stores that are listed in the ITL file.

 

With this new security functionality in mind here are the three problems that we can encounter when moving a phone from one cluster to another cluster.

 

  1. The ITL file of the new cluster is not signed by the phone's existing ITL CCM+TFTP cert, so the phone will not accept the new ITL file or config files.
  2. The TVS servers listed in the phone's existing ITL may not be reachable when the phones are moved to the new cluster.
  3. Even if the TVS servers are reachable for certificate verification, the old cluster TVS servers may not have the new servers' certificates.

 

If these three problems are encountered one possible option is to delete the ITL file manually from all phones being moved between clusters. This is not a desirable solution sine it requires massive effort as the number of phones increases.

 

Symptoms

Any changes that phones receive through TFTP or HTTP of configuration files will not be honored.  Configuration options passed by configuration file partially include:

  • URLs including Authentication URL, Directories URL, Services URL; this includes internal/external directories configuration
  • some locale features
  • callmanager groups for primary and secondary registration

The phone very likely will register, it registers to configured TFTP server by default. The phone likely will not register if the new TFTP server is not running the CallManager service.

 

When a phone has an incorrect ITL file for the current TFTP server the phone console logs show a message similar to:

 

 

1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed </usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not in CTL)

1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED, </usr/ram/SEP00260BD749E9.cnf.xml>

 

Migration Options

With the three previous problems taken into account we can come up with a plan that allows the migration of phones seamlessly from one cluster to the next without requiring manual phone intervention.

 

Bulk Certificate Export

Note

The Bulk Certificate Export method will only work if both clusters are online with network connectivity while the phones are being migrated.

 

Another possible option if both the old and new clusters will be online at the same time is to use the Bulk Certificate migration method.

 

Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.

 

The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate page:

  1. Export certificates from new destination cluster (TFTP only) and original cluster to a central SFTP server.
  2. From original cluster, run Consolidate certificates (TFTP only) on the SFTP server using the Bulk Certificate interface.
  3. On the old origination cluster use the Bulk Certificate function to import the TFTP certificates from the central SFTP server.
  4. Restart TVS services on old origination cluster.
  5. Use DHCP option 150, or some other method, to point the phones to the new destination cluster.
  6. Phones will download the new destination cluster ITL file and attempt to verify it against their existing ITL file.
  7. The cert will not be in the existing ITL file so the phone will ask the old TVS server to verify the signature of the new ITL file. The phone sends a TVS query to the old origination cluster on TCP port 2445 to make this request.
  8. If the certificate export/consolidate/import process worked correctly then TVS returns success, and the phone replaces the in memory ITL file with the newly downloaded ITL file.
  9. The phones can now download and verify the signed configuration files from the new cluster.

 

 

Rollback Enterprise Parameter

Note

This method is only valid if completed before the phone migration is attempted. This cannot be used once phones are already in the "verify file failed" state.

Phones supporting TVS will potentially lose access to secure URL services such as the Corporate Directory before they are migrated to the new cluster after the "Prepare Cluster for Rollback to pre-8.0" parameter is set to True on the original cluster.  Once migrated to the new cluster, the phones will download their new ITLs and Secure URL operation should go back to normal.

 

Another option is to make use of the CUCM Enterprise Parameter "Prepare Cluster for Rollback to pre-8.0." Full documentation of the Rollback parameter can be found here. Once this parameter is set to True, the phones will download a special ITL file that contains empty TVS and TFTP certificate sections.

 

When a phone has an empty ITL file it will accept any unsigned configuration file (for migrations to pre CUCM 8.X clusters), and will also accept any new ITL file (for migrations to different CUCM 8.X clusters).

 

The empty ITL file can be verified by checking "Settings > Security > Trust List > ITL". Empty entries will appear where the old TVS and TFTP servers used to be.

 

The phones must have access to the old CUCM servers only as long as it takes them to download the new empty ITL files. Once the phone has an empty ITL file, the old servers can be decommissioned, powered down, thrown into a river, or rebuilt (depending on business requirements).

 

Using Hardware Security tokens (KEY-CCM-ADMIN-K9=)

If hardware security tokens (product number KEY-CCM-ADMIN-K9=) have been used to generate a CTL (Certificate Trust List) on both the old and new clusters, the phones will be able to freely migrate between clusters as long as at least one of the same hardware tokens was used on both the old and new clusters.

 

When a phone that has a CTL from the old cluster is moved to the new cluster, it will accept the new cluster's CTL since the new CTL contains a security token certificate in common with it's current CTL. Because the CTL also contains the certificate for the CCM+TFTP server, the new cluster's ITL will also be accepted by the phone so there will be no issues in moving the phone between clusters.

 

For phones that don't utilize security by default (ITLs) such as the 7960s and 7940s, you will need to re-run the CTL client on the original cluster first to add new TFTP entries for the TFTP servers of the new cluster prior to moving the phones to the new cluster.  This is due to those phone models not even reaching out for TFTP files for a server not in the CTL.

 

This security token method requires this additional hardware and must be configured on the old cluster. Normally security tokens are used to allow for encrypted signaling and media (SRTP) in a cluster and encrypted / authenticated config files. Also, once a cluster has had security enabled with security tokens, disabling security on that cluster requires manually removing the CTL from each phone on that cluster from the phone itself.

 

Manual ITL Delete

If some catastrophic failure happens and the TFTP key/certificate is no longer available from the old cluster (this is maintained in a DRF backup... TAKE A BACKUP), then the only available option to migrate a phone to a new cluster is to manually delete the ITL file from the phones.

 

This differs for each phone model.  The steps needed for the most common phone models are listed here, but the steps for other models can be found in the Phone Administration Guides.

 

79XX - Settings > Security > Trust List > ITL File > **# (to unlock the settings) > Erase

 

89XX/99XX - Settings > Administrator Settings > Reset Settings > Security Settings

Comments
Michael Berry
Level 1
Level 1

Same exact problem here...no solution yet?

Leonardo Santana
Spotlight
Spotlight

Hello,

I will migrate my CUCM from MCS to UCS, i have version 8.6 i will instal the 8.6 at Vmware and after that i will upgrade to 10.X

My question is i will use the same IP address of my old cluster, i have issues with the ITL Files?

Regards

Leonardo Santana

j.huizinga
Level 6
Level 6

Hi,

Did you ever figure this out? I have the same null message

JH

pkinane
Cisco Employee
Cisco Employee

I filed this defect:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy43181

-------------

Please rate helpful content (i.e. videos, documents, comments) so quality content shows at the top of people's search results. Also, please select the correct answer(s) if any comment(s) answers your question otherwise the question remains on the support forums as unanswered.

-------------

mmoulson1
Level 4
Level 4

We ended up using the Rollback Enterprise Parameter method for our migration which worked fine.

For individual phones that were used for testing we just manually deleted the ITL files.

We ran out of time to troubleshoot the certificate error further. @pkinane I would have been interested to try option 2 from your defect workaround!

Matty

Mohamed Gaffoor
Level 1
Level 1

Hi

Why dont you use Cisco Prime Collaboration , this will migrate your hardware to UCS and upgrade the software automatically.

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/pcdadmin/11_0_1/CUCM_BK_PB6D9005_00_pcd-administration-guide-110/cisco_prime_collaboration_deployment_features.html

thanks

Mohamed

Eduardo Tavares
Level 1
Level 1

Hi Brian !

In my case, I have Pub, 2 Subs and 2 TFTP all servers have TVS started. I need restart TVS in all nodes or only TFTP ?

Thanks,

Eduardo.

pkinane
Cisco Employee
Cisco Employee

Eduardo,

The nodes running CCM will be used to verify the signer of a certificate or file via the TVS. The TFTP can also be used should the servers running CCM not be be able to verify the signer of a certificate/file.

I recommend, as a minimum, restarting the TVS service on any server that is running the CCM service. Personally, I would restart on all nodes.

Craig Dyer
Level 3
Level 3

Luke,

How did you manage the migration in the end?

I am looking at updating a customers domain name and wanted to use the roll back option but am unsure if this will work for the VPN phones.

If any body can guide some advice and guidance that would be very appreciated.

Many Thanks,

Craig

pkinane
Cisco Employee
Cisco Employee

Craig,

If you have more than one server in your CUCM cluster, when done correctly you don't need the roll back parameter.

The first step is to determine how the VPN connection is established. Is it username and password or certificate based? Is it using MIC or LSC?

R/s,

Patrick

Craig Dyer
Level 3
Level 3

Patrick,

My situation is slightly different, the need to change the CUCM platforms domain which will regenerate the ITL and CTL.

Phones are using username and password.

Many Thanks,

Craig

pkinane
Cisco Employee
Cisco Employee

Craig,

As you are using credentials based VPN you don't have to worry about updating the LSC on the phones and uploading the new CAPF to the ASA.

For the ITL and CTL update you first need to mention how many servers you have in the cluster, and how many are running CCM service.

R/s,

Patrick

Craig Dyer
Level 3
Level 3

Patrick,

We have a pair of servers running CuCM.

Craig

pkinane
Cisco Employee
Cisco Employee

Craig,

This is good. First things first, check your cluster for database replication and other health issues (be sure to document and save all of this data you will get). I'll list a few things to check below. Second scan your cluster for phones with ITL issues before you make the domain name change (use the ITL Scanner tool below). Spot check some of the phones for your users by checking the status messages, and debug display (make sure there are not DNS failures, TCP timeouts, ITL Update Failed, etc...).

This is important because you will know if the cluster was healthy, or if their were ITL issues, before you made the change. Without this information you will find the issues after the change and think it was the change that caused the issue when in actuality it might not be the case.

http://www.unifiedfx.com/itl-scanner/

Check these reports in the Cisco Unified Reporting page:

Unified CM Cluster Overview

Unified CM Database Status

Run these commands in the CLI

utils dbreplication runtimestate (publisher only, click for more information)

utils ntp status

show network cluster

show tech network hosts

run sql select name from processnode (all entries shouldhave an exact match for an entry seen in show tech network hosts)

utils diagnose test

show status (make sure the CPU, I/O Wait, and memory is good. It is normal to see I/O Wait in single digits)

show ctl (it is expected you have a CTL if your cluster is in mixed mode or was in mixed mode previously)

show itl (check the date last modified, look at the certicates in the file and make sure nothing seems strange, make sure the file is parsed successfully. NOTE: The ITL will only be on servers running the TFTP service.)

Be sure to analyze all output closely for any errors, discrepancies, or things that seem different between the nodes.

When performing the change be sure to only make the domain name change on one server at a time. By this, I mean enter the set network domain command, let the node do its automated reboot, check to see if the ITL was updated, reboot your phones, then check for ITL issues with the phones. If everything looks fine, move on to the next server, follow the same steps, then move on to the next server in the cluster and so on. It seems like a pain, but ITL issues are more of a pain.

Once you've changed the domain on all servers in the cluster, reboot the whole cluster and check all the same show commands and  reports which were listed earlier.

Be mindful of devices that could be affected by changing the domain such as MGCP enpoints. Also be mindful that this overwrites your certificates on the servers. If you use CA signed certificates, you will need to redo this. If you exported your certs to any other device or cluster you will need to redo that too (i.e. EMCC).

Please let me know the full version of CUCM you are using.

Craig Dyer
Level 3
Level 3

Patrick,

Thanks for you comments but they regard normal migration processes, my question was purely regarding VPN phones and the use of blank ITL/CTL. From memory updating ITL/CTL for VPN phones is blocked to mitigate man in the middle attacks but can't find that documented any more.

I have resolved my customers requirement in another manner so the question regarding rollback and VPN phones is now only out of interest.

Craig

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: