Authenticated fail. Reason:12

[Jean-Christophe Majchrzak]

Hello,

We've done failover tests and have shutdown half of CUCM servers.

We still have Call manager and TFTP subscribers.

Hundreds of phones are working and hundreds of phones are not registering with the following errors :

[11:27:44, 05/18/2013] CTLSEP7081058581C6.tlv updating

[11:28:04, 05/18/2013] TFTP error : download CTLSEP7081058581C6.tlv fail

[11:28:04, 05/18/2013] CTLSEP7081058581C6.tlv updated fail

[11:28:04, 05/18/2013] CTLSEP7081058581C6.tlv updating

[11:28:04, 05/18/2013] CTLSEP7081058581C6.tlv not found in 10.30.1.1

[11:28:04, 05/18/2013] ITLSEP7081058581C6.tlv updating

[11:28:04, 05/18/2013] ITLSEP7081058581C6.tlv (HTTP)

[11:28:49, 05/18/2013] ITLSEP7081058581C6.tlv authenticate fail

[11:28:49, 05/18/2013] SEP7081058581C6.cnf.xml.sgn (HTTP)

[11:29:34, 05/18/2013] invalid file SEP7081058581C6.cnf.xml.sgn, authenticated fail. Reason:12.

[11:29:54, 05/18/2013] TFTP error : download XMLDefault.cnf.xml.sgn fail

[11:29:54, 05/18/2013] can't download configuration file XMLDefault.cnf.xml.sgn from 10.30.1.2

[11:29:54, 05/18/2013] XMLDefault.cnf.xml.sgn (HTTP)

[11:30:39, 05/18/2013] invalid file XMLDefault.cnf.xml.sgn, authenticated fail. Reason:12.

Where 10.30.1.2 is the TFTP that is shutdown and 10.30.1.1 is the TFTP up.

What can I do to solve this ?

JC,

Edit :  I'm running CUCM 8.6.2 and phones are 8945 with lastest published fw.  

[Aman Soi]

Hi Jean,

Can u cross-check whether TVS is running on TFTP server?

regds,

aman

[Jean-Christophe Majchrzak]

Hi Aman

What is Trust Verification Service ?

On the CUCM server I have only Security Service -> Cisco CTL Provider   and it's Deactivated.

JC

[Jean-Christophe Majchrzak]

Moreover I can confirm that we haven't installed and started any security service on CUCM so far.

[Jean-Christophe Majchrzak]

In fact paying more attention to services window I've noticed that Cisco Trust Verification Service is running on the servers.

[Aman Soi]

Hi Jean,

Cucm 8.0 onwards Security by Default [SBD] has been introduced with ITL [Initial Trust List ] getting automatically enabled.

I was trying to figure out the reason behind this but could not find.

If someboday knows,please help out.

regds,

aman

[Jean-Christophe Majchrzak]

I've restarted Publisher, Call Manager Subscriber and TFTP Subscriber and all the failed phones registered again. Thanks to the Primary TFTP.

1100 phones out of 3000 were not able to deal with backup TFTP.

Very strange.

I'll follow this issue with my support within the next weeks.

[Aman Soi]

Hi Jean,

Can u share the output of show itl from both TFTP servers?

regds,

aman

[Jean-Christophe Majchrzak]

Aman, I've exported these informations but I'm afraid they contain sensitive informations. I'm right ?

Maybe I can remove some parts of the listing ? Let me know.

For information, when we do a standard failover, that means I shutdown half of the CUCM cluster, all the phones are registering as expected and the other TFTP does not create any issue as it is normally not involved.

My problem appeared when in addition to the shutdown , I also do a restart of the remaining half cluster. All phones loose connectivity, and when the half cluster is up and running, I'm facing this ITL problem on these 1100 phones.

No specificity regarding their physical location, installation date or firmware version. Impact is everywhere.

JC,

[Aman Soi]

Hi Jean,

What I suggest is open TAC case with Cisco and resolve it?

please share the outcome.

regds,

aman

[Stephen Welsh]

Hi JC,

It looks like you have a number of phones with their ITL Files out-of-sync with certificates used by the TFTP/TVS Service. For the error log you posted as a minumium it's the TFTP Servers (CallManager.pem) file that is out of sync, there are a number of things that can cause this, for example:

• DNS/hostname changes
• Phone firmware upgrades
• CUCM Version upgrade
• Multi-cluster topologies
• 2 or more TFTP Servers in a cluster

An excellent reference on how SBD works is the follownig document form Jason Burns:

Communications Manager Security By Default and ITL Operation and Troubleshooting

https://supportforums.cisco.com/docs/DOC-17679

I also recomend getting Akhil Behl's book for a more complete reference on Cisco's PKI implimentation at the heart of SBD:

Securing Cisco IP Telephony Networks

http://www.amazon.com/dp/1587142953

It may be enough to restart the TFTP Server, if not fundamentally you have two choices, revert the CallManager.pem certificate from a back-up or delete the ITL Files on the all affected phones.

I'm the CTO of UnifiedFX and original author of PhoneView (http://www.unifiedfx.com) a complete endpoint management solution. We originally solved how to delete and manage ITL files remotely when customer first hit issues with Security by Default and as such have extended PhoneView to make managing ITL Files very simple.

I addition to the above references for learnign more about Security by Default, I recommend you trial PhoneView as it's most likely you will have to delete those ITL Files and you really don't want to have to walk round all those phones

You can request a trial from here:

http://www.unifiedfx.com/phoneview/trial

Thanks

Stephen Welsh

CTO

http://www.unifiedfx.com

[Jean-Christophe Majchrzak]

Thanks for your answer Stephen, I'll have a look at all of this.