Mutual TLS is broken in 7.5.7 and 7.5.7s on SPA504 and SPA514

[Philip D'Ath]

We run a secure provisioning server. We mutually authenticate the TLS provisioning requests coming in from the phones (to make sure it is a Cisco phone and that the phone has a certificate in it installed by Cisco) before giving them their configuration file.

This process works across a wide variety of software versions, except the latest 7.5.7 and 7.5.7s releases.

On the server we use Apache/2.4.7. For phones running 7.5.7 and 7.5.7s we get this error in the Apache error log:

[client xx.xx.xx.xx:58598] AH02261: Re-negotiation handshake failed: Not accepted by client!?

Important bits of our Apache config relating to this are:
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:!LOW:!SSLv2:!EXPORT
SSLProtocol -ALL +TLSv1.2 +TLSv1.1 +TLSv1
SSLInsecureRenegotiation on

SSLCACertificateFile /etc/apache2/ca.pem
* This definitely contains the Sipura CA certificates (can't remember, about 5 of them)

<Directory /var/www/https/Cisco>
Options MultiViews
AllowOverride None
Order allow,deny
allow from all
SSLVerifyClient require
SSLVerifyDepth 1
</Directory>

How to we go about gettnig bugs investigated?  Our phones are under small business support.

[Philip D'Ath]

An official Bug ID has now been raised.  CSCuu14196.

https://tools.cisco.com/bugsearch/bug/CSCuu14196

[Dan Lukes]

Thanks. What I asked ? I tried to report a few firmware bugs in the past. Full description, logs, how-to-repeat instructions, but no helpful response from TAC. They didn't tried to repeat (even those very simple to repeat), they neither created "Bug" nor rejected the report - just pure waste of my time ...

So I'm curious what's magic you have your's issue solved within month. Have you a SmartNet or other service contract ?

By the way, the working link to your's bug is: CSCuu14196

[Philip D'Ath · Accepted solution]

Persistence.

I have a SmartNet contract on one of each phone we support.  In my country, it is $15 for a 3 year contract.  I lot of the contracts are actually small business support.  Small business support is no good for this kind of issue.

In this case, I opened 7 cases via the TAC portal.  The first 6 were rejected by the engineer because the phone was covered by a small business support contract.  The last engineer can't have noticed, and let the TAC case open.

Once I had a real person I sent in the Apache config, logs, and when the issue started (7.5.7).  This quickly overwhelmed the first level engineer, and he passed it onto the second level engineer.

To speed the process up I asked the level 2 engineer for a MAC address of a phone in his lab.  I then provisioned that phone using our server.  Now I had a direct line into TAC!

After some convincing that it can't be a config issue because with no changes it works in 7.5.6 he tried to recreate the issue using the phone that was provisioned.

He managed to get the exact same error log as I did.  So now the issue was verified by level 2 support.  The trick was making it easy to verify.  If the engineer had to setup a complete server environment it would probably have died right there.

Then it was escalated to a DE engineer.  The DE was given the physical phone that had been provisioned, so they could easily see the error happening.  This made a big difference as they don't have to go to the trouble of setting up an entire server to reproduce the issue.  They can see the issue sitting on the desk in front of them.

About a week later I had a patched version of the code to test, which worked.

[Dan Lukes]

I have a SmartNet contract on one of each phone we support.
...
In this case, I opened 7 cases via the TAC portal.  The first 6 were rejected by the engineer because the phone was covered by a small business support contract.  The last engineer can't have noticed, and let the TAC case open.

Hm ...

It seems to be the miracle. I have no SmartNet and I'm not willing to pay for the right to report fully analyzed bug to Cisco. But Cisco is not interested to support their SMB customers beyond void marketing claims. No, I will not report the same bug 7 times. I'm skilled enough to workaround most of bugs by self, so it's my goodwill. I will not push Cisco to do their job.

But thank you very much for sharing the experience.

To speed the process up I asked the level 2 engineer for a MAC address of a phone in his lab.  I then provisioned that phone using our server.  Now I had a direct line into TAC!

Yes, I did it several times in the past as well. I have no problem to support their Support. I'm skilled technician, so I know how valuable is a skilled customer that's willing to help with issue analysis. Unfortunately, with current SMB support staff, I'm the most skilled person of the team most of time, the first level support guys doesn't understand even the basics - and I'm not willing to teach them how their products works. And if I'm persistent enough so ticket become created, the depth silence will follow with exception of occasional requests answering again the information provided already. Pure waste of time ...

[BSN]

Hello,

Same issue here, 7.5.7(s) both break my XML application over TLS (on a SPA525G2).

Back to 7.5.6c, it works fine.

Any news regarding the corrected 7.5.7.x version ?

In addition, I have found another bug which perhaps you could talk about to the support through your support case p.dath :

According to my findings, 7.5.6c (and I assume 7.5.7(s)) does not support SHA-256 signed certificates.

It only supports SHA-1 signed certificates...

This is quite annoying because of all security issues around TLS and the discouragement regarding SHA-1.

Thank you !

Regards,

Ben

[Dan Lukes]

You can use workaround described in this thread (modify server's configuration so renegotiation will not occur). You can call TAC - it you will mention ticked number disclosed here they should provide you the patched testing version.

According (in)secure SHA-1 ...

... SPAxxx devices are not secure enough to be exposed in public at all. If you wish for true security, then must be connected to closed LAN. And even inter-device communication needs to be blocked (e.g. phone is allowed to send/receive packets to PBX only, but not to other phone; the feature is called private-vlan on Cisco's switches).

I wish there is no scenario where the SHA-1 will mean true risk unless there has been more severe risks caused by free intruder access to the network.

Of course, I'm not going to claim we should not ask for SHA-256, but the real risk from not to having it is not so high.

Just my $0.02

[BSN]

Regarding SHA-256, the thing is that Certification Authorities now sign their certificates using SHA-256.
Well, this is the case with the one I use, it used to provide SHA-1 certificates, now it provides SHA-256 signed.
My XML applications' certificate still uses SHA-1 (I'm lucky) but in a few months I will have to regenerate it, breaking XML apps if SHA-256 Cisco support is not available.

[Philip D'Ath]

Pretty much every public certificate should be SHA-256 and SHA-1 signed.  The certificate on our provisioning server has both signatures and we have a lot of different SPA phones talking to it just fine.

I would be very surprised if any CA stopped adding SHA-1 signatures at this point in time.  Anything that can use SHA-256 will [should?] in perference to SHA-1.

I suspect that as Cisco use newer and newer openssl/ciscossl versions that the TLSv1.1, TLSv1.2 and SHA256 support will get added automatically.  As nothing more than a re-compile should be needed.

[BSN]

I just found a list of some CAs vs SHA-1/SHA-2 :

https://shaaaaaaaaaaaaa.com

Sounds like pretty all (listed here) have migrated to SHA-2.

Regarding PSIRT, a lot are opened on OpenSSL, as U say, features we need will certainly arrive soon with newer openssl/ciscossl versions compiled.

[Philip D'Ath]

This is where is gets slightly tricky.  The SPA51x (and to my knowledge SPA50x) use openssl (0.9.8zc according to the release notes).  The SPA525G2 however uses an internal library, ciscossl (V1.0.1j.4.6 according to the release notes - it used to use openssl 0.9.8k).  So they are not on the same code base for crypto anymore.

I have our Apache server log the crypto being used in the log with every request.

When a SPA525G2 talks to our server it uses ECDHE-RSA-AES256-SHA.
When a SPA514 talks to our server it uses DHE-RSA-AES256-SHA.

So you can see the SPA525G2 uses much strong crypto.  But yes, still using SHA1.  Also the phones (both of them) use TLSv1 not TLSV1.2.  I tried raising a security vulnerability report over this.  SSLv3 was disabled by default in 7.5.7 because it was considered too weak (thank you POODLE).   TLSv1 also has a serious weakness, and I put forward the reasoning that if SSLv3 was being disabled that TLSv1 should also be disabled (and only use TLSv1.1 or TLSv1.2).

The version of openssl being used supports TLSv1.1 and TLSv1.2 - so it really should be a simple matter of enabling these TLS versions.  Also, the version of openssl supports the SHA2 family- but it is not turned on by default.

This is what it says in the openssl notes:
"Support for SHA-2 was introduced in OpenSSL 0.9.8, but is not enabled by default with SSL_library_init(). In 0.9.8, SHA-2 hash functions must be called specifically or by using OpenSSL_add_all_algorithms() which may not be desired. OpenSSL 0.9.8o enables the SHA-2 hash algorithms in the default configuration."

[BSN]

Thank you for all these technical details !

Do you if TAC / Cisco engineers know all these issues, if they are working on them ?

(TLSv1 to disable, SHA-256 support to add)

Thank you !

[Philip D'Ath]

I do not know sorry.  As I say, support for these should be automatic as newer versions of openssl/ciscossl get used.

To get a "new feature" added requires a business case to be built.  As in spending $x on development will result in revenue of $y.  Just asking for a feature gets no where, and it is pretty much impossible for an external party to get a feature added (unless you are committing to buy a large amount of $y to make the business case).

The only other way is to get a psirt created and have it resolved as a security issue.

http://www.cisco.com/en/US/products/products_security_advisories_listing.html

[BSN]

I sent a mail to TAC, they know about the SHA issue and are going to take a look into it.

[Dan Lukes]

You can't disable TLSv1.0 on server. Brand new phones are distributed with 7.5.2 firmware (and some even with 7.4.4). If you wish for zero-touch deployment then support for TLSv1.0 and even SSLv3 may be disabled in some firmwares, but must be retained on server.

For similar reason you can't use SHA-256 based certificate on provisioning server. Thus even newest firmware must not be tied to SHA256 only.

For the same reason you need to use 1024b key on server - and even DH must not exceed 1024b.

Well, if you give up on zero-touch deployment, then you may upgrade to 2048B key, SHA-256 and TLSv1.2 only. But secure zero-touch configuration is one of most valuable feature of this product. If you are not interested in it, you may consider other vendor's product, you will save the money.

Also, don't forget the TLS can be used on SIP channel as well (and S-RTP can't work withoout it). SPA50x hardware is not fast enough for all those new ciphers. Even SIP/TLS/1024B key make user experience not so good.

[BSN]

Really interesting, due to Logjam attack, I have regenerated my DH group, of course now my XML apps are much more slower...

Private key is 4096B so I moved DH from 1024B to 4096B, I jumped from 1.6s to 4.6s for each XML request...

I then plan to re-think this configuration.

This reminds me the discussion we had here :

Cisco XML Phone applicatiosn over https (SSL)

And performance tests you made on several devices.

I have a related question to what U just said, I'll post there.