Solved: Mutual TLS is broken in 7.5.7 and 7.5.7s on SPA504 and SPA514

Philip D'Ath · ‎04-13-2015

We run a secure provisioning server. We mutually authenticate the TLS provisioning requests coming in from the phones (to make sure it is a Cisco phone and that the phone has a certificate in it installed by Cisco) before giving them their configuration file.

This process works across a wide variety of software versions, except the latest 7.5.7 and 7.5.7s releases.

On the server we use Apache/2.4.7. For phones running 7.5.7 and 7.5.7s we get this error in the Apache error log:

[client xx.xx.xx.xx:58598] AH02261: Re-negotiation handshake failed: Not accepted by client!?

Important bits of our Apache config relating to this are:
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:!LOW:!SSLv2:!EXPORT
SSLProtocol -ALL +TLSv1.2 +TLSv1.1 +TLSv1
SSLInsecureRenegotiation on

SSLCACertificateFile /etc/apache2/ca.pem
* This definitely contains the Sipura CA certificates (can't remember, about 5 of them)

<Directory /var/www/https/Cisco>
Options MultiViews
AllowOverride None
Order allow,deny
allow from all
SSLVerifyClient require
SSLVerifyDepth 1
</Directory>

How to we go about gettnig bugs investigated? Our phones are under small business support.

Philip D'Ath · ‎05-15-2015

A follow up for everyone on this one.

TAC managed to reproduce the problem I was having. A DE engineer has located the problem and built me a new image to load onto the phones which resolves the issue.

So hopefully this bug fix will make it into a mainline image in the future.

View solution in original post

Philip D'Ath · ‎05-17-2015

Persistence.

I have a SmartNet contract on one of each phone we support. In my country, it is $15 for a 3 year contract. I lot of the contracts are actually small business support. Small business support is no good for this kind of issue.

In this case, I opened 7 cases via the TAC portal. The first 6 were rejected by the engineer because the phone was covered by a small business support contract. The last engineer can't have noticed, and let the TAC case open.

Once I had a real person I sent in the Apache config, logs, and when the issue started (7.5.7). This quickly overwhelmed the first level engineer, and he passed it onto the second level engineer.

To speed the process up I asked the level 2 engineer for a MAC address of a phone in his lab. I then provisioned that phone using our server. Now I had a direct line into TAC!

After some convincing that it can't be a config issue because with no changes it works in 7.5.6 he tried to recreate the issue using the phone that was provisioned.

He managed to get the exact same error log as I did. So now the issue was verified by level 2 support. The trick was making it easy to verify. If the engineer had to setup a complete server environment it would probably have died right there.

Then it was escalated to a DE engineer. The DE was given the physical phone that had been provisioned, so they could easily see the error happening. This made a big difference as they don't have to go to the trouble of setting up an entire server to reproduce the issue. They can see the issue sitting on the desk in front of them.

About a week later I had a patched version of the code to test, which worked.

View solution in original post

BSN · ‎10-18-2015

Good news !

I just tested 7.6.1 firmware released a few days ago, XML applications are back into business !

In addition, SHA-256 signed certificates are now supported.

Perfect !

Ben

View solution in original post

BSN · ‎10-18-2015

And it is much more faster !

XML requests' speed do not seem to be affected by DH group size anymore !

I'm back with a 4096-bits DH without impact.

View solution in original post

Dan Lukes · ‎04-14-2015

You can call SMB support for help.

Or we can help you here.

In both cases, you should turn on syslog&debug *highest level) on the phone and catch them. It may reveal critical informations.

In the mean time ...

I'm using the following (even with 7.5.7):

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite HIGH:!ADH:!EXPORT56

I have no

SSLInsecureRenegotiation on

You may try my configuration to narrow issue cause.

This definitely contains the Sipura CA certificates (can't remember, about 5 of them)

No good time for uncertainty. There must be appropriate CA certificate here or the negotiation will fail. So verify you have ...

C=US, ST=California, L=San Jose, O=Cisco Small Business, OU=Cisco Small Business Certificate Authority, CN=Cisco Small Business Client Root Authority 2/emailAddress=ciscosb-certadmin@cisco.com

... certificate here (a.k.a. Cisco 2k Small Business CA).

See also: SPA Certificate Authority (CA) List

Philip D'Ath · ‎04-14-2015

I am highly confident that our server configuration is correct, since we can perform mutual TLS authentication on all over phone software versions *except* 7.5.7 and 7.5.7s.

Attached is a log from a SPA514 running 7.5.6a where mutual TLS is working.

Attached is a log from the same SPA514 running 7.5.7s where mutual TLS is broken.

These logs have been sanitized.

Note that our global spa514.cfg is not mutual TLS authenticated, and always works. The phone specific config, https://xx.xx.xx.xx/Cisco/SPA514G/a4934cfe70fa.cfg, is mutual TLS authenticated and this is the one that has the issue in 7.5.7 and 7.5.7.s.

The phone was factory before taking each log.

When it fails the phone logs:

[0]TCP: RemoteClose

[0]SIP TCP server disconnected -257

The Apache error log shows:

[ssl:error] [pid 13722:tid 139730804971264] [client xx.xx.xx.xx:8520] AH02261: Re-negotiation handshake failed: Not accepted by client!?

Philip D'Ath · ‎03-09-2016

Bummer. I have just been testing out the "new" 8861's with the new 3PCC (third party call controll) running sip88xx.10-3-1-9-3PCC - and the bug is back!!!

BSN · ‎06-20-2016

There are new firmware versions but for the CUCM version.

According to what I found, you can update your device with, but you won't be able to come back to the 3PCC firmware.

Incredible you get the bug back on newer devices !

Philip D'Ath · ‎06-20-2016

Tell me about it. I have taken it up with people in the team that work on this and they promise me the code base is not shared.

I told them someone is telling a lie.

Dan Lukes · ‎06-20-2016

Guys, neither CUCM nor 8861 are SMB class of product, so you are rather off-topic here. Consider new thread in more appropriate community - you may receive comments from experts on matter ...

Philip D'Ath · ‎04-14-2015

I have turned the Apache debugging write up, and caught this happening:

ssl_engine_kernel.c(770): [client xx.xx.xx.xx:1026] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)

[Wed Apr 15 08:49:09.687762 2015] [ssl:info] [pid 14360:tid 140287187810048] [client xx.xx.xx.xx:1026] AH02226: Awaiting re-negotiation handshake

[Wed Apr 15 08:49:19.963124 2015] [reqtimeout:info] [pid 14360:tid 140287187810048] [client xx.xx.xx.xx:1026] AH01382: Request body read timeout

[Wed Apr 15 08:49:19.963329 2015] [ssl:error] [pid 14360:tid 140287187810048] [client xx.xx.xx.xx:1026] AH02261: Re-negotiation handshake failed: Not accepted by client!?

When it wants to do Mutual TLS in 7.5.7/7.5.7s, Apache sends a request to perform a full TLS renegotiation. This request is not responded to by the phone and a timeout occurs. Remember this is only happening in 7.5.7 and 7.5.7s.

Dan Lukes · ‎04-14-2015

With our configuration no renegotiation is required for mutual TLS, so it doesn't occur. It explain why 7.5.7 works for me.

I'm not sure about the Cisco's policy related to renegotiation. You hit either firmware bug or intentional feature.

Phone's syslog&debug may reveal more.

Philip D'Ath · ‎04-14-2015

I've tried the SSL settings you supplied, plus a million other combinations, and a renegotiation still occurs.

As this issue only started happening in the latest 7.5.7 software it has to be a firmware bug. 7.5.7 changed the openssl settings in use on the phone.

Which version of Apache are you using?

The Apache docs say:

"Apache HTTP Server will request SSL renegotiation any time an SSL session is already established but a request is made for a per-location context which requires different security -- for example, if you have the SSLVerifyClient directive in a Directory or Location block.

I only require SSLVerifyClient on one directory in my setup, so an SSL renegotiation has to happen. This is no choice in the matter. Have you got it configured site side? That would be the only way you could avoid the renegotiation.

Philip D'Ath · ‎04-14-2015

OMG, that cracked it.

If I change the site to require global mutual TLS it works. If I use mutual TLS for only a specific directly TLS renegotiation happens, and that is broken in 7.5.7 and 7.5.7s.

Dan Lukes · ‎04-14-2015

Apache 2.4.10, SSLVerifyClient optional site wide. I don't care the results if no authentication required. Otherwise I care.

Our configuration is served by simple script, so I test the authentication results here, but you may consider to use either SSLRequire or FakeBasicAuth

As this issue only started happening in the latest 7.5.7 software it has to be a firmware bug.

Or a former firmware bug related to renegotiation has been corrected. I have no information enough to decide.

Philip D'Ath · ‎05-15-2015

A follow up for everyone on this one.

TAC managed to reproduce the problem I was having. A DE engineer has located the problem and built me a new image to load onto the phones which resolves the issue.

So hopefully this bug fix will make it into a mainline image in the future.

Dan Lukes · ‎05-16-2015

Can you disclose ticket number ?

Philip D'Ath · ‎05-16-2015

My TAC case number is 634465809. I should see if they made this in a bug in the bug navigator.