Cisco SPA112 Firmware 1.4.1(SR3) can't establish TLS

[Tim Harman]

Hi,

My Cisco SPA112 has been working great with my provider, until the new firmware released a few days ago, 1.4.1SR3.  It was on SR1 previous.

The new SR3 has been created it seems to fix a problem where bad actors could submit a false/bad certificate for the TLS session.

But now I can't establish a SIP session to my provider using TLS anymore, I've had to downgrade back to UDP to get it to work.

My provider is 2talk, a well known New Zealand SIP provider.  Here's a log of an SSL tester against their server tls.2talk.co.nz

Testing server defaults (Server Hello)

TLS extensions (standard) "renegotiation info/#65281" "session ticket/#35" "heartbeat/#15"
Session Ticket RFC 5077 hint 300 seconds, session tickets keys seems to be rotated < daily
SSL Session ID support yes
Session Resumption Tickets: yes, ID: yes
TLS clock skew Random values, no fingerprinting possible
Signature Algorithm SHA256 with RSA
Server key size RSA 2048 bits
Server key usage Digital Signature, Key Encipherment
Server extended key usage TLS Web Server Authentication, TLS Web Client Authentication
Serial / Fingerprints 70CBFCC2A2BA4D48D15226C757BD6CDE / SHA1 02624ACFBC81C2C21DA23D3FB5DA275E791FE61B
SHA256 FD1B1B042D4CBEC1F5B46C974ADECE7FA9E530AFB8E788B6B9AF77A60EC4E80E
Common Name (CN) tls.2talk.co.nz
subjectAltName (SAN) tls.2talk.co.nz
Issuer RapidSSL SHA256 CA (GeoTrust Inc. from US)
Trust (hostname) Ok via SAN and CN (same w/o SNI)
Chain of trust Ok
EV cert (experimental) no
"eTLS" (visibility info) not present
Certificate Validity (UTC) 145 >= 60 days (2016-07-21 12:00 --> 2019-09-20 11:59)
# of certificates provided 2
Certificate Revocation List http://gp.symcb.com/gp.crl
OCSP URI http://gp.symcd.com
OCSP stapling not offered
OCSP must staple extension --
DNS CAA RR (experimental) not offered
Certificate Transparency yes (certificate extension)

You can see both the certificate itself and the SNI are the same value.

However, in the logs of the Cisco SPA112 when I try to establish a TLS session, I get the following:

Apr 26 22:12:58 chatterbox [0]SIP/TCP NewLocalPort:5078
Apr 26 22:12:58 chatterbox [0]SIP/TCP NewLocalPort:5078
Apr 26 22:12:58 chatterbox [0: 0]SIP/TCP:Connecting(12)...
Apr 26 22:12:58 chatterbox [0: 0]SIP/TCP:Connecting(12)...
Apr 26 22:12:58 chatterbox [0: 0]SIP/TCP:Connect=0
Apr 26 22:12:58 chatterbox [0: 0]SIP/TCP:Connect=0
Apr 26 22:12:58 chatterbox [0: 0]SIP/TLS:Connecting(12)...
Apr 26 22:12:58 chatterbox [0: 0]SIP/TLS:Connecting(12)...
Apr 26 22:12:58 chatterbox ssl cert err 20
Apr 26 22:12:58 chatterbox [0: 0]SIP/TLS:Connect=-1
Apr 26 22:12:58 chatterbox [0: 0]SIP/TLS:Connect=-1
Apr 26 22:12:58 chatterbox [0: 0]SIP/TLS:Connect Failed
Apr 26 22:12:58 chatterbox [0: 0]SIP/TLS:Connect Failed
Apr 26 22:12:58 chatterbox RSE_DEBUG: getting alternate from domain:tls.2talk.co.nz
Apr 26 22:12:58 chatterbox RSE_DEBUG: All server is down:43
Apr 26 22:12:58 chatterbox RSE_DEBUG: Domain: tls.2talk.co.nz, type=SRVHOST
Apr 26 22:12:58 chatterbox RSE_DEBUG: Total Address:1, up addr:0, ref id:0, ref cnt:1
Apr 26 22:12:58 chatterbox RSE_DEBUG: Current addr:NULL
Apr 26 22:12:58 chatterbox RSE_DEBUG: curr timestamp::42757, pri:600, scnd:600
Apr 26 22:12:58 chatterbox RSE_DEBUG: pri:0, addr: 27.111.14.65:5061, status=DOWN, visited=TRUE, ttl=102757, pri=PRIM
Apr 26 22:12:58 chatterbox [0]SIP/TCP Backoff 1000 ms

Does anyone have any suggestions?  I assume Cisco is using OpenSSL here, and from a bit of Googling (so maybe incorrect) an error 20 is: 

verify error:num=20:unable to get local issuer certificate

But every other unix system I have connects to tls.2talk.co.nz:5061 just fine:

{16:28}~ ➭ openssl s_client -connect tls.2talk.co.nz:5061
CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
verify return:1
depth=0 CN = tls.2talk.co.nz
verify return:1
---
Certificate chain
0 s:/CN=tls.2talk.co.nz
i:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA
1 s:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
---

What's going on?  Is 1.4.1(SR3) buggy??

This was working fine in 1.4.1(SR1)

[Dan Lukes]

---

I assume Cisco is using OpenSSL here, and from a bit of Googling (so maybe incorrect) an error 20 is: 

verify error:num=20:unable to get local issuer certificate

But every other unix system I have connects to tls.2talk.co.nz:5061 just fine:

{16:28}~ ➭ openssl s_client -connect tls.2talk.co.nz:5061
CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
verify return:1
depth=0 CN = tls.2talk.co.nz
verify return:1
---
Certificate chain
0 s:/CN=tls.2talk.co.nz
i:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA
1 s:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
---

Simple question - the certificate used by tls.2talk.co.nz:5061 has been issued by GeoTrust Global CA

Have you configured such CA as trusted in SPA112 (Custom CA option) ? If no, then such connection shall not be trusted - it should be refused. Everything works as designed.

What's going on?  Is 1.4.1(SR3) buggy??

This was working fine in 1.4.1(SR1)

1.4.1SR1 has been buggy. It has been willing to accept any certificate (issued by any CA) which make SSL useless.

Just claim CA in question trusted and it will become work again.

[Tim Harman]

Really appreciate your prompt reply in pointing me in the right direction, however, I'm still stuff.

I've created a CA file, with both the root CA and the intermediate CA file in it.  OpenSSL tells me it's valid if I force it to use that file as the CA:

[21:56:03] root :: micro ➜ /var/www/micro » openssl s_client -CAfile ./chatterbox.pem -connect tls.2talk.co.nz:5061 | grep return
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
verify return:1
depth=0 CN = tls.2talk.co.nz
verify return:1
Verify return code: 0 (ok)

I've uploaded this (via http) to the Cisco SPA112.  I see it collecting the file:

192.168.0.10 - - [27/Apr/2019:21:55:15 +1200] "GET /chatterbox.pem HTTP/1.1" 200 3089 "-" "Cisco/SPA112-1.4.1(SR3) (CCQ173101N7)"

I see in the Information section right at the bottom of the SPA112 Information screen:

Custom CA Status
Custom CA Provisioning Status:	Last provisioning succeeded on 04/27/2019 21:55:13
Custom CA Info:	Installed - /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

Yet still it fails.  This time it fails with Err code 2, which to me indicates that it isn't reading in the Intermediate CA certificate.  But it's there, you can view the CA file here: http://micro.muppetz.com/chatterbox.pem

I've also changed the order of the two certs, in case it wanted the Root First then the Intermediate, or the other way around.

Still always Err 2:

Apr 27 22:00:07 chatterbox [0: 0]SIP/TCP:Connecting(12)...
Apr 27 22:00:07 chatterbox [0: 0]SIP/TCP:Connecting(12)...
Apr 27 22:00:07 chatterbox [0: 0]SIP/TCP:Connect=0
Apr 27 22:00:07 chatterbox [0: 0]SIP/TCP:Connect=0
Apr 27 22:00:07 chatterbox [0: 0]SIP/TLS:Connecting(12)...
Apr 27 22:00:07 chatterbox [0: 0]SIP/TLS:Connecting(12)...
Apr 27 22:00:07 chatterbox ssl cert err 2
Apr 27 22:00:07 chatterbox [0: 0]SIP/TLS:Connect=-1
Apr 27 22:00:07 chatterbox [0: 0]SIP/TLS:Connect=-1
Apr 27 22:00:07 chatterbox [0: 0]SIP/TLS:Connect Failed
Apr 27 22:00:07 chatterbox [0: 0]SIP/TLS:Connect Failed

Is there some other knob/setting I've got to hit here?

Thanks!

[Dan Lukes · Accepted solution]

Well, I have no experience with configuration utilizing Custom CA. My phones are connected to my own PBX ant it uses Cisco issued certificate with no intermediate CA. So I can just guess.

So my advice: don't try to import both certificates (root and intermediate). Import root certificate only. Note that http://micro.muppetz.com/chatterbox.pem contains intermediate certificate. Root certificate is here: https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem

Try and lets me know the results.

[Tim Harman]

Thanks again.

I swear I tried that yesterday and it didn't work, but I must have had the wrong root CA or not done it correctly, because as soon I tried it this time, it worked straight away!!

THANK YOU

[Jorge Canto]

Hello,

I am having the same problem with an SPA122, you say it is working now, question, at the end how did you set the certificate from your website? root and intermediate or root only?

Thank you

[Tim Harman]

Hi Jorge,

I only used the root certificate in the end - adding in an intermediate caused all sorts of problems (it didn't work)

Try that - post back if it doesn't work or you have issues, I spent a lot of time debugging this so can offer you pointers to help debug further if you're stuck.

[Jorge Canto]

Hello Tim,

Thanks so much for your response.

Well, I made it works importing only the root certificate, but then I rebooted the SPA and it is not working anymore, so, maybe there is something else, I read you "swore" to be using the right CA before and suddenly started working, have you tried rebooting your device and verify everything is going smoothly?

Thank you

[Tim Harman]

Hi Jorge, Yes, I've rebooted a number of times after installing the root CA and I'm not having any problems. I no longer even host the certificate on my website, and rebooting it still works. What does it show at the bottom of the Voice screen, does it show provisioned OK and what the root CA is?

[Dan Lukes]

Certificate, once loaded, is persistent until overwritten by another import or reset to factory default.

[Jorge Canto]

Hi Tim, I think something is broken because sometimes things work but sometimes (the most) they don't, the CA seems to be installed (that is what the device GUI shows). I've been able to be registered a couple of times but you just need to change the value of any setting ("Register expires" as an example) and then al crashes again.

Thank you

[Dan Lukes]

One certificate can be imported only (as far as I know). Intermediate certificates should be sent by server during TLS handshaking, so only root CA needs to be imported. If intermediate certificates are not provided by server, it will not work. In such case try import of intermediate certificate only (I never tried this configuration by self, it may or may not work).

[Jorge Canto]

Thanks Dan, importing the CA worked fine but only once, then I rebooted the device and it is not working anymore :(

[Dan Lukes]

Just "not working" is worthless issue description. Nothing to analyze, no way to help. Disclose details (syslog&debug messages of Voice Application at least), please. 

[Jorge Canto]

Hello, sorry, you're absolutely right, I haven't provided any detail, what I mean is I am not able to register to my provider, I've been able to register a few times but randomly, once registered I was able to make calls and all good but if I make any small change or reboot the system never get register again. I've been working very closely with the service provider technicians, if I use a softphone (Zpr) or any other ATA (GS) there is no problem at all with TLS. Regarding logs, I am sorry, but the SPA is set to logging MODE DEBUG and no information appears, the service provider technicians says they receive "unknown CA" error from my account.

Thank you