Re: SPA112 client certificate validation with Apache

Steven Wheeler · ‎04-01-2021

I am attempting to set up client certificate validation for our SPA112 devices using the CA certificate bundle available here. The ATA191 that I am testing with validates just fine, but the SPA112 is always rejected. I'm wondering if anyone else has run into similar issues before and knows what I'm missing. I should also mention that I am using a Cisco signed certificate on the web server. I have attached the CA certificate bundle I am using. And this is my relevant Apache virtual host.

<VirtualHost *:443>
  ServerName devcisco.example.com
  ServerAdmin support@example.com

  ## Vhost docroot
  DocumentRoot "/var/www/"

  ## Directories, there should at least be a declaration for /var/www/
  <Directory "/var/www/">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/onboard-cisco.log"
  LogLevel info
  ServerSignature Off
  CustomLog "/var/log/httpd/onboard-cisco.log" "%h %l %u %t MAC:%{SSL_CLIENT_S_DN_CN}x '%r' %>s %b" 

  ## Proxy rules
  ProxyRequests Off
  ProxyPreserveHost Off
  ProxyPass / http://backend.example.com:8080/onboard/cisco/
  ProxyPassReverse / http://backend.example.com:8080/onboard/cisco/
  ## Rewrite rules
  RewriteEngine On

  ## Server aliases
  ServerAlias devcisco.example.com

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/tls/certs/devcisco.example.com.crt"
  SSLCertificateKeyFile   "/etc/pki/tls/certs/devcisco.example.com.key"
  SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
  SSLCipherSuite          ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
  SSLHonorCipherOrder     on
  SSLOptions +StdEnvVars

  # SSL Proxy directives
  SSLProxyEngine On

  ## Custom fragment
  SSLCACertificateFile /etc/pki/tls/certs/Cisco-Root-CA-Combined.crt
  SSLVerifyClient require
</VirtualHost>

Dan Lukes · ‎04-01-2021

Check Apache LOGs to identify why it rejects the connection.

We are using client certificate for authentication and we have no issues with it. But we are not trying to proxy the communication. According my knowledge of SSL, you can't pass requests thru transparent proxy if client certificates are used.

Steven Wheeler · ‎04-02-2021

Dan, thanks for the reply. Sorry, I had meant to include the Apache log output but forgot about them. They are not terribly helpful.

[Fri Apr 02 09:00:33.839929 2021] [ssl:info] [pid 337] [client xxx.xxx.xxx.xxx:53526] AH01998: Connection closed to child 98 with abortive shutdown (server devcisco.example.com:443)

Based on my experience with Polycom Mutual TLS authentication that's the error message Apache gives when it fails to verify the client certificate. I have also tried turning on debug logging in Apache but it provides no further details. If I disable client verification then the SPA112 is able to connect and download the configuration file. This leads me to suspect that my CA root certificates are wrong but I've tried every Cisco root I could find with no success.

The logs from the ATA are also not very helpful.

create ssl connection failed

I should also mention that I am running the latest firmware (1.4.1 SR5) on my SPA112.

Dan Lukes · ‎04-02-2021

Connection closed to child 98 with abortive shutdown

create ssl connection failed

Then tcpdump/wireshark may be your friend. Catch the SSL stream. It may disclose us what parts of SSL handshake has been successfully completed and which one has been followed by "abortive shutdown". Such abortive shutdown may contain error code.

Or - configure client verification optional then print client certificate details from the script serving the configuration file (or create another one for the purpose). It will disclose you the verification result. Moreover, it will disclose you what CA has issued the client certificate used. According my documentation, it's either

/C=US/ST=California/L=San Jose/O=Cisco Small Business/OU=Cisco Small Business Certificate Authority/CN=Cisco Small Business Client Root Authority 1/emailAddress=ciscosb-certadmin@cisco.com

or

/C=US/ST=California/L=San Jose/O=Cisco Small Business/OU=Cisco Small Business Certificate Authority/CN=Cisco Small Business Client Root Authority 2/emailAddress=ciscosb-certadmin@cisco.com

It depends on manufacturing date of SPA112 ...

If you are using PHP you may use following code fragment:

        $msg = "Provisioning request";
        $msg.= " FROM ".$_SERVER["REMOTE_HOST"];
        $msg.= " IP ".$_SERVER["REMOTE_ADDR"];
        $msg.= " PORT ".$_SERVER["REMOTE_PORT"];
        $msg.= " MAC ".$MAC;
        $msg.= " AGENT ".$_SERVER["HTTP_USER_AGENT"];
        $msg.= " HTTPS ".$_SERVER["HTTPS"];
        $msg.= " SSL_CLIENT_S_DN_L ".$_SERVER["SSL_CLIENT_S_DN_L"];
        $msg.= " SSL_CLIENT_S_DN_ST ".$_SERVER["SSL_CLIENT_S_DN_ST"];
        $msg.= " SSL_CLIENT_S_DN_CN ".$_SERVER["SSL_CLIENT_S_DN_CN"];
        $msg.= " SSL_CLIENT_S_DN ".$_SERVER["SSL_CLIENT_S_DN"];
        $msg.= " SSL_CLIENT_V_START ".$_SERVER["SSL_CLIENT_V_START"];
        $msg.= " SSL_CLIENT_V_END ".$_SERVER["SSL_CLIENT_V_END"];
        $msg.= " SSL_CLIENT_M_SERIAL ".$_SERVER["SSL_CLIENT_M_SERIAL"];
        $msg.= " SSL_CLIENT_I_DN ".$_SERVER["SSL_CLIENT_I_DN"];
        $msg.= " SSL_CLIENT_VERIFY ".$_SERVER["SSL_CLIENT_VERIFY"];
        trigger_error($msg, E_USER_NOTICE);

The resulting message will be printed in Apache error log.

my CA root certificates are wrong
...
... Cisco-Root-CA-Combined

I suspect you are true. Cisco-Root_CA-Combined sounds like filename of the file related to the provisioning SERVER certificate. But clients certificates embedded inside of SPA112 are issued by different authority.

See