on - edited on by
Tyler Langston
- Introduction
- Pre-requisites & Assumptions
- Troubleshooting SSL Certificate Errors: Types, Diagnosis, and Solutions
- Error 1: "SSL certificate verify result: unable to get local issuer certificate"
- Error 2: "SSL certificate problem: self-signed certificate"
- Error 3: "SSL certificate problem: certificate has expired"
- Error 4: "SSL: no alternative certificate subject name matches target host name"
- Conclusion & Resources
Introduction
Welcome to another article in our series about ThousandEyes!
This article will guide you through identifying, diagnosing, and resolving common SSL certificate problems when running ThousandEyes Enterprise Agents on Catalyst 9000 Series switches.
SSL Certificate Errors on Cat 9k – What are they?
Before we get further into it, let’s cover certificates and certificate chains; what they are and why they’re important.
SSL/TLS certificates are digital documents that authenticate the identity of websites and encrypt data transmitted between clients and servers. ThousandEyes Enterprise Agents must validate these certificates to ensure secure communication.
What are Certificate Chains? How Do They Work?
A certificate chain is a series of certificates that provide verification of authenticity and trust at each step. Each certificate is ‘signed’ by the next certificate’s issuer, ensuring an unbroken chain of trust back to the root Certificate Authority (CA). This process is essential for secure communications and preventing those ‘untrusted connection’ warnings that can derail your day.
A typical certificate chain consists of:
- Root CA Certificate - Self-signed and pre-installed in client certificate stores.
- Intermediate Certificate(s) - Issued and signed by the Root CA. There can be multiple intermediate certificates within a chain.
- Server Certificate - The SSL certificate for your website (e.g., www.example.com).
When an Enterprise Agent connects to an HTTPS endpoint:
- The server sends its certificate (and any intermediate certificates).
- The agent verifies each certificate's digital signature using the issuing certificate's public key.
- This process continues until reaching a trusted root CA certificate.
- If any link breaks, validation fails. If any link in the chain is broken or cannot be trusted, the validation fails, resulting in an SSL error being reported in the test results.
Pre-requisites & Assumptions
With our basics covered, let’s review some background you’ll need. This article assumes you have the following knowledge, tools and access:
- Knowledge of, and administrative access to, the ThousandEyes product.
- Familiarity with OpenSSL and using it to get detailed certificate information.
- Familiarity with ThousandEyes Utilities Suite, including how to install the te-agent-utils package.
- Access to your agent’s container within the Catalyst 9000 CLI.
- Awareness of your Enterprise Agent’s operation. If your agent is operating behind a proxy server, you must include the following option in your te-curl and openssl commands for accurate diagnosis.
-x <proxy-ip>:<proxy-port>
Tip: To learn how to verify your proxy settings, please refer to our guide: Verifying Proxy Configurations for ThousandEyes Enterprise Agents.
Important Note: ThousandEyes Docker-based Enterprise Agents on Cisco devices using Cisco Application Hosting Framework (CAF) version 5.1.2 are built on Alpine Linux. The commands and procedures in this article are specifically tailored for that environment.
Note: Throughout this guide, we utilize badssl.com for demonstration purposes. This website is specifically designed to simulate various SSL certificate errors, allowing for consistent and clear reproduction of the issues discussed.
With all that said, let’s get into it!
Troubleshooting SSL Certificate Errors: Types, Diagnosis, and Solutions
To troubleshoot and diagnose SSL certificate issues on Alpine Linux-based Enterprise Agents, we will primarily utilize te-agent-utils (which includes te-curl) for network diagnostics. We will also be using OpenSSL to perform in-depth analysis of certificate chains and extract certificate details (such as issuer, subject, and validity).
Before proceeding with any diagnosis, ensure you have access to your agent's container, and that the necessary tools are installed:
Step 1: Access the agent container from the Cat 9k CLI
app-hosting connect appid <your-agent-name> session /bin/bash
Step 2: Install diagnostic utilities within the agent container
apk update apk add openssl apk add te-agent-utils
Important: To minimize potential risks, we strongly recommend you remove OpenSSL from the agent after troubleshooting is complete.
To remove OpenSSL:
apk del openssl
Error 1: "SSL certificate verify result: unable to get local issuer certificate"
What it means:
The Enterprise Agent received the server's SSL certificate but cannot build a complete chain of trust back to a root certificate in its trusted certificate store. This is the most common SSL error.
When this occurs:
- The target web server is misconfigured and not sending all required intermediate certificates.
- The target server's certificate is issued by a private Public Key Infrastructure (PKI), an organizational CA whose root certificate is not in the agent's trust store.
- The agent's traffic to the target is intercepted by an SSL-decrypting proxy, and the proxy's signing certificate is not trusted by the agent.
How to diagnose:
Step 1: Test the target URL and identify the error
te-curl -v https://your-failing-target.com:port
If using a proxy (see Pre-Reqs 4):
te-curl -v https://your-failing-target.com:port -x <proxy-ip>:<proxy-port>
Example:
Step 2: Display the Certificate Chain:
Building on our understanding from Step 1 that a certificate validation error is occurring, the next step is to review the certificate chain being presented by the server. This will help identify the specific point of failure, such as a missing intermediate certificate or an untrusted issuer. We can do this in a few different ways:
Option A: Using OpenSSL
For detailed certificate chain analysis, OpenSSL is a great way to analyze the server's certificate.
openssl s_client -connect your-target.com:port -servername your-target.com
If using a proxy:
openssl s_client -connect your-target.com:port -servername your-target.com -proxy <proxy-ip>:<proxy-port>
Note: When using openssl s_client, if there are more than two certificates in the chain, it may not display the 3rd+ certificate in PEM format directly in the initial output.
Key information:
From the output of information from this command, check for the following indicators of an incomplete chain:
- verify error:num=20:unable to get local issuer certificate: This error explicitly states that the agent cannot find the issuer of the server's certificate in its local trust store.
- Verify return code: 21 (unable to verify the first certificate): This confirms the failure to establish trust for the initial certificate presented by the server.
- Certificate chain section: In the openssl output, the numbered lines start with the server certificate (#0) followed by the intermediate (#1) and the root (#2). The s: indicates the certificate subject, and i: indicates the issuing certificate's subject.
Option B: Using External Online Tools
For publicly accessible sites, you can utilize online SSL server test tools. These can provide a definitive view of the complete certificate chain as presented by the server. These tools often provide detailed chain validation and identify missing intermediates.
Some of the online tools we’ve used are:
- Qualys SSL Labs Server Test: https://www.ssllabs.com/ssltest/
- DigiCert SSL Certificate Checker: https://www.digicert.com/help/
Option C: Extracting from a Web Browser
Most browsers allow you to inspect certificate details and view the complete certificate chain. This method is useful for both public and internal sites accessible from your workstation. Just browse to the target URL using a web browser and click the padlock icon in the address bar!
With our errors identified, diagnostic information gathered and certificate chain in hand, review the three scenarios below to determine which one is present and how to solve.
| Scenario | Indicators | Solution |
| A: Internal PKI |
|
Add your organization's root CA certificate to the agent |
| B: SSL Decrypting Proxy |
|
Add proxy signing certificate to the agent Installing CA Certificates on Enterprise Agents. |
| C: Server Misconfiguration |
|
Preferred: Fix server to send complete certificate chain. |
Error 2: "SSL certificate problem: self-signed certificate"
What it means:
The target server uses a certificate that was signed by itself rather than a recognized Certificate Authority. The certificate's Issuer and Subject are identical.
When this occurs:
- Internal/development servers
- Appliances with default certificates
How to diagnose:
Step 1: Test the connection and identify the error
te-curl -v https://your-failing-target.com:port
If using a proxy:
te-curl -v https://your-failing-target.com:port -x <proxy-ip>:<proxy-port>
Example:
Step 2: Verify the certificate with OpenSSL:
To confirm the certificate is indeed self-signed, you can extract and examine its details:
echo quit | openssl s_client -showcerts -servername your-target.com \ -connect your-target.com:port > self-signed-cert.pem openssl x509 -text -noout -in self-signed-cert.pem | grep -A2 "Issuer\|Subject"
If using a proxy:
echo quit | openssl s_client -showcerts -servername your-target.com \ -connect your-target.com:port -proxy <proxy-ip>:<proxy-port> > self-signed-cert.pem openssl x509 -text -noout -in self-signed-cert.pem | grep -A2 "Issuer\|Subject"
Example:
Notice that the Issuer and Subject fields are identical, confirming that the certificate is self-signed.
Solution:
To resolve this, you need to add the self-signed certificate to the Enterprise Agent's trusted certificate store.
- Obtain the self-signed certificate directly from the source:
- From a browser: Browse to the target URL, click the padlock icon, and export the certificate.
- From your proxy team: If the self-signed certificate is presented by a proxy, request the certificate from your proxy administration team.
- Copy the .pem file to your Cat 9k's bootflash.
- Install to agent: Follow CA certificate installation guide
Error 3: "SSL certificate problem: certificate has expired"
What it means:
The certificate's validity period has ended. Certificates have a "Not After" date, and the current date is past that date.
How to diagnose:
Step 1: Test the connection and identify the error
te-curl -v https://your-failing-target.com:port
If using a proxy:
te-curl -v https://your-failing-target.com:port -x <proxy-ip>:<proxy-port>
Example:
Step 2: Check expiration with OpenSSL:
To confirm the expiration date, use OpenSSL to extract the dates from the certificate.
echo | openssl s_client -connect your-target.com:port -servername your-target.com 2>/dev/null | openssl x509 -noout -dates
If using a proxy:
echo | openssl s_client -connect your-target.com:port -servername your-target.com -proxy <proxy-ip>:<proxy-port> 2>/dev/null | openssl x509 -noout -dates
Example:
Now that you've confirmed the certificate is expired, the next step depends on who owns the server. You need to determine if the target is a service managed by your organization or an external, third-party service.
- Your Server: An internal application, a company website, or any infrastructure managed by your IT department.
- Third-Party Server: An external service like a SaaS provider, a partner's API, or any website not controlled by your organization.
Once you know who is responsible for the server, select the appropriate action from the table below.
Solution:
| If it's... |
Action |
|
Your server |
Renew the certificate with your CA and install the new one on the server |
|
Third-party server |
Contact the site administrator to renew their certificate |
|
Temporary workaround |
Disable certificate verification in test settings (not recommended for production) |
Error 4: "SSL: no alternative certificate subject name matches target host name"
What it means:
The hostname in your URL doesn't match the Common Name (CN) or Subject Alternative Names (SANs) in the certificate.
How to diagnose:
Step 1: Test the connection and identify the error
te-curl -v https://your-failing-target.com:port
If using a proxy:
te-curl -v https://your-failing-target.com:port -x <proxy-ip>:<proxy-port>
Example:
Step 2: Check the certificate's Subject and SANs with OpenSSL
openssl s_client -connect your-target.com:port -servername your-target.com 2>/dev/null | openssl x509 -noout -text | grep -A5 "Subject:\|Subject Alternative Name"
If using a proxy:
openssl s_client -connect your-target.com:port -servername your-target.com -proxy <proxy-ip>:<proxy-port> 2>/dev/null | openssl x509 -noout -text | grep -A5 "Subject:\|Subject Alternative Name"
Example:
Solutions:
|
Scenario |
Action |
|
Using wrong URL |
Update your ThousandEyes test to use the correct hostname from the certificate |
|
Certificate missing SAN |
Server administrator needs to reissue certificate with all required hostnames in SANs |
|
Load balancer/proxy issue |
Check that load balancer is sending correct SNI (Server Name Indication) |
Conclusion & Resources
We hope you’ve found this guide helpful in troubleshooting some of the most common certificate issues & SSL errors we see on Catalyst9000 Series Switches. We’ve included some additional resources below that we find helpful.
Additional Resources:
ThousandEyes Documentation
SSL/TLS Testing Tools
- Qualys SSL Labs - Comprehensive SSL analysis for public servers
- DigiCert SSL Checker - SSL installation verification
ThousandEyes SSL Certificate Checker:
ThousandEyes provides built-in SSL certificate monitoring. In your test results, click Response Details → TLS Session tab to see:
- Certificate expiration date
- Certificate chain
- Issuer information
Additional Support
For assistance with SSL certificate issues beyond what's covered in this article:
- ThousandEyes Support: Use our in-platform chat for immediate help. Average response time 10 seconds or less!
- Email: support@thousandeyes.com.
- ThousandEyes Discussion Board: Our community page is a good place to get your general knowledge questions answered with help from other ThousandEyes users!
