ThousandEyes Enterprise Agent Proxy Certificate

[dominikhug]

Hi

I've recently installed ThousandEyes Enterprise Agents on some Cat9300-48UXM running IOS-XE 17.9.5 and 17.12.4. Since company proxy is using internal CA certificates, I had to install those certificates in docker container as described here: https://docs.thousandeyes.com/product-documentation/global-vantage-points/enterprise-agents/troubleshooting/installing-ca-certificates-on-enterprise-agents#installing-on-cisco-docker-devices

This was working fine until ThousandEyes updated itself which led to the removal of the installed certificates and loss of connectivity through proxy to ThousandEyes portal. After I reinstalled the certificates, connectivity was restored.

Is there a way to:
- Disable auto-update to prevent certificate removal?
- Disable removal of manually installed certificates?
- Automatically install certificates during auto-update?

Or is there an other solution for this suboptimal behaviour?

Thank you!

[Tyler Langston]

Hi @dominikhug - welcome to the support community! I'm working on getting some input from our experts, but would like to ask you for a bit more detail on one thing first: when you say ThousandEyes updated itself, can you be a bit more specific?

The normal updates of the ThousandEyes service within the container should not remove the installed CA certificates, so it's really unusual to hear that it wiped them. 

Do you have any more specifics about what updated, when, and if there were any other details included?

[dominikhug]

Hi @Tyler Langston 

I currently have ThousandEyes Enterprise Agent installed on 4 Cat9300. All of them were suddenly offline in ThousandEyes dashboard putting following message in switchlog:

108016: Aug 26 07:50:37.499: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE ThousandEyes_Enterprise_Agent LOG: Error calling getController: Curl error: SSL certificate problem: unable to get local issuer certificate
108017: Aug 26 07:50:41.698: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE ThousandEyes_Enterprise_Agent LOG: Error calling checkIn: Curl error: SSL certificate problem: unable to get local issuer certificate
108018: Aug 26 07:51:11.338: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE ThousandEyes_Enterprise_Agent LOG: Error calling getController: Curl error: SSL certificate problem: unable to get local issuer certificate
108019: Aug 26 07:51:14.790: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE ThousandEyes_Enterprise_Agent LOG: Error calling checkIn: Curl error: SSL certificate problem: unable to get local issuer certificate
108020: Aug 26 07:51:45.129: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE ThousandEyes_Enterprise_Agent LOG: Error calling getController: Curl error: SSL certificate problem: unable to get local issuer certificate
108021: Aug 26 07:51:48.636: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE ThousandEyes_Enterprise_Agent LOG: Error calling checkIn: Curl error: SSL certificate problem: unable to get local issuer certificate	

After I've reinstalled SSL certificate again on one of those switches, it was able to reconnect to dashboard. I can now see this reconnected switch has a different agent version (1.195.0) than those which are unable to reconnect (1.191.0). So I now suspect, agent has been updated automatically and somehow deleted the manually installed certificates. The certificates even disappeared from /usr/share/ca-certificates/-directory. Because of this change in agent version, I currently suspect this update was causing my issues.

[Tyler Langston]

Hi @dominikhug - thank you for this information, it's exactly what I needed! I got these details to my experts and they had some thoughts for you:

The normal update of the ThousandEyes service within the container should not remove the installed CA certificates. If the agent container image was automatically updated by some external process (process outside of the agent container), like as part of Catalyst Center or vManage, then this could have removed the CA certificate and it would need to be re-added after the container was reinstalled. Currently CA certificates have to be installed on the agent container after installation, we cannot pass them to the container during install. We note this in our docs here.

An "update" of the container image by an external source is considered a reinstall because an updated image of the container is downloaded, the old container is destroyed, and a new container is created with the new image.

Since they still see the old agent in the platform, that leads us to believe that the agent has been reinstalled, thus creating a new agent. Reinstalling the agent should deploy a new updated agent, thus "removing" any custom changes like a CA cert (it is not "removing" anything, it is installing a new agent without the custom modifications).

This does not mean that the agent updating itself is what removed the CA certs. We'd need to investigate with them why the original containers stopped working, which prompted the reinstall of the new containers/re-adding the CA certificate to get the container online.

Please request the member contact our Support team so we can help them uncover more of what happened.

If you need help contacting support, we have a quick walkthrough of that process here!