- Introduction
- Pre-Requisites & Assumptions
- Automatic Update Components
- Troubleshooting Automatic-Update Problems on Ubuntu
- > Troubleshooting APT auto-updates with ThousandEyes
- > ThousandEyes repository configuration
- >> Checking ThousandEyes APT repository configuration
- >> Checking the ThousandEyes APT GPG key
- >> Verify APT Proxy configuration
- > Missing repository configuration
- > Missing GPG key
- > APT ThousandEyes repository reachability issues
- > APT ThousandEyes repository reachability issues with proxy
- > The agent configuration auto-updates
- Conclusion
Introduction
Welcome to a series of articles on the best way to troubleshoot ThousandEyes like a pro. Want to know what to check first, quick fixes to research or workarounds to consider? You’re in the right place!
ThousandEyes Enterprise Agents offer detailed, end-to-end visibility into application and network performance from within an enterprise environment. Once deployed, an agent keeps up to date by downloading and updating the latest required package from us.
Occasionally, however, something can interrupt this process, leading you to log in and see the yellow warning that an agent is outdated. This article covers what I check first when customers contact me reporting an outdated agent.
Pre-Requisites & Assumptions
This article assumes you have a familiarity with, and adequate permissions for, the following:
- Read Access for ThousandEyes agent settings
- Root Access for Linux distribution
Automatic Update Components
- ThousandEyes repository configuration
- ThousandEyes repository GPG key
- APT proxy configuration (when applicable)
- The agent configuration
Troubleshooting Automatic-Update Problems on Ubuntu
While I’m focusing on the ThousandEyes repository for this article, the same steps may be used to troubleshoot any repository that might have issues.
So what to try first?
> Troubleshooting APT auto-updates with ThousandEyes
1. Create an HTTP test to the destination. You can run an instant test or schedule a test for troubleshooting time.
Note: If you need a proxy for APT, add it to the advanced settings of the test (examples below).
Why this step?
The HTTP server layer will provide you with the HTTP response code and availability. If there are issues with the SSL certificate, they will be shown here with the certificate chain received from the server.
The Agent to Server layer will provide you with path visualization and forwarding loss. When an HTTP layer response is present, forwarding loss might be just an indication that the ICMP response is blocked. If there is an issue with the HTTP layer response, you could start the investigation on the last visible node on path visualization.
> ThousandEyes repository configuration
If everything looks good in the above, the next step is to check that the repository configuration is up to date and not missing anything critical. The ThousandEyes repository is made up of two parts, the repository configuration file and the GPG key.
>> Checking ThousandEyes APT repository configuration
The repository is configured in /etc/apt/sources.list.d/thousandeyes.list. The repo configuration in the file is shown in the output below.
Ubuntu 20 Focal:
deb https://apt.thousaneyes.com/ focal main
Ubuntu 22 Jammy:
deb https://apt.thousandeyes.com jammy main
>> Checking the ThousandEyes APT GPG key
The APT key is saved in /etc/apt/trusted. jpg. The command ‘apt-key list’ shows all installed APT keys.
The ThousandEyes public key is AA5F BA03 CE4E F309 B7E3 D4E1 C99A 15F5 BE71 8900.
$ sudo apt-key list
/etc/apt/trusted.gpg
--------------------
pub rsa2048 2012-08-13 [SC]
AA5F BA03 CE4E F309 B7E3 D4E1 C99A 15F5 BE71 8900
uid [ unknown] ThousandEyes <builduser@ThousandEyes.com>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
790B C727 7767 219C 42C8 6F93 3B4F E6AC C0B2 1F32
uid [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
8439 38DF 228D 22F7 B374 2BC0 D94A A3F0 EFE2 1092
uid [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub rsa4096 2018-09-17 [SC]
F6EC B376 2474 EDA9 D21B 7022 8719 20D1 991B C93C
uid [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
>> Verify APT Proxy configuration
APT proxy configuration should be in /etc/apt/apt.conf.d/. If you need to configure a proxy you can create a file with a name such as 95proxies.
$ ll /etc/apt/apt.conf.d/ total 68 drwxr-xr-x 2 root root 4096 Oct 17 13:24 ./ drwxr-xr-x 7 root root 4096 Oct 15 13:55 ../ -rw-r--r-- 1 root root 92 Apr 9 2020 01-vendor-ubuntu -rw-r--r-- 1 root root 630 Apr 9 2020 01autoremove -rw-r--r-- 1 root root 56 Jun 13 2023 10RecommendsSuggests -rw-r--r-- 1 root root 129 Apr 25 2022 10periodic -rw-r--r-- 1 root root 108 Apr 25 2022 15update-stamp -rw-r--r-- 1 root root 604 Jul 12 2022 20apt-esm-hook.conf -rw-r--r-- 1 root root 85 Apr 25 2022 20archive -rw-r--r-- 1 root root 80 Apr 25 2022 20auto-upgrades -rw-r--r-- 1 root root 1040 Sep 23 2020 20packagekit -rw-r--r-- 1 root root 625 Feb 10 2022 50command-not-found -rw-r--r-- 1 root root 5459 May 19 2022 50unattended-upgrades -rw-r--r-- 1 root root 182 Aug 3 2019 70debconf -rw------- 1 root root 45 Oct 17 13:24 95proxies -rw-r--r-- 1 root root 305 Apr 25 2022 99update-notifier
The content of the file should be:
Acquire::http::proxy "http://username:password@Proxy_fqdn:port";.
Example:
Acquire::http::proxy "http://username:password@my.proxy.cisco.com:80";
> Missing repository configuration
Another common issue I see is a missing repository configuration. While this doesn’t typically happen since agents are designed to identify and add missing repos, some customers have automations that ensure a repository list is a default Linux distribution list.
When this happens, it can overwrite the agent’s default resulting in the outdated notification.
How to identify:
Check to see if ‘https://apt.thousandeyes.com’ is missing in the following output:
sudo apt update
If it is, the repository configuration is missing.
Example:
$ sudo apt-get update Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease Reading package lists... DoneNotice in the above returned information there is no mention of apt.thousandeyes.com. This means our next step is to re-add the repository.
How to resolve:
To add a repository you can run the command:
$ sudo apt-add-repository https://apt.thousandeyes.com [sudo] password for thousandeyes: Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease Hit:5 https://apt.thousandeyes.com focal InRelease Reading package lists... Done
The output above shows that the repository was added, the 'apt-get update' was run, and repository metadata was downloaded.
The agent will add repository configuration to the file:
/etc/apt/sources.list.d/archive_uri-https_apt_thousandeyes_com-jammy.list.
If you still have trouble loading the repository, check file permissions and make sure that read (r) permissions are present.
ls -la /etc/apt/sources.list.d/ total 12 drwxr-xr-x 2 root root 4096 Oct 29 13:21 . drwxr-xr-x 8 root root 4096 Oct 28 2022 .. -rw-r--r-- 1 root root 94 Oct 29 13:21 archive_uri-https_apt_thousandeyes_com-jammy.list
> Missing GPG key
When the ThousandEyes repository GPG key is missing you may see errors in /var/log/te-agent.log logs, such as:
2024-10-30 08:42:53.857 DEBUG [f8ff9640] [te.agent.main] {} (stdout) Err:4 https://apt.thousandeyes.com jammy InRelease
2024-10-30 08:42:53.859 DEBUG [f8ff9640] [te.agent.main] {} (stdout) The following signatures couldn't be verified because the public key is not available: NO_PUBKEY C99A15F5BE718900
How to identify:
To validate that the GPG key is missing we can run 'sudo apt-key list'. Compare your output with the ThousandEyes public key output `AA5F BA03 CE4E F309 B7E3 D4E1 C99A 15F5 BE71 8900`.
How to resolve:
Import the key run the command:
traceroute $ sudo wget -q https://apt.thousandeyes.com/thousandeyes-apt-key.pub -O- | sudo apt-key add - OK
You can run the 'sudo apt-key list' to validate that the key is imported successfully.
> APT ThousandEyes repository reachability issues
Sometimes it’s the network connection itself that’s the problem. While I can’t resolve those issues, I’ve put together a few common places I see customers have issues and suggest those as places to start.
How to identify:
When the ThousandEyes repository is not reachable you may see errors in /var/log/te-agent.log logs, such as:
2024-10-29 10:32:35.961 DEBUG [737fe640] [te.agent.main] {} (stdout) Err:6 https://apt.thousandeyes.com jammy InRelease
2024-10-29 10:32:35.963 DEBUG [737fe640] [te.agent.main] {} (stdout) Could not connect to apt.thousandeyes.com:443 (13.227.219.64). - connect (111: Connection refused) Could not connect to apt.thousandeyes.com:443 (13.227.219.47). - connect (111: Connection refused) Could not connect to apt.thousandeyes.com:443 (13.227.219.74). - connect (111: Connection refused) Could not connect to apt.thousandeyes.com:443 (13.227.219.89). - connect (111: Connection refused) Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:5600:9:9f46:bbc0:93a1). - connect (113: No route to host) Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:9000:9:9f46:bbc0:93a1). - connect (113: No route to host) Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:a200:9:9f46:bbc0:93a1). - connect (113: No route to host) Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:5800:9:9f46:bbc0:93a1), connection timed out Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:3800:9:9f46:bbc0:93a1), connection timed out Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:1000:9:9f46:bbc0:93a1), connection timed out Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:5a00:9:9f46:bbc0:93a1), connection timed out Could not connect to apt.thousandeyes.com:443 (2600:9000:21c7:8600:9:9f46:bbc0:93a1), connection timed out
To check connectivity issues, use ping to get general connectivity information. However, follow up with more specific protocol-related (TCP port 443 by default or TCP + Proxy port) commands. Curl command and traceroute can be useful here.
ping apt.thousandeyes.com curl –vvv https://apt.thousandeyes.com #IF APT PROXY IS USED curl -x <PROXY IP ADDRESS or HOSTNAME>:<PROXY PORT> -U <USERNAME>:<PASSWORD> https://apt.thousandeyes.com sudo traceroute -T -p 443 apt.thousandeyes.com
Ping is a good place to start validating network reachability.
$ ping apt.thousandeyes.com PING d1zh1r0twh1jov.cloudfront.net (13.227.219.89) 56(84) bytes of data. 64 bytes from server-13-227-219-89.ams54.r.cloudfront.net (13.227.219.89): icmp_seq=1 ttl=244 time=34.0 ms 64 bytes from server-13-227-219-89.ams54.r.cloudfront.net (13.227.219.89): icmp_seq=2 ttl=244 time=32.8 ms 64 bytes from server-13-227-219-89.ams54.r.cloudfront.net (13.227.219.89): icmp_seq=3 ttl=244 time=32.6 ms ^C --- d1zh1r0twh1jov.cloudfront.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 32.558/33.125/33.991/0.621 ms
Keep in mind that in some cases ICMP will be blocked by network policy.
To get into protocol/port-specific troubleshooting and confirm HTTPS availability, we can use the curl command.
$ curl -vvv https://apt.thousandeyes.com * Trying 13.227.219.47:443... * connect to 13.227.219.47 port 443 failed: Connection refused * Trying 13.227.219.74:443... * connect to 13.227.219.74 port 443 failed: Connection refused * Trying 13.227.219.89:443... * connect to 13.227.219.89 port 443 failed: Connection refused * Trying 13.227.219.64:443... * connect to 13.227.219.64 port 443 failed: Connection refused * Trying 2600:9000:21c7:aa00:9:9f46:bbc0:93a1:443...
In this scenario, the firewall is resetting the connection and we get 'Connection refused'. It is possible that the firewall will silently drop packets and in those scenarios, you would see a timeout.
To narrow down the issue, we can use a traceroute.
$sudo traceroute -T -p 443 apt.thousandeyes.com 1 192.168.200.1 1.363 ms 1.365 ms 1.356 ms 2 * * * 3 * * *The last device in output or next hop is where we would recommend starting troubleshooting.
> APT ThousandEyes repository reachability issues with proxy
In Ubuntu, the APT proxy is not the same as the HTTP proxy and proxy configuration needs to be added to the APT.
Apt configuration is stored in /etc/apt/apt.conf.d/.
Proxy configuration is in file 'xxproxies', in the below example '95proxies'.
ls -la /etc/apt/apt.conf.d/ total 80 drwxr-xr-x 2 root root 4096 Feb 3 08:27 . drwxr-xr-x 9 root root 4096 Aug 30 16:19 .. -rw-r--r-- 1 root root 630 Apr 8 2022 01autoremove -rw-r--r-- 1 root root 92 Apr 8 2022 01-vendor-ubuntu -rw-r--r-- 1 root root 129 Feb 28 2023 10periodic -rw-r--r-- 1 root root 56 Aug 30 16:29 10RecommendsSuggests -rw-r--r-- 1 root root 108 Feb 28 2023 15update-stamp -rw-r--r-- 1 root root 311 Nov 7 2023 20apt-esm-hook.conf -rw-r--r-- 1 root root 85 Feb 28 2023 20archive -rw-r--r-- 1 root root 80 Feb 19 2021 20auto-upgrades -rw-r--r-- 1 root root 1040 Feb 17 2022 20packagekit -rw-r--r-- 1 root root 625 Dec 8 2021 50command-not-found -rw-r--r-- 1 root root 6155 Jan 14 2022 50unattended-upgrades -rw-r--r-- 1 root root 182 Feb 20 2022 70debconf -rw-r--r-- 1 root root 195 Jul 11 2024 90_te-tcpdump -rw------- 1 root root 47 Feb 3 08:27 95proxies -rw-r--r-- 1 root root 338 May 16 2022 99needrestart -rw-r--r-- 1 root root 305 Feb 28 2023 99update-notifier
HTTP proxy example with 'my.proxy.net' host which uses port '8080'.
$cat /etc/apt/apt.conf.d/95proxies Acquire::http::proxy "http://my.proxy.net:8080";
Now that we know the proxy and port we can try to curl the proxy.
$curl -x my.proxy.net:8080 https://apt.thousandeyes.com
When proxy is used traceroute should be directed to proxy on configured port.
$sudo traceroute -T -p 8080 my.proxy.netThe Thousandeyes agent installs dependencies from default Ubuntu repositories. Same need to be reachable and same troubleshooting can be applied.
> The agent configuration auto-updates
Sometimes an agent is not set up to automatically update. This could be for multiple reasons, from organizational policy, a planned part of your deployment, or as an oversight.
How to identify:
The agent configuration is in /etc/te-agent.cfg and by default does not have an auto-update line. The line is added to stop agent auto-updates.
When the configuration file is set to include 'auto-updates=0' the agent will not perform auto updates. You can change this value to 1 or remove the line to enable the auto updates.
How to resolve:
The agent configuration file is loaded on service start so you will need to restart the service for the change to take effect.
$cat /etc/te-agent.cfg log-path=/var/log log-file-size=10 log-level=DEBUG num-log-files=13 account-token=1234xxx proxy-type=DIRECT proxy-location= proxy-user= proxy-pass= proxy-auth-type= proxy-bypass-list= kdc-user= kdc-pass= kdc-realm= kdc-host= kdc-port=0 kerberos-whitelist= kerberos-rdns=1 crash-reports=1 services-allow-ipv6=0 auto-updates=0 <<< TO ENABLE AUTO UPDATES ENSURE THIS IS CHANGED TO auto-updates=1 #EXIT TEXT EDITOR AND RESTART TE-AGENT SERVICE $sudo systemctl restart te-agent
Conclusion
That brings us to the end of troubleshooting Enterprise Agents with automatic-update issues on Ubuntu.