Application Hosting using Managment Interface (Thousand Eyes)

[Richard G.]

This is our first time deploying app hosting (Thousand Eyes).

I deployed the enterprise agent successfully on an ISR 4431 (through vManage) but am having issues with a 9300

and management interface.

C9300-24T 17.03.04 

Does the Docker interface and Thousand Eyes require their own IP address?

Below is my configuration...as I understand I do not need to configure Ap1/0/1 since I am using the managment interface

interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 10.2.15.10 255.255.255.0
negotiation auto
end

app-hosting appid thousand_eyes
app-vnic management guest-interface 0
guest-ipaddress 10.2.15.200 netmask 255.255.255.0
app-default-gateway 10.2.15.10 guest-interface 0
app-resource docker
run-opts 1 "-e TEAGENT_ACCOUNT_TOKEN=*******************"
run-opts 2 "--hostname "Cisco-9300-SW"
prepend-pkg-opts

name-server0 8.8.8.8

Cisco-9300-SW# sh app-hosting list
App id State
---------------------------------------------------------
thousand_eyes RUNNING

Cisco-9300-SWe#ping vrf Mgmt-vrf cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 72.163.4.185, timeout is 2 seconds:
!!!!!!!!!!

I am not sure what else I am missing but am currently getting this error

IOX_INST_NOTICE: Switch 1 R0/0: ioxman: IOX SERVICE te LOG: Error calling createAgent: Curl error - Couldn't resolve host name resolve host name Error calling createAgent: Cur

[Flavio Miranda]

Hi

 It seems to me that the createAgent is try do exit using the global vrf. When you do the test, you add "ping vrf Mgmt-vrf" but looks like the agent is not doing the same.

Try to not use vrf on the interface just to make sure.

[patrickroberts]

Hi did this get fixed ?

C9300-24U 17.06.03

i have the same issue below is my config

interface GigabitEthernet1/0/13
description Uplink MGMT
switchport access vlan 21
!
interface Vlan21
ip address 10.100.21.1 255.255.255.0
!
interface AppGigabitEthernet1/0/1
switchport trunk allowed vlan 21
switchport mode trunk

enable
app-hosting install appid xxxx package flash:thousandeyes-enterprise-agent-4.2.2.cisco.tar
configure terminal
app-hosting appid xxxx
app-vnic AppGigabitEthernet trunk
vlan 21 guest-interface 0
exit
name-server0 8.8.8.8
name-server1 8.8.4.4
app-resource docker
prepend-pkg-opts
run-opts 1 "-e TEAGENT_ACCOUNT_TOKEN=xxxxxxxx"
run-opts 2 "--hostname xxxx"
exit
exit
exit
write memory
app-hosting activate appid xxxx
app-hosting start appid xxxx

[adam.james]

Hi Richard. 

I too had the same problem.  We are doing a large roll out of TE in our environment.  I have found the fix for this, and it was partly down to my own error.  For TE to run correctly you first need to apply HTTP config.  If you use DNAc to install TE you will be a window pop up referenving to the HTTP configs.  As soon as I applied the HTTP config I could see in the TE dashboard that the devices were now joined.  Just as an FYI - I have had more problems with TE when trying to use DNAc as and the installation process.  Using the CLI is much easier and quicker.  Hope this helps. Thanks Adam

prompt# sh run | sec http
ip http server
ip http authentication local
ip http secure-server
ip http max-connections 16
ip http client source-interface vlan 911
!
** Additional configuration for switches with Cisco IOS XE release earlier than 17.3.3:
ip http secure-active-session-modules dnac
ip http session-module-list dnac NG_WEBUI
ip http active-session-modules none
!
** Additional configuration for switches with Cisco IOS XE 17.3.3 or later:
ip http secure-active-session-modules webui
ip http session-module-list webui NG_WEBUI
ip http session-module-list pki OPENRESTY_PKI
ip http active-session-modules pki
!

[Tyler Langston · Accepted solution]

Adding a response from our engineers on this:

Hi everyone - happy to add some clarification on this issue!  The error message indicates a problem resolving a hostname, the most likely reason is the agent (container) can’t reach the DNS server, hence the hostname resolution doesn’t happen.

I noticed the app-default-gateway in Richard G’s post is configured as the GigabitEthernet0/0 IP address. Instead, this should be the real gateway of the subnet. To validate reachability to the gateway and DNS server working properly you can get into the agent’s shell by running the commands:

#app-hosting connect appid <agent app id> session bash

verify the DNS server config

# cat /etc/resolv.conf

verify the routing table

# route -n

then validate the connectivity with the commands

# te-ping -M icmp <dns-server-ip-address>
# te-ping -M icmp <default-gateway-ip-address>

the customer can also verify the DNS queries work by running the commands

# dig sc1.thousandeyes.com @8.8.8.8
# curl -v sc1.thousandeyes.com

After validating the connectivity to the DNS server and the gateway, if you find problems, you can modify the app-default-gateway and retest.

If you are still running into issues after you’ve attempted the above please feel free to contact Support for additional assistance!

[ahmedaburaihan]

Hi @Tyler Langston very nice answer you have posted, but in my case there are no entries in the resolv.conf file. I am also unable to insert new entries inside the resolv.conf file because it is read-only. Any suggestion?

[Torbjørn]

Hi @ahmedaburaihan, 

Can you post all app-hosting related configuration? 

[ahmedaburaihan]

@Torbjørn Hi. Which specific config do u want? I installed TE Agents using DNA Center. Everything went smoothly, accept that when i check the Interface for IP, does not have any. Then i checked inside using shall, it does have dns server entries. and the permission to change the resolv.conf file is not with root user, it is with nouser nogroup, so with root user i cannot edit the resolv.conf file and create new entries. 

[syed-kamalahmad]

data1.agt.thousandeyes.com
2023-09-20 08:58:01.131 DEBUG [a67fc700] [te.probe.ProbeTaskExecutor.throughput] {} Resources: 0 tasks
2023-09-20 08:58:01.131 DEBUG [a6ffd700] [te.probe.ProbeTaskExecutor.realtime] {} Resources: 0 tasks
2023-09-20 08:58:01.131 DEBUG [acbc8700] [te.probe.ServerTaskExecutor] {} Resources: 0 tasks
2023-09-20 08:58:01.131 DEBUG [a77fe700] [te.probe.ProbeTaskExecutor.bandwidth] {} Resources: 0 tasks
2023-09-20 08:58:06.869 DEBUG [6a7fc700] [te.agent.AgentRelayStreamHandler] {} Establishing control channel connection to c1.thousandeyes.com:443
2023-09-20 08:58:06.879 DEBUG [63fff700] [te.agent.ResolverHandle] {} Name resolution of c1.thousandeyes.com completed after deadline (getaddrinfo() = -3)
2023-09-20 08:58:08.841 ERROR [ade93840] [te.agent] {} Error calling checkIn: Curl error: Could not resolve host: c1.thousandeyes.com
2023-09-20 08:58:08.842 DEBUG [ade93840] [te.agent] {} Commit thread sleeping for 30 seconds
2023-09-20 08:58:10.316 ERROR [97fff700] [te.agent.EndpointClient] {} Error posting results to https://data1.agt.thousandeyes.com: Could not resolve host: data1.agt.thousandeyes.com
2023-09-20 08:58:10.316 WARN [97fff700] [te.agent.DataSubmitter] {} Failed committing data to https://data1.agt.thousandeyes.com/load in 10009 ms. Data for 1 tasks will be resubmitted next time.
2023-09-20 08:58:16.870 DEBUG [6a7fc700] [te.agent.AgentRelayStreamHandler] {} Control channel disconnected after 10 seconds
2023-09-20 08:58:16.870 DEBUG [6a7fc700] [te.agent.AgentRelayStreamHandler] {} Reconnecting control channel in 30 seconds

We are also getting similar error however the most important thing to be noted here with same setup, Same configuraiton, Same IOS/Models of Cedge router where ENT agents are hosted working on 80 % location only 20 % sites reporting this issue.

[Tyler Langston]

Hi @syed-kamalahmad  - I asked our engineering folks about what you've reported and they had some input:

The message itself can mean a whole lot of things, the most likely scenarios are that the gateway or DNS server can't be reached, or their firewall is blocking the endpoints. The first two options are due to misconfigurations on the container/router or switch, and can be dependent on the version of IOS that they're using on the device. We'd recommend the following commands first, but if these don't resolve it we'd want them to open a support case so we can review in more detail.

# te-ping -M icmp <dns-server-ip-address>
# te-ping -M icmp <default-gateway-ip-address>
the customer can also verify the DNS queries work by running the commands
# dig sc1.thousandeyes.com @8.8.8.8
# curl -v sc1.thousandeyes.com
After validating the connectivity to the DNS server and the gateway, if you find problems, you can modify the app-default-gateway and retest.

If you've never opened a support case with us before, it's very simple! This article has more details if you need help. 

Thanks!

[syed-kamalahmad]

Thanks for your quick response @Tyler Langston  We have engaged CISCO TAC on multiple calls and even TE TAC Support but we are still struggling to find out the root cause.

Zscaler which seems to be the gateway to reach to internet for resolving these hostnames has same policy for all location but only few location showing this error and agents are offline. 

Few are intermittently going offline it means it worked for a while then appears offline, channel broken between ThousandEyes portal and Agent hosted Cedge routers.

Either its latency which vary all the time or some bug with Zscaler/Router

By the way we are using 8300, 8500 series routers with latest version on them.

Standard change for IOS upload into vManage. Software code:

vManage - 20.9.2.1

vBond – 20.9.2

vSmart – 20.9.2

cEdges – 17.9.2a for following models:

• C8200-1N-4T
• C8300-1N1S-4T2X​
• C8300-1N1S-6T
• C8300-2N2S-4T2X
• C8500-12X
• C8500-12X4QC
• C8500L-8S4X
• Cat8Kv (Azure Cloud deployment)

[Tyler Langston]

Sorry to hear this has been an ongoing issue for you @syed-kamalahmad! I'd strongly recommend opening a chat or support case with our engineers. They'll be able to review in detail, provide specific advice, and escalate for further assistance if needed.