Integration Guide: ThousandEyes & Splunk Observability Cloud via OTel

migcerva · ‎07-31-2024

Integrating ThousandEyes with Splunk Observability Cloud through OpenTelemetry can enhance your monitoring capabilities by providing real-time visibility into network performance and application delivery.

In this guide, we will walk you through the detailed process of setting up the OpenTelemetry integration, covering both Splunk configuration steps and ThousandEyes configurations. You will learn how to:

Configure Splunk Observability Cloud to receive and process telemetry data.
Set up ThousandEyes to send relevant network performance data through OpenTelemetry.

Step 1: Create an Access Token in Splunk

To begin, you need to create an access token in your Splunk Observability Cloud Instance. This token will be used to authenticate and stream data to Splunk.

Access the Main Menu: The main menu is located on the left side of your screen. If it's hidden, click to expand it.
Navigate to Access Tokens and click New Token:

Tip: If your organization has a long list of access tokens, you might need to scroll down to find this button.

Note: Ensure the “Ingest Token” permission box is checked, and that the “power” role is selected in the API Token with roles field.lll. Save the token string.

lll. Then, get the string details of the token previously created:

Go to Settings > Access Tokens.
Expand the recently created token and click on Show Token.
Make a note of this token as it will be needed later in the process:

Step 2: Gather your Instance Details from Splunk Observability Cloud

Next, you need to retrieve your instance details, specifically the stream endpoint URL:

From your Splunk Observability window, open your profile information.
Go to Settings > View Profile > Organizations.
Take note of the URL displayed. This will be your `streamEndpointURL` and will be used later in ThousandEyes configuration (Step 3, part II)

Step 3: Configure ThousandEyes OpenTelemetry Integration

To stream telemetry data from ThousandEyes to Splunk, we will now go to the ThousandEyes platform to select ThousandEyes tests and configure the OpenTelemetry integration.

Note: For the purposes of this guide we created a new ThousandEyes Test, but you can use any active test (either cloud & enterprise or endpoint) you have already created.

Set up OpenTelemetry Integration:
1. Go to the Integrations page.
2. Click on + New integration and select ThousandEyes for OpenTelemetry.
On the next screen, use the following configuration:
1. Target: HTTP
2. URL: https://<instance URL from step 2.III>/v2/datapoint/otlp
3. From Preset Configurations, select “Splunk Observability”. Add the following configurations:
  1. For X-SF-Token: Use the token created in Step 1
  2. For Content-Type: application/x-protobuf
4. Select the test(s)you’d like to stream data from.
5. Click Save.
  
  The configuration should look like this when complete:

Note: For detailed instructions on ThousandEyes supported tests and metrics to be included for OpenTelemetry streams, refer to this documentation.

Step 4: Visualize ThousandEyes Telemetry Data in Splunk Observability

Finally, you can visualize the telemetry data in your Splunk Observability Cloud instance:

Go back to your Splunk Observability window and click the + sign > New Chart. You will see the streamed metrics from ThousandEyes available to as a chart.

ll. You can then assign the chart to a dashboard and create your report using available metrics. Customize your dashboard to display the data as needed, e.g.:

And that's it! By following these steps, you will successfully integrate ThousandEyes with Splunk Observability Cloud OpenTelemetry, enabling comprehensive monitoring and data visualization.

If you run into any trouble, don’t worry! The ThousandEyes team is here to help - consider opening a chat with one of our engineers to get support.