- Introduction
- Why is this important?
- Pre-requisites & Assumptions
- Step 1: Setting up an OAuth Application Registry in ServiceNow
- Step 2: Integrations 2.0 Configuration on ThousandEyes
- Step 3: Configure ThousandEyes alerts to notify the Operation in ServiceNow
- Step 4: Add a business rule in ServiceNow to resolve an incident automatically when the alert clears (Future you will thank you)
- Troubleshooting ServiceNow
- Conclusion:
- Additional Resources:
Introduction
One of the best parts of the ThousandEyes platform is the ability to integrate and communicate alerts with multiple other softwares. ThousandEyes alerts can be configured to trigger ServiceNow incidents and notify the application and/or network stakeholders in real-time when a service is down.
ServiceNow integration can be achieved in two ways:
- Natively using our classic Integrations 1.0
- Creating a connection using custom webhooks with Integrations 2.0
Previously, custom webhooks were configured differently. However, with the introduction of Integrations 2.0, the integration is broken into two parts: the Connectors (providing the connection with the integration) and the Operations (the actual action being performed). This allows for multiple operations to be performed using the same connector.
For Example: Suppose I wanted to send different parameters to the ServiceNow incident for alerts regarding Network & App Synthetics, Endpoint Experience, and Devices.
Using Integrations 1.0 I would need to configure both the parameters and the integration itself for all three. Using Integrations 2.0, however, while I would need to create separate operations for each, all three could use the same connector.
Why is this important?
If your organization has compliance requirements to use OAuth2.0, you are no doubt familiar with having to use custom webhooks. This document provides a step-by-step guide on how to configure this integration.
Note: Integrations 1.0 is still supported and can be used when you don’t have OAuth 2.0 related compliance requirements, or you do not want to customise the integration by using multiple operations with the same connector.
Pre-requisites & Assumptions
This document assumes that you have a working ServiceNow instance and a ThousandEyes account, with administrative rights in both, and have already configured your ThousandEyes tests and alerts.
Step 1: Setting up an OAuth Application Registry in ServiceNow
- Log in to your ServiceNow instance.
- In the Navigator on the left-hand pane, type “Application Registry” and click on System OAuth -> Application Registry.
- Click the New button.
- Select Create an OAuth API endpoint for external clients.
- Fill in the following details:
- Name: Any descriptive name that meets your org’s standards. We used “ThousandEyes Webhook Client”
- Client ID: This is auto-generated. Make sure to make a note of it!
- Client Secret: This will also be auto-generated. If the tool doesn’t show it when you create the endpoint, click on the endpoint name again and refresh the client secret after the parameters are displayed again. Make a note of this immediately, as it will only be shown once. If you lose it, you will need to regenerate it.
- Redirect URL: https://app.thousandeyes.com/namespace/integrations/AuthCallbackPage.html
- Active: This box should be checked by default, but it’s worth a double check.
- Client Type: Choose Integration as a Service from the drop-down.
- This creates the Application registry entry.
- Note: There is a lot of documentation online that advises testing functionality by running an API call to
/oath_token.do
with the client ID and client secret for testing. We do not advise that as newer versions of ServiceNow such as Xanadu will not allow this. This is because these versions enforce PKCE for all new OAuth clients by default.
Step 2: Integrations 2.0 Configuration on ThousandEyes
- Log in to ThousandEyes and navigate to Manage > Integrations > Integrations 2.0
- Once there, go to Connectors and click on +New Connector. Choose Generic Connector from the options and a window with details will pop up as shown below:
- Fill in the details requested:
- Name: Any descriptive name to distinguish the Connector later as belonging to ServiceNow.
- Target: The base URL of your ServiceNow instance.
- Preset Configuration: Leave it at the default entry.
- Auth Type: OAuth Code.
- Auth URL: https://<your-instance>.service-now.com/oauth_auth.do
- Token URL: https://<your-instance>.service-now.com/oauth_token.do
- Client ID: ServiceNow Client ID from Step 1 (We told you to make a note of it!).
- Client Secret: Auto-generated Client Secret from Step 1. We copied this earlier.
- Token: Click on Get Token to fetch it. You should be taken to the below page (screenshot).
- Note: In some customer environments, there may be an additional login window prior to reaching this page. If so, please log in, click on Allow, and the token will be fetched.
- Once the token is fetched, click on Save & Assign Operation. If the operation is successful, you’ll get a message which reads ‘Connector <NAME> has been saved successfully.’
- The Manage Operations window should pop up automatically, but if it does not you can get there manually by going to the Operations tab manually.
- Once on that page, click +New Operation. Select Custom Webhook from the drop down. A window will open and ask for your input on the following fields:
- Path: /api/now/v1/table/incident
- Custom Headers: +New Header ->content-type:application/json
- URL Query Parameters: Leave blank
- Body:
{ "short_description": "ThousandEyes Alert: '{{alert.rule.name}}' for test {{alert.test.name}}", "description": "An alert was triggered for test '{{alert.test.name}}.\nAlert Type: {{alert.test.testType}}\nSeverity: {{alert.severity.id}}\nView details in ThousandEyes: https://app.thousandeyes.com/alerts/list/?__a={{alert.rule.account.id}}", "impact": "2", "urgency": "2", "state": {{#if (eq type.id 2)}}1{{else}}6{{/if}}, "caller_id": "62826bf03710200044e0bfc8bcbe5df1", "correlation_id": "{{alert.id}}", "correlation_display": "{{alert.id}}", "close_code": "{{#if (eq type.id 1)}}Resolved by ThousandEyes Alert Clear{{/if}}", "close_notes": "{{#if (eq type.id 1)}}Alert cleared automatically by ThousandEyes integration.{{/if}}" }
Note: You do not have to use the caller_id line. However, be aware if it is set to the Client ID from the OAuth Application Registry, the Incident in ServiceNow will populate the Caller accordingly with the username or email corresponding to the caller_id mentioned here.
- Click on Save & Assign Connector. It will open the Manage Connector page. Choose the connector created earlier from the drop down and click Save.
- Go to the Operations tab. Select the created operation and click Test.
- If the test executes correctly you’ll get a message which reads “Testing completed successfully!”.
- If the test does not execute correctly, you would see a ‘Pending’ or ‘Failed’ message.
- If received, verify that the configuration fields have been set properly and re-test.
- Click Save again.
- If the configuration is working as expected, you’ll see that the status for the connector shows “Connected”
Step 3: Configure ThousandEyes alerts to notify the Operation in ServiceNow
- In ThousandEyes, go to Manage > Alert Rules.
- Choose an alert for which you want to configure the notification and click on it.
- Go to the Notifications tab as shown below.
- Under Integrations choose the Operation you have created (SNOW in our example).
- Click on Save Changes
Step 4: Add a business rule in ServiceNow to resolve an incident automatically when the alert clears (Future you will thank you)
This step ensures that incidents are created with the State field set to New when an alert triggers, as well as Resolved when the alert clears.
- In the left-hand pane navigator of ServiceNow, type “Business Rules”.
- Go to System Definition > Business Rules.
- Click New in the top right and name the business rule. We recommend something descriptive like “Process ThousandEyes Alerts”.
- Configure the following fields as indicated:
- Table: Incident
- Advanced: Checked
- When: Before
- Insert: Checked
- Update: Unchecked
- Delete: Unchecked
- In the Advanced tab, leave Condition blank and add the following script
(function executeBR(current, previous /*null on insert*/) {
gs.info('Alert correlation_id: ' + current.correlation_id + ', state: ' + current.state);
if (current.state == 1) {
// Trigger alert: always allow creation
gs.info('Trigger alert received → creating new incident.');
return;
}
// Clear alert: search for existing unresolved incident
var gr = new GlideRecord('incident');
gr.addQuery('correlation_id', current.correlation_id);
gr.addQuery('state', '!=', 6,7); // only unresolved incidents
gr.addQuery('caller_id', current.caller_id); // add if you want to clear incidents created by the ThousandEyes user only
gr.query();
if (gr.next()) {
gs.info('Existing unresolved incident found: ' + gr.number + ', state: ' + gr.state);
// Resolve the existing incident
gr.state = 6;
gr.close_code = 'Solved (Permanently)';
gr.close_notes = 'Automatically resolved by ThousandEyes alert clear.';
gr.update();
gs.info('Incident ' + gr.number + ' resolved automatically.');
// Prevent creating a new incident
current.setAbortAction(true);
gs.info('New incident creation aborted for clear alert.');
} else {
// No existing unresolved incident found → skip creating a resolved incident
current.setAbortAction(true);
gs.info('Clear alert received but no unresolved incident exists → abort creation.');
}
})(current, previous);
Note: What does this do?
This script checks for the current state of an alert and if it is a new alert that is triggered as indicated by the type.id from ThousandEyes being set to 1, a new incident is created, otherwise, if the type.id is 2 indicating that an earlier alert is now cleared, it will look for the correlation id to identify the correct incident and mark it as ‘Resolved’.
- Click Update in the top right to update the Business Rule.
Testing Protocols
- Trigger an alert on ThousandEyes: Create an alert which will get triggered in the next test run such as HTTP Response Code of 200 for tests that are working fine. Set it to Manual and choose it to be triggered if at least 1 agent is affected 1 of 1 times in a row. In the Notifications tab of the alert make sure that you map the alert to the integration you just created.
- Go to ServiceNow. Click All and search for “Incident”. Choose Open Incidents and check whether the alert that was triggered is visible.
- If the incidents were created as expected you’ll see a table that looks like below:
- Verify the details being passed through by clicking on one of the incidents to get more information. An example is shown below:
Troubleshooting ServiceNow
- If Incidents are not getting created, go to System Logs > All and look for logs related to your webhook.
- If you want to see what ThousandEyes is sending, go to System Definition > Business Rules and add a rule by clicking New in the top right. Alternatively you can update the script in the Business rule created in Step 4 by searching for the rule name under Business Rules.
- If you decide to add a new rule, add the following details.
- Table: Incident
- When: Before
- Insert: Checked
- Condition: Leave blank
- Advanced: Checked
- In the tab named Advanced, leave Condition blank and add the following under Script:
-
(function executeRule(current, previous) { try { gs.log("TE Business Rule fired for incident: " + current.number); gs.log("Full incident record: " + JSON.stringify({ short_description: current.short_description, description: current.description, source: current.source, event_class: current.event_class, resource: current.resource, node: current.node, metric_name: current.metric_name, u_source: current.u_source, u_event_class: current.u_event_class, u_resource: current.u_resource, u_node: current.u_node, u_metric_name: current.u_metric_name, u_type: current.u_type, u_severity: current.u_severity, u_additional_info: current.u_additional_info })); } catch (e) { gs.error("TE Logging Business Rule error: " + e.message); } })(current, previous);
- If you decide to add a new rule, add the following details.
- Go to System logs > Script Log Statements as shown below to see the messages logged from ThousandEyes.
Conclusion:
If you’re able to validate that your alerts are triggering properly congratulations! You have successfully integrated ThousandEyes and ServiceNow.