I have recently presented on how to build a JSON Schema Validation Adapter for Crosswork Workflow Manager (CWM) 1.1 during a customer PoC for bulk provisioning of L3VPN endpoints. After completing the project, I experimented with how AI-assisted code generation could have helped sped up coding a CWM adapter in Golang for performing utility functions in a workflow. I chose GitHub Copilot with VS Code as it provides the best programmer's experience. The GitHub Copilot extension for VS Code is described as an AI pair programmer tool that helps you write code faster and smarter. It allows you to use natural language through its chat interface to generate code.
I'm sharing my experience working with GitHub Copilot on my journey towards basic Network Automation programming with CWM and NSO. A recent Automation Developer Days' session on The Power of CWM 1.1 with NSO proved the point of how and why NSO works better with CWM. My primary focus is on code generation though GitHub Copilot, though I also briefly compare other AI tools.
Project Overview
- JSON Schema Validation Adapter: Exploring how GitHub Copilot can generate Golang code for a JSON Schema Validation function.
- NSO Integration: Experimenting with AI tools to generate code for retrieving information from NSO via its RESTCONF API.
- Serverless Workflow DSL: Using GitHub Copilot to generate Serverless Workflow DSL code for CWM workflows that incorporate NSO RESTCONF calls.
JSON Schema Validation Adapter
Let's start with code generation of Golang code for the JSON Schema Validation adapter for CWM.The details on using the CWM SDK to build an adapter are available in my Cisco DevNet Code Exchange Repository on cwm-json-schema-validator. Note that I'm using GitHub Copilot v1.214.0.
Given the following protobuf messages defined for the adapter:
// CWM SDK NOTE: Messages here e.g.
/* Documentation for Request */
message Request {
google.protobuf.Value json_input = 1; // json input data to be validated
google.protobuf.Value json_schema = 2; // json schema to be used for validation
}
/* Documentation for Response */
message Response {
string output = 1; // json input data validation result
}
The CWM SDK generated the following code for gathering the JSON input and Schema, and the data type for the response output:
func (x *Request) GetJsonInput() *structpb.Value {
if x != nil {
return x.JsonInput
}
return nil
}
func (x *Request) GetJsonSchema() *structpb.Value {
if x != nil {
return x.JsonSchema
}
return nil
}
// json input data validation result
type Response struct {
Output string `protobuf:"bytes,1,opt,name=output,proto3" json:"output,omitempty"`
}
And the minimal skeleton for the validation function:
func (adp *Adapter) Validate(ctx context.Context, req *Request, cfg *common.Config) (*Response, error) {
var res *Response
var err error
// CWM SDK NOTE: Implement your activity logic here...
return res, err
}
My objective was to get GitHub Copilot to generate working code for this CWM adapter. My first prompt was:
implement Validate using open source library to validate GetJsonInput() against GetJsonSchema() that are passed in through the Request object argument and return response as Response with an output string of validation results
Copilot selected the same open-source library I had chosen previously but generated code that didn't compile as the JSON objects were used as strings. Copilot should have generated the right code for handling the input data types, especially given the file cisco.jsonschema.validate.adapter.pb.go was open in VS Code.
I then refined my prompt:
Revise the generated code and convert req.GetJsonSchema and req.GetJsonInput from *structpb.Value as protobuf JSON objects to strings
A demo of using GitHub Copilot with the CWM SDK generated JSON Schema Validation adapter project is available here.
After many prompt attempts, a more effective prompt was:
Implement Validate using gojsonschema to validate req.GetJsonInput() against req.GetJsonSchema() and return response as Response with an output string of validation results.
Pick a loader from gojsonschema that supports JSON object types for both the JSON input and the JSON schema with the fewest lines of Go code.
This generated code closely matched my original hand-code version. Note that Copilot may have generated multiple solutions from your prompt, you may need to hit the re-run button to get the results that you want.
Python Code for NSO's RESTCONF API
Next, I generated Python code to access NSO's RESTCONF API. My prompt was:
Write python code to retrieve the software version of nso through its restconf api
For debugging, I added:
Add debug statements in the generated code to allow all responses from NSO to be shown for troubleshooting
The initial code had an incorrect RESTCONF API endpoint. Using GitHub Copilot's inline chat, I fixed the issue:
Following is the corrected version of the Python script:
import requests
from requests.auth import HTTPBasicAuth
# Replace these variables with your NSO details
nso_host = "localhost"
nso_port = "8080"
username = "admin"
password = "admin"
# Construct the URL for the RESTCONF API endpoint that provides NSO version information
url = f"http://{nso_host}:{nso_port}/restconf/data/tailf-ncs-monitoring:ncs-state/tailf-ncs-monitoring:version"
# Set up the headers. Adjust if your NSO version expects a different content type.
headers = {
"Accept": "application/yang-data+json",
}
# Make the GET request with basic authentication
response = requests.get(url, auth=HTTPBasicAuth(username, password), headers=headers)
# Debug: Print the entire response for troubleshooting
print(f"Response Status Code: {response.status_code}")
print("Response Headers:")
for key, value in response.headers.items():
print(f" {key}: {value}")
print("Response Body:")
print(response.text)
# Check if the request was successful
if response.status_code == 200:
# Parse the JSON response
data = response.json()
# Extract and print the software version
nso_version = data.get("tailf-ncs-monitoring:version")
print(f"NSO Software Version: {nso_version}")
else:
print(f"Failed to retrieve NSO version, status code: {response.status_code}")
Sample run of the script:
% python cp-get-nso-version.py
Response Status Code: 200
Response Headers:
Date: Fri, 19 Jul 2024 22:32:05 GMT
Last-Modified: Thu, 27 Jun 2024 02:41:03 GMT
Cache-Control: private, no-cache, must-revalidate, proxy-revalidate
Etag: W/"1719-456063-579682+json"
Content-Type: application/yang-data+json
Transfer-Encoding: chunked
X-Cisco-NSO-Trace-ID: dc1c6bfa-b38c-4575-a638-147ace1d6165
Pragma: no-cache
Content-Security-Policy: default-src 'self'; block-all-mixed-content; base-uri 'self'; frame-ancestors 'none';
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Response Body:
{
"tailf-ncs-monitoring:version": "6.1.9"
}
NSO Software Version: 6.1.9
Retrieving IOS-XR Device Software Version
To retrieve the IOS-XR device software version, I used the prompt:
write python code to retrieve device software version of IOS-XR through NSO's RESTCONF API by sending a cli command to the device through the CLI NED
Include debug statements in the code for troubleshooting
The generated code required endpoint and key corrections:
import requests
from requests.auth import HTTPBasicAuth
import json
# Replace these variables with your NSO and device details
nso_host = "localhost"
nso_port = "8080"
username = "admin"
password = "admin"
device_name = "xr-1"
# Construct the URL for the RESTCONF API endpoint
url = f"http://{nso_host}:{nso_port}/restconf/operations/tailf-ncs:devices/device={device_name}/live-status/exec/any"
# Set up the headers
headers = {
"Accept": "application/yang-data+json",
"Content-Type": "application/yang-data+json",
}
# Prepare the payload with the CLI command
payload = {
"input": {
"args": ["show version"]
}
}
# Make the POST request with basic authentication
response = requests.post(url, auth=HTTPBasicAuth(username, password), headers=headers, data=json.dumps(payload))
# Debug: Print the entire response for troubleshooting
print(f"Response Status Code: {response.status_code}")
print("Response Body:")
print(response.text)
# Check if the request was successful
if response.status_code == 200:
# Parse the JSON response
data = response.json()
# Extract and print the relevant information, e.g., software version
# The exact key might vary, adjust according to your NSO version and response structure
#output = data.get("tailf-ncs:output", {}).get("result")
output = data.get("tailf-ned-cisco-ios-xr-stats:output", {}).get("result")
if output:
print("Device Software Version:")
print(output)
else:
print("Failed to extract software version from the response.")
else:
print(f"Failed to retrieve device software version, status code: {response.status_code}")
The data resource was incorrectly being used to invoke the rpc call through the CLI NED. When I attempted the same query from Perplexity AI, the initial generated code was similar. However, when I asked it to use the operations resource, it then realized that it was using the wrong resource in its initial generated code and that the operations resource was the right resource to use for executing CLI commands on the device.
Sample run of the script:
% python cp-get-xr-version.py
Response Status Code: 200
Response Body:
{
"tailf-ned-cisco-ios-xr-stats:output": {
"result": "\r\n\rMon Jul 22 20:22:17.113 UTC\r\nCisco IOS XR Software, Version 7.9.2 LNT\r\nCopyright (c) 2013-2023 by Cisco Systems, Inc.\r\n\r\nBuild Information:\r\n Built By : \r\n Built On : Thu Jun 29 03:07:05 UTC 2023\r\n Build Host : bdb7eb8f4e82\r\n Workspace : /auto/srcarchive16/prod/7.9.2/8000/ws\r\n Version : 7.9.2\r\n Label : 7.9.2\r\n\r\ncisco 8000 (VXR)\r\ncisco 8201 (VXR) processor with 32GB of memory\r\nPE2 uptime is 10 weeks, 4 days, 2 hours, 36 minutes\r\nCisco 8201 1RU Chassis\r\n\r\nRP/0/RP0/CPU0:PE2#"
}
}
Device Software Version:
Mon Jul 22 20:22:17.113 UTC
Cisco IOS XR Software, Version 7.9.2 LNT
Copyright (c) 2013-2023 by Cisco Systems, Inc.
Build Information:
Built By :
Built On : Thu Jun 29 03:07:05 UTC 2023
Build Host : bdb7eb8f4e82
Workspace : /auto/srcarchive16/prod/7.9.2/8000/ws
Version : 7.9.2
Label : 7.9.2
cisco 8000 (VXR)
cisco 8201 (VXR) processor with 32GB of memory
PE2 uptime is 10 weeks, 4 days, 2 hours, 36 minutes
Cisco 8201 1RU Chassis
RP/0/RP0/CPU0:PE2#
I have also experimented with other AI tools to accomplish the same task. In some cases, the quality of the generated code was slightly better with Perplexity AI, especially with the debug statements. ChatGPT has produced similar results as Copilot. The output was easier to read and I didn't have to inject any missing debug statements when debugging the RESTCONF output responses from NSO.
Serverless Workflow DSL for NSO RESTCONF API
Finally, I generated Serverless Workflow DSL code for invoking the NSO RESTCONF API through the CWM adapter. The prompt was:
generate serverless workflow dsl code in json format to invoke the NSO RESTCONF api for retrieving the XR device software version.
The initial code required some adjustments which I have made to produce the version on the right:
As I'm able to use the actionDataFilter to process the output response from NSO within the same state as the one that invoked the NSO RESTCONF API call, I didn't need a second state to do the output processing.
When running the corrected version of the Serverless Workflow DSL code inside of CWM, I was able to get the following workflow output:
{
"eventId": "21",
"eventTime": "2024-07-22T21:11:49.944864234Z",
"eventType": "WorkflowExecutionCompleted",
"taskId": "6291504",
"workflowExecutionCompletedEventAttributes": {
"result": {
"payloads": [
{
"metadata": {
"encoding": "json/plain"
},
"data": {
"Data": {
"output": {
"tailf-ned-cisco-ios-xr-stats:output": {
"result": "\r\n\rMon Jul 22 21:48:27.324 UTC\r\nCisco IOS XR Software, Version 7.9.2 LNT\r\nCopyright (c) 2013-2023 by Cisco Systems, Inc.\r\n\r\nBuild Information:\r\n Built By :\r\n Built On : Thu Jun 29 03:07:05 UTC 2023\r\n Build Host : bdb7eb8f4e82\r\n Workspace : /auto/srcarchive16/prod/7.9.2/8000/ws\r\n Version : 7.9.2\r\n Label : 7.9.2\r\n\r\ncisco 8000 (VXR)\r\ncisco 8201 (VXR) processor with 32GB of memory\r\nPE2 uptime is 10 weeks, 4 days, 4 hours, 2 minutes\r\nCisco 8201 1RU Chassis\r\n\r\nRP/0/RP0/CPU0:PE2#"
}
},
"status": "success"
}
}
}
]
},
"workflowTaskCompletedEventId": "20"
}
}
]
Given the above CLI output in the workflow output, my next attempt was to get Github Copilot to generate the jq expression for extracting only the software version string information from the CLI output. Unfortunately, Copilot wasn't able to provide me with the correct jq expression that worked. However, I was able to use Perplexity AI to generate the correct jq expression with the following two prompts as shown in the screen captures:
Following is the final version of the workflow for printing the software version information of a XR device retrieved through NSO's RESTCONF API:
{
"id": "pai-get-xr-version-workflow",
"name": "pai-get-xr-version-workflow",
"states": [
{
"end": true,
"name": "InvokeNSORESTCONF",
"type": "operation",
"actions": [
{
"name": "invokeNSOAPI",
"functionRef": {
"refName": "callNSORESTCONF",
"arguments": {
"input": {
"data": {
"args": "show version"
},
"path": "restconf/operations/devices/device=xr-1/live-status/exec/any"
},
"config": {
"resourceId": "NSO"
}
}
},
"actionDataFilter": {
"results": "${ if .status == 200 and (.data) then { \"status\": \"success\", \"output\": (.data.\"tailf-ned-cisco-ios-xr-stats:output\".result| capture(\"Cisco IOS XR Software, Version (?<version>\\\\d+\\\\.\\\\d+\\\\.\\\\d+)\") | .version)} else { \"status\": \"error\", \"output\": \"nil\"} end }"
}
}
]
}
],
"version": "1.0",
"startsAt": "InvokeNSORESTCONF",
"functions": [
{
"name": "callNSORESTCONF",
"operation": "cisco.nso.v1.0.1.restconf.Post"
}
],
"description": "Workflow to retrieve the software version of an XR device via NSO RESTCONF API",
"specVersion": "0.8"
}
Following is the workflow output:
[
{
"eventId": "21",
"eventTime": "2024-07-23T00:34:13.207053989Z",
"eventType": "WorkflowExecutionCompleted",
"taskId": "1048624",
"workflowExecutionCompletedEventAttributes": {
"result": {
"payloads": [
{
"metadata": {
"encoding": "json/plain"
},
"data": {
"Data": {
"output": "7.9.2",
"status": "success"
}
}
}
]
},
"workflowTaskCompletedEventId": "20"
}
}
]
Conclusion
In conclusion, while GitHub Copilot provides a good starting point with code generation, achieving accurate and high-quality results requires detailed prompts and iterative refinement. As warned by GitHub Copilot Chat: "I'm powered by AI, so surprises and mistakes are possible. Make sure to verify any generated code or suggestions.". With free or open source AI tools that have its own cycles of updating the training of their LLMs with the most up-to-date information, you also run the risk that it won't work the latest and greatest of what you would like to use it for. I would assume that I would likely run into more hurdles as I try to use the tool to help solve more advanced and sophisticated problems. However, I am sure that the code generation capabilities of AI assisted tools will continue to improve. It will certainly help us to improve our productivity in coding and more time to work on the more difficult problems. It also lowers the barrier of entry for users of the network automation tools that require some level of programming skills.
2 Comments