Introduction
One of the challenging topics of network automation is to test the routers to see if their current operational state match the expected operational state, or in the case you are changing configuration compare pre-operational states with post-operational states .
The traditional way to do it for the network engineer is to use the CLI and explore the data returned on the screen and retrieve the relevant states, counters ... and put them in an Excel document.
As there is hundreds of those states and counters, verifications become difficult if the operation has lots of checks.
Moreover if you have all those checks multiplied by a large number of devices, it is impossible to cover without automation.
You have multiple ways to automate checks on your network, you can use custom scripts, open source solutions like Netmiko, Ansible , you can also use PyATS which is a very good testing framework from Cisco.
The aim of this post is to show a way to automate checks with NSO and create an Excel document as audit.
This is not THE way to do it, just a way, but I have applied it on thousands of devices and run millions of checks with a similar method as the one presented here.
Along with NSO I will also use the PyATS parser, historically I was using TextFSM homemade templates for doing the parsing part of my workflow, but I tend to progressively migrate to the very rich parsers list of PyATS.
Note : The code provided only works with CLI NEDs and Cisco devices.
Prerequisites
Update Feb23 : seems there is an incompatibility with python > 3.9 due to pyats, and it won't work on arm Mac (M1,M2,Mx).
On the Python installation running alongside your NSO let's install some libraries, I've prepared for this demonstration :
pip install pyats-parserThis library, as described in the documentation, bring the power of the huge PyATS parsers collection to every piece of python you can write, the downside is, because of dependencies, you will have to install all the PyATS libraries and not only the parsing part.
pip install nso-live-statusIn fact you can directly install this library because it depends on pyats-parser to run, so pyats-parser will be installed automatically.
nso-live-status is a wrapper to the live status of NSO, I just wanted to have a convenient function : run_live_status(root_nso_object, device, str_cli_command)
Not only this function will execute the live status but if the command is recognised by PyATS it will be parsed and the object returned will contain a structured output
pip install nso-restconfThis will be useful for manipulating NSO through the Restconf API
And finally the popular openpyxl for generating excel documents:
pip install openpyxlExample of check
For this example we will simply check the version of IOS-XR to see if it match a target version.
I've prepared a NSO Package which can be used as a "checks starting pack" at : https://github.com/rtrjl/check_device
so all you have to do is to :
git clone git@github.com:rtrjl/check_device.gitin your NSO packages directory, and :
cd check_device
cd src
make clean
make in the NSO CLI session :
# packages reloadonce the package reload is done, we can try the action :
in the NSO CLI session :
# check_device check_version device Paris-75 target_version 7.5.1
check_status NOK
check_message [check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Paris-75
Industrialization of checks
You now have an action to check the version and you can call it with the CLI which is nice, but NSO bring a very powerful built-in feature with an auto-generated RESTConf API on top of all your actions and services.
Using this RESTConf API we can launch the checks on a set of devices:
import json
from nso_restconf.restconf import RestConf
from concurrent.futures import ThreadPoolExecutor, as_completed
nso_api = RestConf(address="http://127.0.0.1", port=8080,
username="admin", password="admin")
# for the example credentials are in the code :-)
devices_list = ["Kiev-47",
"Madrid-40",
"Napoli-50",
"Paris-66",
"Riga-46",
"Stuttgard-64"]
#the numbers have no significations they are just the last bytes of the IP of the routers on the lab I use.
def nso_check_version(device, target_version):
data = json.dumps({'input': {'device': f"{device}", "target_version": f"{target_version}"}})
result = nso_api.action(data, "check_device/check_version")
return result.json()
The script should print something like that:
{'check_device:output': {'device': 'Kiev-47', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Kiev-47"}}
{'check_device:output': {'device': 'Madrid-40', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Madrid-40"}}
{'check_device:output': {'device': 'Napoli-50', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Napoli-50"}}
{'check_device:output': {'device': 'Paris-66', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Paris-66"}}
{'check_device:output': {'device': 'Riga-46', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Riga-46"}}
{'check_device:output': {'device': 'Stuttgard-64', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Stuttgard-64"}}
In parallel
The script is running well but with this approach it can take a lot of time because it will connect to devices one after the other, it will be fine on a small network but on a bigger ... Let's use threads to accelerate the process :
from concurrent.futures import ThreadPoolExecutor, as_completed
processes = []
with ThreadPoolExecutor(max_workers=10) as executor:
for device in devices_list:
processes.append(executor.submit(nso_check_version, device, "7.5.1"))
for task in as_completed(processes):
print(f"{task.result()}")
The same output as the sequential one will be printed but in this version we leverage another interesting feature of NSO:
devices can be reached in parallel to run show commands, with limits , it will depend on the ammount of RAM on the server on which NSO is installed, but we can run 10 workers without problems.
Now we have retrieved the checks we can put the results in a good old excel document :
checks_list = []
for task in as_completed(processes):
result = task.result()["check_device:output"]
checks_list.append([result["device"], result["operating_system"], result["current_version"], result["check_status"], result["check_message"]])
from openpyxl import Workbook
wb = Workbook()
ws = wb.active
headers = ["Device", "OS", "OS version", "Check status", "Check message"]
ws.append(headers)
for data in checks_list:
ws.append(data)
wb.save("devices-version-audit.xlsx")
That's all, in this demo we have seen different topics like using pyATS parsers with NSO or doing NSO live status (show commands) in parallel leveraging the auto-generated RESTCONF API, we have also seen how we can write a check using actions in NSO.
Explore the pyATS parsers page : https://pubhub.devnetcloud.com/media/genie-feature-browser/docs/#/parsers and imagine all the checks you can do