gNMI (gRPC Network Management Interface) was developed as a higher performance and lighter weight alternative to NETCONF for network management and configuration. gNMI was developed as part of the OpenConfig initiative to create vendor-neutral, model-driven, and open-source network management and configuration standards.
gNMI is lacking in its support of making configuration changes as compared to NETCONF as it doesn’t provide full support of transactions.
However, gNMI’s strengths lie in its support of streaming telemetry. It provides enhanced capabilities for subscribing to real-time telemetry updates from network devices, allowing for proactive monitoring and alerting. While NETCONF also supports some streaming capabilities through the YANG Push protocol, gNMI's gRPC foundation makes it more suited for handling large volumes of streaming data. gNMI telemetry seems to be gaining adoption in the industry.
gNMI was developed by the OpenConfig Working Group which consisted of various major network operators and vendors. The OpenConfig Working Group operates in an informal and community-driven manner. NETCONF was developed by IETF which operates under a formal standardization process with defined procedures and guidelines.
Although NETCONF was developed with a formal structure, certain areas in the RFC remained open to interpretation by the reader. That’s why we have developed the NYAT (NETCONF & YANG Automation Testing) program to help network equipment vendors and service providers to identify and address NETCONF/YANG implementation issues that are getting in the way of enabling network automation for multi-vendor network devices. In my opinion, there is a need to perform interoperability testing of gNMI telemetry implementations to facilitate seamless multi-vendor automation.
gNMI Telemetry Interop Test Tool
Due to the various weaknesses in the gNMI specifications, there exist implementation variations amongst the gNMI devices available in the market. To avoid vendor-specific coding that may be required when working with gNMI telemetry servers from different vendors, we need to strive for normalized gNMI telemetry server implementations.
We have developed a gNMI Telemetry Testing Tool for the automated testing of the telemetry capabilities and behaviors of gNMI servers. Our initial focus of the tool is on telemetry as that is the strength of gNMI and where it is the most widely used. It is my belief that this is where gNMI shines.
By leveraging the real-time data provided through gNMI telemetry, closed-loop automation can be realized to dynamically optimize the network by continuously observing the network’s state and analyzing the telemetry data.
Our test tool is still work in progress. The automated testing is developed using the Robot Framework. The gNMI test client was developed using grpcio-tools. It performs compliance testing of the gNMI Capabilities, Get and Subscribe method calls.
There are two broad categories of test cases that come with the test tool:
- model-agnostic test cases are for basic testing of gNMI operations on the Capabilities, Get, Set and Subscribe requests.
- OpenConfig model specific test cases are designed to test the vendor’s gNMI implementation of the OpenConfig YANG models.
The initial focus of OpenConfig is for testing the interfaces model. We plan to expand to cover more areas of OpenConfig, ideally to cover the equivalent of SNMP’s MIB II capabilities.
The test tool also allows the end user to add additional test cases including vendor-specific model testing.
At the completion of the automated test, a test report with the summary of the test results and a set of log files that help with analyzing and troubleshooting the test cases are produced. I have presented on gNMI Telemetry Interoperability Testing using this test tool during the recent Automation Developer Days in Stockholm.
Interoperability testing has been performed using this test tool against several Cisco and 3rd party gNMI telemetry servers.
Observed gNMI Server Implementation Variations
During our interoperability testing, we’ve run into several interesting variations between the implementations of the tested devices:
- Only proprietary instead of OpenConfig models are supported. Lack of support of OpenConfig impedes both interoperability and ease of automation between various network devices.
- Get requests with using the root path aren’t supported by some implementations.
- There are differences in interpretation of the origin, prefix, and path fields. The gNMI specifications lacks a standard way of disambiguating identical paths between supported models. Using the YANG module names in either the prefix or path fields would seem reasonable to achieve that. However, very few gNMI servers support this.
- The gNMI specifications calls for JSON encoding to always be supported. All tested servers support JSON_IETF except for some that support JSON in addition. JSON_IETF is a more restricted extension of JSON meant to maximize interoperability and ease mapping into other data formats. One can argue that support of JSON_IETF should be made mandatory instead of JSON.
- Some YANG model paths can only be specified with the Subscribe calls, but not with the Get calls.
- The capabilities response from some servers advertises incorrect supported encodings that aren’t really supported by the gNMI server.
- There are a few cases in the gNMI specifications where the gNMI server must send a sync_response. However, I’ve often seen missing sync_responses in the gNMI server implementations.
- Within the initial set of updates of a Subscribe response (such as ON_CHANGE stream subscriptions), I’ve seen multiple updates of the same leaf element which isn’t allowed according to the gNMI specifications.
- Update messages are being aggregated when not permitted. Update messages are supposed to be independent and, for resource efficiency reasons on the server, shouldn’t be aggregated. Some server implementations have chosen to aggregate them anyway.
- The Operational data type isn't usually supported. Operational here refers to data that is related to software processes operating on the device.
These are just a few of the variations that we have noticed. As we get to complete development of the rest of test cases that we have planned for, I expect that we'll find even more.
Conclusion
My objective of this project is to promote the normalization of the gNMI telemetry server implementations in network devices with the help of our gNMI Telemetry Testing Tool. This will help avoid custom coding during network automation when working with devices from different vendors. The test passing criteria used by our test tool is certainly by no means the only correct interpretation of the gNMI specifications.
I encourage you to try out our test tool with your gNMI telemetry servers and give us feedback in the discussions section of our gNMI Telemetry Testing Tool github repo. If you don’t have a gNMI telemetry server handy, you can try out our test tool using ConfD along with our ConfD gNMI adapter. If it ever makes sense to bring everyone on the same page, we may attempt to develop a reference implementation of a gNMI telemetry server using ConfD with its gNMI adapter. A version of the gNMI adapter for NSO can be found in the NSO-developer github repo. This adds gNMI support to NSO’s northbound.
