Porting an existing service to use a NETCONF NED, part 1

Jan Lindblad · ‎08-25-2017

As I noted in the root post (Porting an existing service to use a NETCONF NED), the first step in our journey of porting an existing NSO service from using a CLI NED to also supporting a NETCONF NED to that same device consists of the following:

1) Establish NETCONF communications
- Set up device, NSO and service meta data
- Update NSO configuration to plug in the new NETCONF device(s)
- Build a NETCONF NED for the device
- Adjust NEDs to co-exist (should ideally not be needed, but currently it is)
- Ensure sync-from works

For service to be ported from CLI to NETCONF NED, I'm using one of the NSO 4.4.2 standard L3VPN examples, nso/4.4.2/examples.ncs/service-provider/virtual-mpls-vpn . I actually made a copy of the example in order to not mess up the original, but that's up to you ;-)

$ cd virtual-mpls-vpn

For device, I'm using a csr1000v-universalk9.16.06.01.iso image running in VirtualBox on my laptop. A very nice solution for testing towards "real" equipment if you ask me. If you're interested in how to set that up, make a comment asking for it, and I'll make a post about that some day. Here, I'll just assume it's running at 192.168.50.50, with NETCONF on port 830 and credentials vagrant/vagrant.

We'll be needing the Pioneer package. I'm using Pioneer 2.23.0. Install it in packages/

$ cd packages/

$ cp -r <path to downloaded packages>/pioneer .

For NED, I'm using the currently latest IOS CLI NED, i.e. cisco-ios 5.4.4. So I need to discard the default NED in the example and replace with the proper CLI NED.

$ rm -rf cisco-ios
$ cp -r <path to downloaded packages>/cisco-ios .

I will need a NETCONF NED for the XE device as well, but that one I'll build myself. Much of this blog post will be about how to do that properly.
Now, let's build all the packages.

$ cd ..

$ make

Since this system is using a lot of Java packages, we'll need to raise the JavaVM memory ceiling a bit. Set this environment variable before you start NSO, and you should be fine.

$ export NCS_JAVA_VM_OPTIONS="-Xmx1G"

This system is using many netsim devices. Now, let's start both the NETSIMs and NSO itself.

$ make start

Now, let's log in and replace the ce0 netsim device with the "real" CSR device instead. First we need to add it to the device list, with IP, port, auth details, and fetch the SSH host key.

$ ncs_cli -Cu admin
admin connected from 127.0.0.1 using console on JLINDBLA-M-W0J2

JLINDBLA-M-W0J2# con
Entering configuration mode terminal
JLINDBLA-M-W0J2(config)# devices authgroups group vagrant
JLINDBLA-M-W0J2(config-group-vagrant)# default-map remote-name vagrant remote-password vagrant
JLINDBLA-M-W0J2(config-group-vagrant)# top
JLINDBLA-M-W0J2(config)# devices device ce0

JLINDBLA-M-W0J2(config-device-ce0)# address192.168.50.50JLINDBLA-M-W0J2(config-device-ce0)# device-type netconf 
JLINDBLA-M-W0J2(config-device-ce0)# authgroup vagrant
JLINDBLA-M-W0J2(config-device-ce0)# port 830
JLINDBLA-M-W0J2(config-device-ce0)# comm
Commit complete.
JLINDBLA-M-W0J2(config-device-ce0)# ssh fetch-host-keys
result updated
fingerprint {
    algorithm ssh-rsa
    value b0:d4:d4:f8:01:b6:c9:07:6f:ad:a1:82:bc:b3:d0:3d
}
JLINDBLA-M-W0J2(config-device-ce0)# connect
result false
info Device ce0 does not advertise any known YANG modules
JLINDBLA-M-W0J2(config-device-ce0)# 

"No known YANG modules" is expected, we don't have any NETCONF NED for the device yet.

The CSR device is a trifle different from the IOS devices the system was built for, in particular the interface names are different. So we also need to update the topology information in the system to the interface names the CSR uses.

JLINDBLA-M-W0J2(config)# topology connection ce0
JLINDBLA-M-W0J2(config-connection-ce0)# endpoint-1 device ce0 interface GigabitEthernet3 ip-address 192.168.1.1/30
JLINDBLA-M-W0J2(config-connection-ce0)# comm
Commit complete.

Next step is to build the NED. We'll use Pioneer (even if other possibilities exist) for this.

JLINDBLA-M-W0J2(config-connection-ce0)# top
JLINDBLA-M-W0J2(config)# devices device ce0
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang fetch-list

Retrieving module list from deviceDevice supports netconf-monitoring
Found out the names for a total of 375 modules

JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang download
Downloading 375 modules to /tmp/download/ce0

Now, these 375 YANG modules actually pertain to four separate "views" of the devices. This XE device offers the Native, the IETF, the OpenConfig and the SNMP views. The SNMP view is pretty useless and OpenConfig is broken, so only Native or IETF remain as reasonable choices right now. I believe the Native view has most functionality, so that's what I will use in today. 

The immediate question is then which YANG files that belong to the Native view. Unfortunately, there is currently no easy way of knowing. Some original research has taught me. So here follows the recipe for a Native XE NETCONF NED:
 

JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang disable name-pattern *
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang enable name-pattern "Cisco-IOS-XE-* cisco-self-mgmt cisco-ia cisco-odm ietf-inet-types ietf-key-chain ietf-netconf* ietf-restconf* ietf-yang* iana-crypt-hash"
 
One of the YANG modules is mis-named. Cisco-IOS-XE-mpls-ldp is actually part of the IETF view but named like a native YANG. Let's remove it manually.
 
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang disable name-pattern Cisco-IOS-XE-mpls-ldp                     
success Disabling module Cisco-IOS-XE-mpls-ldp
 
Time to build the NETCONF NED.
 
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang build-netconf-ned                         
Cleaning up existing ned-directory
Starting build of 120 YANG modules, this may take some time
 
Build complete. Run install-netconf-ned, then run 'packages reload' to use the packagened-directory /tmp/packages/ce0
JLINDBLA-M-W0J2(config-device-ce0)#
Built fine, good. Now install and load the package.
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang install-netconf-ned
Copying new package into /Users/jlindbla/nso/4.4.2/examples.ncs/service-provider/vvpn-xe/packages/ce0
success Installed -- now you need to: packages reload
JLINDBLA-M-W0J2(config-device-ce0)# exit
JLINDBLA-M-W0J2(config)# exit
JLINDBLA-M-W0J2# packages reload
>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has been cancelled.
Error: Failed to load ./state/packages-in-use/1/ce0/load-dir/Cisco-IOS-XE-route-map.fxs: The element /devices/template/config/interface/Loopback/tx-queue/bandwidth/unit (in namespace urn:ios) and the element /devices/device/config/native/route-map/route-map-seq/set/extcommunity/cost/community-id (in namespace http://cisco.com/ns/yang/Cisco-IOS-XE-route-map) have been assigned the same value. Use the YANG statement tailf:id-value to resolve the issue.
 

Oh, wow. (Those were Steve Jobs' final words) What's this?
 

This is a hash collision between two YANG symbols. This happens rarely, but as we are loading hundreds of YANGs with thousands of symbols, the risk increases. Ask me in the comments, and I'll post a full explanation. Tail-f engineering is working to automate the hash collision resolution a bit, but for now, we need to do this manually.
 
So the hash value for the symbols "unit" and "community-id" happens to be the same with the hash algorithm used by NSO, so we need to manually assign a different value for one of them. By some grepping in the models, I know "unit" is used in many places, and "community-id" very rarely. So I'll prefer to change community-id.

$ grep -rils community-id packages/*/src/yang

packages/ce0/src/yang/Cisco-IOS-XE-route-map.yang
$

 
Ok, so only a single file. Very good. So I look for all occurrences of community-id in those files (that file). There are two, and they're both next to each other:
 
    list community-id {
      key "community-id cost-value";
      leaf community-id {
        description
          "Community ID";
        type uint8 {
          range "0..255";
 
So what I want to do is annotate the YANG elements with a tailf:id-value. The value 21474 is a random value I picked out of thin air. It's like I picked a random UUID. I just hope it's not colliding with anything else. I would see the same error again if I made a bad choice, but with a different symbol than unit. My chances are pretty good, tho. I'd estimate the collision risk as 1 in 100,000 or so.
 
Editing /tmp/download/ce0/Cisco-IOS-XE-route-map.yang to look like this:
 
    list community-id {
      tailf:id-value 21474;
      key "community-id cost-value";
      leaf community-id {
        tailf:id-value 21474;
        description
          "Community ID";
        type uint8 {
          range "0..255";
 
We also need an import statement near the top of the file.
 
  import tailf-common { prefix tailf; }
 
So let's rebuild the NED again. Then install and load.
 
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang build-netconf-ned
 
Build complete. Run install-netconf-ned, then run 'packages reload' to use the packagened-directory /tmp/packages/ce0
JLINDBLA-M-W0J2(config-device-ce0)# pioneer yang install-netconf-ned
Old package exists, moving to /Users/jlindbla/nso/4.4.2/examples.ncs/service-provider/vvpn-xe/old-packages/ce0-20170825-153137
Copying new package into /Users/jlindbla/nso/4.4.2/examples.ncs/service-provider/vvpn-xe/packages/ce0
success Installed -- now you need to: packages reload
JLINDBLA-M-W0J2(config-device-ce0)# exit
JLINDBLA-M-W0J2(config)# exit
JLINDBLA-M-W0J2# packages reload
>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has been cancelled.
Error: Duplicate prefix "ios" used in 'http://cisco.com/ns/yang/Cisco-IOS-XE-native' and 'urn:ios'
 
Oh, wow, again. What's this now?
 
This is a prefix collision between two YANG modules. With so many YANG modules and some choosing quite short prefixes, this is bound to happen sooner or later. So obviously the Tail-f team is working on a fix to this. Right now, we have to intervene with another manual edit:
 
module Cisco-IOS-XE-native {
  namespace "http://cisco.com/ns/yang/Cisco-IOS-XE-native";
  prefix ios;
 
Editing /tmp/download/ce0/Cisco-IOS-XE-native.yang to look like this:
 
module Cisco-IOS-XE-native {
  namespace "http://cisco.com/ns/yang/Cisco-IOS-XE-native";
  prefix ios-native;

 
Redoing the build, install and reload (exactly as above) again. This time all packages load fine:
 

JLINDBLA-M-W0J2# packages reload 
>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.
reload-result {
    package ce0
    result true
}
reload-result {
    package cisco-ios

 
After loading, I have the habit of checking that all packages are ok. They are.
 

JLINDBLA-M-W0J2# JLINDBLA-M-W0J2# show packages package oper-status 
                                                                                        PACKAGE                
                         PROGRAM                                                        META     FILE          
                         CODE     JAVA           BAD NCS  PACKAGE  PACKAGE  CIRCULAR    DATA     LOAD   ERROR 
NAME                 UP  ERROR    UNINITIALIZED  VERSION  NAME     VERSION  DEPENDENCY  ERROR    ERROR  INFO   
---------------------------------------------------------------------------------------------------------------
ce0                  X   -        -              -        -        -        -           -        -      -      
cisco-ios            X   -        -              -        -        -        -           -        -      -      
cisco-iosxr          X   -        -              -        -        -        -           -        -      -      
esc                  X   -        -              -        -        -        -           -        -      -      
id-allocator         X   -        -              -        -        -        -           -        -      -      
ipaddress-allocator  X   -        -              -        -        -        -           -        -      -      
juniper-junos        X   -        -              -        -        -        -           -        -      -      
l3vpn                X   -        -              -        -        -        -           -        -      -      
l3vpnui              X   -        -              -        -        -        -           -        -      -      
pioneer              X   -        -              -        -        -        -           -        -      -      
resource-manager     X   -        -              -        -        -        -           -        -      -      
vm-manager           X   -        -              -        -        -        -           -        -      -      
vm-manager-esc       X   -        -              -        -        -        -           -        -      -      
weblog               X   -        -              -        -        -        -           -        -      -      

 
Let's see if we can sync-from, and display the device config?
 

JLINDBLA-M-W0J2# devices device ce0 sync-from 
result true
JLINDBLA-M-W0J2# show running-config devices device ce0 config ios-native:native interface GigabitEthernet 
devices device ce0
 config
  ios-native:native interface GigabitEthernet 1
   ip address primary address 192.168.50.50
   ip address primary mask 255.255.255.0
   mop disabled
   negotiation auto true
  !
  ios-native:native interface GigabitEthernet 2
   description "skoda local network"
   ip address primary address 10.0.0.1
   ip address primary mask 255.255.255.0
   mop disabled
   negotiation auto true
  !
  ios-native:native interface GigabitEthernet 3
   ip address primary address 10.0.4.15
   ip address primary mask 255.255.255.0
   mop disabled
   negotiation auto true
  !
 

Yay! Finally we got the NED layer to play along. Now we're ready to get started with the actual porting of the service.
You'll find the next part of the story here Porting an existing service to use a NETCONF NED, part 2
 

Porting an existing service to use a NETCONF NED, part 1

Get NSO for Free!

Collective Intelligence resolved my SSH problems with NSO on the latest macOS Mojave!

Secrets of the NSO Transaction Phases