Reconciling out-of-band changes with NSO Service Configuration

[Dan.Sullivan]

As an NSO Software Solution Architect, I've spent quite a bit of time working on NSO deployments with service provider customers. One challenge that seems to pop up over and over again has been handling out-of-band configuration changes i.e changes made to network devices outside the purview of NSO. Of course the problem becomes even more complicated if an operator actually changes the service intent directly on the network devices and the changes isn't recorded within an NSO service configuration. In this case, the network has the correct configuration while NSO no longer reflects the correct service intent. The purpose of this blog post is to determine how we can effectively reconcile the NSO service intent with device changes that occur out-of-band.

I've seen a few solutions attempted to "reconcile" service intent with network changes and often times the solutions consist of a lot of messy, difficult to maintain code. Customers soon see that the cost of maintaining this reconciliation code out weighs the benefits of being able to actually handle these out-of-band changes. So what do many customers wind up doing? They in fact deploy services using NSO but don't actually use NSO to modify services after the initial activation. How can we solve this issue without developing a lot of difficult to maintain java/Python code? Is it possible to utilize NSO's capabilities to help with this issue? This issue becomes a bit more complicated when LSA services are in use.

First off, in many cases service provider network operators are making very limited changes to existing services. For example, increasing/decreasing bandwidth associated with a VPN, changing a VLAN value, etc. If you can identify a few common changes its possible to handle most of the out-of-band changes. For these cases, we can utilize the NSO template engine to help solve this issue without a lot of complicated service specific code. How exactly can this be done?

First, a reconcile action will need to be defined and support for each service type. Both the top level LSA service instance and the device node LSA service will need a similarly defined action. Consider the following example in which after the definition of the top level service I simply insert a reconcile action:

           list myservice{

                    key "name";

                    container metro {

                        list devices {

                            leaf device {

                                type string;

                            }

                            leaf int {

                                type string;

                            }

                           leaf description {

                              tailf:info "UNI base interface description";

                              type string;

                          }

                          leaf bandwidth {

                             type uint64 {

                                 tailf:info "Total interface bandwidth";

                                 range 1000..100000000000;

                            }

                          }

                          leaf service-instance {

                             type uint16 {

                                tailf:info "Interface service instance number [1..4000]";

                                range 1..4000;

                            }

                        }

                        leaf evc-id {

                           tailf:info "Customers EVC-ID - policy name";

                           type string;

                       }

                       leaf service-bandwidth {

                          tailf:info "rate set in the policer for the evc";

                          type uint32;

                       }

                       leaf service-burst {

                           tailf:info "burst rate (based on service-bandwidth)";

                           type uint32;

                       }

                       leaf p-bit {

                          tailf:info "set-cos-transmit (p-bit) value for the policer";

                           type uint32;

                       }

                      leaf qos {

                          tailf:info "qos-transmit value for the policer";

                        type uint32;

                        default 1;

                      }

                      list c-vlans {

                          key c-vlan;

                          leaf c-vlan {

                             tailf:info "Customer VLAN (55) (55-70) for ranges enter each VLAN separately";

                                type string;

                          }     

                      }

                      leaf s-vlan {

                          type uint16 {

                              tailf:info "s-vlan [0..4094]";

                              range 0..4094;

                          }

                      }

                  } // devices

              } // metro

              tailf:action reconcile {                           <== Reconcile action definition

                    tailf:actionpoint sn-reconcile-point;

                    input {

                     }

                     output {

                          leaf status {

                             type string;

                          }

                          leaf error-message {

                            type string;

                      }

                  }

           } // service

You will need an identical action defined for the lower layer LSA service on the device node and of course that action will also be part of the LSA device NED so it can be called from the top level service node. The action definition is fairly generic.

For example consider the following:At a high level the steps to reconcile an LSA service instance are the following:

Top level LSA service node reconcile action:

1. Invoke the per LSA device reconcile actions for all device nodes on which the service is currently deployed
2. Issue sync-from actions to all LSA device nodes in the previous step
3. Reconcile top level service with configuration changes from all LSA nodes
4. Commit changes for top level reconcile action

On each LSA device node:

1. Apply service specific reconcile template
2. Execute a dry-run to see if service is any changes to the service intent will cause the service to be out-of-sync
3. If the dry-run succeeds, execute a re-deploy dry-run and check for any unreconcilable changes

So you'll see that only the actions in red are actually unique per service type. The majority of the action is not service dependent.

Lets examine Step(3) for the Top Level service node reconcile action. The purpose of step 3 is to insure any changes made to the lower LSA service are reflected in the upper level service configuration. In fact, this step is the exact opposite of what the typical top level LSA service instance dispatch operation.

Consider the following snippet from a template used to dispatch the top level service basically determining which of the LSA nodes require the service be configured :

The template will look up the remote node and then use the lsa-<remote-node> as the LSA node name and copy the service configuration only to those LSA nodes which manages the device.

   <device foreach="{/metro/devices}">

      <name>{concat('lsa-', deref(device)/../remote-node)}</name>

      <config>

        <myservice xmlns="http://example/conn-lsa">

          <name>{string(/name)}</name>

          <metro>

            <devices>

              <device>{device}</device>

              <int>{int}</int>

              <description>{description}</description>

              <bandwidth>{bandwidth}</bandwidth>

              <service-instance>{service-instance}</service-instance>

              <evc-id>{evc-id}</evc-id>

              <service-bandwidth>{service-bandwidth}</service-bandwidth>

              <service-burst>{service-burst}</service-burst>

              <p-bit>{p-bit}</p-bit>

              <qos>{qos}</qos>

              <c-vlans>

                <c-vlan>{c-vlans/c-vlan}</c-vlan>

              </c-vlans>

              <s-vlan>{s-vlan}</s-vlan>

            </devices>

          </metro>

        </myservice>

      </config>

    </device>

Now the template used to reconcile the LSA node service configuration with the top level service configuration is simply the reverse of the above which would be the following:

    <services xmlns="http://tail-f.com/ns/ncs">

       <myservice xmlns="http://example/conn">

           <name>{$NAME}</name>

           <metro>

                <devices tags="delete">

                <device>{../../ncs:devices/device[name=$LSAN]/config/myservice[name=$NAME]/metro/devices/device}</device>

                <device>{../../ncs:devices/device[name=$LSAN]/config/myservice[name=$NAME]/metro/devices/device}</device>

                <int>{int}</int>

                <description>{description}</description>

                <bandwidth>{bandwidth}</bandwidth>

                <service-instance>{service-instance}</service-instance>

                <evc-id>{evc-id}</evc-id>

                <service-bandwidth>{service-bandwidth}</service-bandwidth>

                <service-burst>{service-burst}</service-burst>

                <p-bit>{p-bit}</p-bit>

                <qos>{qos}</qos>

                <c-vlans>

                    <c-vlan>{c-vlans/c-vlan}</c-vlan>

                </c-vlans>

                <s-vlan>{s-vlan}</s-vlan>

           </devices>

        </metro>

    </myservice>

</services>

Now for each of the LSA nodes the reconcile action will be needed to actually pull the modified configuration from the device configuration in the CDB and then update the service configuration on the LSA node.

Since most operators change only the bandwidth, service burst, or c-vlans we will only attempt to reconcile those in the following example. Any other changes to the service attempt will not be reconcilable and the operator will have to determine how to handle those conditions manually.

For the above example, the reconcile template might look like the following the color coding below is intended to show where each parameter is handled by the template:

<config xmlns="http://tail-f.com/ns/config/1.0">

  <service xmlns="http://example/conn-lsa">

    <name tags="nocreate">{/name}</name>

    <metro>

      <devices foreach="{/metro/devices}">

        <device>{device}</device>

         <service-bandwidth>{string(../../../ncs:devices/device[name=current()/device]/config/ios:policy-map[name=current()/evc-id]/class-default/class[name='class-default']/police-policy-map/police/cir)}</service-bandwidth>

        <service-burst>{string(../../../ncs:devices/device[name=current()/device]/config/ios:policy-map[name=current()/evc-id]/class-default/class[name='class-default']/police-policy-map/police/bc)}</service-burst>

        <c-vlans tags="delete" when="{starts-with(current()/int, 'GigabitEthernet')}">

          <c-vlan>{c-vlans/c-vlan}</c-vlan>

        </c-vlans>

        <c-vlans when="{starts-with(current()/int, 'GigabitEthernet')}">

          <c-vlan>{../../../ncs:devices/device[name=current()/device]/config/ios:interface/GigabitEthernet[name=substring-after(current()/int, 'GigabitEthernet')]/service/instance[id=current()/service-instance]/encapsulation/dot1q/id}</c-vlan>

        </c-vlans>

        <c-vlans tags="delete" when="{starts-with(current()/int, 'TenGigabitEthernet')}">

          <c-vlan>{c-vlans/c-vlan}</c-vlan>

        </c-vlans>

        <c-vlans when="{starts-with(current()/int, 'TenGigabitEthernet')}">

          <c-vlan>{../../../ncs:devices/device[name=current()/device]/config/ios:interface/TenGigabitEthernet[name=substring-after(current()/int, 'TenGigabitEthernet')]/service/instance[id=current()/service-instance]/encapsulation/dot1q/id}</c-vlan>

        </c-vlans>

      </devices>

   </metro>

</service>

</config>

In summation,  you can easily build a reconcile solution with a few templates to allow for the service intent to be update based on out-of-band changes if needed. Hopefully this helps clarify the tasks you might need to undertake a solution like I've outlined above. Additionally, I've attached the two reconcile actions I developed and you can use them as a guideline when building your own. In the interest of brevity, I didn't include the LSA device node model or NED but of course your action has to be defined in the LSA NED for this solution to work.

-Dan

ReconcileActionSn.py.zip

ReconcileActionDn.py.zip

[rogaglia]

excellent

[Dan.Sullivan]

Roque, Thanks for taking a look!

-Dan

[gschudel]

really nice Dan - thanks

[ZAhmad04890]

Hi Dan,

Thanks for sharing, I have a question regarding developing python code, where I am using YANG service model and XML vendor templates and calling these service variables from(YANG and XML) through my python code. The package runs without any issues, However if I load the service.py file into IPython , I cannot print the values of service variables i.e YANG+XML, but just I can print the root variables lint root.connection  or device.

My question is is there a way that I can print the values of  service variables by loading the python code in iPython  editor? or there is another way which is used by developers?