cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
1126
Views
5
Helpful
6
Replies
jmaruschak
Beginner

Differences Between NSO CDB and Device Running Config Due to Device Interpretation

Greetings,

 

We've run into a number of examples of differences appearing between the NSO CDB and the running config on certain devices. This appears to be caused by the device accepting a certain syntax, but storing a different but equivalent syntax in its running config. We see this most frequently with Cisco IOS-XR devices such as the ASR9K.

 

Services are sometimes brought out of sync by this behavior. It also causes non-obvious issues for users attempting to modify the affected configuration in some cases. Is this expected behavior, or should we be in touch with Cisco to have this corrected? If it's expected, does anyone have tips or best practices for dealing with this?

 

For Cisco IOS-XR especially, the device does not immediately appear to be out of sync. I believe the way the XR NED performs a check-sync is by comparing the commit ID from the device to the commit ID NSO received during its last push to the device. Naturally, if the last change to the device configuration was made through NSO, these IDs would match and the device would show in-sync. A compare-config command would reveal the difference, however.

 

A couple of examples we've seen recently:

 

A problem with the subnet mask. The user told NSO to push "mask /31", but the device added "255.255.255.254" to the actual running config:

             cisco-ios-xr:interface {
                 HundredGigE 0/4/0/1 {
                     ipv4 {
                        address {
-                            mask /31;
+                            mask 255.255.255.254;
                         }
                     }
                 }

The version stored in the CDB is visible in the commit dry-run outformat native output, but this appears to have been transformed by the device itself:

ncs(config)# commit dry-run outformat native
native {
    device {
        name -----
        data interface HundredGigE 0/4/0/1
              ipv4 address 172.16.10.8/31
             exit

 

A problem with a certain 100G interface name. The user told NSO to push "Hu0/4/0/1" but the device added "HundredGigE0/4/0/1" to the running config. This caused issues for another user, who read the configuration on the device itself and attempted to push a modified configuration through NSO. NSO behaved as though it was adding the second user's config and not modifying the first user's config.

             cisco-ios-xr:mpls {
                 ldp {
-                    interface Hu0/4/0/1 {
-                    }
+                    interface HundredGigE0/4/0/1 {
+                    }

Again, the version from the CDB was passed along to the device, but the device seems to have made its own interpretation:

ncs(config)# commit dry-run outformat native
native {
    device {
        name -----
        data mpls ldp
              interface Hu0/4/0/1
              exit
             exit
    }

Again, more or less I'm just looking for feedback from the community if this is something that I should work with TAC on or if we expect it to behave this way.

1 ACCEPTED SOLUTION

Accepted Solutions
vleijon
Cisco Employee

Hi,

This is an interesting, and slightly complicated, question and I'll try to give a decent answer. As you noticed, a device does certain kinds of normalization in the CLI, which means that the input you give to NSO will not always match what is actually on the device.

 

Now, since we want NSO to reflect your input we don't expect NSO to modify its configuration unless you issue an explicit sync-from. An even more complicated example is auto-encrypted passwords, what should NSO do when a password is plaintext in NSO but automatically encrypted on the device? (The answer is what is often called "secret handling", where NSO keeps track of the encrypted version of the passwords in a shadow tree.)

 

So what we do? We add normalization of various kinds to the check-sync process. If you have the NED you can unpack it and take a look at how the sausage is made, look at the method modifyInput in cisco-iosxr-cli-7.18/src/java/src/com/tailf/packages/ned/iosxr/IosxrNedCli.java to see the modifications we make to try to get some normalization in the check sync process.

 

Overall, it is a tricky situation where we are forced to make trade-offs when devices have these kinds of ambigious/auto-normalizing behaviour. My general recommendation is to first try and live with it, if you use services or templates you should try to use the canonical names for everything to avoid the problems, but in cases where is not possible or where it becomes unreasonable I recommend reaching out to Cisco - there are already some mechanisms to support you and the NED developers might be able to either give you additional tips or make certain improvements for you, but it is hard to find a "perfect" behaviour.

View solution in original post

6 REPLIES 6
vleijon
Cisco Employee

Hi,

This is an interesting, and slightly complicated, question and I'll try to give a decent answer. As you noticed, a device does certain kinds of normalization in the CLI, which means that the input you give to NSO will not always match what is actually on the device.

 

Now, since we want NSO to reflect your input we don't expect NSO to modify its configuration unless you issue an explicit sync-from. An even more complicated example is auto-encrypted passwords, what should NSO do when a password is plaintext in NSO but automatically encrypted on the device? (The answer is what is often called "secret handling", where NSO keeps track of the encrypted version of the passwords in a shadow tree.)

 

So what we do? We add normalization of various kinds to the check-sync process. If you have the NED you can unpack it and take a look at how the sausage is made, look at the method modifyInput in cisco-iosxr-cli-7.18/src/java/src/com/tailf/packages/ned/iosxr/IosxrNedCli.java to see the modifications we make to try to get some normalization in the check sync process.

 

Overall, it is a tricky situation where we are forced to make trade-offs when devices have these kinds of ambigious/auto-normalizing behaviour. My general recommendation is to first try and live with it, if you use services or templates you should try to use the canonical names for everything to avoid the problems, but in cases where is not possible or where it becomes unreasonable I recommend reaching out to Cisco - there are already some mechanisms to support you and the NED developers might be able to either give you additional tips or make certain improvements for you, but it is hard to find a "perfect" behaviour.

View solution in original post

Interesting.

What about an opposite situation like this: password must be passed to the device as plain text but should not be kept as such in the NSO. It is easy to encrypt it in the service config, however it has to be decrypted before pushed to the device, which means that device manager will store plain text config in the CDB (which is not desired). Is there any way around that?

You can encrypt in the CDB using the same hash mechanism used by the device and send the password encrypted to the device.

In python you may use passlib.hash as it contains several hash options used by Cisco devices.

You may do the same for other vendors.

 

HTH, Gustavo

Password (it is not really a password but for the sake of discussion I will keep it simple) has to be sent to the device in clear text since that is the only format the device will accept it. On the other hand the same password must not be clear text throughout the CDB. It is simple to do it in the service model, but when it comes to the device configuration, the parameter has to be decrypted and pushed into the template to get the device config. Device config gets pushed with clear text config  but NSO also stores it in the CDB under devices device XYZ config... and that is the problem.

Could you please provide a clear example of what you are trying to achieve and preferably open a new thread on this? (so we don't hijack this one).

I have implemented what I have described to manage local passwords on ASA devices and it works. Maybe working through examples would be a better way to understand what the problem is for you.

 

Cheers

Andre Gustavo Albuquerque
Cisco Employee

Hi, first thing is to make sure you are using the latest NED version for the IOS-XR.

If doing so, you should open a TAC case to have it fixed. The NSO config should reflect device config and there shouldn't be such issues.

 

Cheers