Out-of-sync of tailf:aes-256-cfb-128-encrypted-string leafs in LSA

[lweddewer]

Hi,

when using tailf:aes-256-cfb-128-encrypted-string leafs in a LSA enviroment a device sync-from cfs node always leads to an out-of-sync for the encrpyted leafs. Normal using of the services using such encrypted leafs works beside the out-of-sync issue.

Is this behavior expected?
What's the best practice for this?
I didn't to find anything regarding the combination of LSA and encrypted in the NSO documentations.  

Regards
Lothar

[cohult]

Hi, That will also happen if a device implements the tailf encrypted string type YANG extension. See the NETCONF NED Development documentation. You need to add the "tailf:ned-ignore-compare-config" YANG extension to the leaf, at least on the upper NSO/CFS node. Example:

leaf my-encrypted-password {
        tailf:ned-ignore-compare-config;
        type tailf:aes-256-cfb-128-encrypted-string;
}

How to create an annotation file for the"tailf:ned-ignore-compare-config" YANG extension from the NETCONF NED Development chapter:
https://cisco-tailf.gitbook.io/nso-docs/development/advanced-development/developing-neds/netconf-ned-development#adding-yang-annotation-files
Regards

[lweddewer]

The lsa-neds are generated by the tools coming with NSO. 
Having a deaper look into it I can confirm that the tailf:ned-ignore-compare-config is automaticly inserted for enrypted leafs when generating the ned. During my tests I never seen a lsa-ned itself gone out-of-sync. It is the service feeding the lsa-ned that  results a in-sync false after a device sync-from.
A re-deploy on the service updates the encrypted leafs in ned and the related service in the rfs-node but no not any further service or ned in the rfs-node. So the overall behavior seams to be fine. 
During my testing I've done a check-sync outformat native. 
To be honest I was really surprised to see the content of the content of the encrypted leafs in cleartext in the output.

native {
    device {
        name rfs_piu_nso
        data <rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
                   message-id="1">
                <edit-config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
                  <target>
                    <running/>
                  </target>
                  <test-option>test-then-set</test-option>
                  <error-option>rollback-on-error</error-option>
                  <with-inactive xmlns="http://tail-f.com/ns/netconf/inactive/1.0"/>
                  <config>
                    <rfs-generic-device-actions xmlns="http://dt.com/rfs-generic-device-actions">
                      <presets>
                        <authgroup>
                          <name>netsim</name>
                          <password>*****</password>
                          <private>
                            <re-deploy-counter>-1787450893</re-deploy-counter>
                          </private>
                        </authgroup>
                      </presets>
                    </rfs-generic-device-actions>
                  </config>
                </edit-config>
              </rpc>

    }
}

The password leaf is the encrypted leaf and I replacing the cleartext password by ***** manualy.
Didn't expect to see the cleartext by using a simple build in action!
Is this really the result we should see?

[cohult]

When sending, for example, a tailf:aes-256-cfb-128-encrypted-string to a device over NETCONF, here, another NSO instance, the string is decrypted by NSO and sent to the device. The device then immediately encrypts it before storing it in its database. The decrypted string is sent from the CFS upper NSO to the RFS lower NSO over an authenticated and encrypted SSH link. 
The string is decrypted before it is sent because NSO and the device, here CFS NSO and RFS NSO, should not share init vectors for security reasons. Thus, the lower RFS NSO cannot decrypt the string if CFS sends it to the string it encrypted.

[lweddewer]

I'm not shure if that is fully correct. When running a device sync-to dry-run { outformat native } I also see encrypted leafs with encrpyted strings:

native <rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
             message-id="2">
          <edit-config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
            <target>
              <running/>
            </target>
            <test-option>test-then-set</test-option>
            <error-option>rollback-on-error</error-option>
            <with-inactive xmlns="http://tail-f.com/ns/netconf/inactive/1.0"/>
            <config>
              <devices xmlns="http://tail-f.com/ns/ncs">
                <authgroups>
                  <group>
                    <name>lab</name>
                    <default-map>
                      <remote-password>$9$pBiIKUTSROfujss0x/NmCrvofCrHWTYvvJEr8mdgcoc=</remote-password>
                    </default-map>
                  </group>
 .....
                </authgroups>
              </devices>
              <rfs-generic-device-actions xmlns="http://dt.com/rfs-generic-device-actions">
                <presets>
                  <authgroup>
                    <name>netsim</name>
                    <password>**cleartest**</password>
                  </authgroup>

When looking in the originial yang file there is no different on the leaf level:

Yang in cisco-nso-nc-6.1 ned:

                leaf remote-password {
                  tailf:info "Specify the remote password";
                  tailf:suppress-echo "true";
                  type tailf:aes-256-cfb-128-encrypted-string;
                  description
                    "Remote password.";
                }

Our own yang:

        leaf password {
          type tailf:aes-256-cfb-128-encrypted-string;
          tailf:suppress-echo "true";
          mandatory true;
        }

In both cases the tailf:ned-ignore-compare-config is added during the make of the ned on the cfs-node and can be find in the autogenerated files in the ncsc-out folder.
May be there is a missing flag in the Makefile or something else. 
Any hint would be fine.

[cohult]

Regarding "a device sync-from cfs node always leads to an out-of-sync for the encrypted leafs", but "tailf:ned-ignore-compare-config is added during the make of the ned on the cfs-node".

Then there is a build problem. The built FXS files used by the CFS node are not built using "tailf:ned-ignore-compare-config" on the encrypted leafs. 

[lweddewer]

I'm not sure if that is the right conclusion because both parts are processed with the same tools provided by Cisco coming with the NSO software. I just doublecheck building the cisco-nso-nc-6.1 ned new by using make clean all. Result is that the behavior is unchanged: encrypted belonging the the cisco-nso-nc-6.1 ned are transfered in there encrypted form. Encrypted strings in the other neds build for the rfs-services are transfered as uncrypted strings.
The question is still open if this is expected or it's as failure.

[beredis802]

The behavior you observe, where tailf:aes-256-cfb-128-encrypted-string sheets cause synchronization problems in an LSA environment, is most likely due to NSO not being able to correctly detect changes to encrypted data because it is treated as binary content. This behavior is not expected, but it can occur when encrypted data is involved in synchronization operations. As for react vs react native, I've heard this more than once, it will be interesting to read. It is best to avoid direct synchronization of encrypted sheets and instead manage the decrypted values before re-encrypting them on the target device. In addition, consider implementing custom synchronization logic for encrypted data and ensure consistent encryption and decryption on different nodes.

[lweddewer]

To be honest I hoped for a better support of encrypted strings in a NSO and specialy LSA environment.

Every time your writing the same cleartext string into an encrypted leaf type tailf:aes-256-cfb-128-encrypted-string you gets a new content due to the random part used for the encryption. 
You can preventing the rfs-neds getting out-of-sync by using  tailf:ned-ignore-compare-config.
But you'll get all upper services in in-sync state false whenever you are using sync-from on the rfs-node device.
And if the seen dry-run output is correct that parts of the encrypted strings are actually send as uncrypted strings we are using both methods. But both don't work as expected.
If you are using a random salt for encrypting strings comparing the encrypted strings will only given very limit informations regarding the original string.

[cohult]

I think we are getting to the core of the issue here. You are not supposed to change the device configuration from the CFS NSO node. The CFS NSO node only mounts the RFS from the RFS NSO nodes under the CFS NSO nodes /devices tree.
You seem to access more than that, which is not supported. You get out of sync when the CFS sends the encrypted remote-password to the RFS NSO node, as you are not supposed to access that configuration on the RFS NSO node from the CFS NSO node. You are only supposed to access the Resource Facing Service configuration. 

After compiling the $NCS_DIR/packages/lsa/cisco-nso-nc-6.x/src/ncsc-out/modules/yang/tailf-ncs-devices.yang module, you will notice how the NSO compiler has, among other things, changed the remote-password leaf type from 

tailf:aes-256-cfb-128-encrypted-string to string. Hence, that configuration can be read from the remote RFS NSO node by the CFS NSO node, but you are not supposed to change it.

[lweddewer]

There is no plan to change configurations of devices onboarded to an RFS-Node directly via the representation in the cisco-nc-6.x ned. Due to the fact that only the common device features are part of the rfs-ned it's not possible in most cases.
We currently only plan to use the generic actions (check-sync, connect, ...) which all works fine so far.
But when using a rfs based action to onboard the device to the rfs-node this part of the cisco-nc-6.1 ned will be out of sync untill the next rfs-node device sync-from on the cfs-node.

But if you do a sync-from on the rfs-node device on the cfs-node every encrypted string in any service providing at least one will be not in-sync even the sync-state of all rfs-neds are fine. I guess the reason is the NSO is not able to check that the content of both encrypted strings are the same.
This may be the expected behavior but I couldn'f find anything regarding this in any documentation.

The second point is that it is possible to see any content of any encrypted string by simple using a standard action which shouldn't happen. Meanwhile I've seen this unexpected behavior in two mostly independed projects in our company. Both us the standard tooling provided by CISCO for building rfs-neds for LSA.

[cohult · Accepted solution]

Regarding the documentation, I agree it can be improved to explain when the CFS node changes tailf:aes-256-cfb-128-encrypted-string to string type for non-RFS config encrypted leafs, and how tailf:ned-ignore-compare-config is used based on what I wrote earlier in this thread:

When sending, for example, a tailf:aes-256-cfb-128-encrypted-string to a device over NETCONF, here, another NSO instance, the string is decrypted by NSO and sent to the device. The device then immediately encrypts it before storing it in its database. The decrypted string is sent from the CFS upper NSO to the RFS lower NSO over an authenticated and encrypted SSH link. 
The string is decrypted before it is sent because NSO and the device, here CFS NSO and RFS NSO, should not share init vectors for security reasons. Thus, the lower RFS NSO cannot decrypt the string if CFS sends it to the string it encrypted.

---

I assume your second point refers to the device NETCONF trace log containing the decrypted string?

[lweddewer]

Regarding your answer. 
That's not the point. The tailf:ned-ignore-compare-config stuff is done automatic by the LSA tooling provided by Cisco. And that's seems to work as expected. But if this is the expected behavior you have two options I see:

1. Using sync-from on the rfs-node instances on cfs-node having the device instance in sync but all service not in sync which are providing encrypted strings
2. Not using sync-from on the rfs-node instances on cfs-node so handling of the encrypted strings is fine but all config data is unreliable in the rfs-node instances. 

And from your answers I guess that there is currently no final recommendation how to deal with that!?

The second point is a little bit different. I didn't looked into the logs so I can't say if the the clear text is also presented there.
I was talking about the direct output from a commit dry-run outformat native or a re-deploy dry-run { outformat native } on the cfs-node. In both cases the clear text is part of the normal output of the command.

[cohult]

I am trying to reproduce your issue without success. What you describe is not expected behavior. You should not get out of sync If a CFS configures an RFS that configures device(s). Please describe in detail:
- Where your encrypted leafs are located. Device configuration? RFS node service configuration? CFS node service configuration?
- Describe what the templates and/or code for service mapping are doing.
- What commands you are running on which node to get out of sync.