NSO - Reboot IOS-XE device leads to unsync configuration

LoicChabert27318 · ‎05-12-2022

Hi,

When i commit configuration from NSO, config is applied on IOS-XE device. Next, i try to reload the router.

First of all, the router ask to save configuration to startup config. Is it normal, after NSO commit, configuration is not saved to startup config ?

During my test scenario, i not confirm and reload the router without saving to startup (to simulate an electrical power issue).

When router is UP and confd/netconf server running, i try to check if configuration is sync, but i have the following output:

noc@ncs> request devices device ar01-vpn.mar92 sync-from dry-run { outformat cli }
cli
[ok][2022-05-12 16:52:37]
noc@ncs> request devices device ar01-vpn.mar92 check-sync
result out-of-sync
info got: 1652-367116-179638 expected: 1652-366528-305780

[ok][2022-05-12 16:52:43]
noc@ncs> *** ALARM out-of-sync: got: 1652-367116-179638 expected: 1652-366528-305780

With router uptime:

ar01-vpn.mar92 uptime is 6 minutes

Router version: Cisco CSR1000v 17.3.3

Best regards.

Alex Stevenson · ‎05-24-2022

Hello @LoicChabert27318,

I'm not an NSO expert but I thought you might want to try recreating the issue on one of our DevNet sandboxes. We have NSO, as well as CSR1000v available ---> https://devnetsandbox.cisco.com/RM/Topology

You might also want to check out the Cisco Bug Search Tool --> https://bst.cloudapps.cisco.com/bugsearch/

Hope this helps!

Nabsch · ‎05-24-2022

Hello,

Can you specify NSO version and also NED Name/version ?

ygorelik · ‎05-24-2022

I am not an expert in NSO, but think that this is expected behavior for this specific use case.

First. When configuration change is committed, it is not get written to the startup datastore automatically, you need to do that manually when shutting down or rebooting the device.

Second. The check-sync does not do actual comparison of the current device configuration with CDB image. It simply checks matching of commit ID on the device and in CDB. The output of the command shows that they are different, which is expected, because you did not save the configuration to startup datastore, hence the last committed ID on the router got lost. You will need to do a sync-to operation to write last committed changes to the device.

Yan

LoicChabert27318 · ‎05-27-2022

Hi,

I have also get in touch with NSO support teams, and they give me the same answer. The statup datastore should be commited with an external action inside the orchestrator.

I'm confused about this behaviour, because, on some devices, netconf commit save it as statupconfig too (ios-xr for example). I have also some NXOS devices, using NXOS-CLI ned, and no running to startup config is mandatory (probably hard coded on NXOS-CLI ned).

Maybe it's because IOS-XE does not have any candidate/commit native functions (Old IOS behaviour) ?

If some of you have some experience on how netconf works on Cisco IOS-XE, maybe some words/comments could be added below ? I'm also curious how cisco make it works on SDWAN/SD-Fabric solutions, running with IOS-XE devices. As far i read on documentation, Cisco DNA center send configuration through netconf on IOS-XE devices.

Regards.

rogaglia · ‎08-05-2022

Let me clarify some points.

NSO keeps for every device a "transaction-id" that is calculated every time it communicates with devices. When you perform a "check-sync", you communicate with the device and re-calculate the "transaction-id" to check if it matches what NSO had in CDB. That is why you get this sort of messages:

*** ALARM out-of-sync: got: 1652-367116-179638 expected: 1652-366528-305780

it basically means that check-sync get a transaction-id "1652-367116-179638" but the last time it stored in CDB "1652-366528-305780".

So, how are transaction-ids calculated? We would hope that there is a universal way to do so and that devices will calculate it themselves. Actually, there is a draft in the IETF to get just this: https://www.ietf.org/archive/id/draft-lindblad-netconf-transaction-id-02.html

Now, the current reality is that every NED has a different way to calculate the transaction IDs, depending on the device API/CLI. Moreover, some devices have more than one way to do it and it is up to you to configure the one works for you with different "ned-settings".

All the info is included in the NED README file. For Example, for IOS-XE, you have 6 different methods (see below) as some may or may not work depending on platforms and OS versions. The default method is the config-hash which is a sort of comparison of the full config (contradicting the previous comment). Now, in large configurations, the "config-hash" may be slow and therefore you may want to use the other methods. It is true that in other devices like IOS-XR or Juniper you have support for commits and a transaction-id is basically the hash of the time of the last commit.

```

6.2. ned-settings cisco-ios read
--------------------------------

This section lists the read ned-settings used with sync-from and
check-sync:

- read transaction-id-method <enum>

The method to use by the NED for calculating transaction ID is a
configurable option. The default method is quite slow since it
uses output of show running-config and a software calculated
MD5. The advantage though is that it does not change even if the
device reboots. Another advantage is that it works on all
platforms. If you do not care about the transaction id changing if
the device reboots you may increase performance significantly by
changing the method the transaction id is calculated.

Five different methods are supported:

config-hash - Calculate MD5 on a snapshot of the entire running
config for calculation. (Default)

last-config-change - Use the 'Last configuration change' timestamp
in running config only (WARNING: changed at reboot)

config-id - Use the 'show configuration id' command to determine
if config changed. (WARNING: changed at reboot)

config-history - Use the 'show configuration history' command to
determine if config changed. (WARNING: changed at reboot)

config-hash-cached - Same as config-hash except that transaction
id method will reuse the last unmodified
config from show (sync-from).

config-hash-modeled - Same as config-hash except transaction id is
only calculated on modeled config.

Note: 'show config id|history' is not supported on some platforms,
e.g. 3550, cat4500, cat6500 etc. But if the option is not supported,
you will get to know this by use of an exception.

```

rogaglia · ‎08-05-2022

Just to complement this long thread, there is another ned-setting that sets the "write memory (or copy to startup)" NSO behavior. It is also described in the README file:

```

6.3. ned-settings cisco-ios write
---------------------------------

This section lists the write ned-settings used with commit:

- write memory-setting <enum>

This ned-setting is used to configure when/if an applied
configuration is saved to the device. Three values are supported:

on-commit - Save configuration immediately after the config has been
successfully applied on the device. If an error occurs when
saving the whole running config will be rolled back (default).

on-persist - Save configuration during the NED persist handler. Called
after the config has been successfully applied and commited
If an error occurs when saving an alarm will be triggered.
No rollback of the running config is done.

disabled - Disable saving the applied config to persistent memory.

Default: on-commit

```