Introduction
In our project, NSO interactive with devices by CLI NED, whatever is Cisco IOX or HuaWei VRP. Customer have a concern that is "out-of-sync" issue in their network. Because they couldn't avoid other management tools and only control devices by NSO now. There are more management tools to controler devices, as follow example that not include all:
- SNMP by NMS
- Telnet/SSH by NOC
- Cisco Security Manager for Firepower
- NSO
- Other Controller
Due to customer couldn't limit access devices, so that should possible configuration change before NSO deliver services, that will cause "out-of-sync". Do we have any solution for this scenario? For this article, will discuss the scenario and try to resolve the issue.
Solution
Netconf NED
Why discuss the Netconf NED at this scenario? That due to Netconf have "lock" function, and there is a session between orchestrator and devies, so similar function should be achieved. For "lock" function, can reference as follow link:
7.5. <lock>
Description: The <lock> operation allows the client to lock the
entire configuration datastore system of a device. Such locks are
intended to be short-lived and allow a client to make a change
without fear of interaction with other NETCONF clients, non-
NETCONF clients (e.g., SNMP and command line interface (CLI)
scripts), and human users.
Example in ASr9k
RP/0/RSP0/CPU0:ASR9001-A#show netconf-yang clients
Wed Jul 26 15:36:30.756 UTC
Netconf clients
client session ID| NC version| client connect time| last OP time| last OP type| <lock>|
3270008865 | 1.1| 0d 0h 0m 57s| 02:43:18| lock| Yes|
RP/0/RSP0/CPU0:ASR9001-A#
RP/0/RSP0/CPU0:ASR9001-A#config ter
Wed Jul 26 15:36:37.470 UTC
Current Configuration Session Line User Date Lock
00000001-00196153-0000000a netconf cisco Wed Jul 26 15:35:33 2017 *
RP/0/RSP0/CPU0:ASR9001-A(config)#
RP/0/RSP0/CPU0:ASR9001-A(config)#int lo345
RP/0/RSP0/CPU0:ASR9001-A(config-if)#ip add 3.3.3.3 255.255.255.255
RP/0/RSP0/CPU0:ASR9001-A(config-if)#commit
Wed Jul 26 15:36:53.246 UTC
% Failed to commit .. Another configuration session
had a lock on the running configuration.
If you still want to commit your changes,
you may try the 'commit' command again.
Attention for Netconf NED
Refer netconf ned that should only include yang module, no any logical function, so need build by ourselves. In future, NSO team will not release XR netconf ned too. You can check all released NED should CLI type, and juniper ned that is special, that not only include yang module, but have some other in NED.
No-out-of-sync-check
At some times, customer need commit service succeed firstly and ignore "out-of-sync" alarm temporary. That will have follow risk:
If the service should create a resource on a device, and the device config in NSO's CDB contains that resource, but the device config doesn't (config changed by other tools), then it will not be created.
So we can do "sync-from devices" before NSO deliver the service and "commit no-out-of-sync-check". That will reduce risk, but not 100%.
No-out-of-sync-check + No-overwrite
Our have another feature "no-overwrite" that released from 3.x:
NCS has a commit-flag that will restrict the device configuration to not overwrite data that NCS did not create: commit no-overwrite.
But acctually, I don't known how to single use the "on-overwrite"... but based on the test, no-overwrite should useful when it interactive with "no-out-of-sync-check", that will reduce out-of-sync risk. As follow example:
admin@ncs# config t
Entering configuration mode terminal
admin@ncs(config)# services loopback test loopback-intf 1100 ip-address 10.100.1.4 device 9001-A
admin@ncs(config-loopback-test)# commit
Aborted: Network Element Driver: device 9001-A: out of sync
admin@ncs(config-loopback-test)#
admin@ncs(config-loopback-test)# commit no-out-of-sync-check no-overwrite
Aborted: Network Element Driver: device 9001-A: out of sync
admin@ncs(config-loopback-test)#
Due to some special reason, customer have change the loopback 1100 address to 10.100.1.3 by other management tools, if customer direct use "no-out-of-sync-check" in NSO, the device configuration will be changed that maybe cause some issue in customer network. If add "no-overwrite", NSO will check if the out-of-sync configuration is in the service module range.
As above example, out-of-sync due to loopback1100's address changed, and this part configuration in services loopback. So out of sync and commit failed. In same scenario, if out-of-sync due to other configuration that not in service, commit will be succeed.
No-out-of-sync-check + No-overwrite + Partial sync from
From NSO 4.4.2's release notes:
The headline feature in NSO 4.4.2 is the ability to do partial sync-from. This is extremely useful in situations where there are multiple management systems and/or manual configuration going on. This enables NSO to synchronise only that subset of configuration that is touching, and this can speed up operations considerably.
When use the partial sync form?
As above example (loopback service), if customer said they frequency to change the address for lo1100 for testing by diff management tools, that will very useful for partial sync from. You can design default commit behavior by "No-out-of-sync-check + No-overwrite". For special requirement, you can add partial sync from.
In-Sync behavior in NSO?
Refer to NSO default behavior, check sync by "transaction ID" that is calculated after every commit in every device. And different NED maybe have different way to calculate transaction ID. As it is in CDB and as part of the commit asks the NED what the current transaction ID is. If they are different, the device config must have changed on the device compared to what is in CDB, and NSO says it is out of sync.
If config no-out-of-sync + no-overwrite,that shuold not check sync by "transaction ID". It just reads those config nodes, from the device that are to be written to, and compares the current values of those nodes. In other words, it doesn't check any values that are not written to.
Partial sync Limitation?
If you want to use the partial sync from, you must point "path" that is xpath in yang module, if no path, that should no any effect. How to config path? You can reference follow example.
Example for partial-sync-from
>>> direct change lo1100 address to 3.3.3.3/24 in the asr9k that cause out of sync.
admin@ncs# devices check-sync
sync-result {
device 9001-A
result out-of-sync
info got: 81ac7c9b4bf57c134495168cfc1f8c28 expected: 39ec9448dc61526921eb748d1152cbf4
}
admin@ncs# config
Entering configuration mode terminal
admin@ncs(config)# services loopback test loopback-intf 1100 ip-address 10.100.1.4 device 9001-A
admin@ncs(config-loopback-test)# commit no-out-of-sync-check no-overwrite
Aborted: Network Element Driver: device 9001-A: out of sync
admin@ncs(config-loopback-test)#
admin@ncs(config)# show full-configuration devices device 9001-A config cisco-ios-xr:interface Loopback | display xpathd
/devices/device[name='9001-A']/config/cisco-ios-xr:interface/Loopback[id='0']/ipv4/address/ip 1.1.1.1
/devices/device[name='9001-A']/config/cisco-ios-xr:interface/Loopback[id='0']/ipv4/address/mask 255.255.255.255
/devices/device[name='9001-A']/config/cisco-ios-xr:interface/Loopback[id='1100']/ipv4/address/ip 2.2.2.2
/devices/device[name='9001-A']/config/cisco-ios-xr:interface/Loopback[id='1100']/ipv4/address/mask 255.255.255.255
admin@ncs(config)#
admin@ncs(config)# devices partial-sync-from path [ /devices/device[name='9001-A']/config/cisco-ios-xr:interface/Loopback[id='1100']/ipv4/address/ ]
sync-result {
device 9001-A
result true
}
admin@ncs(config)#
admin@ncs(config)# commit no-out-of-sync-check no-overwrite
Commit complete.
admin@ncs(config)#
admin@ncs(config)# exit
admin@ncs# devices check-sync
sync-result {
device 9001-A
result unknown
}
There are another example, you can compare to check that:
ncs-netsim create-network ./cisco-iosxr 1 xr
ncs-setup --dest ncs-1
ncs-netsim start
(cd ncs-1; ncs -c ncs.conf)ncs-netsim cli-c xr0
config
enable secret 0 on-device-old
commit
exit
exitncs_cli -u admin -C
devices sync-from
exitncs-netsim cli-c xr0
config
enable secret 0 on-device-new
commit
exit
exitncs_cli -u admin -C
config
devices device xr0 config cisco-ios-xr:enable secret 0 on-nso
commit # <- this fails due to device being out-of-sync
commit no-out-of-sync-check no-overwrite # <- also fails since the device is out of sync on the written path
devices partial-sync-from path [ /devices/device[name='xr0']/config/enable ]
commit no-out-of-sync-check no-overwrite # <- thanks to the partial sync above, it show now succeed