advice on managing brownfield devices using both NSO and manual input

[jmsherry]

I am seeking advice on the best practice in managing configuration using both NSO and a third party system, where the third party system adds partial configuration under the container/context created by NSO.

Our final goal is to have NSO manage 100% of configuration inside a container. But we are not quite there yet, hence looking for best practice to deal with the two situations during the transition period.

The first problem I encounter:
1. after manually adding a configuration line to the container created by NSO service, the NSO service becomes out of sync. Upon re-deploy, NSO service tries to delete the manaully added line

for example:
NSO service prod-service-endpoint[prod-service-endpoint:id='ESE000204'] configure below two lines on a juniper router:

! Refcount: 4
! Backpointer: [ /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000204'] /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000206'] /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000208'] /ncs:services/prod-virtual-access-circuit:prod-virtual-access-circuit[prod-virtual-access-circuit:id='VAC000153'] ]
configuration interfaces interface xe-0/0/10
! Refcount: 1
! Backpointer: [ /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000204'] ]
unit 2819
! Refcount: 1
description "userservice_jqos EPP000126"
! Refcount: 1
vlan-id 2819

### It translates to the below router CLI output:

user@lab-test-pe1-re0> show configuration interfaces xe-0/0/10 unit 2819
description "userservice_jqos EPP000126";
vlan-id 2819;

### nso service is in sync at this point
user@nso-lab# services prod-service-endpoint ESE000204 check-sync
in-sync true
user@nso-lab# services prod-service-endpoint ESE000204 re-deploy dry-run
cli {
}

### manually add a new line via router CLI

user@lab-test-pe1-re0# set interfaces xe-0/0/10 unit 2819 family inet accounting source-class-usage input
user@lab-test-pe1-re0> show configuration interfaces xe-0/0/10 unit 2819
description "userservice_jqos EPP000126";
vlan-id 2819;
family inet {
accounting {
source-class-usage {
input;
}
}

### then sync-from to make NSO aware of the new line

user@nso-lab# devices device lab-test-pe1 sync-from
result true
user@nso-lab#

###
user@nso-lab# show running-config devices device lab-test-pe1 config configuration interfaces interface xe-0/0/10 unit 2819 | display service-meta-data
devices device lab-test-pe1
config
! Refcount: 4
! Backpointer: [ /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000204'] /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000206'] /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000208'] /ncs:services/prod-virtual-access-circuit:prod-virtual-access-circuit[prod-virtual-access-circuit:id='VAC000153'] ]
configuration interfaces interface xe-0/0/10
! Refcount: 1
! Backpointer: [ /ncs:services/prod-service-endpoint:prod-service-endpoint[prod-service-endpoint:id='ESE000204'] ]
unit 2819
! Refcount: 1
description "userservice_jqos EPP000126"
! Refcount: 1
vlan-id 2819
family inet accounting source-class-usage input
!

### nso service is now out of sync
user@nso-lab# services prod-service-endpoint ESE000204 check-sync
in-sync false

## nso service tries to delete the non-service data upon re-deploy
user@nso-lab# services prod-service-endpoint ESE000204 re-deploy dry-run
cli {
local-node {
data devices {
device lab-test-pe1 {
config {
configuration {
interfaces {
interface xe-0/0/10 {
unit 2819 {
family {
- inet {
- accounting {
- source-class-usage {
- input;
- }
- }
- }

## what is the best option to keep the nso service in sync and also keep the new line in NSO database?
one option I can think of:
* whenever a third party system touches the router, after a sync-from, issue services xxx re-deploy reconcile { keep-non-service-config }.
This can be hard to manage, there are lots of services to reconcile each time a third party system touches the configuration.
*

This leads to the second issue:
2. Now the nso service [prod-service-endpoint:id='ESE000204'] owns the container unit 2819.
When this nso service is deleted, NSO will delete vlan-id and description. But it is rejected by the router due to the line added manually becomes orphan and invalid line, as an interface with unit must have a vlan-id value.

One option to overcome both problem 1 and problem 2, is
if the third party system and NSO configure the same container, the third party system must configure the configuration on the router ahead of NSO. In the above example, we need to manually add "vlan-id" + "description" + "accounting". After that, a sync-from from NSO will suck in the configuration and gets a refcount 1.
Then we use NSO service to deploy the vlan-id and description. For these two lines, the NSO refcount increases to 2 after nso service re-deploy
When it is done in such sequence, NSO service will always in sync, provided the third party system does not change the vlan-id nor the description.
When NSO service is deleted, those two lines of configuration will be reverted back to its original value on the router rather than failed deletion. The drawback of this is that the router configuration never gets deleted post nso service deletion until human intervention to manually delete them from the router.

Is this how others deal with a brown field network where both NSO and third party system must co-exist to manage the configuration inside a same configuration container/context?
Our final goal is to have NSO manage 100% of configuration inside a container. But we are not quite there yet, hence looking for best practice to deal with the above two situations during the transition period.

[huayyang]

the 2 problems can be solved by the new brownfield service protection in 6.5:

starting with a service configuring 1 if:

% show devices device ex0 config sys interfaces interface eth1 | display service-meta-data
/* Refcount: 1 */
description service;
/* Refcount: 1 */
mtu 222;
/* Refcount: 1 */
/* Backpointer: [ /fms:fms[fms:name='test'] ] */
unit 1 {
/* Refcount: 1 */
description by-service;
/* Refcount: 1 */
vlan-id 2;
}

introduce oob on the netsim device:

admin@ex0%set sys interfaces interface eth1 unit 1 family inet address 1.2.4.5
[ok][2025-10-02 15:57:32]

[edit]
admin@ex0%commit
Commit complete.
[ok][2025-10-02 15:57:34]

[edit]
admin@ex0%show sys interfaces interface eth1
description service;
mtu 222;
unit 1 {
description by-service;
vlan-id 2;
family {
inet {
address 1.2.4.5;
}
}
}

with this out-of-band-policy:

admin@ncs% show services out-of-band
policy fms-servicepoint {
rule serv-win {
path .;
priority 2;
at-create manage-by-service;
at-delete manage-by-service;
at-value-set manage-by-service;
}
}

you can do 

admin@ncs% request fms test re-deploy confirm-network-state
[ok][2025-10-02 15:58:55]

[edit]
admin@ncs%
System message at 2025-10-02 15:58:55...
Commit performed by admin via console using cli.
admin@ncs% show devices device ex0 config sys interfaces interface eth1 | display service-meta-data
/* Refcount: 1 */
description service;
/* Refcount: 1 */
mtu 222;
/* Refcount: 1 */
/* Backpointer: [ /fms:fms[fms:name='test'] ] */
unit 1 {
/* Refcount: 1 */
description by-service;
/* Refcount: 1 */
vlan-id 2;
family {
inet {
/* Refcount: 1 */
/* Backpointer: [ /fms:fms[fms:name='test'] ] */
/* Out-of-band: [ /fms:fms[fms:name='test'] ] */
address 1.2.4.5;
}
}
}
[ok][2025-10-02 15:59:00]

[edit]

now the oob is a part of the service meta-data and will be lifecycled with this service, e.g. service update:

admin@ncs% set fms test device [ ex0 ] iface eth1 unit 1 vid 9
[ok][2025-10-02 16:10:05]

[edit]
admin@ncs% commit confirm-network-state

on device:

admin@ex0%
System message at 2025-10-02 16:10:12...
Commit performed by admin via ssh using netconf.
admin@ex0%show sys interfaces interface eth1
description updated-by-service;
mtu 222;
unit 1 {
description by-service;
vlan-id 9;
family {
inet {
address 1.2.4.5;
}
}
}
[ok][2025-10-02 16:10:15]

and for service deletion:

admin@ncs% commit
Commit complete.
[ok][2025-10-02 16:11:20]

on device:

admin@ex0%show sys interfaces interface eth1
----------------------------------------^
syntax error: element does not exist
[error][2025-10-02 16:11:22]

for more info:

https://nso-docs.cisco.com/guides/operation-and-usage/operations/out-of-band-interoperation

https://www.youtube.com/watch?v=BRxdQJntjZo&list=PLhTPrPcGzO7FNN228f4NOMZ84aYfYAlFi&index=41

[jmsherry]

Thanks Huayyang. I will read more and test the new feature in NSO 6.5. It seems a good solution to our problems. However, our environment is NSO6.3, I wonder how people manage these problems before NSO6.5? It will take time to move to NSO6.5 due to NED upgrade, NSO service template testing, and impact to production services etc. 

[huayyang]

there is no out-of-the-box support for you to manage brownfield, customers must have solutions tailored for the use cases.

the 2 problems in your use case requires contradicting solutions: 1 indicates you don't want the service to own the device config but 2 indicates you want the opposite(given you want the device config removed when service is deleted). so I don't think there's a good solution.

[snovello]

Actually without those 6.5 features there are no easy fixes, and a good recommendation is to decide which objects are managed by services and which are not managed, and so avoid having mixed configuration objects, 

When that recommendation cannot be followed, and it seems to be your case, you can do some specific code, so for the example you could in the post-modification, explicitly delete the unit when deleting the whole service. Here you will encounter the issue that the service has already been deleted when that post-modification callback is executed, but you can  store the relevant data -interface name and unit number - in the opaque parameter (proplist) which is still survives at that point.

It is ugly and only solves the specific issue you mentioned with deletion - that's why the new feature is so useful.