Maintaining a device “golden” config. or its compliancy over time.

ron.seguin · ‎03-13-2018

Using service package(s) to build various components of a network element in order to achieve a “golden” config, I’d like to get the communities thoughts on the best practices on maintaining this “golden” config. over time.

Referring to the Juniper snippet below; this service package pushes both a host-name & domain-name and assigns the associated refcounters. Because there are no associated refcounters or backpointers for <system>, I found that configuration can be added (either out-of-band or using the NSO client) over time; under “system”, and a re-deploy and/or re-deploy reconcile { discard-non-service-config } will not trigger the configuration for deletion and thus making the element non-compliant.

I’ve briefly played with using tags=“replace” in <system> with some success; however, found that using the “replace” tag in various locations in a large service package containing large amounts of configuration resulted in long commit times and application timeouts.

<name>router</name>

<host-name refcounter="1" >router</host-name>

<domain-name refcounter="1" >somewhere.com</domain-name>

</system>

</configuration>

</config>

</device>

</devices>

</config>

Thanks!

gschudel · ‎03-19-2018

Hi Ron

Comments/replies to yours...

Assuming the XML you show is presumably the output of "show devices device <nsodevname> config junos:configuration system | display service-meta-data

correct?

COMMENT: you write:

"Referring to the Juniper snippet below; this service package pushes both

a host-name & domain-name and assigns the associated refcounters."

[gs] "...assigns... the refcount" -- this is _automatic_ as a consequence of running the service...

not _you_ assigning them in the service...

"Because there are no associated refcounters or backpointers for <system>"

[gs] which there cannot be - in Juniper you cannot every get rid of "system" --- it's part of the OS...

"I found that configuration can be added (either out-of-band or using the NSO client) over time; under “system”,

[gs] sure "out-of-band" changes can happen...

"and a re-deploy and/or re-deploy reconcile { discard-non-service-config } will not trigger the configuration

for deletion and thus making the element non-compliant."

[gs] I'm not understanding this...

** in general, a _device_ can be in-sync with CDB or not; this can be tested, and corrected

(EXAMPLE BELOW)

** in general, if a config was written by a service, the _service_ can be in-sync or not; this can be tested, and corrected

(EXAMPLE BELOW)

--EXAMPLE-

I made a quite "template-only" service model (called "tempFoo")

//-----

list tempFoo {

key name;

uses ncs:service-data;

ncs:servicepoint "tempFoo";

leaf name {

type string;

}

leaf-list device {

type leafref {

path "/ncs:devices/ncs:device/ncs:name";

}

leaf hname {

type string;

}

leaf dname {

type string;

}

//-----

<config-template xmlns="http://tail-f.com/ns/config/1.0"

servicepoint="tempFoo">

<devices xmlns="http://tail-f.com/ns/ncs">

<device>

<name>{/device}</name>

<config>

<configuration xmlns="http://xml.juniper.net/xnm/1.1/xnm">

<system>

<host-name>{/hname}</host-name>

<domain-name>{/dname}</domain-name>

</system>

</configuration>

</config>

</device>

</devices>

</config-template>

=================

ORIGINAL SITUATION:

===============

*** on device:

j0# show running-config configuration system

configuration system time-zone UTC

(skip)

j0#

*** on NSO:

admin@ncs% show devices device j0 config junos:configuration system | display service-meta-data

time-zone UTC;

(skip)

[ok][2018-03-19 09:08:10]

[edit]

admin@ncs%

=================

DEPLOY SERVICE:

===============

*** on NSO:

admin@ncs% set tempFoo tix01 device j0 hname router dname somewhere.com

[ok][2018-03-19 09:10:28]

[edit]

admin@ncs% commit

Commit complete.

[ok][2018-03-19 09:10:30]

[edit]

admin@ncs% show devices device j0 config junos:configuration system | display service-meta-data

/* Refcount: 1 */

host-name router;

/* Refcount: 1 */

domain-name somewhere.com;

time-zone UTC;

(skip)

[ok][2018-03-19 09:10:33]

[edit]

admin@ncs% show tempFoo

tempFoo tix01 {

device [ j0 ];

hname router;

dname somewhere.com;

}

[ok][2018-03-19 09:10:42]

[edit]

admin@ncs%

*** on device:

j0#

System message at 2018-03-19 09:10:30...

Commit performed by admin via ssh using netconf.

j0# show running-config configuration system

configuration system host-name router

configuration system domain-name somewhere.com

configuration system time-zone UTC

(skip)

j0#

=================

TRY THINGS:

===============

now we can try a few things...

a) make an out-of-band change to device...

j0# config

Entering configuration mode terminal

j0(config)# configuration system host-name newfooname

j0(config)# commit

Commit complete.

j0(config)# exit

j0# show running-config configuration system

configuration system host-name newfooname

configuration system domain-name somewhere.com

configuration system time-zone UTC

(skip)

j0#

** now what?

let's look at NSO...

First -- device "chec-sync"

admin@ncs% request devices device j0 check-sync

result out-of-sync

info got: 1521-475923-340415 expected: 1521-475830-386426

[ok][2018-03-19 09:13:09]

[edit]

admin@ncs% *** ALARM out-of-sync: got: 1521-475923-340415 expected: 1521-475830-386426

admin@ncs%

Ok - it's "out-of-sync" --- this means certain _other things_ NEED to be done...

Second - because "out-of-sync" you should "compare-config":

** if it was "in-sync" there would be _no_ difference in "compare-config" since it's

simply "device" config comparison between what's in CDB and _on the device_

admin@ncs% request devices device j0 compare-config

diff

devices {

device j0 {

config {

junos:configuration {

system {

- host-name router;

+ host-name newfooname;

}

[ok][2018-03-19 09:15:54]

[edit]

admin@ncs%

OK - that looks good

- here you have choices: "sync-to" or "sync-from" -- or do nothing for for now

Third - but what about the service?

** recall, 'service' is stored separately in CDB...

admin@ncs% show tempFoo

tempFoo tix01 {

device [ j0 ];

hname router;

dname somewhere.com;

}

[ok][2018-03-19 09:19:08]

[edit]

admin@ncs%

tempFoo tix01 wants "hname" == router ---- but it's newfooname on the device..

(and no sync-from or sync-to yet...)

admin@ncs% request tempFoo tix01 ?

Possible completions:

check-sync - Check if device config is according to the service

commit-queue -

deep-check-sync - Check if device config is according to the service

get-modifications - Get the data this service created

log -

re-deploy - Run/Dry-run the service logic again

reactive-re-deploy - Reactive redeploy of service logic

touch - Touch a service

un-deploy - Undo the effects of this service

admin@ncs%

admin@ncs% request tempFoo tix01 check-sync

in-sync true

[ok][2018-03-19 09:22:45]

[edit]

admin@ncs% request tempFoo tix01 deep-check-sync

sync-result {

local-node {

in-sync false

}

[ok][2018-03-19 09:22:53]

[edit]

admin@ncs%

IMPORTANT!

** service "check-sync" is "in-sync true" because this asks only if the service matches CDB,

but recall the device is out-of-sync with CDB.

** service "deep-check-sync" is "in-sync false" because this asks if the service matches

all the way to the device. since the _service_ is actually the affected portion of the config

on the device, the _service_ is also out of sync...

=================

TO RESTORE THE SERVICE:

===============

So now what can get the service "host-name" back on the device...

** you can:

--------

a) do a "sync-to" and this will put CDB device config back on the device...

-- in this case no further actions would be required...

(but some analysis was required to know this...)

b) do a "sync-from" and this will put existing device config into CDB...

-- but then you need to "re-deploy" the service...

let's try the second one..

on NSO...

** for the device...

admin@ncs% request devices device j0 sync-from

result true

[ok][2018-03-19 09:31:11]

[edit]

admin@ncs% request devices device j0 check-sync

result in-sync

[ok][2018-03-19 09:31:18]

[edit]

admin@ncs% request devices device j0 compare-config

[ok][2018-03-19 09:31:22]

[edit]

admin@ncs%

This all makes sense - "sync-from" but CDB and device "in-sync", so then "compare-config" has no delta

** but then for the service...

admin@ncs% request tempFoo tix01 check-sync

in-sync false

[ok][2018-03-19 09:32:31]

[edit]

admin@ncs% request tempFoo tix01 deep-check-sync

sync-result {

local-node {

in-sync false

}

[ok][2018-03-19 09:32:42]

[edit]

admin@ncs%

This makes sense too - BOTH are out-of-sync (service) because CDB and the device both have

a config that does not match "the service"...

so...

admin@ncs% request tempFoo tix01 re-deploy dry-run

cli {

local-node {

data devices {

device j0 {

config {

junos:configuration {

system {

- host-name newfooname;

+ host-name router;

}

[ok][2018-03-19 09:33:14]

[edit]

admin@ncs% request tempFoo tix01 re-deploy

[ok][2018-03-19 09:33:19]

[edit]

admin@ncs%

System message at 2018-03-19 09:33:19...

Commit performed by admin via console using cli.

admin@ncs%

*** on device:

j0#

System message at 2018-03-19 09:33:19...

Commit performed by admin via ssh using netconf.

j0# show running-config configuration system

configuration system host-name router

configuration system domain-name somewhere.com

configuration system time-zone UTC

(skip)

j0#

All good now!

==========

Note: NSO has a "compliance report" function

available checks:

-- look at devices current and/or historic device sync status

-- looks at one or a number of stored device templates against devices/device-groups

-- looks at deployed services sync status against related devices

Couple of other points:

- regarding tags="replace" -- this likely makes no sense for the elements shown. there can only by _one_ host-name anyway..

so not sure tags="replace" is useful here (or on domain-name)

- "compliance" (likely) requires looking at "device" and "service" comparisons (a service _could_ stretch across many devices)

- golden configs for a device may not best be handled in a "Service" model

configs on "devices" general happen in three parts:

<1> lines of configuration that are:

** included “one time” per device,

** “common” across all devices (or class of devices, - e.g. P, PE, RR) and immutable (no variables)

<2> lines of configuration that are:

** “one time” per device,

** but that have variable/unique parts per device (e.g., loopback IPs etc.)

<3> lines of configuration that are:

** repeated “many times” per device, — e.g. service instances (L3 VPN's)

** have variable/unique parts per "repetition" -- e.g., per L3 VPN instance

items in <1> best handled in device-templates (no variables required) -- these are likely never deleted or modified

items in <2> best handled in device-templates w/ variables, or actions or other (no state) - also likely limited delete/modify

items in <3> best handled in service models (due to likely high modification/delete requirements)

hope this helps

gregg

ron.seguin · ‎03-20-2018

Hi Gregg,

Thanks for the examples. So, what I'm gathering is that you can't easily identify and remove all rogue configuration.

Thanks,

-Ron

gschudel · ‎03-20-2018

Hi Ron

I probably didn't fully address what you have in mind.

your model should be capable of adding configs and ID'ing if something is incorrect (as compared to the template)

using a "service" to do this _may not be the best choice_ (in this case)...

configuring routers w/ golden config (typically not done via "service" model)

* other choices for applying golden configs w/o a service model (i can show examples) -- e.g. device-templates, or action applying template (a nice option, even allows dry-runs)

The use of tags is appropriate for this as well. You said you looked at tags="replace" in <system> and other places, and that it worked. I showed adding config "out-of-band" and it _absolutely_ was picked up by NSO as not belonging (see "compare-config" example)...

The first results i showed you came from a simple "template-only" service model - but i also ran it as an "action" (which seems preferable for GoldenConfig applications) - but aside from how they are called, the results are the same.

configs not addressed by the template (assuming tags are applied) are flagged.

I also just amended my model to add

to the juniper XML template ... re-ran the model i showed you and it picked up _everything_ that was never included in the initial goldenconfig XML... (note all the "-" which ID parts of the config i did not add via template)

--[begin dry-run cli output]------------

Device: j0 devices {

device j0 {

config {

junos:configuration {

system {

- host-name NewCoolRouter;

+ host-name OMGcouldbe;

- time-zone UTC;

- name-server 10.10.10.1;

- name-server 10.10.20.1;

- name-server 1.1.1.3;

login {

- message "This is j0";

}

- services {

- ssh {

- root-login deny;

- protocol-version [ v2 ];

- }

- netconf {

- ssh {

- }

ntp {

- peer 1.1.1.1;

- server 10.20.10.110;

- server 10.30.10.120;

}

--[end dry-run output]--------------

hth

gregg