Existing Services Discovery - avoiding re-deploy reconcile issues

[ken.mcnamara1]

This is a proposal for managing initial Services Discovery in an unorthodox manner.

Comments are welcome and invited.

Problem - Existing services that don't reconcile correctly

Services created from scratch in the system have correct refcounters and can be life-cycle managed without problems.  (Move, Add, Change, Delete)

But existing services present a management problem when the reconciliation process does not set the refcounters correctly.  A change to a service deployed this way produces unexpected results when NSO thinks part of the service is shared with another service or the original configuration. i.e. The Service does not OWN that piece of the configuration.

Service Discovery has 4 steps:

1. Parse existing configuration(s) for service parts

2. Write those parts to the /root/services/<service> tree

3. Commit

4. Re-deploy reconcile

In addition all of this must be done with absolutely NO impact on the routers or the network.  This is a trivial requirement when there are a few services (say less than 50) - but when there are several 1,000 services it is not trivial at all.

Proposal (applies to initial Service Discovery, not ongoing out-of-band additions):

1. Create a simulated network with devices names equal to the live network.

2. Load the simulated network with the real network configurations.

3. Parse the services into an external data structure

4. Reset the simulated routers - and load basic configurations (some parts of the services may not be allowed to create parts of the configuration, so these parts need to be created in the configuration before commit)

4. Write and commit the services (no need to re-deploy reconcile, these are new services)

5. Delete the simulated devices, connect to the real devices (same names) and sync-from

If you read this far - you probably have an opinion - or can see why this is not possible or ill-advised.

Please comment.

Thank you.

Ken McNamara

Applied Computer Solutions

[alam.bilal]

Is the main concern the scaling of the reconciliation process when doing it all in NSO?

In addition all of this must be done with absolutely NO impact on the routers or the network

Wouldn't FASTMAP should ensure this. As part of service reconciliation, NSO should not be sending anything to the network given the configs are already there on the devices and NSO computes minimum delta diffs.

This is a trivial requirement when there are a few services (say less than 50) - but when there are several 1,000 services it is not trivial at all.

If this becomes an issue (CPU/Compute intensive???), perhaps can be done in an incremental manner. Either via some NB System or an Action can be put in for the service that can reconcile a subset (indicated via the input to the Action).

The proposed alternative may be cumbersome to setup. Creating a simulated network and ensuring appropriate "base" configs are present.

A couple more comments:

a. Sometimes the service instance data can be retrieved from an external system. As customers keep that in some shape or form (e.g. Excel sheets, CSV etc). Need to be extra cautious to infer that from device configs. Ok for simple services but for service instances that span multiple devices can get tricky.

b. It is advised not to hang services off /services/<my-service>

[ken.mcnamara1]

Bilal -

Thanks for your reply and comments.

In this instance the main concern is not the scaling of reconciliation but the failure of reconcile to work correctly.

There is a part of the configuration which has a leaf-list, in this case the refcounter is left at '2' when it should be '1'. So a 'service delete' deletes a part of that section, then reinstates it.

A fresh service commit does not have this problem.

It is trivial to review 50 'dry-run's to insure there is not impact, but not 1,000.  In this case 'no-networking' would probably be used if the commit were being done on a live network (probably being overly cautious here, but...).

Creating a simulated network with 'base' configurations has already been done during development. The main issue here is that the templates prevent the 'creation' of some specific sections of the configuration.

Yes, parsing out the services from device configurations across multiple devices has not been easy.  But there is no reliable external data (what data exists is collected and reported by an external script that parses the raw 'show running-config's).  Parsing the CDB is much more reliable.

Please explain 'it is not advised to hang services off /services/<my-service>'?

To be more formal -- /root/services/<my-service>  (<my-service> looks like servicename__servicename[uniqueID})

Thank you again.

KenMc

[alam.bilal · Accepted solution]

I have heard of some work being done with respect to backpointers for leaf-list elements to make them full fledged nodes. Would that help? I can find out more details. Best to raise this with Engineering and see if something can be worked out.

Please explain 'it is not advised to hang services off /services/<my-service>'?

One instance where augmenting a service under /services becomes problematic is when using Layered Services Architecture (LSA). That is, when a higher-level-service needs to map to a lower-level-service on a different NSO node. There might be other reasons. As a result the instrumentation (ncs-make-package) has been updated to not augment service-skeleton under /services.

[JimBoucher]

I had a similar Discussion but showed how to resolve it without using netsims as an intermediary.

https://community.cisco.com/t5/nso-developer-hub-discussions/why-is-commit-no-deploy-redeploy-reconcile-commit-no-networking/m-p/4142715#M5615