Rollback and Pre- and post-modification callbacks in NSO

[snovello]

Overview

This post is describes how the NSO rollback file feature interacts with the pre- and post-modification callbacks. These callbacks are outside of the scope of FASTMAP so changes they make to the NSO database (CDB) are not automatically undone when the service is deleted, unlike the changes in the create callback which are recorded in a reverse diff-set datastructure.

Rollback

When you look into a rollback file, which is a text file that describes how to undo the changes in a transaction, you will see, that if you modify a service, for example by deleting it, the rollback file contains the opposite change, so in the delete case, creating the service with its parameters in the last state before deletion. What you do not see are any of the effects of the transformations it has created. In other words it does not include include any of the changes it made in any of the pre-modification post-modification and create/nanoservice-create callbacks.

This can be seen also in the steps NSO takes when creating a transaction. The rollback file is created before any of the service transforms are called, so it at that point NSO does not know any of the changes made by those transforms.

# commit | details very-verbose
applying transaction for running datastore usid=66 tid=100 trace-id=673d380adb474eb31016e006ca80e1e1
 2025-04-23T08:03:01.257 waiting to apply... ok (0.000 s)
entering validate phase
 2025-04-23T08:03:01.258 creating out-of-band checkpoint... ok (0.000 s)
 2025-04-23T08:03:01.258 creating rollback checkpoint... ok (0.000 s)
 2025-04-23T08:03:01.258 creating rollback file... ok (0.001 s).                        <— rollback file created
 2025-04-23T08:03:01.259 creating pre-transform checkpoint... ok (0.000 s)
 2025-04-23T08:03:01.259 run pre-transform validation... ok (0.001 s)
 2025-04-23T08:03:01.260 creating transform checkpoint... ok (0.000 s)
 2025-04-23T08:03:01.260 run transforms and transaction hooks...
 2025-04-23T08:03:01.261 taking service write lock... ok (0.000 s)
 2025-04-23T08:03:01.261 holding service write lock...
 2025-04-23T08:03:01.261 service /vpn-endpoints/vpn-endpoint[name='s1']: creating service...
 2025-04-23T08:03:01.261 service /vpn-endpoints/vpn-endpoint[name='s1']: pre-modification... ok (0.014 s) <—— pre-modification changes added to the transaction
 2025-04-23T08:03:01.276 service /vpn-endpoints/vpn-endpoint[name='s1']: create... ok (0.017 s)
 2025-04-23T08:03:01.293 service /vpn-endpoints/vpn-endpoint[name='s1']: saving reverse diff-set and applying changes... ok (0.002 s)
 2025-04-23T08:03:01.295 service /vpn-endpoints/vpn-endpoint[name='s1']: post-modification... ok (0.000 s)
 2025-04-23T08:03:01.295 service /vpn-endpoints/vpn-endpoint[name='s1']: creating service: ok (0.034 s)
 2025-04-23T08:03:01.295 run transforms and transaction hooks: ok (0.035 s)

So in NSO, if I create a service, commit that, then apply the rollback (which contains instructions to delete the service), and commit that rollback,  the state of CDB will be identical to the state if I had created it manually first, then manually deleted the service. This design ensures the service is in full control of what happens when it is created deleted or modified and does not have to know whether a rollback fille was loaded as part of a transaction. Of course typically you want the net effect of create/delete to be no effect at all, and that is the default behaviour that the create callback and the FASTMAP algorithm provide. pre- and post-modification callbacks are sometimes used to alter that default behaviour a little.

Relying on the service to make its transforms, rather than recording the output of a previous service execution makes rollback very flexible. You can apply a rollback file selectively, after other transactions have changed cdb, or load a few rollback files into a transaction, for example all changes made in a workflows that subsequently failed and needs to be undone, and know that the same service logic is executed just as if you had entered all the changes manually.

Use cases

What does this mean for typical use cases of pre-modification and post-modification callbacks?

One typical use of pre-modification is pre-requisite configuration. The service logic may rely on the existence of some configuration, for example it may reference an access list that should be present on the device. The access list should always be there. You can make my service more robust by always creating that access list in pre-modification. You use pre-modification because that access list should actually always be there, so you do not want to remove it when the service is removed.  In this example creating and deleting a service instance, will have the side effect of creating that access list if it was not present. The side effect is there if you create and delete with two explicit requests, or if you create and load the rollback file. In this case the side effect is desired. Having that access list prepared on the device is actually intended.

Another example is a service that manages a resource e.g. a physical interface, when removing the service you want to ensure the resource is configured in a way that it is clear the resource is free. For example the service adds the resource's name to a free-list data structure in CDB, or sets a particular description in the configuration which is then interpreted as marking the resource as free. In the interface case you might also shut it down. Here if you create the service, taking a resource that was not  properly marked inititally, and then delete the service the side-effect is that now the resource is properly marked as free. Again that is as intended, even if the deletion is through a rollback.

It is  a different behaviour, if you compare rollback in NSO and rollback in a typical database. In a typical DB there there are no transformations, and the rollback is essentially taking all the transaction changes and reversing them, making a change a rolling it back is guaranteed to take you to the previous DB state.  So for these examples, removing the access list and taking the resource to the state where it was not marked as free, NSO give you the option through the pre and post-modification to take more control and have behaviours on rollback that are not necessarily undoing all changes.

Best practices

Given the way NSO works, post- and pre-modifications which make changes that will survive the creation and then deletion of a service, should only be done for changes that are actually intended to be permanent. Ideally pre-requisite configuration would be handled through a day1 service or device templates, so that services can just rely on the presence of pre-requisites. Marking unused resources as free, can also be done outside of service implementions so that the interface is returned to it's free state simply by removing the service with no extra logic. So it might in some cases be worth investing in cleaning up the network a bit more rather than putting extra checks in a service so that it can work in a network that is not as clean and regular as the ideal case.

Often pre-modification can be used for doing things that don't result in changes to the database if you create and then delete a service. The use cases have no interaction with the rollback feature. For example I could have a resource intensive algorithm to search through all interfaces at a physical location and pick a free interface. I would only want to execute it when I create the service. In the pre-modification I can test whether the service is being created, and run that procedure. The result is then stored in the proplist, and the create callback can safely access that result, knowing it will always be there.

To give a counter-example or anti-pattern, I could use pre-modification to deal with the case of two services which are often created together but afterwards need to be independent with independent life-cycles. For example I need to create 2 VPN endpoints that provide redundant access, they have many parameters in common, but after the creation I want their lifecycle to be independent. I could have a flag to mark the need to create the second interface, and create it in the pre-modification, then I remember in the opaque proplist that the second access has been created.

If I implemented such a service this way, this is basically creating a side effect inside the transforms that the service computes, and  the effect will be that if I create and delete such a service, the redundant interface that was created as a side effect will, not be deleted. This may be what I want in the case of two requests, one to create one to delete - evn though it is somewhat counter-intuituve, it is even less intuituve in the case of  create followed by rollback. To the user the look and feel is very close to a stacked service and they may expect a behaviour where on deleting just after creating, NSO deletes both instances. There are a few better ways to implement this

• Since they are two independent services with independent lifecycles, just request them separately, your request from a northbound system can create two instances of a service with similar parameters. This works if there is a northbound system, but if there is an operator using NSO directly, it implies repeating those same parameters, so a more complex request.
• Using a stacked service pattern, and having the parameters relating to the backup interface as optional, defaulting to the value of the primary if any parameter is not set. This has the downside of requiring the northbound system to be always aware of whether it is updating a primary or secondary interface, even in cases where that may not be so relevant. It does however clearly model the redundancy and reflects more cleearly your intent, than if you model the two interfaces separately.
• Creating an action that creates the two services, with similar parameters This can lead to a simpler interface for the operator on creation, but afterwards they are just independent endpoint services. The action can be built to attach itself to the current transaction, so that you can still use it from the NSO CLI in combination with other modifications to make a larger change as a single transaction, or to dry run a change.