How to use Stacked Services to improve performance?

[Mark Swanborough]

Sometime back, I had a discussion about NSO performance with a one or two engineers from the BU. There was a suggestion that Stacked Services can be used to improve NSO performance. I took this on faith, and repeated it a few times... but then someone asked me to explain how this would work in detail... and I have to confess, now I am not sure!

Please can someone comment on the accuracy of the below explanation as to how stacked services can be used to improve performance? Thank you!

Using Stacked Services to improve NSO Service Performance

Example Scenario:
The customer has a pair of ASR5Ks for EPC, and a pair of ASR1Ks for APN VPNs. The full service has a 1000s lines of configuration on the ASR5K EPCs, but these only change very occasionally, and a few 10s of lines on the ASR1Ks that change frequently.

Problem:

NSO diff calculation speed is dependent on the number of lines of configuration in your change set.

When the full service is created, your service templates result in 10000 elements of config. This is unavoidable. However, when any change is made to the Service, FASTMAP reverts those changes, your service re-applies those changes, and each element of the change set is compared to calculate the diffs. This is a full object tree walk, and that takes a lot of time with 10000 elements.

Solution:

To avoid this, the infrequently changed lines can be placed in a stacked service, which means that every time fastmap is ran for a Update to the service, you don’t need recalculate the diffs on the ASR5K, just for the stacked service. So far so good.

However, if this is a classic parent-child stacked service (composition), whenever you modify the parent service, fastmap applies the service reverse diff. I assume that fastmap must also apply the reverse diff of the child service(s), and so on.

So a parent-child stacked service, will not help with performance issues!

But what about a side by side stacked service (aggregation) that we will call aggregator-aggregated? The aggregated service is importantly NOT created by the parent service inside fastmap (this would be composition, parent-child) - so if the config that doesn't change very often is in a separate service, then obviously the diff's wouldn't get calculated!

So, this could help, but now I have two services, not one... so how is this stacked services? If the stacked aggregated service parameters are set and changed by the aggregator, then we have stacked services. Specifically, this works well if the aggregator is creating a list in an entry in the aggregated service, because these parameters will then be fastmap'ed.

Now, this might sound a bit messy to implement, and yes, potentially using preModification to create the aggregated service/delete it upon service deletion, and adding an entry into the aggregated service list, is not very clean. But it could result in good performance improvement.

[Jan Lindblad · Accepted solution]

Good question, thanks for putting it out there! I particularly appreciate you taking the time to explain the whole mechanism so that I don't have to ;-) Your description is very precise except one detail that I will add:

I assume that fastmap must also apply the reverse diff of the child service(s), and so on.

That's right. When the parent service create() function runs, it will re-compute the input configs for the child services, and invoke each child's create() etc, rippling all the way down to the bottom of the dependency structure. Except -- and here's where the savings come in -- fastmap won't call the child's create() function if the input config data exactly matches the previously stored input config for that service. In this case, fastmap already knows what the child service will result in (same as last time, i.e. what's stored in the database), and just pulls that in. Since what's pulled in is exactly what's already stored in the database, it won't add anything to the diff that will need to be traversed in the end, so saves (potentially a lot of) CPU cycles.

[mmalysz]

While reading this discussion one concern popped up in my mind related to stacked services related to this. Jan Lindblad - what happens in case the parent service calculated that there is a change required to the child's create() function in case parent was executed as a result of reactive-re-deploy. In the documentation we can read that:

Also the reactive-re-deploy will make a "shallow" re-deploy in the sense that underlying stacked services will not be re-deployed. This "shallow" feature is important when stacked services are used for performance optimization reasons

Does it mean that in case of RFM combined with Stacked Services in case the parent calculates a change to the child the child is actually not called?

[frjansso]

With shallow re-deploy, changed child services will be invoked.

With an deep re-deploy, all child services would be re-deployed regardless of changes or not. Hence the possible speedup with shallow re-deploy, only changed sub services will be invoked.