NSO memory usage and scaling

[ThreeFire]

Dear community,

as I am planning an NSO setup, I need some hints as to how many devices a single server can support. I am especially interested in memory usage. Can anyone here help me out with information about how many devices they manage using one NSO server and how much memory is used? I know that this probably strongly depends on the configuration size per device, but just to have some kind of idea what to expect would help tremendously.

Thanks!

[uavsec001]

Hey,

Nso setup in what kind of context? Dev env, lab or production env?

You got it right regarding memory and device config... however it is not only that, there is also service configurations that have to be taken into consideration, other resource pools that can run in the NSO etc.

I don't have the data for the prod but for example I have my dev container with 4 junos devices with about 120000 lines of xml config each. the VM itself has 14 GB of ram (which is generous to be honest), then I am running a docker container which actually runs NSO (will take a look tomorrow at how much resources that container uses).

[ThreeFire]

Hey uavsec001,

thanks for replying!

---

Nso setup in what kind of context? Dev env, lab or production env?

It would be a production environment with roundabout 1000 devices.

How much of the memory in your lab setup is actually used? Do you run a production environment with more devices? The device count/configuration size/memory consumption info would be very valuable.

Thanks!

[uavsec001]

So, I did some counting. :)

Idle ubuntu (official) docker container uses 82 MB of RAM.

When I start the NSO (local installation) RAM utilization jumps to 3.05 GB while the ting is idling.

CDB contains:

- 4 devices with 178292 configuration lines total (this is purely just device configuration)

- 5510 lines of service config

- 28214 lines of auxiliary data structure that helps to onboard brown field pre NSO services into the NSO

Those stats are our customer feeding us their actual config and the rest is the service we designed for them. We don't have any prod env.

This is really a case specific situation since here we have only a line count and not how long the lines are which is also an important thing to consider.

[ThreeFire]

Thanks for the counting! Definitely already helpful information. I will probably have to include a few production machines measure the memory jump and update this thread with the data.

Thanks for your efforts!

[uavsec001]

No problem.

P.S.

There is a rough estimate floating around that says:

100 devices, 5000 services = 32 GB RAM 4vCPU

5000 devices, 20000 services = 64 GB RAM 6vCPU

20000 devices, 150000 services = 96 GB RAM 8vCPU

It is an old estimate so take it with a grain of salt as NSO went through considerable changes since then.

[lmanor]

Right, this is always an ask... but always hard to answer for any specific situation.

Given that every deployment is going to be different... #devices, # services, complexity of services, amount of config per device, etc... it is hard to even attempt a ball-park number...

What I have seen is customers test and monitor the resources of their specific solution (much like is discussed in this thread), starting with a VM with sizable memory and vcpu allocations - so as not to run out of a resource, which usually presents as some strange behavior and not a nice message like 'out of memory'.  Then the VM can be tuned 'down' to what you think is required with of course growth consideration.

In addition to memory and CPU processing (clock rate, # vcpu's, etc), in VM deployments you also need to consider storage media access performance, as NSO will journal CDB info to disk.

[ThreeFire]

Thanks for your helpful input. Will keep that in mind when planning the rollout.