Managing large scale NSO installation with NCT

[sunge]

Managing a system installation of NSO for an sysadmin is not that

complicated, as NSO is shipped with a useful commands and actions

that lets you start, stop and restart the NSO daemon,

create - and restore - CDB backups, apply engineering patches on a

running instance, fetch, install, load and unload packages and more.

And upgrading from one NSO version to another is nothing more than

installing the newer NSO version and change a symlink pointer.

All the above steps for the sysadmin quickly goes from not-so

cumbersome to quite cumbersome when you move from one NSO instance

to two, or more, NSO instances - not unusual if you utilize HA, NSO

clusters or NSO LSA - or a mix of all three.

Imagine you need to manage 16 NSO nodes - in a 8 node cluster setup with

HA, adding 8 more slave-nodes to the mix.

Now imagine upgrading NSO to a newer version.

16 nodes need to install a new NSO version,

16 nodes need to make sure it has the same packages

for the new version of NSO so that the upgrade phase goes smooth, and

16 nodes where HA needs to be activated in the correct order.

Thats for examples 16 symlink pointers that need to change...

The savvy sysadmin will of course not do all the above by hand, most

likely a cleaver script will handle it, but the script would need to

know quite a lot on the NSO system install structure, commands and

actions.

If only there were a bunch of *nix programs to hide some of

the logic that the sysadmin could use...

Well, luckily there are: NCT - a collection of tools for installing

and managing NSO nodes.

The purpose of this toolset is to aid the operational team, managing

and running a large scale NSO installation made ad-hoc for NSO.

NCT has been shipped with NSO since NSO 4.1, and consists of the

following tool set:

• backup
  Create/restore CDB backups on nodes
• check          
  Check NSO status, such as version, northbound api availability and more
• cli-cmdPerform a CLI command on the nodes
• copy
  Copy files to the nodes
• ha
  Interface the tailf-hcc functionality
• install
  Install NSO on nodes
• load-config
  Load initial config, running config or ncs.conf to nodes
• move-device
  Move a managed device from one NSO node to another
• packages
  Handle package installation/upgrades etc on nodes
• patch
  Provide nodes with an engineering patch
• ssh-cmd
  Issue a custom ssh command on the nodes
• start
  Start NSO on the nodes
• stop
  Stop NSO on the nodes
• upgrade
  Upgrade/Downgrade NSO
• get-logs
  Fetch all the logs from the nodes

The commands are run from a terminal on the form

$ nct <COMMAND> --hostsfile <FILE> [Additional flags]

Where the <FILE> contains address information etc for the nodes in the

setup.

Each tool has its own man-page, and the easiest way to read its

documentation is using the 'help' command to nct, such as

# For general documentation

$ nct help

# For command specific documentation

$ nct help cli-cmd

...

$ nct help install

...

$ nct help <CMD>

NCT can be used both in scripts, as they behave like you'd expect a *nix command to behave,

with status codes etc, or as a power tool to manually supervise and operate NSO installations.

The tool-set is installed out-of-the-box with NSO on both system and

local installations.

Note:

the 16 node scenario described above is not a far fetched example

invented by my, but an actual customer use-case that successfully

used NCT to upgrade NSO and its packages to a newer version, all in

a live system.

For additional information and examples, I'd refer you to the NCT man

pages.

There is also an old blog post written by on of the tail-f

founders on his experiences with NCT here:

https://communities.cisco.com/people/cwikstro/blog/2016/01/07/introducing-nso-cluster-tools

[camoberg]

Very cool!

[rogaglia]

Hi Simon,

A customer asked me today a question about NCT and the NSO upgrade process.

Let's say I want to upgrade NSO to a new major version (maybe in a lab). Can NCT upgrade NSO and also re-compile all the existing packages for the newly install version without having to perform any pre-provisioning work?

Regards,

Roque

[sunge]

Hey Roque,

No, NCT can't do that currently. It was designed for system installations, with packages pre-compiled and stored in the install-directory - normally thats /opt/ncs/packages.

The procedure when upgrading for NCT is then, when you upgrade from say 4.2 to 4.3:

• is 4.3 installed?
• For each package loaded in the current running nso installation, 4.2, is there an 4.3 compiled counterpart?

If no to the above the upgrade will abort.

This approach gives us correctness and robustness, but the drawback is that each customer-built package needs to conform to the "proper" way to load, name and package (tar.gz) the packages.

[scott.barvick]

I like nct for what it does, but do you think it should be invested in by the BU or should we have the community develop Ansible-based solutions?   This seems like a good example of where the community would move much faster than the product could.

[viskansa]

Good stuff to manage multiple nodes.

[Robert Moss]

This is really great, it will save me a week of effort writing my own Ansible playbooks

Could there be a Standalone version just for deployment?

I would normally create a deployment server which is separate from the NSO instances in order to build out an environment

[Jan Lindblad]

One approach doesn't have to preclude the other. I agree this would be a field where community contributions might be particularly interesting. Until there are a few good solutions out there, the NSO team will have to offer at least one default way of handling larger NSO installations, therefore NCT.