cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Managing large scale NSO installation with NCT

1260
Views
18
Helpful
7
Comments
Cisco Employee

Managing a system installation of NSO for an sysadmin is not that

complicated, as NSO is shipped with a useful commands and actions

that lets you start, stop and restart the NSO daemon,

create - and restore - CDB backups, apply engineering patches on a

running instance, fetch, install, load and unload packages and more.

And upgrading from one NSO version to another is nothing more than

installing the newer NSO version and change a symlink pointer.

All the above steps for the sysadmin quickly goes from not-so

cumbersome to quite cumbersome when you move from one NSO instance

to two, or more, NSO instances - not unusual if you utilize HA, NSO

clusters or NSO LSA - or a mix of all three.


Imagine you need to manage 16 NSO nodes - in a 8 node cluster setup with

HA, adding 8 more slave-nodes to the mix.


Now imagine upgrading NSO to a newer version.

16 nodes need to install a new NSO version,

16 nodes need to make sure it has the same packages

for the new version of NSO so that the upgrade phase goes smooth, and

16 nodes where HA needs to be activated in the correct order.

Thats for examples 16 symlink pointers that need to change...


The savvy sysadmin will of course not do all the above by hand, most

likely a cleaver script will handle it, but the script would need to

know quite a lot on the NSO system install structure, commands and

actions.


If only there were a bunch of *nix programs to hide some of

the logic that the sysadmin could use...

Well, luckily there are: NCT - a collection of tools for installing

and managing NSO nodes.


The purpose of this toolset is to aid the operational team, managing

and running a large scale NSO installation made ad-hoc for NSO.

NCT has been shipped with NSO since NSO 4.1, and consists of the

following tool set:

  • backup
    Create/restore
    CDB backups on nodes
  • check          
    Check
    NSO status, such as version, northbound api availability and more
  • cli-cmdPerform a CLI command on the nodes
  • copy
    Copy files to the nodes
  • ha
    Interface the
    tailf-hcc functionality
  • install
    Install
    NSO on nodes
  • load-config
    Load initial config, running config or ncs.conf to nodes
  • move-device
    Move a managed device from one
    NSO node to another
  • packages
    Handle package installation/upgrades etc on nodes
  • patch
    Provide nodes with an engineering patch
  • ssh-cmd
    Issue a custom ssh command on the nodes
  • start
    Start
    NSO on the nodes
  • stop
    Stop
    NSO on the nodes
  • upgrade
    Upgrade/Downgrade
    NSO
  • get-logs
    Fetch all the logs from the nodes


The commands are run from a terminal on the form

$ nct <COMMAND> --hostsfile <FILE> [Additional flags]

Where the <FILE> contains address information etc for the nodes in the

setup.

Each tool has its own man-page, and the easiest way to read its

documentation is using the 'help' command to nct, such as

# For general documentation

$ nct help

# For command specific documentation

$ nct help cli-cmd

...

$ nct help install

...

$ nct help <CMD>


NCT can be used both in scripts, as they behave like you'd expect a *nix command to behave,

with status codes etc, or as a power tool to manually supervise and operate NSO installations.


The tool-set is installed out-of-the-box with NSO on both system and

local installations.

Note:

the 16 node scenario described above is not a far fetched example

invented by my, but an actual customer use-case that successfully

used NCT to upgrade NSO and its packages to a newer version, all in

a live system.

For additional information and examples, I'd refer you to the NCT man

pages.


There is also an old blog post written by on of the tail-f

founders on his experiences with NCT here:

https://communities.cisco.com/people/cwikstro/blog/2016/01/07/introducing-nso-cluster-tools

7 Comments
Cisco Employee

Very cool!

Cisco Employee

Hi Simon,

A customer asked me today a question about NCT and the NSO upgrade process.

Let's say I want to upgrade NSO to a new major version (maybe in a lab). Can NCT upgrade NSO and also re-compile all the existing packages for the newly install version without having to perform any pre-provisioning work?

Regards,

Roque

Cisco Employee

Hey Roque,

No, NCT can't do that currently. It was designed for system installations, with packages pre-compiled and stored in the install-directory - normally thats /opt/ncs/packages.

The procedure when upgrading for NCT is then, when you upgrade from say 4.2 to 4.3:

  • is 4.3 installed?
  • For each package loaded in the current running nso installation, 4.2, is there an 4.3 compiled counterpart?

If no to the above the upgrade will abort.

This approach gives us correctness and robustness, but the drawback is that each customer-built package needs to conform to the "proper" way to load, name and package (tar.gz) the packages.

Beginner

I like nct for what it does, but do you think it should be invested in by the BU or should we have the community develop Ansible-based solutions?   This seems like a good example of where the community would move much faster than the product could.

Not applicable

Good stuff to manage multiple nodes.

Cisco Employee

This is really great, it will save me a week of effort writing my own Ansible playbooks

Could there be a Standalone version just for deployment?

I would normally create a deployment server which is separate from the NSO instances in order to build out an environment

Cisco Employee

One approach doesn't have to preclude the other. I agree this would be a field where community contributions might be particularly interesting. Until there are a few good solutions out there, the NSO team will have to offer at least one default way of handling larger NSO installations, therefore NCT.