Managing a system installation of NSO for an sysadmin is not that
complicated, as NSO is shipped with a useful commands and actions
that lets you start, stop and restart the NSO daemon,
create - and restore - CDB backups, apply engineering patches on a
running instance, fetch, install, load and unload packages and more.
And upgrading from one NSO version to another is nothing more than
installing the newer NSO version and change a symlink pointer.
All the above steps for the sysadmin quickly goes from not-so
cumbersome to quite cumbersome when you move from one NSO instance
to two, or more, NSO instances - not unusual if you utilize HA, NSO
clusters or NSO LSA - or a mix of all three.
Imagine you need to manage 16 NSO nodes - in a 8 node cluster setup with
HA, adding 8 more slave-nodes to the mix.
Now imagine upgrading NSO to a newer version.
16 nodes need to install a new NSO version,
16 nodes need to make sure it has the same packages
for the new version of NSO so that the upgrade phase goes smooth, and
16 nodes where HA needs to be activated in the correct order.
Thats for examples 16 symlink pointers that need to change...
The savvy sysadmin will of course not do all the above by hand, most
likely a cleaver script will handle it, but the script would need to
know quite a lot on the NSO system install structure, commands and
actions.
If only there were a bunch of *nix programs to hide some of
the logic that the sysadmin could use...
Well, luckily there are: NCT - a collection of tools for installing
and managing NSO nodes.
The purpose of this toolset is to aid the operational team, managing
and running a large scale NSO installation made ad-hoc for NSO.
NCT has been shipped with NSO since NSO 4.1, and consists of the
following tool set:
- backup
Create/restore CDB backups on nodes - check
Check NSO status, such as version, northbound api availability and more - cli-cmdPerform a CLI command on the nodes
- copy
Copy files to the nodes - ha
Interface the tailf-hcc functionality - install
Install NSO on nodes - load-config
Load initial config, running config or ncs.conf to nodes - move-device
Move a managed device from one NSO node to another - packages
Handle package installation/upgrades etc on nodes - patch
Provide nodes with an engineering patch - ssh-cmd
Issue a custom ssh command on the nodes - start
Start NSO on the nodes - stop
Stop NSO on the nodes - upgrade
Upgrade/Downgrade NSO - get-logs
Fetch all the logs from the nodes
The commands are run from a terminal on the form
$ nct <COMMAND> --hostsfile <FILE> [Additional flags]
|
Where the <FILE> contains address information etc for the nodes in the
setup.
Each tool has its own man-page, and the easiest way to read its
documentation is using the 'help' command to nct, such as
# For general documentation
$ nct help
# For command specific documentation
$ nct help cli-cmd
...
$ nct help install
...
$ nct help <CMD>
|
NCT can be used both in scripts, as they behave like you'd expect a *nix command to behave,
with status codes etc, or as a power tool to manually supervise and operate NSO installations.
The tool-set is installed out-of-the-box with NSO on both system and
local installations.
Note:
the 16 node scenario described above is not a far fetched example
invented by my, but an actual customer use-case that successfully
used NCT to upgrade NSO and its packages to a newer version, all in
a live system.
For additional information and examples, I'd refer you to the NCT man
pages.
There is also an old blog post written by on of the tail-f
founders on his experiences with NCT here:
https://communities.cisco.com/people/cwikstro/blog/2016/01/07/introducing-nso-cluster-tools