Service package in HA deployment

[douglasfir]

Hello,

I am trying to install a custom service package in deployment with two NSOs in HA mode enabled, but trying to do "package reload" on the master node I am getting the error:

>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has been cancelled.
Error: Cannot do data model upgrade when HA is active

Is there a procedure/workflow how to onboard a service package in this case? NSO is version 5.4.x and the service package is copied in /var/opt/ncs/packages on both NSO nodes.

Thanks in advance!

Dragan

[tsiemers1]

You can use the following command

request packages ha sync and-reload

See this excerpt from the NSO admin guides. Some caveats are that the commit queue is empty before running and locked. This essentially sets the primary node to read only and sync packages from the primary load-dir to the slave.

Scroll down to the Package Upgrade section and find the relevant HA information. It even has a script example if you want to build upon that and use it.

https://developer.cisco.com/docs/nso/guides/#!nso-deployment/upgrading:~:text=mix%20them%20up.-,Package%20upgrade,-Package%20upgrades%20are

[douglasfir]

Hi @tsiemers1 ,

unfortunately /packages/ha... commands don't seem to be supported in my environment. Could it be because the commands are from the 5.8 guide, since my version is 5.4? I have to use this version because of NSO being part of CNC 2.0.

Thanks in advance!

[tsiemers1]

Looks that way. From the 5.4 deployment.pdf in the guides, you can use this method, which looks to do the following order in the script example. You would have to read the man page for the nct commands to verify for sure.

1. Set master to read-only
2. Deactivate the slave from HA
3. Upload/Uninstall/Install package
4. Run 'request packages reload' on slave
5. Set slave to Master
6. Set new master to read-only
7. Upload/Uninstall/Install package
8. Set current master to slave and activate

Package upgrade
Similarly to upgrading NSO, upgrading packages is also a staged process. We have the exact same sequence of steps as in upgrading NSO. This must also be scripted.
Here is a small script to do this.
#!/bin/bash
set -ex
  package=$1
  fromver=$2
  tover=$3
  ncsver=$4
  restpass=$5
  file=ncs-${ncsver}-${package}-${tover}.tar.gz
  frompack=ncs-${ncsver}-${package}-${fromver}
  topack=ncs-${ncsver}-${package}-${tover}
  args="--rest-pass $restpass "
  function pkg() {
      grp=$1
      nct packages --group ${grp} --file ${file} -c fetch ${args}
      nct packages --group ${grp} --package ${frompack} -c deinstall ${args}
      nct packages --group ${grp} --package ${topack} -c install ${args}
      nct cli-cmd -c 'request packages reload' --group ${grp}
}
  nct backup
  nct ha --group master --action readonly ${args} --mode true
  nct ha --group slave --action deactivate ${args}
  pkg slave
  nct ha --group master --action deactivate ${args}
  nct ha --group slave --action role-override ${args} --role master
  nct ha --group slave --action activate ${args}
  nct ha --group master --action readonly ${args} --mode false
  pkg master
  nct ha --group master --action role-override ${args} --role slave
  nct ha --group master --action activate ${args}
This script also leaves the system with reversed HA roles, same argument as in previous section apply.
The script can be expanded to handle multiple packages in one go. It's more efficient to upgrade several packages in one go than in several steps. Another important efficiency note here is the use of nct cli-cmd -c 'request packages reload'. There are two ways to load new data models into NSO. This is one, the other is to invoke:
  $ sudo /etc/init.d/ncs restart-with-package-reload
The former is considerably more efficient than the latter. If the amount of data in CDB is huge, the time difference to upgrade can be considerable.

[douglasfir]

Hi @tsiemers1 ,

in the end I didn't use nct tool, but the commands that are bult in ncs_cli for high-availability (high-availability enable/disable) and installing packages (software packages install...).

It seemed simpler and it did the work in the end, so I prefered it over nct, but thanks for your answer too!

[douglasfir]

Hi @tsiemers1,

thanks a lot, I will follow your approach, I just need to wait for the approval from the team owning the system, hopefully by the end of this week and then I can try to implement the steps.

Kind regads,

Dragan