Packages doesn't get reloaded when requested

Scarfido · ‎03-22-2022

Hi,

I'm a newcomer to NSO and I'm currently trying to build my first custom package using templates and python (very simple at the moment). After making a change to my code, when I issue a "request package reload" command, everything seems to be fine (see output below) but the code used by NSO doesn't get updated.

I've looked around and saw that the files in the state/packages-in-use directory doesn't get updated and this seems the be the problem. Also, even after the reload has finished, there still is a packages-in-use.new directory.

lrwxrwxrwx 1 alain alain        19 Mar 22 10:16 packages-in-use -> packages-in-use.cur/
drwxr-xr-x 1 alain alain       512 Mar 21 22:45 packages-in-use.cur/
drwxr-xr-x 1 alain alain       512 Mar 22 10:15 packages-in-use.new/

My workaround for the moment is to :

Stop NSO : ncs --stop
Delete packages in state directory : rm -r state/packages-in-use*
Start NSO : ncs

This is very cumbersome as you might imagine and may cause deeper scars to the NSO database. I've tried to created a fresh instance of NSO multiples times. I'm using NSO 5.7

Output of reload command :

admin@ncs> request packages reload force
reload-result {
    package cisco-iosxr-cli-7.38
    result true
}
reload-result {
    package juniper-junos-nc-4.3
    result true
}
reload-result {
    package my-package
    result true
}

Also, this reload is very quick, almost instant and much quicker more that I was used to at the beginning and I don't get the

>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.

prompt.

Does someone knows what is happening to my setup ?

RichardD2 · ‎03-22-2022

In which directory did you create your packages?

Scarfido · ‎03-22-2022

I've tried two setups with identical results :

directly inside the ~/nso-5.7/nso-instance/packages directory
inside the ~/nso-5.7/packages/services directory and including my package in the ncs-setup command

RichardD2 · ‎03-22-2022

The workflow I've followed that gives me consistent results:

Make sure to source ~/nso-5.7/nso-instance/ncsrc

Create my packages inside ~/nso-5.7/nso-instance/packages directory (put any NED's there as well)

[When NSO starts it looks for packages to load using a list of directories listed under ncs-config/load-path in the ncs.conf file. NSO copies the files to a private directory defined in ncs-config/state-dir (also in ncs.conf), loads the packages, which is why it's better not to touch that directory]

Forgot to ask if this is a local or system-install as directories look a bit different. Admin-guide says:

In a "system install" of NSO, packages are always installed (normally by means of symbolic links) in the
packages subdirectory of the "run directory", i.e. by default /var/opt/ncs/packages, and the
private directory tree is created in the state subdirectory, i.e. by default /var/opt/ncs/state.

Does show packages package oper-status show everything is up?

Scarfido · ‎03-23-2022

Thanks for your help

For the source command, do you mean "source ~/nso-5.7/ncsrc " ? I don't have a ncsrc file in my instance directory.

I'm using a local install.

Here is the output of show packages package oper-status

                                                                                                        PACKAGE
                          PROGRAM                                                                       META     FILE
                          CODE     JAVA           PYTHON         BAD NCS  PACKAGE  PACKAGE  CIRCULAR    DATA     LOAD   ERROR
NAME                  UP  ERROR    UNINITIALIZED  UNINITIALIZED  VERSION  NAME     VERSION  DEPENDENCY  ERROR    ERROR  INFO
-------------------------------------------------------------------------------------------------------------------------------
cisco-iosxr-cli-7.38  X   -        -              -              -        -        -        -           -        -      -
juniper-junos-nc-4.3  X   -        -              -              -        -        -        -           -        -      -
my-package            X   -        -              -              -        -        -        -           -        -      -

u.avsec · ‎03-22-2022

Hey.

I have almost no good experiences messing with packages-in-use dir.

First check that packages are correctly placed, then that they are compiled with 'make clean all'. Then reload again.

If that doesn't do the trick.

What I would do next is remove all packages, except NEDs that you didn't mess with code wise, from .../packages and do a reload. If affording that luxury is possible. (Yes, you will wreck your NSO config.) Then I would check custom packages for any abnormalities. Place them back in and reload.

Scarfido · ‎03-23-2022

Hi,

It's also seems to me that it is a bad idea to mess with the state directory !

All my packages are detected by NSO so I assume they're placed correctly. I've tried to use symlinks and do the following :

Delete the simlink to my package in the nso-instance/packages directory
Run "request packages reload force" (NSO force me to use the "force" parameter so I seems that it knows that I've removed the package"
Create the simlink again
Run "request packages reload"

Doing so gets me the

>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.

prompt but the package isn't properly reloaded.

What should I do to compile my package ? When I modify my yang model, I run "make" in the "src" directory but I don't have a Makefile for the python directory.

I can absolutely burn my NSO config and I have already tried to completely delete and recreate an instance. I'm only using 2 NEDs on top of my package

u.avsec · ‎03-23-2022

For yang changes what you do is fine. Apart from make you can also do 'make clean all'. This will compile package from scratch (depending on the changes in yang I sometimes find that make reuses stuff from previous compiles which can sometimes break stuff in my experience)

For python changes you can just packages reload or packages package my-package redeploy and python changes will be loaded in.

When you say package is not properly reloaded, what criteria you base that observation on? It doesn't pop up when you try to configure a serviceand commit it or something else?

Scarfido · ‎03-23-2022

I usually check if my modifications are taken into account by adding a log message in my code and look if I see it when I create a service.

I still have problems with packages reload or packages package my-package redeploy but I found out that if I do ncs --stop and ncs --with-package-reload(-force), it works reliably

RichardD2 · ‎03-23-2022

This looks similar to issues others have had. Do you have all the prerequisites installed? As in Java JDK 8.0 or later, Python 3.7 or higher, ANT etc?

Check out the NSO Installation Guide in the docs folder.

Scarfido · ‎03-24-2022

I've checked and it seems that I meet the requirements

alain@LAPTOP-IUDEM59:~/nso-5.7/nso-instance$ java --version
openjdk 11.0.13 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.13+8-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
alain@LAPTOP-IUDEM59:~/nso-5.7/nso-instance$ python --version
Python 3.8.10
alain@LAPTOP-IUDEM59:~/nso-5.7/nso-instance$ ant -version
Apache Ant(TM) version 1.10.7 compiled on October 24 2019

Also, I'm running NSO on top of WSL and I may have found a issue with files that are copied from Windows : I had 2 templates files that I had stored in my Windows download directory. I've copied them to the package folder and NSO gave me an error about not been able to load them.

Then, I tried creating 2 new files, directly inside the package (inside WSL) and copy-pasting the content and NSO was able to use them without issues