NSO and Dockers

[mcardani]

Hi Team, I'm looking at guidelines or any documentation to have NSO running into a Docker, especially how to manage CDB with respect to the underneath host filesystem.

Thanks,

Marco.

[Dan.Sullivan]

I don't think there is any official NSO documentation regarding Docker.

-Dan

[frjansso · Accepted solution]

I've started to look into how to run NSO in Docker/K8s and there are a few things to consider.

Run NSO in the foreground to get logs to stdout

Mount either the entire project as a persistent volume or just the ncs-cdb and state directories.

You could argue that the logs directory should be persistent, OTOH if you capture logs from stdout already you may no need that.

If you use k8s, it's good practice to specify limits and requests as well as specifying a readinessProbe and a livenessProbe.

[Sean Chandler]

I've been running NSO in docker for some time now but have recently (in version 4.5.3) run into an issue where including cisco-ios package causes package loading to hang indefinitely.  show packages package oper-status shows the java vm uninitialized.  Have you run into this issue?

[mcardani]

No, we haven’t, we are not using cisco-ios package, we are developing a generic ned based on corba.

[adam.allen]

Did you work out the problem, I have a very messy TAC case and I would love to ship a Dockerfile to get the environments stuff aligned. Unfortunately we only see the problem on system installs.

My first attempt was - for small packages this works, however for cisco-ios the fetch package command closes with a timeout.
echo "request software packages fetch package-from-file /tmp/vendor45/$file" | ncs_cli -u admin --noaaa
echo "request software packages install allow-version-mismatch replace-existing package $file" | ncs_cli -u admin --noaaa


So I use the dockerfile to get every package loaded except cisco-ios and do the above commands interactively - which works.

Then if I do request packages reload or /etc/init.d/ncs stop /etc/init.d/ncs restart-with-package reload I get to the following.

Bad configuration: /etc/ncs/ncs.conf:0: "/var/ncs/nso-4.4.x/state/packages-in-use.cur: Failed to rename to /var/ncs/nso-4.4.x/state/packages-in-use.old: cross-domain link"
Bad configuration: /etc/ncs/ncs.conf:0: "/var/ncs/nso-4.4.x/state/packages-in-use.cur: Failed to rename to /var/ncs/nso-4.4.x/state/packages-in-use.old: cross-domain link"
Daemon died status=21

I'm new to Docker but this feels something I haven't understood done right.

[adam.allen]

Notes for future folk stumbling across this.

If I'd used a persistent volume I might have not run into this issue. However I wanted to avoid persisting things as that makes more things to distribute.

It's nothing to do with the IOS ned - it just looked that way.

Summary.

- Package reloads fail, NCS packages - during a `request packages reload` or `/etc/init.d/ncs start-with-package-reload` NCS will move directories under `$NCS_RUN_DIR/state` and Docker throws cross-domain sym-link errors. Workaround is to manipulate packages without letting NCS do the reloads.
- `/etc/init.d/ncs stop`
- `tar xvfz /tmp/ncs-4.5.5.1-stockholm-0.0.7.tar.gz -C $NCS_RUN_DIR/state/packages-in-use.cur/1/`
- `/etc/init.d/ncs start`

- NCS commands piped in bash - if we do `echo 'request software ' | ncs_cli -u admin` with a big package get `*** CLOSE TIMEOUT ***` from ncs_cli. It's any command that takes a long time. This is what led me to install everything except cisco-ios, and this is why I was blaming the IOS package.

[xiaoxi.cheng]

We have been running NCS inside Docker for the last two years, externally mounting volume for directories under /var/opt/ncs except the packages dir.

The hang CLI issue is related to a bug in 4.5. The work around is to add a sleep after the echo command:

bash -c "(echo -e 'your cli command here'; sleep 2) | /opt/ncs/current/bin/ncs_cli -u admin ..."

[vleijon]

Moving files directly into  packages-in-use is not recommended. But you will have to have either an external mount with the files or make sure they are copied into the container in some other way. 

Copying the packages into a local packages directory has worked well for me.