NSO Docker package reload hanging with cisco-ios

[Sean Chandler]

Hi Folks,

Since version 3.4 or so I have been using NSO in a docker container to great success.  A recent issue with cisco-ucs ned caused me to try upgrading to latest 4.5.3 (also tried 4.5.2) and discovered that package loading would hang on cisco-ios.  Removing this package alleviates the issue entirely.  During the probelm 'show packages package oper-status' shows the java vm uninitialized.  Looking in the log ncs-java-vm.log I see the following:

<INFO> 30-Jan-2018::14:05:41.921 NcsMain JVM-Launcher: - LOADING PACKAGE ["cisco-asa"] -->

<WARN> 30-Jan-2018::14:05:42.195 ResourceManager JVM-Launcher: - Resource annotated field :ASANedCli::mm already set, no resource injection performed

<WARN> 30-Jan-2018::14:05:42.195 ResourceManager JVM-Launcher: - Resource annotated field :ASANedCli::cdb already set, no resource injection performed

<INFO> 30-Jan-2018::14:05:42.195 NcsMain JVM-Launcher: - LOADING PACKAGE ["cisco-asa"] --> LOADED

<INFO> 30-Jan-2018::14:05:42.195 NcsMain JVM-Launcher: - DONE COMMAND --> LOAD_PACKAGE

<INFO> 30-Jan-2018::14:05:42.196 NcsMain JVM-Launcher: - READ SOCKET =>

<INFO> 30-Jan-2018::14:05:42.196 NcsMain JVM-Launcher: -  <= GOT LOAD_PACKAGE

<INFO> 30-Jan-2018::14:05:42.197 NcsMain JVM-Launcher: - FOUND TYPE: callback

<INFO> 30-Jan-2018::14:05:42.197 NcsMain JVM-Launcher: - FOUND TYPE: ned

<INFO> 30-Jan-2018::14:05:42.198 NcsMain JVM-Launcher: - LOADING PACKAGE ["cisco-ios"] -->

Has anyone run into this issue in general use?  One caveat I have is that I have modified /etc/init.d slightly (below) but everything else I see appears the same.

root@innopod-ncs:/src# cat package-links.sh

#!/bin/bash

cd /var/opt/ncs/packages/

for d in `find /opt/ncs/packages/ -maxdepth 1 -type d`; do

    ls $d/README > /dev/null 2>&1

    if [ $? == 0 ]; then

        echo "Adding link from $d to /var/opt/ncs/packages/"

        ln -sf $d

    fi

done

root@innopod-ncs:/src# diff ncs /etc/init.d/ncs

38,43d37

< ARG=$2

< case "$ARG" in

<     daemon)

<         daemon='--foreground'

<         ;;

< esac

48c42

<         $ncs --cd ${rundir} ${heart} ${conf} ${daemon}

---

>         $ncs --cd ${rundir} ${heart} ${conf}

77,78d70

<         /src/package-links.sh

<

83c75

<             $ncs --cd ${rundir}  ${heart} ${conf} ${daemon}

---

>             $ncs --cd ${rundir}  ${heart} ${conf}

88,89d79

<         /src/package-links.sh

<

96c86

<             $ncs --cd ${rundir}  ${heart} ${conf} ${daemon}

---

>             $ncs --cd ${rundir}  ${heart} ${conf}

[frjansso · Accepted solution]

This is most certainly a memory issue where the JVM is killed.

There can be two (AFAIK) reasons for this

JVM limit

You probably have to up the memory limit for NSO's JVM, please pass this environment var to your container: NCS_JAVA_VM_OPTIONS=-Xmx2G

Container limit

If pure Docker, see the "--memory bytes" flag.

If k8s, see "resources" for the pod, e.g.:

          resources:

            limits:

              memory: 4Gi

            requests:

              memory: 256Mi

[Sean Chandler]

Instant fix!  Thanks so much.  I would never have figured that out on my own   4.5.3 is now working perfectly.

-s