Install Cisco IOS and Cisco IOS XR NEDs - Package reload failed

kiran kotari · ‎04-11-2018

Hi Experts,

I am trying a new lab setup using vbox and vagrants with ubuntu/trusty64. At "Install Cisco IOS and Cisco IOS XR NEDs" facing an issue on package reload.

How to debug the issue and where couldn't it went wrong.

admin@ncs> request packages reload | debug service

>>> System upgrade is starting.

>>> Sessions in configure mode must exit to operational mode.

>>> No configuration changes can be performed until upgrade has completed.

>>> System upgrade has completed successfully.

reload-result {

package cisco-ios

result false

info [LOAD_PACKAGE[com.tailf.packages.ned.ios.IOSNedCli]]

}

reload-result {

package cisco-iosxr

result false

info [LOAD_PACKAGE[com.tailf.packages.ned.iosxr.IosxrNedCli]]

}

[ok][2018-04-11 09:53:56]

admin@ncs>

kiran kotari · ‎04-16-2018

Adding logs:-

==> ncs-java-vm.log <==

<INFO> 16-Apr-2018::13:19:03.141 NcsMain JVM-Launcher: - <= GOT UNLOAD_ALL

<INFO> 16-Apr-2018::13:19:03.141 NcsMain JVM-Launcher: - UNLOAD PACKAGE ["cisco-ios"] -->

<INFO> 16-Apr-2018::13:19:03.141 NcsMain JVM-Launcher: - UNLOAD PACKAGE ["cisco-ios"] --> OK

<INFO> 16-Apr-2018::13:19:03.142 NcsMain JVM-Launcher: - DONE COMMAND --> UNLOAD_ALL

<INFO> 16-Apr-2018::13:19:03.142 NcsMain JVM-Launcher: - READ SOCKET =>

<INFO> 16-Apr-2018::13:19:03.142 NcsMain JVM-Launcher: - <= GOT RELOAD_SCHEMA

<INFO> 16-Apr-2018::13:19:03.386 NcsMain JVM-Launcher: - DONE COMMAND --> RELOAD_SCHEMA

<INFO> 16-Apr-2018::13:19:03.387 NcsMain JVM-Launcher: - READ SOCKET =>

<INFO> 16-Apr-2018::13:19:03.387 NcsMain JVM-Launcher: - <= GOT LOAD_SHARED_JARS

<INFO> 16-Apr-2018::13:19:03.387 NcsMain JVM-Launcher: - LOAD SHARED PACKAGE ["cisco-ios"] -->

<INFO> 16-Apr-2018::13:19:03.388 NcsMain JVM-Launcher: - loading shared jar: file:///home/vagrant/ncs-run/state/packages-in-use/1/cisco-ios/shared-jar/ios-ns.jar

<INFO> 16-Apr-2018::13:19:03.388 NcsMain JVM-Launcher: - LOAD SHARED PACKAGE ["cisco-ios"] --> OK

<INFO> 16-Apr-2018::13:19:03.388 NcsMain JVM-Launcher: - DONE COMMAND --> LOAD_SHARED_JARS

<INFO> 16-Apr-2018::13:19:03.388 NcsMain JVM-Launcher: - READ SOCKET =>

<INFO> 16-Apr-2018::13:19:03.388 NcsMain JVM-Launcher: - <= GOT LOAD_PACKAGE

<INFO> 16-Apr-2018::13:19:03.389 NcsMain JVM-Launcher: - FOUND TYPE: upgrade

<INFO> 16-Apr-2018::13:19:03.389 NcsMain JVM-Launcher: - FOUND TYPE: ned

<INFO> 16-Apr-2018::13:19:03.389 NcsMain JVM-Launcher: - LOADING PACKAGE ["cisco-ios"] -->

<ERROR> 16-Apr-2018::13:19:03.403 NedMuxManager JVM-Launcher: - Exception while executing an LOAD_PACKAGE transition

com.tailf.ncs.NcsException: com.tailf.packages.ned.ios.IOSNedCli

at com.tailf.ncs.ctrl.AbstractPDEntry.load(AbstractPDEntry.java:255)

at com.tailf.ncs.ctrl.AbstractPDEntry$1.execute(AbstractPDEntry.java:84)

at com.tailf.ncs.ctrl.fsm.FSM$Transition.execute(FSM.java:286)

at com.tailf.ncs.ctrl.fsm.FSM.event(FSM.java:174)

at com.tailf.ncs.ctrl.NedMuxManager.loadPackageEvent(NedMuxManager.java:166)

at com.tailf.ncs.NcsMain.loadPackage(NcsMain.java:956)

at com.tailf.ncs.NcsMain.access$1000(NcsMain.java:88)

at com.tailf.ncs.NcsMain$8.execute(NcsMain.java:303)

at com.tailf.ncs.ctrl.fsm.FSM$Transition.execute(FSM.java:286)

at com.tailf.ncs.ctrl.fsm.FSM.event(FSM.java:174)

at com.tailf.ncs.NcsMain.mainLoop(NcsMain.java:694)

at com.tailf.ncs.NcsMain.run(NcsMain.java:490)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.tailf.packages.ned.ios.IOSNedCli

at javax.management.loading.MLet.findClass(MLet.java:988)

at javax.management.loading.MLet.findClass(MLet.java:922)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at com.tailf.ncs.ctrl.AbstractPDEntry.load(AbstractPDEntry.java:244)

... 12 more

<ERROR> 16-Apr-2018::13:19:03.404 NedMuxManager JVM-Launcher: - Exception while executing an EXCEPTION transition for exception com.tailf.packages.ned.ios.IOSNedCli

com.tailf.ncs.NcsException: com.tailf.packages.ned.ios.IOSNedCli

at com.tailf.ncs.ctrl.AbstractPDEntry.load(AbstractPDEntry.java:255)

at com.tailf.ncs.ctrl.AbstractPDEntry$1.execute(AbstractPDEntry.java:84)

at com.tailf.ncs.ctrl.fsm.FSM$Transition.execute(FSM.java:286)

at com.tailf.ncs.ctrl.fsm.FSM.event(FSM.java:174)

at com.tailf.ncs.ctrl.NedMuxManager.loadPackageEvent(NedMuxManager.java:166)

at com.tailf.ncs.NcsMain.loadPackage(NcsMain.java:956)

at com.tailf.ncs.NcsMain.access$1000(NcsMain.java:88)

at com.tailf.ncs.NcsMain$8.execute(NcsMain.java:303)

at com.tailf.ncs.ctrl.fsm.FSM$Transition.execute(FSM.java:286)

at com.tailf.ncs.ctrl.fsm.FSM.event(FSM.java:174)

at com.tailf.ncs.NcsMain.mainLoop(NcsMain.java:694)

at com.tailf.ncs.NcsMain.run(NcsMain.java:490)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.tailf.packages.ned.ios.IOSNedCli

at javax.management.loading.MLet.findClass(MLet.java:988)

at javax.management.loading.MLet.findClass(MLet.java:922)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at com.tailf.ncs.ctrl.AbstractPDEntry.load(AbstractPDEntry.java:244)

... 12 more

<ERROR> 16-Apr-2018::13:19:03.405 NcsMain JVM-Launcher: - catched exception in mainLoop

com.tailf.ncs.ctrl.NcsCtrlException: LOAD_PACKAGE[com.tailf.packages.ned.ios.IOSNedCli]

at com.tailf.ncs.ctrl.MuxManager.addToDeployException(MuxManager.java:37)

at com.tailf.ncs.NcsMain.loadPackage(NcsMain.java:965)

at com.tailf.ncs.NcsMain.access$1000(NcsMain.java:88)

at com.tailf.ncs.NcsMain$8.execute(NcsMain.java:303)

at com.tailf.ncs.ctrl.fsm.FSM$Transition.execute(FSM.java:286)

at com.tailf.ncs.ctrl.fsm.FSM.event(FSM.java:174)

at com.tailf.ncs.NcsMain.mainLoop(NcsMain.java:694)

at com.tailf.ncs.NcsMain.run(NcsMain.java:490)

at java.lang.Thread.run(Thread.java:748)

<INFO> 16-Apr-2018::13:19:03.406 NcsMain JVM-Launcher: - READ SOCKET =>

<INFO> 16-Apr-2018::13:19:03.406 NcsMain JVM-Launcher: - <= GOT DONE_LOADING

<INFO> 16-Apr-2018::13:19:03.407 NcsMain JVM-Launcher: - DONE LOADING -->

<INFO> 16-Apr-2018::13:19:03.411 NcsMain JVM-Launcher: - DONE LOADING --> OK

<INFO> 16-Apr-2018::13:19:03.411 NcsMain JVM-Launcher: - DONE COMMAND --> DONE_LOADING

<INFO> 16-Apr-2018::13:19:03.412 NcsMain JVM-Launcher: - READ SOCKET =>

Jan Lindblad · ‎04-16-2018

Maybe you could try one NED at a time? If they work separately, maybe you are running out of Java runtime memory? (search this forum for ways to increase). If they don't work even separately, you might have a build problem. Maybe a make clean all in each NED would fix the problem, or at least reveal the issue if you look at the compiler messages.

kiran kotari · ‎04-20-2018

Hi Jan,

Yes, they are working separately and I realized the Java run-time memory was 512 MB.

Now I have increased the java run-time memory to 4096 MB, still the issue not resolved. I am facing the same issue on package reload

Heap size after inc.

vagrant@vagrant-ubuntu-trusty-64:~/ncslab/services/packages$ java -XX:+PrintFlagsFinal -version | grep -iE 'heapsize'

Picked up _JAVA_OPTIONS: -Xmx4096m -Xmx4096m

uintx ErgoHeapSizeLimit = 0 {product}

uintx HeapSizePerGCThread = 87241520 {product}

uintx InitialHeapSize := 65011712 {product}

uintx LargePageHeapSizeThreshold = 134217728 {product}

uintx MaxHeapSize := 4294967296 {product} --> 4GB

java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

vagrant@vagrant-ubuntu-trusty-64:~/ncslab/services/packages$

I found the below log.

==> ncs.log <==

<ERR> 20-Apr-2018::15:14:10.723 vagrant-ubuntu-trusty-64 ncs[1870]: - Failed to setup the shared memory schema

Can you please point the discussion the way to increase java runtime memory. That might help little bit.

tepearso · ‎04-20-2018

Stefano explained how to increase NSO's java heap size in a recent post. Is this how you increased the java heap in your setup?

Re: Sync from error(Message) By Daniel Frey, posted Apr 8 2018

What version of the cisco-ios NED are you running? Earlier this week, one of my customers tried to install NSO 4.6 and ran into a similar issue - running out of java heap while loading NEDs. I had him increase the java heap, but it still failed. We eventually realized that he incorrectly installed the NED. Instead of pointing to the latest cisco-ios NED, he created a symbolic link to the old, default cisco-ios NED that comes with the 4.6 distribution. Once he correctly installed the NED, he was off and running. I'm not sure if you have the same issue, but figured I would share... maybe you'll get lucky.