Upgrading to NSO 5.3.2.3: (ncs.log)The NCS Java VM synchronization failed (timeout) - stopping jvm

[JimBoucher]

I am trying to upgrade to NSO 5.3.2.3 from 5.2.0.3 and I am getting failures starting the Java VM. Does anyone know what this is about?

(ncs.log)

<DEBUG> 10-Aug-2020::16:50:48.191 sv0a0171 ncs[90606]: - Loading file /apps/opt/ncs/ncs-5.3.2.3/etc/ncs/ncs.ccl
<DEBUG> 10-Aug-2020::16:50:48.201 sv0a0171 ncs[90606]: - Loading file /apps/opt/ncs/ncs-5.3.2.3/etc/ncs/smart-license.ccl
<DEBUG> 10-Aug-2020::16:50:48.818 sv0a0171 ncs[90606]: - Loading file /apps/opt/ncs/ncs-5.3.2.3/scripts/command/device_save.sh
<INFO> 10-Aug-2020::16:50:48.915 sv0a0171 ncs[90606]: - Starting to listen for Internal IPC on 127.0.0.1:4569
<INFO> 10-Aug-2020::16:50:50.515 sv0a0171 ncs[90606]: - Starting the NCS Java VM
<DEBUG> 10-Aug-2020::16:50:51.906 sv0a0171 ncs[90606]: - Loading file /apps/var/opt/ncs/state/packages-in-use/1/slf-port-channels/templates/slf-port-channels-template.xml
<INFO> 10-Aug-2020::16:50:52.085 sv0a0171 ncs[90606]: - Starting to listen for NETCONF SSH on 0.0.0.0:2022
<INFO> 10-Aug-2020::16:50:52.306 sv0a0171 ncs[90606]: - Starting to listen for NETCONF TCP on 127.0.0.1:2023
<INFO> 10-Aug-2020::16:50:52.753 sv0a0171 ncs[90606]: - Starting to listen for WebUI SSL on 0.0.0.0:443
<CRIT> 10-Aug-2020::16:50:52.753 sv0a0171 ncs[90606]: - Cannot bind to WebUI socket 0.0.0.0:443 : unknown reason
<INFO> 10-Aug-2020::16:50:52.998 sv0a0171 ncs[90606]: - Starting to listen for CLI SSH on 0.0.0.0:2024
<ERR> 10-Aug-2020::16:51:25.389 sv0a0171 ncs[90606]: - The NCS Java VM synchronization failed (timeout) - stopping jvm
<INFO> 10-Aug-2020::16:51:25.735 sv0a0171 ncs[90606]: - NCS started vsn: 5.3.2.3

[perander]

This log message occurs either because of a timeout or a crash.

It is documented in the NSO Development book in the chapter The NSO Java VM,
see particularly section Debugging Startup.

Snippet from above mentioned section:

If you have problems with the Java VM crashing during startup, a common pitfall is running out of memory (either total memory on the machine, or heap in the JVM). If you have a lot of Java code (or a loaded system) perhaps the Java VM did not start in time. Try to determine the root cause, check ncs.log and ncs-java-vm.log, and if needed increase the timeout.

For complex problems, for example with the class loader, try logging the internals of the startup:

admin@ncs(config)# java-vm java-logging logger com.tailf.ncs level level-all
admin@ncs(config-logger-com.tailf.maapi)# commit
Commit complete.

Setting this will result in a lot more detailed information in ncs-java-vm.log during startup.

[JimBoucher · Accepted solution]

Sorry for the false alarm on this one. I had missed that I needed to update the /var/opt/ncs/current/ncsrc file with my java vm options when I changed the link to currrent. Thing started after that. Thanks.

NCS_JAVA_VM_OPTIONS="-Xss8192K -Xms2G -Xmx4G";

export NCS_JAVA_VM_OPTIONS