Python VM Uninitialized in NSO 6.1 Docker Container with M2 Processor

[casbarre]

Hi everyone!

My team has decided to create a Docker container with NSO 6.1.7. The image contains a NSO System Install and a few NEDs and packages built with Python. NSO 6.1 seems to perfectly work with M2 outside of the container.  So we are sure this only happens inside the Docker container.  Everything seems to be fine but after running a packages reload, all packages get the Python VM Uninitialized error:

admin@ncs# show packages package oper-status   
packages package package-2
 oper-status python-uninitialized
packages package package-1
 oper-status python-uninitialized
packages package cisco-ios-cli-6.106
 oper-status up
packages package cisco-iosxr-cli-7.54
 oper-status up
packages package resource-manager
 oper-status up

Logs are not really showing any error.

ncs-python-vm.log file:

root@2fe84da1bc4c:/var/log/ncs# tail -f  ncs-python-vm.log
<INFO> 28-May-2024::10:02:38.892 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2857>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2856>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2855>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2854>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2853>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2852>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2851>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2849>
<INFO> 28-May-2024::10:02:38.893 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2848>
<INFO> 28-May-2024::10:02:38.900 2fe84da1bc4c ncs[148]: Got EXIT on Port= #Port<0.2847>

NCS.log file:

root@2fe84da1bc4c:/var/log/ncs# tail -f   ncs.log 
<INFO> 28-May-2024::10:02:07.438 2fe84da1bc4c ncs[148]: - Starting the NCS Python VM package-1
<INFO> 28-May-2024::10:02:07.587 2fe84da1bc4c ncs[148]: - Starting the NCS Python VM package-1
<ERR> 28-May-2024::10:02:37.707 2fe84da1bc4c ncs[148]: - The NCS Python VM package-1, watchmen startup timeout reached, terminating
<INFO> 28-May-2024::10:04:15.587 2fe84da1bc4c ncs[148]: - Smart Licensing Entitlement Notification: type = "notifyEnforcementMode", agentID = "sa1", notificationTime = "May 28 10:04:10 2024", version = "1.0", requestedDate = "May 28 10:04:07 2024", tag = "regid.2015-10.com.cisco.NSO-platform-production,1.0_d1445dab-9d96-4593-99f2-6f633b8a759c", enforceMode = "eval", daysLeft = 88, expiryDate = "Aug 24 06:08:07 2024", requestedCount = 1

One of the package's Python log file:

root@2fe84da1bc4c:/var/log/ncs# tail -f ncs-python-vm-package-1.log
<DEBUG> 28-May-2024::10:02:08.462 package-1 MainThread: - cname=:package-1
<DEBUG> 28-May-2024::10:02:08.463 package-1 MainThread: - classes=[':package-1main.Main']
<DEBUG> 28-May-2024::10:02:08.597 package-1 MainThread: - Imported: :package-1.main
<DEBUG> 28-May-2024::10:02:08.598 package-1 MainThread: - *** main_class: Main
<DEBUG> 28-May-2024::10:02:08.598 package-1MainThread: - Getattr Main from: :package-1.main
<DEBUG> 28-May-2024::10:02:08.598 package-1r MainThread: - Creating class: Main
<DEBUG> 28-May-2024::10:02:08.598 package-1 MainThread: - --- Init NcsPyVM object
<DEBUG> 28-May-2024::10:02:08.598 package-1 MainThread: - Created class: Main
<DEBUG> 28-May-2024::10:02:08.598 package-1MainThread: - Created Thread: :package-1.main.Main
<DEBUG> 28-May-2024::10:02:08.615 batch-executor ComponentThread:package-1: - Starting Main Class: package-1.main.Main

 What do you think that should be done to fix the issue?

Thanks in advance!

[Torbjørn]

Can you post the Dockerfile?

[Islam_Hassanin]

packages reload is a heavy NSO task which reloads all the packages files, usually Uninitialized errors occurs due to memory/cpu thresholds. so make sure that your container has no limitation for the resources.

Also you can check python-VM log file for your package under /var/log/ncs folder. but most properlly it's a resources related issue. 

[casbarre · Accepted solution]

The issue was fixed after adding this line to my ncs.conf file:

<python-vm>
    <start-timeout>PT240S</start-timeout>
</python-vm>

Thank you all for your kind advice.  

[cohult]

Note that as of NSO 6.3 there is an arm64 variant of the NSO containers. For example, the performance improvement is significant on Apple silicon Mx processors over instead of running x86_64 (Intel) Docker images on Apple silicon with Rosetta 2.