NSO python3 inheritance error - ImportError: No module named

[rainnomm56]

I put all my most used helper scripts in "/var/opt/ncs/ext".For services who need those external scripts i make symlink in service python folder.

     rain@ohoo:/var/opt/ncs/devstuff/hostlist/python/hostlist$ ll

     lrwxrwxrwx 1 root ncsadmin   16 mai   15 17:02 ext -> /var/opt/ncs/ext

     -rw-rwxr-- 1 root ncsadmin    0 mai   14 00:22 __init__.py

     -rw-rwxr-- 1 root ncsadmin 6030 mai   15 21:42 main.py

Inside main.py I just import necessary module from that folder.

     from ext.newhostlist import Newhostlist

Thus far I've used default python 2.7.something and this solution has worked adequately. But i'm tired of working around python problems (last one was that enum implementation sucks), so i tried to switch NSO to python3. I did this: NSO: Running Python code with a different python version

Now with python3 NSO is not able to pick up those symlinks actually its not able to import anything from ".../python/hostlist" folder,:

<INFO> 15-May-2018::22:12:52.441 hostlist MainThread: - Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]

<INFO> 15-May-2018::22:12:52.441 hostlist MainThread: - Starting...

<INFO> 15-May-2018::22:12:52.454 hostlist MainThread: - Started

<ERROR> 15-May-2018::22:12:52.569 hostlist MainThread: - Traceback (most recent call last):

  File "/opt/ncs/ncs-4.6.1/src/ncs/pyapi/ncs_pyvm/startup.py", line 246, in start_components

    pname=pname, cname=cname, main_q=self.msg_sink)

  File "/opt/ncs/ncs-4.6.1/src/ncs/pyapi/ncs_pyvm/ncsthreads.py", line 153, in __init__

    mod = __import__(modname)

  File "/var/opt/ncs/state/packages-in-use/3/hostlist/python/hostlist/main.py", line 10, in <module>

    from ext.newhostlist import Newhostlist

ImportError: No module named 'ext'

I'm not a python programmer, NSO is my second project where I do some python, so i made test folder and this inheritance thingy works with python3 just fine outside of NSO.

rain@ohoo:~/project/tmp/1$ ll

-rw-rw-r-- 1 rain rain   98 mai   15 22:16 123.py

lrwxrwxrwx 1 rain rain   16 mai   15 21:19 ext -> /var/opt/ncs/ext

rain@ohoo:~/project/tmp/1$ cat 123.py

from ext.newhostlist import Newhostlist

obj=Newhostlist()

print(obj.find_sr('kjj-mgw1','vpnsr'))

rain@ohoo:~/project/tmp/1$ python3 123.py

noe-sr1

I did this python3 switch on my own laptop NSO. I also have test NSO server with python2.7 so i compared "sys.path" variables, but as far as I can see they are mostly same:

NSO server with python 2.7 - print(sys.path):

['/opt/ncs/ncs-4.5.2/src/ncs/pyapi/ncs_pyvm'

'/opt/ncs/ncs-4.5.2/src/ncs/pyapi'

'/var/opt/ncs/state/packages-in-use/2/hostlist/python'

'/usr/lib/python2.7'

'/usr/lib/python2.7/plat-x86_64-linux-gnu'

'/usr/lib/python2.7/lib-tk'

'/usr/lib/python2.7/lib-old'

'/usr/lib/python2.7/lib-dynload'

'/usr/local/lib/python2.7/dist-packages'

'/usr/lib/python2.7/dist-packages']

My laptop with python 3 NSO - print(sys.path):

['/opt/ncs/ncs-4.6.1/src/ncs/pyapi/ncs_pyvm'

'/opt/ncs/ncs-4.6.1/src/ncs/pyapi'

'/var/opt/ncs/state/packages-in-use/3/hostlist/python'

'/usr/lib/python35.zip'

'/usr/lib/python3.5'

'/usr/lib/python3.5/plat-x86_64-linux-gnu'

'/usr/lib/python3.5/lib-dynload'

'/usr/local/lib/python3.5/dist-packages'

'/usr/lib/python3/dist-packages']

What else is wrong?

[tbjurman · Accepted solution]

Hi Rain.

The problem here is not related to running under NSO (well, yes, indirectly) and has nothing to do with symbolic links.

Python 3 has a completely new module import system which need some special treatment. If we have a structure like this (trying to mimic yours):

  hostlist

  └── python

      └── hostlist

          ├── ext

          │  ├── __init__.py

          │  └── newhostlist.py

          ├── __init__.py

          └── main.py

main.py looks like this:

  from ext.newhostlist import Newhostlist

Now, suppose our PYTHONPATH contains hostlist/python just as it would when running under NSO (we can simulate that by making that our current directory). Let's see what happens if we try to import hostlist.main (just as NSO does):

$ cd hostlist/python

$ python3

>>> import hostlist.main

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/tmp/hostlist/python/hostlist/main.py", line 1, in <module>

    from ext.newhostlist import Newhostlist

ImportError: No module named 'ext'

>>>

It's trying to find ext somewhere in PYTHONPATH without success. It did work with Python 2.7, but with Python 3 we explicitly need to tell it to do an import relative to the current module (main.py). So let's change our main.py to this:

  from . ext.newhostlist import Newhostlist

Let's see what happens now:

$ python3

>>> import hostlist.main

>>>

And there you have you answer. Good Luck!

/Tomas