topic Re: lxml 3.4.4 really required? in Tools

lxml 3.4.4 really required?

dvulovic — Fri, 24 Mar 2017 15:01:47 GMT

Hi,

I have tried today to install YDK on Windows and it couldn't do it because of lxml. YDK (including just released 0.54) depends on exact lxml version 3.4.4 from 2015. As I have newer version by default, I had to use virtualenv but pip could not build lxml from source, as it requires libxml2 and libxslt which I could not install easily (I'm not saying that it cannot be done but it certainly required above-average knowledge). On Windows, lxml is usually installed as a binary, however version 3.4.4 is quite old and there is no binary for this version and Python 3.5 that I could find (only up to 3.2 is available). I could install different version of Python etc. but I'm wondering is this really necessary i.e. is YDK really so dependent of lxml 3.4.4 ? If not, it would be a life-saver to have lxml dependency that would not be constrained to single (yet old) version.

Any advice (and of course corrections) is greatly appreciated!

Best regards,

Djordje

Re: lxml 3.4.4 really required?

abhirame — Fri, 24 Mar 2017 15:12:38 GMT

Hi,

We have got ydk installed on windows by following these instructutions. Can you try this?

Thanks

Re: lxml 3.4.4 really required?

dvulovic — Fri, 24 Mar 2017 15:18:56 GMT

Hi,

these instructions are the just the basic requirements.

See what happens when I try to install YDK:

(YDK) C:\Users\dvulovic\virtualenv\YDK>pip install ydk

Collecting ydk

Using cached ydk-0.5.4-py2.py3-none-any.whl

Collecting lxml==3.4.4 (from ydk)

Using cached lxml-3.4.4.tar.gz

Collecting ncclient>=0.4.7 (from ydk)

Using cached ncclient-0.5.3.tar.gz

Collecting enum34==1.1.3 (from ydk)

Using cached enum34-1.1.3-py2.py3-none-any.whl

Requirement already satisfied: pybind11==2.0.1 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from ydk)

Requirement already satisfied: setuptools>0.6 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages\setuptools-18.1-py3.5.egg (from ncclient>=0.4.7->ydk)

Requirement already satisfied: paramiko>=1.15.0 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from ncclient>=0.4.7->ydk)

Requirement already satisfied: six in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from ncclient>=0.4.7->ydk)

Requirement already satisfied: cryptography>=1.1 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: pyasn1>=0.1.7 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: asn1crypto>=0.21.0 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: idna>=2.1 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: packaging in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: cffi>=1.4.1 in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: pyparsing in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from packaging->cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Requirement already satisfied: pycparser in c:\users\dvulovic\virtualenv\ydk\lib\site-packages (from cffi>=1.4.1->cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk)

Installing collected packages: lxml, ncclient, enum34, ydk

Running setup.py install for lxml ... error

Complete output from command C:\Users\dvulovic\virtualenv\YDK\Scripts\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\dvulovic\\AppData\\Local\\Temp\\pip-build-yquark_5\\lxml\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\dvulovic\AppData\Local\Temp\pip-r67p4r74-record\install-record.txt --single-version-externally-managed --compile --install-headers C:\Users\dvulovic\virtualenv\YDK\include\site\python3.5\lxml:

Building lxml version 3.4.4.

Building without Cython.

ERROR: b"'xslt-config' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n"

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt

C:\Users\dvulovic\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'bugtrack_url'

warnings.warn(msg)

running install

running build

running build_py

creating build

creating build\lib.win-amd64-3.5

creating build\lib.win-amd64-3.5\lxml

copying src\lxml\builder.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\cssselect.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\doctestcompare.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\ElementInclude.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\pyclasslookup.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\sax.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\usedoctest.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\_elementpath.py -> build\lib.win-amd64-3.5\lxml

copying src\lxml\__init__.py -> build\lib.win-amd64-3.5\lxml

creating build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\__init__.py -> build\lib.win-amd64-3.5\lxml\includes

creating build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\builder.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\clean.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\defs.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\diff.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\ElementSoup.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\formfill.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\html5parser.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\soupparser.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\usedoctest.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\_diffcommand.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\_html5builder.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\_setmixin.py -> build\lib.win-amd64-3.5\lxml\html

copying src\lxml\html\__init__.py -> build\lib.win-amd64-3.5\lxml\html

creating build\lib.win-amd64-3.5\lxml\isoschematron

copying src\lxml\isoschematron\__init__.py -> build\lib.win-amd64-3.5\lxml\isoschematron

copying src\lxml\lxml.etree.h -> build\lib.win-amd64-3.5\lxml

copying src\lxml\lxml.etree_api.h -> build\lib.win-amd64-3.5\lxml

copying src\lxml\includes\c14n.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\config.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\dtdvalid.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\etreepublic.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\htmlparser.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\relaxng.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\schematron.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\tree.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\uri.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\xinclude.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\xmlerror.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\xmlparser.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\xmlschema.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\xpath.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\xslt.pxd -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\etree_defs.h -> build\lib.win-amd64-3.5\lxml\includes

copying src\lxml\includes\lxml-version.h -> build\lib.win-amd64-3.5\lxml\includes

creating build\lib.win-amd64-3.5\lxml\isoschematron\resources

creating build\lib.win-amd64-3.5\lxml\isoschematron\resources\rng

copying src\lxml\isoschematron\resources\rng\iso-schematron.rng -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\rng

creating build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl

copying src\lxml\isoschematron\resources\xsl\RNG2Schtrn.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl

copying src\lxml\isoschematron\resources\xsl\XSD2Schtrn.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl

creating build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_abstract_expand.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_dsdl_include.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_schematron_message.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_schematron_skeleton_for_xslt1.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_svrl_for_xslt1.xsl -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\readme.txt -> build\lib.win-amd64-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1

running build_ext

building 'lxml.etree' extension

creating build\temp.win-amd64-3.5

creating build\temp.win-amd64-3.5\Release

creating build\temp.win-amd64-3.5\Release\src

creating build\temp.win-amd64-3.5\Release\src\lxml

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\dvulovic\AppData\Local\Temp\pip-build-yquark_5\lxml\src\lxml\includes -IC:\Users\dvulovic\virtualenv\YDK\include -IC:\Users\dvulovic\AppData\Local\Programs\Python\Python35\include -IC:\Users\dvulovic\AppData\Local\Programs\Python\Python35\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /Tcsrc\lxml\lxml.etree.c /Fobuild\temp.win-amd64-3.5\Release\src\lxml\lxml.etree.obj -w

cl : Command line warning D9025 : overriding '/W3' with '/w'

lxml.etree.c

C:\Users\dvulovic\AppData\Local\Temp\pip-build-yquark_5\lxml\src\lxml\includes\etree_defs.h(14): fatal error C1083: Cannot open include file: 'libxml/xmlversion.h': No such file or directory

error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit status 2

----------------------------------------

Command "C:\Users\dvulovic\virtualenv\YDK\Scripts\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\dvulovic\\AppData\\Local\\Temp\\pip-build-yquark_5\\lxml\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\dvulovic\AppData\Local\Temp\pip-r67p4r74-record\install-record.txt --single-version-externally-managed --compile --install-headers C:\Users\dvulovic\virtualenv\YDK\include\site\python3.5\lxml" failed with error code 1 in C:\Users\dvulovic\AppData\Local\Temp\pip-build-yquark_5\lxml\

Any idea how to overcome this and I will be very grateful.

Best regards,

Djordje

Re: lxml 3.4.4 really required?

abhirame — Fri, 24 Mar 2017 15:23:20 GMT

Unfortunately, I do not have access to a windows machine at the moment where I can reproduce this problem. An alternative you could maybe consider is using YDK's Vagrant sandbox, which will give you a ubuntu VM with YDK pre-installed.

Re: lxml 3.4.4 really required?

dvulovic — Fri, 24 Mar 2017 15:31:02 GMT

This is an interesting proposition for e.g. someone doing hands-on lab, but I want to use IDE on my PC. And to install IDE on Vagravt VM running on my PC is a no-go, sorry.

But to get back on the main question, is this really really necessary:

lxml==3.4.4

or it can become

lxml>=3.4.4

Best regards,

Djordje

Re: lxml 3.4.4 really required?

einarnn — Fri, 24 Mar 2017 15:38:26 GMT

Djordje,

We typically use “==“ in pip requirements because the version listed represents what we have actually tested with prior to release. It’s entirely possible that “lxml>=3.4.4” may be fine, but we haven’t verified it.

Cheers,

Einar

Re: lxml 3.4.4 really required?

abhirame — Fri, 24 Mar 2017 16:13:16 GMT

I tried upgrading to lxml 3.7.3 and so far, I do not see any issues with ydk (I am running ydk 0.5.4).

Re: lxml 3.4.4 really required?

saalvare — Tue, 28 Mar 2017 22:54:36 GMT

Following the instructions at:

http://ydk.cisco.com/py/docs/getting_started.html#windows

LXML 3.4.4 breaks YDK installation on Windows with latest Python 3 (3.6.1). Installation works just fine on Windows with latest Python 2 (2.7.13).

C:\Python27\Scripts>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
setuptools (28.8.0)


C:\Python27\Scripts>pip install ydk-models-cisco-ios-xr
Collecting ydk-models-cisco-ios-xr
  Using cached ydk_models_cisco_ios_xr-6.2.1-py2.py3-none-any.whl
Collecting enum34==1.1.3 (from ydk-models-cisco-ios-xr)
  Using cached enum34-1.1.3-py2.py3-none-any.whl
Collecting ydk-models-ietf>=0.1.1 (from ydk-models-cisco-ios-xr)
  Using cached ydk_models_ietf-0.1.1-py2.py3-none-any.whl
Collecting ydk-models-openconfig>=0.1.2 (from ydk-models-cisco-ios-xr)
  Using cached ydk_models_openconfig-0.1.2-py2.py3-none-any.whl
Collecting ncclient>=0.4.7 (from ydk-models-cisco-ios-xr)
  Using cached ncclient-0.5.3.tar.gz
Collecting ydk>=0.5.2 (from ydk-models-cisco-ios-xr)
  Using cached ydk-0.5.4-py2.py3-none-any.whl
Collecting lxml==3.4.4 (from ydk-models-cisco-ios-xr)
  Downloading lxml-3.4.4-cp27-none-win_amd64.whl (3.3MB)
    100% |################################| 3.3MB 290kB/s
Requirement already satisfied: setuptools>0.6 in c:\python27\lib\site-packages (from ncclient>=0.4.7->ydk-models-cisco-ios-xr)
Collecting paramiko>=1.15.0 (from ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached paramiko-2.1.2-py2.py3-none-any.whl
Collecting six (from ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pybind11==2.0.1 (from ydk>=0.5.2->ydk-models-cisco-ios-xr)
  Using cached pybind11-2.0.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached pyasn1-0.2.3-py2.py3-none-any.whl
Collecting cryptography>=1.1 (from paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading cryptography-1.8.1-cp27-cp27m-win_amd64.whl (1.3MB)
    100% |################################| 1.3MB 556kB/s
Collecting idna>=2.1 (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached idna-2.5-py2.py3-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading cffi-1.10.0-cp27-cp27m-win_amd64.whl (157kB)
    100% |################################| 163kB 1.3MB/s
Collecting packaging (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached packaging-16.8-py2.py3-none-any.whl
Collecting ipaddress (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading ipaddress-1.0.18-py2-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached asn1crypto-0.22.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached pycparser-2.17.tar.gz
Collecting pyparsing (from packaging->cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Installing collected packages: enum34, pyasn1, idna, pycparser, cffi, six, pyparsing, packaging, ipaddress, asn1crypto, cryptography, paramiko, lxml, ncclient, pybind11, ydk, ydk-models-ietf, ydk-models-openconfig, ydk-models-cisco-ios-xr
  Running setup.py install for pycparser ... done
  Running setup.py install for ncclient ... done
Successfully installed asn1crypto-0.22.0 cffi-1.10.0 cryptography-1.8.1 enum34-1.1.3 idna-2.5 ipaddress-1.0.18 lxml-3.4.4 ncclient-0.5.3 packaging-16.8 paramiko-2.1.2 pyasn1-0.2.3 pybind11-2.0.1 pycparser-2.17 pyparsing-2.2.0 six-1.10.0 ydk-0.5.4 ydk-models-cisco-ios-xr-6.2.1 ydk-models-ietf-0.1.1 ydk-models-openconfig-0.1.2


C:\Python27\Scripts>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.22.0)
cffi (1.10.0)
cryptography (1.8.1)
enum34 (1.1.3)
idna (2.5)
ipaddress (1.0.18)
lxml (3.4.4)
ncclient (0.5.3)
packaging (16.8)
paramiko (2.1.2)
pip (9.0.1)
pyasn1 (0.2.3)
pybind11 (2.0.1)
pycparser (2.17)
pyparsing (2.2.0)
setuptools (28.8.0)
six (1.10.0)
ydk (0.5.4)
ydk-models-cisco-ios-xr (6.2.1)
ydk-models-ietf (0.1.1)
ydk-models-openconfig (0.1.2)


C:\Python27\Scripts>

Will add an issue on GitHub to have the LXML requirement updated. In the meantime, you can either install with Python 2 or update the dependency in https://github.com/CiscoDevNet/ydk-py/blob/master/core/setup.py and install from source as explained at: http://ydk.cisco.com/py/docs/getting_started.html#installing-from-source

Re: lxml 3.4.4 really required?

dvulovic — Mon, 03 Apr 2017 13:31:08 GMT

Hi Santiago,

as you advised, I changed setup.py files to require lxml >= 3.4.4 and YDK was installed successfully.

Thanks,

Djordje