YDK-Gen Questions

[brfoster]

Hi I have a couple of questions about this tool.

1) What is the difference (or relationship) between profiles and bundles?  Its not clear to me the difference in these from the explanation on Github.

2) If I just have a set of yang files in a directory and I want to build the SDK for those what would the sample profile or bundle JSON file look like?

Thanks!

Brent

[xiaoqinz · Accepted solution]

Hi Brent,

Sorry for the confusing explanation about the difference between profile and bundles in GitHub.

For the first question,

The YDK profile package combines the generated APIs(language bindings like ydk-py before version 0.5.0) with runtime core library(such as CRUDService and CodecService). And the YDK bundle package separate the generated APIs from the runtime core library.

So, if you are using YDK generated using `--profile` option, by using:

$ pip install <ydk-gen-home>/gen-api/python/dist/ydk*.tar.gz

You are good to go.

And, if you are using YDK generated using `--bundle` option, you need to generate and install YDK runtime core library before you install bundle package:

$ python generate.py --core

$ pip install <ydk-gen-home>/gen-api/python/ydk/dist/ydk*.tar.gz

$ python generate.py --python --bundle <path-to-profile-file>

$ pip install <ydk-gen-home>/gen-api/python/<path-to-bundle-directory>/dist/ydk*.tar.gz

And if you want to install another bundle package, write a profile file for this bundle package, generate it and install:

$ python generate.py --python --bundle <path-to-another-profile-file>

$ pip install  <ydk-gen-home>/gen-api/python/<path-to-another-bundle-directory>/dist/ydk*.tar.gz

Those two bundle packages are Python namespace packages, and if you check pip installed pacakges, you will find those entries:

ydk (0.5.0)

ydk-models-<bundle-package-name> (<version>)

ydk-models-<another-bundle-package-name> (<version>)

To unintall bundle packages:

$ pip uninstall ydk-models-<bundle-package-name>

We try to use this approach to add flexibility to YDK, so you could install/uninstall/organize your YDK packages.

If you want to use a local YANG directory to generate YDK, please make sure all the YANG files and its dependency modules are been added to that directory(use "$ pyang *.yang" in that directory and no errors) and write a profile file like:

{

   "name":"<your-package-name>",

   "version":"<your-version>",

   "ydk_version":"0.4.1",

   "Author":"<author>",

   "Copyright":"<copyright>",

   "Description":"<description>",

   "models":{

      "dir":[

         "/<path-relative-to-ydk-gen-home>"   // <-- please make sure the leading slash is there.

      ]

   }

}

The syntax for bundle package is the same.

We have example JSON files under <ydk-gen-home>/profiles. And if you want to add remote GitHub repositories to your JSON file(such as example files under <ydk-gen-home>/profiles/ydk) or add dependencies for your bundle packages in pip(such as example files under <ydk-gen-home>/profile/bundles), feel free to check them.

Thanks,

Xiaoqin

[brfoster]

Thanks for the great explanation!

I tried running one of the samples and it errors out. Any idea what’s happening?

(mypython) $ python generate.py --python --profile profiles/test/ydktest.json

Traceback (most recent call last):

File "generate.py", line 278, in

'profile').generate(options.profile))

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 82, in generate

return self._generate_profile(description_file)

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 107, in generateprofile

api_pkgs = self._get_api_pkgs(resolved_models_dir)

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 164, in getapi_pkgs

modules = pyang_builder.parse_and_return_modules()

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/builder/_pyang_model_builder.py", line 53, in parse_and_return_modules

self._validate_pyang_modules(filenames)

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/builder/_pyang_model_builder.py", line 259, in validatepyang_modules

raise YdkGenException(err_msg)

ydkgen.common.YdkGenException

[saalvare]

If you want to try a profile, I'd suggest you use the ydk-0.4.2 profile.

https://github.com/CiscoDevNet/ydk-gen/blob/master/profiles/ydk/ydk_0_4_2.json

$ python generate.py --python --profile profiles/ydk/ydk_0_4_2.json -v

[brfoster]

Well, that one errored out too. Something must be wrong with my environment.

Traceback (most recent call last):

File "generate.py", line 278, in

'profile').generate(options.profile))

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 82, in generate

return self._generate_profile(description_file)

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 107, in generateprofile

api_pkgs = self._get_api_pkgs(resolved_models_dir)

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 164, in getapi_pkgs

modules = pyang_builder.parse_and_return_modules()

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/builder/_pyang_model_builder.py", line 53, in parse_and_return_modules

self._validate_pyang_modules(filenames)

File "/home/gibson/PythonProjects/ydk-gen/ydkgen/builder/_pyang_model_builder.py", line 259, in validatepyang_modules

raise YdkGenException(err_msg)

ydkgen.common.YdkGenException

Killed

[xiaoqinz]

It seems like the models in your directory are not complete, please use additional verbose option '-v' to populate pyang errors. We use ydktest.json in development and ydk_0_4_2.json to generate ydk-py(0.4.2). Maybe you could reset your git repository to master HEAD, and regenerate it. If it is still broken, please copy errors from pyang here, so we could help you to find the missing models.

Thanks,

Xiaoqin

[brfoster]

Xiaoqin,

It is still failing, so I completely removed ydk-gen and started from scratch.  I pasted the complete log so you can see that I checked the starting environment (pip list) and also checked for the dependencies (yum list).  Then I cloned the repository, ran the install.sh script, and tried to build one of the canned profiles (ydktest.json).  It failed the same way.  What is wrong?

[gibson@rickenbacker PythonProjects]$ python2.7 -m pip list

bitarray (0.8.1)

ecdsa (0.13)

enum34 (1.1.3)

lxml (3.4.4)

ncclient (0.5.2)

paramiko (1.15.2)

pip (8.1.2)

protobuf (3.0.0b2.post2)

pyang (1.7)

pyangbind (0.5.5)

pycrypto (2.6.1)

requests (2.10.0)

sample (0.0.1)

setuptools (1.4.2)

six (1.10.0)

Twisted (16.3.0)

virtualenv (15.0.2)

wheel (0.29.0)

zope.interface (4.2.0)

[gibson@rickenbacker PythonProjects]$

[gibson@rickenbacker PythonProjects]$ sudo yum list python-pip python-devel libxml2-devel libxslt-devel libssh-devel boost-devel boost-python libcurl-devel libtool gcc-c++

[sudo] password for gibson:

Loaded plugins: fastestmirror, security

Loading mirror speeds from cached hostfile

epel/metalink                                                                                                                                                                  | 7.9 kB    00:00

* base: centos.eecs.wsu.edu

* epel: mirror.metrocast.net

* extras: reflector.westga.edu

* updates: mirror.rackspace.com

base                                                                                                                                                                          | 3.7 kB    00:00

epel                                                                                                                                                                          | 4.3 kB    00:00

epel/primary_db                                                                                                                                                                | 5.9 MB    00:01

extras                                                                                                                                                                        | 3.4 kB    00:00

updates                                                                                                                                                                        | 3.4 kB    00:00

Installed Packages

boost-devel.x86_64                                                                              1.41.0-28.el6                                                                                @base

boost-python.x86_64                                                                              1.41.0-28.el6                                                                                @base

gcc-c++.x86_64                                                                                  4.4.7-17.el6                                                                                @base

libcurl-devel.x86_64                                                                            7.19.7-52.el6                                                                                @base

libssh-devel.x86_64                                                                              0.5.5-5.el6                                                                                  @epel

libtool.x86_64                                                                                  2.2.6-15.5.el6                                                                              @base

libxml2-devel.x86_64                                                                            2.7.6-21.el6_8.1                                                                            @updates

libxslt-devel.x86_64                                                                            1.1.26-2.el6_3.1                                                                            @base

python-devel.x86_64                                                                              2.6.6-66.el6_8                                                                              @updates

python-pip.noarch                                                                                7.1.0-1.el6                                                                                  @epel

Available Packages

boost-devel.i686                                                                                1.41.0-28.el6                                                                                base

boost-python.i686                                                                                1.41.0-28.el6                                                                                base

libcurl-devel.i686                                                                              7.19.7-52.el6                                                                                base

libssh-devel.i686                                                                                0.5.5-5.el6                                                                                  epel

libxml2-devel.i686                                                                              2.7.6-21.el6_8.1                                                                            updates

libxslt-devel.i686                                                                              1.1.26-2.el6_3.1                                                                            base

python-devel.i686                                                                                2.6.6-66.el6_8                                                                              updates

[gibson@rickenbacker PythonProjects]$

[gibson@rickenbacker PythonProjects]$ git clone https://github.com/CiscoDevNet/ydk-gen.git

Initialized empty Git repository in /home/gibson/PythonProjects/ydk-gen/.git/

remote: Counting objects: 2504, done.

remote: Total 2504 (delta 0), reused 0 (delta 0), pack-reused 2504

Receiving objects: 100% (2504/2504), 2.07 MiB, done.

Resolving deltas: 100% (1479/1479), done.

[gibson@rickenbacker PythonProjects]$ cd ydk-gen

[gibson@rickenbacker ydk-gen]$ ls

Copyright  gen-api  generate.py  install.sh  LICENSE  profiles  README.md  requirements.txt  sdk  test  yang  ydkgen

[gibson@rickenbacker ydk-gen]$ source ./install.sh

Creating/activating virtual environment

Already using interpreter /usr/bin/python2.7

New python executable in /home/gibson/PythonProjects/ydk-gen/mypython/bin/python2.7

Also creating executable in /home/gibson/PythonProjects/ydk-gen/mypython/bin/python

Installing setuptools, pip, wheel...done.

Installing python dependencies..

Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 1))

Requirement already satisfied (use --upgrade to upgrade): enum34==1.1.3 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 2))

Collecting future>=0.15.2 (from -r requirements.txt (line 3))

Collecting Jinja2==2.8 (from -r requirements.txt (line 4))

  Using cached Jinja2-2.8-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): lxml==3.4.4 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 5))

Collecting MarkupSafe==0.23 (from -r requirements.txt (line 6))

Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 7))

Collecting pyang==1.6 (from -r requirements.txt (line 8))

  Using cached pyang-1.6-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 9))

Collecting Sphinx==1.4a1 (from -r requirements.txt (line 10))

  Using cached Sphinx-1.4a1-py2.py3-none-any.whl

Collecting sphinx-rtd-theme==0.1.9 (from -r requirements.txt (line 11))

  Using cached sphinx_rtd_theme-0.1.9-py2-none-any.whl

Collecting XlsxWriter==0.7.3 (from -r requirements.txt (line 12))

  Using cached XlsxWriter-0.7.3-py2.py3-none-any.whl

Collecting GitPython==1.0.1 (from -r requirements.txt (line 13))

Requirement already satisfied (use --upgrade to upgrade): Twisted>=16.0.0 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 14))

Requirement already satisfied (use --upgrade to upgrade): protobuf==3.0.0b2.post2 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 15))

Requirement already satisfied (use --upgrade to upgrade): ncclient>=0.4.7 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 16))

Collecting alabaster<0.8,>=0.7 (from Sphinx==1.4a1->-r requirements.txt (line 10))

  Using cached alabaster-0.7.9-py2.py3-none-any.whl

Collecting snowballstemmer>=1.1 (from Sphinx==1.4a1->-r requirements.txt (line 10))

  Using cached snowballstemmer-1.2.1-py2.py3-none-any.whl

Collecting docutils>=0.11 (from Sphinx==1.4a1->-r requirements.txt (line 10))

Requirement already satisfied (use --upgrade to upgrade): six>=1.4 in /usr/lib/python2.7/site-packages (from Sphinx==1.4a1->-r requirements.txt (line 10))

Collecting Pygments>=2.0 (from Sphinx==1.4a1->-r requirements.txt (line 10))

  Using cached Pygments-2.1.3-py2.py3-none-any.whl

Collecting babel!=2.0,>=1.3 (from Sphinx==1.4a1->-r requirements.txt (line 10))

  Using cached Babel-2.3.4-py2.py3-none-any.whl

Collecting imagesize (from Sphinx==1.4a1->-r requirements.txt (line 10))

  Using cached imagesize-0.7.1-py2.py3-none-any.whl

Collecting gitdb>=0.6.4 (from GitPython==1.0.1->-r requirements.txt (line 13))

Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/lib/python2.7/site-packages (from Twisted>=16.0.0->-r requirements.txt (line 14))

Requirement already satisfied (use --upgrade to upgrade): setuptools in ./mypython/lib/python2.7/site-packages (from protobuf==3.0.0b2.post2->-r requirements.txt (line 15))

Collecting pytz>=0a (from babel!=2.0,>=1.3->Sphinx==1.4a1->-r requirements.txt (line 10))

  Using cached pytz-2016.6.1-py2.py3-none-any.whl

Collecting smmap>=0.8.5 (from gitdb>=0.6.4->GitPython==1.0.1->-r requirements.txt (line 13))

Installing collected packages: future, MarkupSafe, Jinja2, pyang, alabaster, snowballstemmer, docutils, Pygments, pytz, babel, imagesize, Sphinx, sphinx-rtd-theme, XlsxWriter, smmap, gitdb, GitPython

  Found existing installation: pyang 1.7

    Not uninstalling pyang at /usr/lib/python2.7/site-packages, outside environment /home/gibson/PythonProjects/ydk-gen/mypython

Successfully installed GitPython-1.0.1 Jinja2-2.8 MarkupSafe-0.23 Pygments-2.1.3 Sphinx-1.4a1 XlsxWriter-0.7.3 alabaster-0.7.9 babel-2.3.4 docutils-0.12 future-0.15.2 gitdb-0.6.4 imagesize-0.7.1 pyang-1.7 pytz-2016.6.1 smmap-0.9.0 snowballstemmer-1.2.1 sphinx-rtd-theme-0.1.9

Setup completed..

To exit virtualenv, run the below command from your shell/terminal:

deactivate

To generate YDK model APIs for your profile, run the below command from your shell/terminal:

python generate.py --profile <profile-file>

(mypython) [gibson@rickenbacker ydk-gen]$

(mypython) [gibson@rickenbacker ydk-gen]$ python generate.py --python --profile profiles/test/ydktest.json -v

Resolving a dir tag with params: [

  "/yang/ydktest",

  "/yang/ietf"

]

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-sanity-augm@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/main-aug3@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-filterread@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-sanity-deviations-types@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-sanity-submodule@2016-04-25.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/inherit@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-sanity-deviations-another@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/main@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/oc-pattern@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-sanity@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/main-aug1@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/main-aug2@2015-11-17.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-sanity-types@2016-04-11.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ydktest/ydktest-types@2016-05-23.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ietf/ietf-netconf@2011-06-01.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ietf/ietf-yang-types@2013-07-15.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ietf/ietf-inet-types@2013-07-15.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ietf/ietf-netconf-acm@2012-02-22.yang to /tmp/tmp8R7obx.models

Copying /home/gibson/PythonProjects/ydk-gen/yang/ietf/ietf-netconf-with-defaults@2011-06-01.yang to /tmp/tmp8R7obx.models

Parsing file /tmp/tmp8R7obx.models/ydktest-sanity-augm@2015-11-17.yang format <built-in function format> name ydktest-sanity-augm revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/main-aug3@2015-11-17.yang format <built-in function format> name main-aug3 revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ietf-netconf@2011-06-01.yang format <built-in function format> name ietf-netconf revision 2011-06-01

Parsing file /tmp/tmp8R7obx.models/ydktest-filterread@2015-11-17.yang format <built-in function format> name ydktest-filterread revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ydktest-sanity-deviations-types@2015-11-17.yang format <built-in function format> name ydktest-sanity-deviations-types revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ietf-yang-types@2013-07-15.yang format <built-in function format> name ietf-yang-types revision 2013-07-15

Parsing file /tmp/tmp8R7obx.models/ietf-inet-types@2013-07-15.yang format <built-in function format> name ietf-inet-types revision 2013-07-15

Parsing file /tmp/tmp8R7obx.models/ydktest-sanity-submodule@2016-04-25.yang format <built-in function format> name ydktest-sanity-submodule revision 2016-04-25

Parsing file /tmp/tmp8R7obx.models/inherit@2015-11-17.yang format <built-in function format> name inherit revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ydktest-sanity-deviations-another@2015-11-17.yang format <built-in function format> name ydktest-sanity-deviations-another revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/main@2015-11-17.yang format <built-in function format> name main revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ietf-netconf-acm@2012-02-22.yang format <built-in function format> name ietf-netconf-acm revision 2012-02-22

Parsing file /tmp/tmp8R7obx.models/oc-pattern@2015-11-17.yang format <built-in function format> name oc-pattern revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ydktest-sanity@2015-11-17.yang format <built-in function format> name ydktest-sanity revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/main-aug1@2015-11-17.yang format <built-in function format> name main-aug1 revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/main-aug2@2015-11-17.yang format <built-in function format> name main-aug2 revision 2015-11-17

Parsing file /tmp/tmp8R7obx.models/ietf-netconf-with-defaults@2011-06-01.yang format <built-in function format> name ietf-netconf-with-defaults revision 2011-06-01

Parsing file /tmp/tmp8R7obx.models/ydktest-sanity-types@2016-04-11.yang format <built-in function format> name ydktest-sanity-types revision 2016-04-11

Parsing file /tmp/tmp8R7obx.models/ydktest-types@2016-05-23.yang format <built-in function format> name ydktest-types revision 2016-05-23

/tmp/tmp8R7obx.models/ydktest-sanity@2015-11-17.yang:337: the value "[u'auto-sense-speed']" does not match its base type - bit not defined for the default value

Traceback (most recent call last):

  File "generate.py", line 278, in <module>

    'profile').generate(options.profile))

  File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 82, in generate

    return self._generate_profile(description_file)

  File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 107, in _generate_profile

    api_pkgs = self._get_api_pkgs(resolved_models_dir)

  File "/home/gibson/PythonProjects/ydk-gen/ydkgen/__init__.py", line 164, in _get_api_pkgs

    modules = pyang_builder.parse_and_return_modules()

  File "/home/gibson/PythonProjects/ydk-gen/ydkgen/builder/_pyang_model_builder.py", line 53, in parse_and_return_modules

    self._validate_pyang_modules(filenames)

  File "/home/gibson/PythonProjects/ydk-gen/ydkgen/builder/_pyang_model_builder.py", line 259, in _validate_pyang_modules

    raise YdkGenException(err_msg)

ydkgen.common.YdkGenException

(mypython) [gibson@rickenbacker ydk-gen]$

[xiaoqinz]

Hi Brent,

Thanks so much for your complete log!

In the last line just before Python trackback, pyang complains about:

/tmp/tmp8R7obx.models/ydktest-sanity@2015-11-17.yang:337: the value "[u'auto-sense-speed']" does not match its base type - bit not defined for the default value

It seems like this error causes ydk-gen to throw exception and exit execution, and I was able to reproduce this issue on my Mac with pyang 1.7.

Maybe you could try to downgrade pyang to 1.6(maybe your system has pyang 1.7 installed, but as specified in requirements.txt, ydk-gen needs pyang 1.6. Sorry, we might need to update install.sh):

$ pip install pyang==1.6

And try to regenerate the package.

Thanks,

Xiaoqin

[brfoster]

That worked!

One more question:  Is there any plan to add RESTCONF support?  Seems that shouldn't be hard to add!

Thanks,

[xiaoqinz]

Yes! RESTCONF is on our roadmap.

Thanks,

Xiaoqin