Issues using ydk gNMI docker

fredrik.gronlund · ‎02-16-2022

Hi, I have been having some issues using the gNMI parts of the YDK.
Because I run a non-Ubuntu Linux distro, I'm setting up a build env inside of the official YDK docker, which works OK. Running my code, I get the following error when running go vet:

go vet...# github.com/CiscoDevNet/ydk-go/gnmi/ydk/providers
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/providers/gnmi_provider.go:51:21: undefined: path.GnmiServiceProviderConnect
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/providers/gnmi_provider.go:62:2: undefined: path.GnmiServiceProviderDisconnect
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/providers/gnmi_provider.go:74:52: undefined: path.GnmiSession
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/providers/gnmi_provider.go:75:13: undefined: path.GnmiServiceProviderGetSession
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/providers/gnmi_provider.go:91:10: undefined: path.GnmiServiceGet
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/providers/gnmi_provider.go:105:9: undefined: path.ExecuteGnmiRPC
# github.com/CiscoDevNet/ydk-go/gnmi/ydk/services
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:45:38: undefined: providers.GnmiServiceProvider
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:51:18: undefined: path.ExecuteGnmiRPC
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:55:9: undefined: operationSucceeded
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:59:38: undefined: providers.GnmiServiceProvider
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:64:18: undefined: path.ExecuteGnmiRPC
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:71:47: undefined: providers.GnmiServiceProvider
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:72:20: undefined: path.GnmiServiceGetCapabilities
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:77:44: undefined: providers.GnmiServiceProvider
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/services/gnmi_service.go:94:14: undefined: path.GetSubscribeDataPayload
# github.com/CiscoDevNet/ydk-go/gnmi/ydk/path
/root/go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/gnmi/ydk/path/gnmi_path.go:24:27: fatal error: ydk/ydk_gnmi.h: No such file or directory
 // #include <ydk/ydk_gnmi.h>
                           ^
compilation terminated.

I'm using the company build system, but in essence it just runs simple go commands, in this case "go vet".

Code is imported as follows (some unnecessary code removed:)

ackage upstream

import (
	"fmt"
	session "github.com/CiscoDevNet/ydk-go/gnmi/ydk/path"
	"github.com/CiscoDevNet/ydk-go/gnmi/ydk/providers"
	"github.com/CiscoDevNet/ydk-go/gnmi/ydk/services"
	"github.com/CiscoDevNet/ydk-go/ydk"
	"github.com/CiscoDevNet/ydk-go/ydk/types"
	"testing"
	"time"
)

func TestSomething(t *testing.T) {
	repo := types.Repository{Path: "./models"}
	provider := providers.GnmiServiceProvider{
		Repo:    repo,
		Address: "127.0.0.1",
		Port:    50051}
	provider.Connect()

	subscribeStream(&provider)
	provider.Disconnect()
}
// More code and local functions below

Seeing as this is inside the official Docker, I expected everything to be installed already, but am I missing some important step?

ygorelik · ‎02-16-2022

Hi Fredric

What YDK docker image did you pulled?
What is the YDK version?

Thanks,
Yan

fredrik.gronlund · ‎02-17-2022

I pulled it with the following command, per the GitHub README file:

  docker run -it ydkdev/ydk-go

The resulting image has the hash c62814c15bab, and is tagged as "latest" in docker.

Inside the docker, the last commit in the ydk-go git project is "c245318d82dd1abfb372aef070462f16f6f882e2", version 0.8.3 from 2019-05-31.

EDIT: I tried running "git pull" in the YDK project inside the docker, checking out commit fac11668ff819ea9ae9271849cbd6c2b3f1a6ed1. That doesn't seem to help.

ygorelik · ‎02-18-2022

Hi Fredrik

Indeed that docker image does not have gNMI related software installation. Besides, it is very old and outdated.

I suggest you to try the latest docker image that definitely has gNMI working:

docker run -it ydksolutions/ydk-gen:0.8.6

Let me know if you have any issue working with this docker image.

Yan Gorelik

@cisco

fredrik.gronlund · ‎02-21-2022

Hi, I've looked into this a bit more. I haven't got it to work this time either, this is what I did:

I fetched the docker using

docker run -it ydksolutions/ydk-gen:0.8.6

To make it easier, I immediately ran

bash

And added these to the environment:

export CGO_LDFLAGS_ALLOW="-fprofile-arcs|-ftest-coverage|--coverage"

Then I cloned our project, and ran a build, which ends up running "go vet":

go vet...# github.com/CiscoDevNet/ydk-go/ydk/path
../go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/ydk/path/path.go:480:18: could not determine kind of name for C.NetconfServiceProviderGetCapabilityByIndex
../go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/ydk/path/path.go:477:10: could not determine kind of name for C.NetconfServiceProviderGetNumCapabilities
../go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/ydk/path/path.go:457:7: could not determine kind of name for C.NetconfServiceProviderInitWithOnDemand
../go/pkg/mod/github.com/!cisco!dev!net/ydk-go@v0.0.0-20191017213541-fac11668ff81/ydk/path/path.go:453:7: could not determine kind of name for C.NetconfServiceProviderInitWithOnDemandRepo
# github.com/CiscoDevNet/ydk-go/gnmi/ydk/path
/usr/bin/ld: //usr/local/lib/libydk_gnmi.a(gnmi_session.cpp.o): undefined reference to symbol '_ZNSo9_M_insertIdEERSoT_@@GLIBCXX_3.4.9'
//usr/lib/x86_64-linux-gnu/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
mi-utils/common.go.mk:158: recipe for target 'vet' failed
make: *** [vet] Error 2

My code is unchanged from the original post. Is there something else I should be doing to use the YDK?

I noticed there are some scripts in the "/root/scripts/" directory, some of which seem to be tests, and some are env setup scripts. Should I run those?

ygorelik · ‎02-22-2022

Hi Fredrik

I cannot see what your program is doing. Could you give me simple sample of your main GO program, which is failing in the docker?

Thanks,

Yan

fredrik.gronlund · ‎02-22-2022

This is the file in its entirety. It is just supposed to do some preliminary tests, but it doesn't even pass compilation.

package upstream

import (
	"fmt"
	session "github.com/CiscoDevNet/ydk-go/gnmi/ydk/path"
	"github.com/CiscoDevNet/ydk-go/gnmi/ydk/providers"
	"github.com/CiscoDevNet/ydk-go/gnmi/ydk/services"
	"github.com/CiscoDevNet/ydk-go/ydk"
	"github.com/CiscoDevNet/ydk-go/ydk/types"
	"testing"
	"time"
)

func TestSomething(t *testing.T) {
	repo := types.Repository{Path: "./models"}
	provider := providers.GnmiServiceProvider{
		Repo:    repo,
		Address: "127.0.0.1",
		Port:    50051}
	provider.Connect()

	subscribeStream(&provider)
	provider.Disconnect()
}


func getSession(provider *providers.GnmiServiceProvider) *session.GnmiSession {
	return provider.GetSession()
}

func makeSub(provider *providers.GnmiServiceProvider) {
	previous := ""
	gs := getSession(provider)
	for true {
		response := gs.GetLastSubscribeResponse(previous)
		if len(response) > 0 && response != previous {
			// Do anything with received response
			ydk.YLogDebug(fmt.Sprintf("%s:  Last received subscribe response:\n%s\n", time.Now().Format(time.RFC850), response))
			previous = response
		}
		if !gs.SubscribeInProgress() {
			break
		}
		time.Sleep(100 * time.Millisecond)
	}
}

func subscribeStream(provider *providers.GnmiServiceProvider) {
	// Build subscription request
	sub := services.GnmiSubscription{}

	operData := Cisco_IOS_XEWirelessAccessPointOper_AccessPointOperData{}
	// TODO generate yang -> go models for rfid, accesspoint and location oper
	sub.Entity = &operData

	// type sample (we dont want every update)
	sub.SampleInterval = 10 * 1000000000 // in ns
	sub.SubscriptionMode = "SAMPLE"

	var subList []services.GnmiSubscription
	subList = append(subList, sub)

	// Start subscription and result monitoring
	service := services.GnmiService{}

	// TODO: log capabilities
	fmt.Printf("Server capabilities %v", service.Capabilities(provider))
	go service.Subscribe(provider, subList, 10, "STREAM", "JSON")
	time.Sleep(100 * time.Millisecond)
	makeSub(provider)
}

ygorelik · ‎02-27-2022

Hi Fredrik

I am not sure, but you might be missing setting of environment variables for Go:

export CXX=/usr/bin/g++
export CC=/usr/bin/gcc
export GOPATH=$HOME/go
export GOROOT=/usr/local/go
export CGO_ENABLED=1
export CGO_LDFLAGS_ALLOW="-fprofile-arcs|-ftest-coverage|--coverage"

See script /ydk-gen/test/run_go_test.sh for an example. You can run the script to see the YDK go package is working properly in the docker.

Also, when setting encoding the only supported one is JSON_IETF, meaning JSON is not supported. Please correct.

Also, in your example there is missing main function, so I could not run your program.

Regards,

Yan

fredrik.gronlund · ‎02-28-2022

Hi, regarding the function: As you can tell from importing the "testing" package and the function call on the form

func TestXxx(*testing.T)

This is a test function using the standard Go testing package.

In regards to those environment variables, adding them in the "ydksolutions/ydk-gen:0.8.6" docker did not help, the code still doesn't compile, with the exact same error message as in my post from 2022-02-21 above. For clarity, everything is run inside the docker unless otherwise stated.

Changing "JSON" to "JSON_IETF" did not help either.

I feel like I should reiterate - the issue isn't yet that my code doesn't compile, it is that the imported libraries don't seem to compile.

Since you mentioned the script "/root/scripts/run_go_tests.sh", I tried running it inside a completely fresh docker of the same version. Below is the complete log printout:

fredrik@fredrik-pc ~ $ docker run -it ydksolutions/ydk-gen:0.8.6
# bash
root@7c8391d4c540:/# cd ~
root@7c8391d4c540:~# ll
total 92
drwx------  1 root root  4096 Oct 31 19:08 ./
drwxr-xr-x  1 root root  4096 Feb 28 10:30 ../
-rw-------  1 root root  7828 Oct 31 19:54 .bash_history
-rw-r--r--  1 root root  3106 Apr  9  2018 .bashrc
drwx------  4 root root  4096 Dec 15  2020 .cache/
drwxr-xr-x  3 root root  4096 Sep 30 17:14 .config/
drwx------  3 root root  4096 Dec 15  2020 .local/
-rw-r--r--  1 root root   148 Aug 17  2015 .profile
-rw-------  1 root root   403 Sep 30 17:06 .python_history
-rw-------  1 root root 11376 Oct 31 19:06 .viminfo
-rw-r--r--  1 root root   165 Dec 15  2020 .wget-hsts
drwxr-xr-x  2 root root  4096 Sep 30 17:17 .ydk/
drwxr-xr-x 13 root root  4096 Dec 15  2020 confd/
drwxr-xr-x  4 root root  4096 Dec 15  2020 go/
drwxr-xr-x 22 root root  4096 Dec 15  2020 grpc/
drwxr-xr-x 13 root root  4096 Dec 15  2020 protobuf-3.5.0/
drwxr-xr-x  2 root root  4096 Jan 11  2021 scripts/
drwxr-xr-x  7 root root  4096 Oct 31 19:41 venv/
drwxr-xr-x  2 root root  4096 Dec 15  2020 ydk-gen/
root@7c8391d4c540:~# ./scripts/init_test_env.sh 

*** Mon Feb 28 10:30:41 UTC 2022: init_test_env.sh: Running OS type: Linux

*** Mon Feb 28 10:30:41 UTC 2022: init_test_env.sh: YDKGEN_HOME is set to /root/github/ydk-gen

*** Mon Feb 28 10:30:41 UTC 2022: init_test_env.sh: Python location: /usr/bin/python
./scripts/init_test_env.sh: line 77: cd: /root/github/ydk-gen: No such file or directory
./scripts/init_test_env.sh: line 21: cd: /root/github/ydk-gen/sdk/cpp/core/tests/confd/ydktest: No such file or directory

*** Mon Feb 28 10:30:41 UTC 2022: init_test_env.sh: Initializing confd in /root

*** Mon Feb 28 10:30:41 UTC 2022: init_test_env.sh: Found confd resource file: /root/confd/confdrc
make: *** No rule to make target 'stop'.  Stop.
root@7c8391d4c540:~# ./scripts/run_go_tests.sh  
*** Mon Feb 28 10:30:50 UTC 2022: run_go_tests.sh: YDKGEN_HOME is set to /root/github/ydk-gen
./scripts/run_go_tests.sh: line 158: cd: /root/github/ydk-gen: No such file or directory
./scripts/run_go_tests.sh: line 159: ../venv/bin/activate: No such file or directory
*** Mon Feb 28 10:30:50 UTC 2022: run_go_tests.sh: Initializing Go environment
*** Mon Feb 28 10:30:50 UTC 2022: run_go_tests.sh: GOPATH = /root/golang
*** Mon Feb 28 10:30:50 UTC 2022: run_go_tests.sh: GOROOT = /usr/local/go
./scripts/run_go_tests.sh: line 36: cd: /root/github/ydk-gen: No such file or directory
*** Mon Feb 28 10:30:50 UTC 2022: run_go_tests.sh: CC: /usr/bin/cc
*** Mon Feb 28 10:30:50 UTC 2022: run_go_tests.sh: CXX: /usr/bin/c++
# cd .; git clone -- https://gopkg.in/yaml.v3 /root/golang/src/gopkg.in/yaml.v3
Cloning into '/root/golang/src/gopkg.in/yaml.v3'...
fatal: unable to access 'https://gopkg.in/yaml.v3/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
package gopkg.in/yaml.v3: exit status 128
*** Mon Feb 28 10:30:54 UTC 2022: run_go_tests.sh: Command 'go
root@7c8391d4c540:~#

As you can see, running the tests fails.

So, my questions are as follows at this point:

Is there documentation for the newer version of the Docker image I can refer to?
If not, what do I need to do to make the test scripts run?
Is there documentation for the newer version of the ydk-go (that has gNMI support) that I can refer to?
After making the test scripts work, how should I then import the ydk-go gNMI module?

I ask, because the documentation on the public GitHub was last updated for version 0.8.4, 2019-10-06, which you said is very old and updated.

ygorelik · ‎02-28-2022

Hi Fredrik

I found multiple issue:

1. The docker image did not have loaded ydk-gen directory. Although it is not necessary to have it in the docker, I added it for completeness and pushed new docker image. It is actually needed for model bundles generation. No changes in the code though were introduced.

2. Your code had multiple issues:

- name of the file for the tests must end with suffix '_test'

- missing model bundle (need generate one)

- the package name must be 'test'

I attached my code that is compiled and ran perfectly fine, although panicking on connection, which is expected. Please compare and make corrections for your tests.

Regards,

Yan