Problem with Docker App Deployment on IE4000

[nie93]

Hello All,

Greetings!

I am trying to deploy a docker app following the tutorial from https://learninglabs.cisco.com/lab/iot-iox-app-docker/, and it works well after I deployed on the Cisco Sandbox (x86_64 architecture). But when I try to deploy the same app following the same instruction on Cisco IE4000 (IE4K) device, I did change the "cpuarch" to "ppc" in the YAML file "package.yaml", and rebuild the docker image and repackage the ioxclient "package.tar" file. After I use the command "ioxclient application install myappname package.tar" to deploy the packaged application, after around 15 minutes, it turns out the following error messages:

Currently active profile :  ie4000
Command Name:  application-install
Saving current configuration
Could not complete your command :  Error. Server returned 500
{
 "description": "Error during app installation: [Errno 95] Operation not supported",
 "errorcode": -1,
 "message": "Error during app installation"
}

Then I looked into the Device Manager Webpage (https://{MyIOxIPAddress}:8443), and found the following related error messages:

2019-01-29 16:47:06,056 [command_wrapper:ERROR] [Thread-10] [commandwrappers.py:1124 - tar_extractall()] Error executing command : tar -xvpf /ios/tmp/tmpExtractvtJ3UB/rootfs/326636140e631eefc6432dc65c94734e2bfca8b53109f3adf07911fc5d213d18/layer.tar -C /ios/caf/work/layers/d6da3c54c8f3a43215d51cc0d43eff9759624996e40aaf11e4be5a555c038c32/layer_contents
    
Traceback (most recent call last):
  File "/tmp/tmpaPCE2W/ie4k/caf/src/appfw/utils/commandwrappers.py", line 1118, in tar_extractall
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['tar', '-xvpf', u'/ios/tmp/tmpExtractvtJ3UB/rootfs/326636140e631eefc6432dc65c94734e2bfca8b53109f3adf07911fc5d213d18/layer.tar', '-C', '/ios/caf/work/layers/d6da3c54c8f3a43215d51cc0d43eff9759624996e40aaf11e4be5a555c038c32/layer_contents']' returned non-zero exit status 2
2019-01-29 16:47:06,065 [runtime.hosting:DEBUG] [Thread-10] [layer_registry.py:263 - add_app_layer()] Extraction of tar file /ios/tmp/tmpExtractvtJ3UB/rootfs/326636140e631eefc6432dc65c94734e2bfca8b53109f3adf07911fc5d213d18/layer.tar using tar binary failed. Reason: Command '['tar', '-xvpf', u'/ios/tmp/tmpExtractvtJ3UB/rootfs/326636140e631eefc6432dc65c94734e2bfca8b53109f3adf07911fc5d213d18/layer.tar', '-C', '/ios/caf/work/layers/d6da3c54c8f3a43215d51cc0d43eff9759624996e40aaf11e4be5a555c038c32/layer_contents']' returned non-zero exit status 2 ret_code: -1
2019-01-29 16:48:08,001 [runtime.hosting:ERROR] [Thread-10] [layer_registry.py:281 - add_app_layer()] Error while adding a APP layer to registry. Cause: 
Traceback (most recent call last):
  File "/tmp/tmpaPCE2W/ie4k/caf/src/appfw/runtime/layer_registry.py", line 264, in add_app_layer
  File "/usr/lib/python2.7/tarfile.py", line 2070, in extractall
    self.extract(tarinfo, path)
  File "/usr/lib/python2.7/tarfile.py", line 2107, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "/usr/lib/python2.7/tarfile.py", line 2191, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "/usr/lib/python2.7/tarfile.py", line 2269, in makelink
    os.symlink(tarinfo.linkname, targetpath)
OSError: [Errno 95] Operation not supported
2019-01-29 16:48:08,010 [runtime.hosting:ERROR] [Thread-10] [controller.py:2206 - _deployConnector()] Exception while deploying connector:testapp. Exception:[Errno 95] Operation not supported

I also tried to deploy the "package.tar" file directly through the Device Manager, and it returns the following error message:

Application Deployment Failed: Error while attempting to upload connector: [Errno 32] Broken pipe

 Please let my know if there is any other information you need to proceed this issue.

Thanks a lot!

Jack

[Steve Zhang]

It seems there is some problem for your Docker app. You'd better use a correct one for your experiment. You may refer to the below for generating the related apps.
https://developer.cisco.com/docs/iox/#tutorial-build-sample-docker-type-python-with-c-app
https://developer.cisco.com/docs/iox/#tutorialbuild-sample-docker-type-c-based-simple-app

[nie93]

Thanks Steve for your prompt reply!

I go through every steps in your second link (https://developer.cisco.com/docs/iox/#tutorialbuild-sample-docker-type-c-based-simple-app) and try to deploy on IOx Sandbox, and it works fine with the c++ based codes.

But when I turn to deploy on the IE4000 switch, I change the url in `Dockerfile` to `FROM devhub-docker.cisco.com/iox-docker/ie4k/base-rootfs` to get the correct info. Then I try to test it by running the development tools image locally. When I use it to compile the c++ codes, the error messages shows missing libraries or links, as shown below:

sgdril@ARPAE:~/nie/CiscoIoT/simple-cpp-app-ie4k$ pwd
/home/sgdril/nie/CiscoIoT/simple-cpp-app-ie4k
sgdril@ARPAE:~/nie/CiscoIoT/simple-cpp-app-ie4k$ docker run -v ${pwd}/apps:/opt/share -it ie4k-tools /bin/sh
/ # cd /opt/src/
/opt/src # ls -l
-rw-rw-r--    1 root     root           276 Jan 30 18:27 loop_app.cpp
-rw-rw-r--    1 root     root          8399 Jan 30 18:27 test-cpp-app.cpp
/opt/src # which g++
/usr/bin/g++
/opt/src # ls -l /usr/bin/g++
lrwxrwxrwx    1 root     root            22 Mar  9  2017 /usr/bin/g++ -> powerpc-poky-linux-g++
/opt/src # g++ test-cpp-app.cpp -o test-cpp-app
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__ltdf2@GLIBC_2.4'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__gtdf2@GLIBC_2.4'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__adddf3@GLIBC_2.3.2'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__fixunsdfsi@GLIBC_2.3.2'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__ltsf2@GLIBC_2.4'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__gtsf2@GLIBC_2.4'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__subdf3@GLIBC_2.3.2'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__eqdf2@GLIBC_2.3.2'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__floatunsidf@GLIBC_2.4'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__muldf3@GLIBC_2.3.2'
/usr/lib/gcc/powerpc-poky-linux/4.9.1/../../../../lib/libstdc++.so: undefined reference to `__extendsfdf2@GLIBC_2.3.2'
collect2: error: ld returned 1 exit status
/opt/src #

I also create an issue on Github page here: https://github.com/CiscoIOx/docker-demo-apps/issues/1#issue-404940017

Jack

[nie93]

The missing library issue has been fixed by @hgovind on the Github page: https://github.com/CiscoIOx/docker-demo-apps/issues/1

[nie93]

I think the problem may locate at the docker layer. I try to deploy the IE4K switch with the "test-cpp-app" application in the turotial. But it also displays some similar error "[Errno 95] Operation not supported" :

OSError: [Errno 17] File exists: '/ios/caf/work/layers/1badee7bbaf6dec41d17fcef39daa53e91ea810b66ed204bb431cf280348d3e2'

This issue is also appeared in another user's experience: https://community.cisco.com/t5/cisco-iox-discussions/problem-with-docker-deploy-in-ie4000/td-p/3457687 . I have reconfigured the device from scratch but still have the same problem.

Please let me know if there are configuration related to that. I did notice there is a chapter "Docker Layer" in the iox-doc, but none of the tutorial mention something related to it.

Thanks!

[suressan · Accepted solution]

IE4000 platform doesn't support docker layers. Hence you need to repackage your application with unified rootfs (instead of split into docker layers).

Use this command to build your docker image as IOx app with unified rootfs.

$ ioxclient docker pkg -p ext2 <docker_image> <projectdir>

If you already have package.yaml created in projectdir, then change the "rootfs" attribute value to "rootfs.ext2" (instead of rootfs.tar). Now repackage the IOx app and deploy it on IE4k. 

[nie93]

Thanks for your reply!

I tried to package the image into ext2 as you said using the following commands:

$ sudo /home/nie/ioxclient_1.8.1.0_linux_amd64/ioxclient docker package -p ext2 -a nie/iox-python-test:latest .

Then it returns the following error message:

```
Currently active profile : default
Command Name: docker-package
Using the package descriptor file in the project dir
Validating descriptor file package.yaml with package schema definitions
Parsing descriptor file..
Found schema version 2.2
Loading schema file for version 2.2
Validating package descriptor file..
File package.yaml is valid under schema version 2.2
Generating IOx docker package with a single layer
checking for -d option
Docker image rootfs size in 1M blocks: ~ 60
Computed headroom in 1M blocks: ~ 6
Creating iox package with rootfs size in 1M blocks: ~ 66
Unable to copy contents into rootfs image, 67518+0 records in
67518+0 records out
69138432 bytes (69 MB, 66 MiB) copied, 0.213294 s, 324 MB/s
mke2fs 1.42.13 (17-May-2015)
cp: error writing 'rootfs-20190408151858/./bin/busybox': No space left on device
cp: cannot create directory 'rootfs-20190408151858/./dev': No space left on device
cp: error writing 'rootfs-20190408151858/./main.py': No space left on device

Unable to generate Flat rootfs file from docker image
Error while exporting docker image and creating rootfs image
Error occurred : exit status 1

```

[nie93]

I also tried packaging into ext3, ext4 file system, but only ext4 works. Do you think I can deploy it using ext4 file system on IE4000?

[suressan]

I see "No space left on device" error? Did you free up some disk space and try with -p ext2 again ? 

[nie93]

My disk space should be enough. I think this error is due to incorrect configuration. I have tried your method with IE4000 dockerfile configuration, package.yaml file (with rootfs.ext2), then it could be deployed and run on the device right now.
Thanks a lot!!!
Jack