ISR4331 IOx VirtualPortGroup issue

[VladimirAkhmarov]

Hello

Tried to configure networking for custom Docker container according to https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKARC-1002.pdf and notes from https://developer.cisco.com/docs/iox/#!isr-4000asr-1000-series-application-development/iox-application-life-cycle-ios-cli-commands but failed. Container itself is a regular Squid application under Apline Linux x86_64 (latest)

IOS version

#sh ver | i RELEASE
Cisco IOS Software [Cupertino], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.9.4a, RELEASE SOFTWARE (fc3)

IOX version

#sh iox-service

IOx Infrastructure Summary:
---------------------------
IOx service (CAF) : Running
IOx service (HA) : Not Supported 
IOx service (IOxman) : Running 
IOx service (Sec storage) : Not Supported 
Libvirtd 5.5.0 : Running

IOX network config

interface VirtualPortGroup4
 ip address 172.19.0.137 255.255.255.252
 no ip redirects
 ip nat inside
 no mop enabled
 no mop sysid
!
app-hosting appid IOX
 app-vnic gateway0 virtualportgroup 4 guest-interface 0
  guest-ipaddress 172.19.0.138 netmask 255.255.255.252
 app-default-gateway 172.19.0.137 guest-interface 0
 name-server0 172.19.0.137
!

IOX container status

#sh app-hosting detail appid IOX
App id : IOX
Owner : iox
State : RUNNING
Application
Type : docker
Name : test_iox
Version : 1.0
Description : Test for IOx
Author : Vladimir Akhmarov
Path : harddisk:/IOX/test_iox.1.0.x86_64.tar.gz
URL Path : 
Multicast : yes
Activated profile name : custom

Resource reservation
Memory : 256 MB
Disk : 1 MB
CPU : 800 units
CPU-percent : 6 %
VCPU : 1

Platform resource profiles
Profile Name CPU(unit) Memory(MB) Disk(MB)
--------------------------------------------------------------

Attached devices
Type Name Alias
---------------------------------------------
serial/shell iox_console_shell serial0
serial/aux iox_console_aux serial1
serial/syslog iox_syslog serial2
serial/trace iox_trace serial3

Network interfaces
---------------------------------------
eth0:
MAC address : 52:54:dd:04:c4:5e
IPv4 address : 172.19.0.138
IPv6 address : ::
Network name : VPG4


Docker
------
Run-time information
Command : 
Entry-point : /root/startup.sh 
Run options in use : 
Package run options : 
Application health information
Status : 1
Last probe error : nsenter: cannot open /proc/29596/ns/ipc: No such file or directory

Last probe output : [ ]

Container is in RUNNING state but it's IP address in not reachable

#ping 172.19.0.138 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.19.0.138, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

And ARP record is Incomplete

#sh ip arp virtualportGroup 4
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.19.0.137 - bc5a.56c2.c980 ARPA VirtualPortGroup4
Internet 172.19.0.138 0 Incomplete ARPA 

I tried also not to configure IP address under IOS CLI using app-vnic command and use IOx Local Manager through WebUI (assign IP address using GUI) but the outcome is the same

Troubleshooting guide https://developer.cisco.com/docs/iox/#!troubleshooting-guide/troubleshooting-guide says that when application is running but not accessible we should check "The network information on the pop up should display the IP address" under IOx Local Manager. Screenshot attached shows that there is no network information available for eth0. The same scenario observed for configuring container's network parameters under IOS CLI and under IOS WebUI (I saw one topic here says that container networking under IOS CLI should be removed and WebUI used instead)

There is strange error for health check probe:

Last probe error : nsenter: cannot open /proc/29596/ns/ipc: No such file or directory

There are CAF errors also:

[lxccontainer.py:583 - _get_nsenter_ifconfig()] nsenter:did not got app ipaddress info : b'nsenter: cannot open /proc/29596/ns/net: No such file or directory\n'

[lxccontainer.py:578 - _get_nsenter_ifconfig()] nsenter: /sbin/ifconfig, get app ipaddress info failed: b'nsenter: cannot open /proc/29596/ns/net: No such file or directory\n'

Could you please help me to understand why container cannot receive assigned IP address?

[wharrison021]

I have been going in circles trying to resolve a similar problem. Container running and has the appropriate IP on the switch subnet but cannot see the outside world. How did you resolve this? The documentation is not helpful.

[Torbjørn]

Your config looks fine. Seems like the namespace isn't created as it should for some reason. Are you able to run other containers? What version of IOS-XE are you running?

Easiest way to check if you can run other containers would be to spin up a guestshell:

interface VirtualPortGroup9
 ip address 192.0.2.1 255.255.255.0
app-hosting appid guestshell
 app-vnic gateway1 virtualportgroup 9 guest-interface 0 
  guest-ipaddress 192.0.2.2 netmask 255.255.255.0
  exit
 app-default-gateway 192.0.2.1 guest-interface 0 
 end
guestshell enable

[VladimirAkhmarov]

Hello @Torbjørn 

Thanks for the guide. I successfully created guestshell container with IP address per config below

app-hosting appid guestshell
 app-vnic gateway0 virtualportgroup 4 guest-interface 0 
  guest-ipaddress 172.19.0.138 netmask 255.255.255.252
 app-default-gateway 172.19.0.137 guest-interface 0
 name-server0 172.19.0.137

Here is an IOS-XE output

#guestshell enable
Interface will be selected if configured in app-hosting
Please wait for completion
guestshell installed successfully
Current state is: DEPLOYED
guestshell activated successfully
Current state is: ACTIVATED
guestshell started successfully
Current state is: RUNNING
Guestshell enabled successfully

And a guestshell output

#guestshell
[guestshell@guestshell ~]$ uname -a
Linux guestshell 5.4.231 #1 SMP Fri Jun 2 05:35:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[guestshell@guestshell ~]$ hostname -I
172.19.0.138

I removed guestshell and tried to use the same networking config for my custom container. Unfortunately original error is still there

Maybe you have other ideas how this issue could be fixed?

[balaji.bandi]

is the ping only the issue ?

i used IOSx in Cat 9300 works as expected.

https://www.balajibandi.com/?p=1621

Note : have cert error - it expired i am expected today to get cert and installed.

[Torbjørn]

Hmm, that's odd. If you share the image you are attempting to run I'll troubleshoot it in the lab.