Catalyst 8300 - Virtual Service will not activate after code upgrade

[Stephanie Tingey]

Hey,

I have a catalyst 8300 router and I upgraded the code from 17.09.04a to 17.13.1a.  Now my ubuntu kvm virtual service will not activate.  I've uninstalled and attempted to reinstall the ova file, i even tried an earlier version i created that i know worked fine on earlier code.  Here's the debug from the install and activation:

VIRTUAL-SERVICE [PlexShark]: Sending install req for [PlexShark], path=harddisk:/plexoriginal.ova, uri= uid=1

Jan 27 09:09:32.313: VIRTUAL-INSTANCE: Message sent for INSTALL TDL request: Virtual-instance name: PlexShark, UID: 1VIRTUAL-SERVICE: Started response timer for tid 99000006 - 30 minutes

Teurerouter#
Jan 27 2024 02:09:47.367 GMT: %VMAN-5-PACKAGE_SIGNING_LEVEL_ON_INSTALL: R0/0: vman: Package 'plexoriginal.ova' for service container 'PlexShark' is 'unsigned', signing level cached on original install is 'unsigned'
Teurerouter#VIRTUAL-SERVICE: Install response handler: VM[PlexShark]: Owner IOSd trans_id 2566914054
VIRTUAL-SERVICE [PlexShark]: vm[PlexShark] set owner [IOSd] pkg_path [/vol/harddisk/plexoriginal.ova]
VIRTUAL-SERVICE [PlexShark]: application_name: 'ubuntu' application_vendor: '' application_version: '1.1'
VIRTUAL-SERVICE [PlexShark]: Default profile info: license_name: '', license_ver: ''
VIRTUAL-SERVICE: Install pkg response for tid 99000006: rc=0, descr=Install Success
VIRTUAL-SERVICE [PlexShark]: License type: none, no license needed

Jan 27 2024 02:10:00.308 GMT: %VIRT_SERVICE-5-INSTALL_STATE: Successfully installed virtual service PlexShark
Teurerouter#VIRTUAL-SERVICE: Received local transport activation request
VIRTUAL-SERVICE: Enabling vman local transport
VIRTUAL-SERVICE: Local transport 'activation' request processed

Teurerouter#VIRTUAL-SERVICE [PlexShark]: Activate CLI: appl->owner [IOSd]

Jan 27 09:10:20.088: VIRTUAL-INSTANCE: Message sent for ACTIVATE TDL request: Virtual service name: PlexShark, UID: 1VIRTUAL-SERVICE [PlexShark]: Started virtual service (1) activate response timer - 30 minutes

Teurerouter#
Jan 27 2024 02:10:23.035 GMT: %VMAN-2-START_FAILED: R0/0: vman: Virtual Service[PlexShark]::Start failed::Failed to start the virtual service
Teurerouter#VIRTUAL-SERVICE: clnt_type 0: Interface counter is '1'
VIRTUAL-SERVICE: Information for virtual port grp '0' is received
VIRTUAL-SERVICE [PlexShark]: Deliver intf response, vm =PlexShark, counter=1
VIRTUAL-SERVICE [PlexShark]: Received interface id=0, type=1, state=0
VIRTUAL-SERVICE [PlexShark]: Received virtual port group interface 0 with service MAC 44b6.be91.7eae, state: down

Jan 27 09:10:26.177: VIRTUAL-INSTANCE: Message sent for IF MTU TDL message: appliance 'PlexShark'VIRTUAL-SERVICE [PlexShark]: Activate response handler: rsp_rc 1
VIRTUAL-SERVICE [PlexShark]:
Deliver response: appliance_state 3 rsp_rc 1 if_notify name PlexShark clnt_type 0 act_state 1

Jan 27 2024 02:10:26.177 GMT: %VIRT_SERVICE-5-ACTIVATION_STATE: Failed to activate virtual service PlexShark
Teurerouter#VIRTUAL-SERVICE [PlexShark]: Stopped virtual service (1) response timer
VIRTUAL-SERVICE: Delivered Virt-manager response message to virtual service 'PlexShark' - Response: 'FAIL'

Current configuration : 205 bytes
!
interface VirtualPortGroup0
ip address 10.0.0.1 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
zone-member security ZBFW_inside
no mop enabled
no mop sysid
end

virtual-service
signing level unsigned
!
!
virtual-service PlexShark
vnic gateway VirtualPortGroup0

[omihut1]

I ran into this issue too.  Did you find a solution please?

[Stephanie Tingey]

No, never did.  I've given up at this point.

[omihut1]

Let me try to help you a bit, even though I don't have a final solution.  When you upgrade IOS, is possible the underlying router OS changed too.  With that is possible the interface names changed.  virtual-service install command extracts files from ova archive to harddisk:virtual-instance directory.  The supplied package.yaml file gets translated into an xml file that vman looks at.  I suspect interface name "net1" is not a valid name anymore, but not sure yet what the correct name should be.

I'm surprised how little interest is out there in this otherwise great potential feature.

[Stephanie Tingey]

Yes, that could be part of the issue.  What bothers me the most is the lack of information in the debug.  In earlier versions of code, the debug would give more information than just FAIL.  I don't have a service contract of any kind, so i can't engage TAC or anyone at cisco for assistance directly.  I'd like to find an official package that will run and try to activate that in an attempt to decipher what has changed.  

[Stephanie Tingey]

I tried installing it again on 17.15.3

Teurerouter(config)#virtual-service PlexShark
Teurerouter(config-virt-serv)#vnic gateway virtualportGroup 0
Teurerouter(config-virt-serv-vnic)#VIRTUAL-SERVICE: Appliance service engine create name=VirtualPortGroup0
VIRTUAL-SERVICE [PlexShark]: Virtual port group 0 ref counter 1 to be added

Teurerouter(config-virt-serv-vnic)#act
% Activating virtual-service 'PlexShark', this might take a few minutes. Use 'show virtual-service list' for progress.

Teurerouter(config-virt-serv)#VIRTUAL-SERVICE [PlexShark]: Activate CLI: appl->owner [IOSd]

Aug 5 20:25:58.236: VIRTUAL-INSTANCE: Message sent for ACTIVATE TDL request: Virtual service name: PlexShark, UID: 1VIRTUAL-SERVICE [PlexShark]: Started virtual service (1) activate response timer - 30 minutes

Teurerouter(config-virt-serv)#
Aug 5 2025 14:26:01.656 MDT: %VMAN-2-START_FAILED: R0/0: vman: Virtual Service[PlexShark]::Start failed::Failed to start the virtual service
Teurerouter(config-virt-serv)#VIRTUAL-SERVICE: clnt_type 0: Interface counter is '1'
VIRTUAL-SERVICE: Information for virtual port grp '0' is received
VIRTUAL-SERVICE [PlexShark]: Deliver intf response, vm =PlexShark, counter=1
VIRTUAL-SERVICE [PlexShark]: Received interface id=0, type=1, state=0
VIRTUAL-SERVICE [PlexShark]: Received virtual port group interface 0 with service MAC 44b6.be91.7eaf, state: down

Aug 5 20:26:04.832: VIRTUAL-INSTANCE: Message sent for IF MTU TDL message: appliance 'PlexShark'VIRTUAL-SERVICE [PlexShark]: Activate response handler: rsp_rc 1
VIRTUAL-SERVICE [PlexShark]:
Deliver response: appliance_state 3 rsp_rc 1 if_notify name PlexShark clnt_type 0 act_state 1

Aug 5 2025 14:26:04.832 MDT: %VIRT_SERVICE-5-ACTIVATION_STATE: Failed to activate virtual service PlexShark
Teurerouter(config-virt-serv)#VIRTUAL-SERVICE [PlexShark]: Stopped virtual service (1) response timer
VIRTUAL-SERVICE: Delivered Virt-manager response message to virtual service 'PlexShark' - Response: 'FAIL'

for reference, here is my config.yaml:

manifest-version: 1.0

info:

  name: PlexShark

  description: "Ubuntu Plex Server with Wireshark"

  version: 1.0

app:

  # Indicate app type (vm, paas, lxc etc.,)

  apptype: vm

  resources:

   cpu: 40

   memory: 4194304

   vcpu: 4

   disk:

    - target-dev: hdc

      file: plex.qcow2

   interfaces:

    - target-dev: eth0

   serial:

    - console

    - aux

  # Specify runtime and startup

  startup:

    runtime: kvm

    boot-dev: hd

---

i changed interfaces target-dev from net1 to eth0..  so it's not that i guess.

[almitch101]

I have ran into this same issue and Im receiving the same error messages as you. I have been spending countless hours trying to get it to work. I have found some resources that may can help. Looks like the package.yaml needs to contain other mandatory values based on a newer schema which I believe the newer IOS requires. I am attempting to build out a new package.yaml file based on the template in the link below. If that doesnt work I'll try to build a .ova/.tar with ioxclient and see how far I get. From my readings Cisco IOx is essentially the evolution and replacement for the older "Virtual Services" framework on routers/devices.

https://developer.cisco.com/docs/iox/tutorial-build-sample-vm-type-iox-app/#tutorial-build-sample-vm-type-iox-app

https://developer.cisco.com/docs/iox/package-descriptor/#iox-package-descriptor

[Stephanie Tingey]

The problem with that is, at least with earlier versions of the iox system is the overall tar file can only be a certain size.  But i think you're on the right track with the yaml file.  

I was able to get a copy of the snort ids package to look at how the yaml file is built.  

descriptor-schema-version: "2.17"

info:
name: UTD-Snort-Feature
description: "Unified Threat Defense"
version: "1.0.12_SV3.1.81.0_XE17.15"
author-link: "http://www.cisco.com"
author-name: "Cisco Systems, Inc."

app:
type: lxc
cpuarch: x86_64

resources:
profile: custom-map
custom-map:
-
name: low
cpu-percent: 25
memory: 1024
disk: 210
-
name: medium
cpu-percent: 50
memory: 2048
disk: 210
-
name: high
cpu-percent: 75
memory: 4096
disk: 210
-
name: urlf-low
cpu-percent: 25
memory: 3072
disk: 1710
-
name: urlf-medium
cpu-percent: 50
memory: 4096
disk: 1710
-
name: urlf-high
cpu-percent: 75
memory: 6144
disk: 1710
-
name: cloud-low
cpu-percent: 25
memory: 2048
disk: 360
-
name: cloud-medium
cpu-percent: 50
memory: 3072
disk: 610
-
name: cloud-high
cpu-percent: 75
memory: 5120
disk: 610
-
name: onbox-low
cpu-percent: 25
memory: 3072
disk: 2110
-
name: onbox-medium
cpu-percent: 50
memory: 5120
disk: 2310
-
name: onbox-high
cpu-percent: 75
memory: 6144
disk: 2310
-
name: default
cpu-percent: 25
memory: 1024
disk: 210

network:
-
interface-name: eth0
-
interface-name: eth1
-
interface-name: eth2
-
interface-name: ieobc

host_mounts:
-
host_mount_path: "/bootflash/SHARED-IOX"
target_mount: "/bootflash/SHARED-IOX"
description: ""
-
host_mount_path: "/tmp/HTX-IOX"
target_mount: "/tmp/HTX-IOX"
description: ""
-
host_mount_path: "/tmp/xml"
target_mount: "/tmp/xml"
description: ""
-
host_mount_path: "/tmp/binos-IOX"
target_mount: "/tmp/binos-IOX"
description: ""
-
host_mount_path: "/tmp/psv"
target_mount: "/tmp/psv"
description: ""

startup:
rootfs: utd-snort.ext2
target: /sbin/init

[omihut1]

We're looking for app type vm.  I tried few variations, still no luck.

[almitch101]

I was able to get my alpine linux vm with my custom app running on IOS-XE 17.15 using IOx app-hosting. I packaged the vm using the ioxclient tool and it output a 6GB .tar file I was able to install, activate and start the vm and access my app on the vm with no errors. I used the two links I posted and this one below to put it all together. The github link is a great resource to get a basic vm up and running. You can model your package.yaml after that example.

https://github.com/CiscoIOx/alpine37-docker-kvm

[Stephanie Tingey]

i manually copied it to my harddisk: file system and got it to install, however, when i try to activate i get this:

% Error: Error while changing app state: [Errno 13] Permission denied: '/var/iox'

[omihut1]

I'm using 8300 router and copy the file to bootflash filesystem.  From there I use "virtual-service install name <name> package bootflash:filename" and this seems to work until the point where it complains about failure to activate with very little details.  I believe it's about the yaml syntax issue.

[Stephanie Tingey]

sorry, i didn't make myself clear on my last post.  I tried repackaging for the iox system..  That's where i'm getting the permission denied.  Yes, for the virtual-service side, it's probably a change with the yaml structure, but i still can't find any info on what has changed.  

[omihut1]

Glad someone got a step forward.  I was able to get guestshell working using Webex gateway onboarding script.  It gets installed as app-hosting.  Separately, what I really wanted to accomplish was to install a virtual machine using virtual-service method.  My understanding is that app-hosting virtuals share host kernel, where with virtual-service method you can supply your own complete virtual machine, including kernel.  In other words, you could install a Windows virtual using virtual-service, but not using app-hosting.  Please correct me if I'm wrong.

virtual-service calls for .ova filename, which is a tar file also.  I didn't use ioxclient, just the following lines:
openssl sha1 *.qcow2 *.ver *.yaml > package.yaml
tar -cvf filename.ova *.qcow2 *.ver *.yaml *.mf

Please share your package.yaml file, list filenames going into your tar file, ioxclient command line and router config relevant to app-hosting.  Thank you.