This how-to is a step-by-step guide to create and configure an app for IOx on IE3400 which can monitor traffic flowing through the switch.
Introduction
IOx on IE3400 offers the capability to send a copy of any traffic flowing through the switch to a custom app hosted on the switch itself. This can be useful for troubleshooting as well as de-centralized monitoring and security analysis. For example, the Cisco Cybervision Sensor makes extensive use of this feature.
For details about setting up IOx and app hosting on IE3400 please see this article:
https://community.cisco.com/t5/internet-of-things-documents/how-to-setup-iox-on-ie3400-and-run-a-simple-docker-container/ta-p/4067665
Prerequisites
RSPAN vs. ERSPAN
There are two methods of sending traffic to the application - RSPAN and RSPAN with ERSPAN headers. Plain RSPAN (remote SPAN) uses a dedicated remote-span VLAN which has MAC learning disabled and mirrors all packets as-is. RSPAN with ERSPAN (encapsulated remote SPAN) headers also uses a dedicated remote-span VLAN, but on top of that encapsulates all packets with an ERSPAN GRE header.
The three benefits of ERSPAN encapsulation are:
- The ingress interface is recorded in the header
- The exact time at which the packet was received is recored
- The application can be packaged as a vanilla docker container (plain RSPAN requires the application to be packaged in a special way)
The main disadvantages are that the application must be able to decapsulate the ERSPAN header and that this can incur a (small) performance hit.
This how-to focuses on plain RSPAN. For RSPAN with ERSPAN headers, please see this article.
Step-by-step instructions to configure traffic monitoring using plain RSPAN
These steps assume that IOx is already configured.
1. Create IOx package from docker container
At first we must re-package our docker container into an IOx package. The reason for this is, that in order to be able to use plain RSPAN traffic monitoring without ERSPAN encapsulation, the IOx app needs to have a special "mirroring" flag set in its package.yaml file.
First, let's unpack the ioxclient archive, create a new directory and a new package.yaml file:
thulsdau@ubuntu1:~$ tar -xzf Downloads/ioxclient_1.10.1.0_linux_amd64.tar.gz
thulsdau@ubuntu1:~$ mkdir iperf3_ioxpackage
thulsdau@ubuntu1:~$ cd iperf3_ioxpackage/
thulsdau@ubuntu1:~/iperf3_ioxpackage$ cat <<EOF >package.yaml
> descriptor-schema-version: "2.10"
> info:
> name: iperf3_tcpdump
> version: latest
> app:
> cpuarch: aarch64
> env:
> PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> resources:
> cpu: "700"
> disk: "20"
> memory: "64"
> network:
> - interface-name: eth0
> ports:
> tcp:
> - "5201"
> udp:
> - "5201"
> - interface-name: eth1
> mirroring: yes
> profile: custom
> startup:
> rootfs: rootfs.tar
> target:
> - iperf3 -s
> type: docker
> EOF
thulsdau@ubuntu1:~/iperf3_ioxpackage$ cat package.yaml
descriptor-schema-version: "2.10"
info:
name: iperf3_tcpdump
version: latest
app:
cpuarch: aarch64
env:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
resources:
cpu: "700"
disk: "20"
memory: "64"
network:
- interface-name: eth0
ports:
tcp:
- "5201"
udp:
- "5201"
- interface-name: eth1
mirroring: yes
profile: custom
startup:
rootfs: rootfs.tar
target:
- iperf3 -s
type: docker
thulsdau@ubuntu1:~/iperf3_ioxpackage$
In this example, eth1 will be the interface inside the app which is supposed to receive the RSPAN traffic.
Next, we need to import the docker container which we want to re-package (if it is not already in the image repository):
thulsdau@ubuntu1:~/iperf3_ioxpackage$ zcat ../iperf3_dockerimage_aarch64.tar.gz | docker load
8ccb5a80ea7d: Loading layer [==================================================>] 3.54MB/3.54MB
Loaded image: iperf3_dockerimage_aarch64:latest
And finally, we can re-package using the ioxclient command line tool:
thulsdau@ubuntu1:~/iperf3_ioxpackage$ ../ioxclient_1.10.1.0_linux_amd64/ioxclient docker package --name iperf3_ioxpackage iperf3_dockerimage_aarch64 .
Currently active profile : default
Command Name: docker-package
Timestamp at DockerPackage start: 1591258547599
Using the package descriptor file in the project dir
Validating descriptor file package.yaml with package schema definitions
Parsing descriptor file..
Found schema version 2.10
Loading schema file for version 2.10
Validating package descriptor file..
File package.yaml is valid under schema version 2.10
Generating IOx package of type docker with layers as rootfs
Replacing symbolically linked layers in docker rootfs, if any
No symbolically linked layers found in rootfs. No changes made in rootfs
Removing emulation layers in docker rootfs, if any
The docker image is better left in it's pristine state
Parsing Package Metadata file : /home/thulsdau/iperf3_ioxpackage/.package.metadata
Updated package metadata file : /home/thulsdau/iperf3_ioxpackage/.package.metadata
No rsa key and/or certificate files provided to sign the package
-------------------------------------------------------------------------
Generating the envelope package
-------------------------------------------------------------------------
Checking if package descriptor file is present..
Skipping descriptor schema validation..
Created Staging directory at : /tmp/912613205
Copying contents to staging directory
Timestamp before CopyTree: 1591258548643
Timestamp after CopyTree: 1591258548654
Creating artifacts manifest file
Creating an inner envelope for application artifacts
Including rootfs.tar
Generated /tmp/912613205/artifacts.tar.gz
Parsing Package Metadata file : /tmp/912613205/.package.metadata
Updated package metadata file : /tmp/912613205/.package.metadata
Calculating SHA256 checksum for package contents..
Timestamp before SHA256: 1591258549079
Timestamp after SHA256: 1591258549079
Path: .package.metadata
SHA256 : 4d406c366842c1a952e45567f0fcd6009889e1de82b8fb702198792e04dd2bcf
Timestamp before SHA256: 1591258549079
Timestamp after SHA256: 1591258549079
Path: artifacts.mf
SHA256 : 41962a9f5e83f6b462921277dff4fc938add3e6b8b30aeb66fc414d8691e50f9
Timestamp before SHA256: 1591258549079
Timestamp after SHA256: 1591258549091
Path: artifacts.tar.gz
SHA256 : 60ed1b4eacf6cad8fb26b7e5f73891c824fcbc132188e9e436f42fcf2366200e
Timestamp before SHA256: 1591258549091
Timestamp after SHA256: 1591258549091
Path: envelope_package.tar.gz
SHA256 : 6a2009ef7c196e7ce2e650915c9c38a91f193692fc50c7e70d4428c2e39e5a74
Timestamp before SHA256: 1591258549091
Timestamp after SHA256: 1591258549092
Path: package.yaml
SHA256 : d4d158c4691456a4b782e333df326222f00afc29e6f4fa0a6b04d38b5ba1d6fa
Generated package manifest at package.mf
Generating IOx Package..
Package docker image iperf3_dockerimage_aarch64 at /home/thulsdau/iperf3_ioxpackage/iperf3_ioxpackage.tar
thulsdau@ubuntu1:~/iperf3_ioxpackage$ ls -lh
total 3.8M
-rw-r--r-- 1 thulsdau thulsdau 3.7M Jun 4 08:15 iperf3_ioxpackage.tar
-rw-r--r-- 1 thulsdau thulsdau 501 Jun 4 07:53 package.yaml
thulsdau@ubuntu1:~/iperf3_ioxpackage$
2. Configure a dedicated remote-span VLAN
A dedicated and otherwise unused VLAN which is configured for remote-span is needed. No interfaces other than AppGigabitEthernet1/1 are allowed to carry this VLAN.
IE3400#conf t
Enter configuration commands, one per line. End with CNTL/Z.
IE3400(config)#vlan 2
IE3400(config-vlan)#name RSPAN
IE3400(config-vlan)#remote-span
IE3400(config-vlan)#exit
3. Configure Interface AppGigabitEthernet1/1
The interface AppGigabitEthernet1/1 connects the application to the outside. It needs to carry the remote-span VLAN.
IE3400(config)#interface AppGigabitEthernet1/1
IE3400(config-if)#switchport mode trunk
IE3400(config-if)#switchport trunk allowed vlan 1,2
IE3400(config-if)#exit
4. Configure the IOx App
The IOx app needs to be configured. The app will typically need at least two interfaces - one for management and another one to receive the ERSPAN packets. The interface for ERSPAN needs to have an IP configured. Other than that, no special configuration is required here to support ERSPAN.
IE3400(config)#app-hosting appid iperf3_tcpdump
IE3400(config-app-hosting)#app-vnic AppGigabitEthernet trunk
IE3400(config-config-app-hosting-trunk)#vlan 1 guest-interface 0
IE3400(config-config-app-hosting-vlan-access-ip)#guest-ipaddress 100.64.1.12 netmask 255.255.255.0
IE3400(config-config-app-hosting-vlan-access-ip)#exit
IE3400(config-config-app-hosting-trunk)#vlan 2 guest-interface 1
IE3400(config-config-app-hosting-vlan-access-ip)#mirroring
IE3400(config-config-app-hosting-vlan-access-ip)#exit
IE3400(config-config-app-hosting-trunk)#app-default-gateway 100.64.1.1 guest-interface 0
IE3400(config-app-hosting)#exit
5. Configure traffic monitoring
Traffic monitoring itself is configured via the well-known "monitor session" commands. The ERSPAN destination IP needs to be the IP of the app configured in the previous step.
! Setup RSPAN
IE3400(config)#monitor session 1 source vlan 1 rx
IE3400(config)#monitor session 1 destination remote vlan 2
6. Install, Activate & Start App
IE3400#term mon
IE3400#copy http://192.168.3.3/tftp/iperf3_ioxpackage.tar flash:
Destination filename [iperf3_ioxpackage.tar]?
Accessing http://192.168.3.3/tftp/iperf3_ioxpackage.tar...
Loading http://192.168.3.3/tftp/iperf3_ioxpackage.tar !!!!!!!!!!!!!!!!!!!!
3878400 bytes copied in 1.940 secs (1999175 bytes/sec)
IE3400#app-hosting install appid iperf3_tcpdump package flash:iperf3_ioxpackage.tar
Installing package 'flash:iperf3_ioxpackage.tar' for 'iperf3_tcpdump'. Use 'show app-hosting list' for progress.
*Jun 4 08:40:41.696: %IM-6-INSTALL_MSG: R0/0: ioxman: app-hosting: Install succeeded: iperf3_tcpdump installed successfully Current state is DEPLOYED
IE3400#app-hosting activate appid iperf3_tcpdump
iperf3_tcpdump activated successfully
Current state is: ACTIVATED
IE3400#
*Jun 4 08:42:58.443: %IM-6-ACTIVATE_MSG: R0/0: ioxman: app-hosting: Activate succeeded: iperf3_tcpdump activated successfully Current state is in ACTIVATED
IE3400#app-hosting start appid iperf3_tcpdump
iperf3_tcpdump started successfully
Current state is: RUNNING
IE3400#
6. Verify that traffic is received by the app
!Login into shell of our application
IE3400#app-hosting connect appid iperf3_tcpdump session
/ # tcpdump -qnl -i eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:52:10.077484 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 198, length 64
08:52:11.078575 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 199, length 64
08:52:12.079843 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 200, length 64
08:52:13.081512 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 201, length 64
08:52:14.083247 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 202, length 64
08:52:15.084530 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 203, length 64
08:52:16.086386 IP 100.64.1.3 > 100.64.1.10: ICMP echo request, id 38225, seq 204, length 64
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel
/ #
Congratulations, you are running an application on your IE3400 switch which can monitor traffic!
