How to configure traffic monitoring with IOx on IE3400 using ERSPAN encapsulation

[thulsdau]

This how-to is a step-by-step guide to create and configure an app for IOx on IE3400 which can monitor traffic flowing through the switch.

Introduction

IOx on IE3400 offers the capability to send a copy of any traffic flowing through the switch to a custom app hosted on the switch itself. This can be useful for troubleshooting as well as de-centralized monitoring and security analysis. For example, the Cisco Cybervision Sensor makes extensive use of this feature.

For details about setting up IOx and app hosting on IE3400 please see this article:
https://community.cisco.com/t5/internet-of-things-documents/how-to-setup-iox-on-ie3400-and-run-a-simple-docker-container/ta-p/4067665 

Prerequisites

• IE3400 or IE3400H switch
• 4GB SD-Card (SD-IE-4GB=)
• IOS-XE version 17.2.1 installed on flash: (not on the SD-CARD!), see release notes for installation instructions
• Connection to the CLI, either via console or network (i.e. telnet, ssh, ...)
• A docker container image with binaries compiled for aarch64 and a traffic monitoring application like tcpdump (a sample container image is attached to this document) 

RSPAN vs. ERSPAN

There are two methods of sending traffic to the application - RSPAN and RSPAN with ERSPAN headers. Plain RSPAN (remote SPAN) uses a dedicated remote-span VLAN which has MAC learning disabled and mirrors all packets as-is. RSPAN with ERSPAN (encapsulated remote SPAN) headers also uses a dedicated remote-span VLAN, but on top of that encapsulates all packets with an ERSPAN GRE header.

The three benefits of ERSPAN encapsulation are:

• The ingress interface is recorded in the header
• The exact time at which the packet was received is recored
• The application can be packaged as a vanilla docker container (plain RSPAN requires the application to be packaged in a special way)

The main disadvantages are that the application must be able to decapsulate the ERSPAN header and that this can incur a (small) performance hit.

This how-to focuses on RSPAN with ERSPAN headers. For plain RSPAN, please see this article.

Step-by-step instructions to configure traffic monitoring with ERSPAN encapsulation

These steps assume that IOx is already configured.

1. Configure a dedicated remote-span VLAN

A dedicated and otherwise unused VLAN which is configured for remote-span is needed. No interfaces other than AppGigabitEthernet1/1 are allowed to carry this VLAN.

​IE3400#conf t​
Enter configuration commands, one per line.  End with CNTL/Z.​
IE3400(config)#vlan 2​
IE3400(config-vlan)#name ERSPAN​
IE3400(config-vlan)#remote-span​
IE3400(config-vlan)#exit​

 2. Configure Interface AppGigabitEthernet1/1

The interface AppGigabitEthernet1/1 connects the application to the outside. It needs to carry the remote-span VLAN.

IE3400(config)#interface AppGigabitEthernet1/1
IE3400(config-if)#switchport mode trunk
IE3400(config-if)#switchport trunk allowed vlan 1,2
IE3400(config-if)#exit

3. Configure the IOx App

The IOx app needs to be configured. The app will typically need at least two interfaces - one for management and another one to receive the ERSPAN packets. The interface for ERSPAN needs to have an IP configured. Other than that, no special configuration is required here to support ERSPAN.

IE3400(config)#app-hosting appid iperf3​_tcpdump
IE3400(config-app-hosting)#app-vnic AppGigabitEthernet trunk​
IE3400(config-config-app-hosting-trunk)#vlan 1 guest-interface 0​ !place eth0 in Vlan 1 for mgmt
IE3400(config-config-app-hosting-vlan-access-ip)#guest-ipaddress 100.64.1.11 netmask 255.255.255.0​
IE3400(config-config-app-hosting-vlan-access-ip)#exit ​
IE3400(config-config-app-hosting-trunk)#vlan 2 guest-interface 1​ !place eth1 in Vlan 2 for ERSPAN
IE3400(config-config-app-hosting-vlan-access-ip)#guest-ipaddress 192.168.2.3 netmask 255.255.255.0​
IE3400(config-config-app-hosting-vlan-access-ip)#exit
IE3400(config-config-app-hosting-trunk)#exit​
IE3400(config-app-hosting)#app-default-gateway 100.64.1.1 guest-interface 0​
IE3400(config-app-hosting)#app-resource profile custom​
IE3400(config-app-resource-profile-custom)#cpu 1400​
IE3400(config-app-resource-profile-custom)#memory 256​
IE3400(config-app-resource-profile-custom)#vcpu 2​
IE3400(config-app-resource-profile-custom)#exit
IE3400(config-app-hosting)#exit

4. Configure traffic monitoring

Traffic monitoring itself is configured via the well-known "monitor session" commands. The ERSPAN destination IP needs to be the IP of the app configured in the previous step.

! Setup RSPAN with ERSPAN encapsulation
IE3400(config)#monitor session 1 source vlan 1 rx​
IE3400(config)#monitor session 1 destination remote vlan 2​
IE3400(config)#monitor session 1 destination format-erspan 192.168.2.3

5. Install, Activate & Start App

IE3400#term mon​
IE3400#copy http://192.168.3.3/tftp/iperf3_eft_dockerimage_aarch64.tar flash:​
Destination filename [iperf3_eft_dockerimage_aarch64.tar]?​
Accessing http://192.168.3.3/tftp/iperf3_eft_dockerimage_aarch64.tar...​
Loading http://192.168.3.3/tftp/iperf3_eft_dockerimage_aarch64.tar !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!​
7281152 bytes copied in 3.901 secs (1866195 bytes/sec)​
IE3400#app-hosting install appid iperf3_tcpdump package flash:iperf3_eft_dockerimage_aarch64.tar​
Installing package 'flash:iperf3_eft_dockerimage_aarch64.tar' for 'iperf3'. Use 'show app-hosting list' for progress.​

IE3400#​
*Apr  9 05:28:59.605: %IM-6-INSTALL_MSG: R0/0: ioxman: app-hosting: Install succeeded: iperf3 installed successfully Current state is DEPLOYED​

IE3400#app-hosting activate appid iperf3​_tcpdump
iperf3 activated successfully​
Current state is: ACTIVATED​

IE3400#​
*Apr  9 05:30:32.088: %IM-6-ACTIVATE_MSG: R0/0: ioxman: app-hosting: Activate succeeded: iperf3 activated successfully Current state is in ACTIVATED​
IE3400#app-hosting start appid iperf3​_tcpdump
iperf3 started successfully​
Current state is: RUNNING​

IE3400#

6. Verify that traffic is received by the app

!Login into shell of our application
IE3400#app-hosting connect appid iperf3_tcpdump session
/ # tcpdump -qln -i eth1 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
05:46:41.525108 IP 127.0.0.1 > 192.168.2.3: GREv0, seq 24624, length 170: gre-proto-0x22eb
05:46:41.628391 IP 127.0.0.1 > 192.168.2.3: GREv0, seq 24625, length 104: gre-proto-0x22eb
05:46:43.524850 IP 127.0.0.1 > 192.168.2.3: GREv0, seq 24626, length 170: gre-proto-0x22eb
05:46:45.067140 IP 127.0.0.1 > 192.168.2.3: GREv0, seq 24627, length 92: gre-proto-0x22eb
05:46:45.373117 IP 127.0.0.1 > 192.168.2.3: GREv0, seq 24628, length 92: gre-proto-0x22eb
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
/ # exit
IE3400#

Congratulations, you are running an application on your IE3400 switch which can monitor traffic!