Nexus 5K and ERSPAN

Steven Williams · ‎03-20-2013

So the documentation says that the 5k does not support RSPAN. So ERSPAN it is, but source only. So the question is, can you only use a device for the destination that supports ERSPAN? For instance I could not create a source session on my 5K's and shot that traffic to my destination host on 2960S.

Steve Fuller · ‎03-23-2013

Hi,

This is entirely possible provided the host has an IP address that can be used as the destination IP address in the ERSPAN configuration.

The following is the ERSPAN configuration on my Nexus 5548.

[...]
interface loopback0
ip address 192.168.2.133/32
[...]
monitor session 1 type-erspan-source
erspan-id 11
vrf default
destination ip 172.17.1.101
source interface Ethernet1/31 both
source interface Ethernet1/32 both
no shut
monitor erspan origin ip-address 192.168.2.133 global

I have a Red Hat Linux server configured with an IP address 172.17.1.101 which is reachable from my Nexus switch:

ocs5548-1# ping 172.17.1.101
PING 172.17.1.101 (172.17.1.101): 56 data bytes
64 bytes from 172.17.1.101: icmp_seq=0 ttl=62 time=4.295 ms
64 bytes from 172.17.1.101: icmp_seq=1 ttl=62 time=0.868 ms
64 bytes from 172.17.1.101: icmp_seq=2 ttl=62 time=0.978 ms
64 bytes from 172.17.1.101: icmp_seq=3 ttl=62 time=7.211 ms
64 bytes from 172.17.1.101: icmp_seq=4 ttl=62 time=9.57 ms
--- 172.17.1.101 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.868/4.584/9.57 ms

And now when I run tcpdump on my Linux server I can see the GRE packets sourced from the ERSPAN session on the Nexus.

[sfuller@rhel5 ~]$ sudo tcpdump -i bond0 proto gre -c 10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 96 bytes
09:07:27.047805 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13102, length 50: gre-proto-0x88be
09:07:27.047891 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13103, length 50: gre-proto-0x88be
09:07:27.049757 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13104, length 122: gre-proto-0x88be
09:07:27.050813 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13105, length 127: gre-proto-0x88be
09:07:27.051899 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13106, length 50: gre-proto-0x88be
09:07:27.052054 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13107, length 50: gre-proto-0x88be
09:07:27.052211 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13108, length 50: gre-proto-0x88be
09:07:27.052645 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13109, length 50: gre-proto-0x88be
09:07:27.052852 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13110, length 50: gre-proto-0x88be
09:07:27.053009 IP ocs5548-1-loop0.ntilab.net > rhel5-bond0.ntilab.net: GREv0, seq 13111, length 50: gre-proto-0x88be
10 packets captured
10 packets received by filter
0 packets dropped by kernel

You'll obviously need a capture device that understand GRE, but that shouldn't be an issue.

The other thing to be mindful of with ERSPAN is the MTU size if you're trying to capture the entire packet. An ERSPAN packet comprises the encapsulated IPv4 header (20-bytes), GRE header (8-bytes), ERSPAN header (8-bytes) and the original packet and so a 1500-byte packet that is being captured would have to be truncated to make it through a network with a 1500-byte MTU. There's a section on Configuring Truncated ERSPAN in the Nexus System Management guide if you want to control the truncation.

Regards

Steven Williams · ‎03-23-2013

Where is your sniffer connected to? The Nexus?

I have done the exact config you have and can also ping my IDS interface, but it is not seeing the traffic.

I have my internet router connected to my 5K's, then a vPC to a stack 2960.

I am monitoring one port on each N5K and trying to send it to a port on my 2960. 2960's do not support ERSPAN.

Steve Fuller · ‎03-23-2013

Hi,

The sniffer is connected directly to the Nexus 5K, but it works equally well to one that's connected to another switch.

So here I'm using another Linux server that you can see is not directly connected:

ocs5548-1# traceroute 192.168.1.102
traceroute to 192.168.1.102 (192.168.1.102), 30 hops max, 40 byte packets
1 ocs7206-1-gi0-3.ntilab.net (192.168.2.136) 1.797 ms 0.846 ms 2.661 ms
2 ocs7206-2-tun31051.ntilab.net (192.168.2.10) 2.338 ms 1.646 ms 2.687 ms
3 ocs6504-2-gi4-48.ntilab.net (192.168.15.162) 2.28 ms 1.711 ms 8.74 ms
4 ocs4948-1-gi1-48.ntilab.net (192.168.1.154) 8.92 ms 5.592 ms 8.922 ms
5 rhel3-eth0.ntilab.net (192.168.1.102) 8.929 ms 5.589 ms 8.939 ms

And a tcpdump on this server sees the ERPSAN traffic from the Nexus 5500:

[sfuller@rhel3 ~]$ sudo tcpdump -i eth0 proto 47 -c 10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
15:05:08.915890 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 88, length 48: gre-proto-0x88be
15:05:09.459487 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 89, length 44: gre-proto-0x88be
15:05:09.460342 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 90, length 90: gre-proto-0x88be
15:05:09.552179 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 91, length 44: gre-proto-0x88be
15:05:09.752264 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 92, length 58: gre-proto-0x88be
15:05:09.839423 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 93, length 70: gre-proto-0x88be
15:05:09.839425 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 94, length 70: gre-proto-0x88be
15:05:09.839426 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 95, length 46: gre-proto-0x88be
15:05:10.210534 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 96, length 58: gre-proto-0x88be
15:05:10.211444 IP ocs5548-1-loop0.ntilab.net > rhel3-eth0.ntilab.net: GREv0, seq 97, length 104: gre-proto-0x88be
10 packets captured
10 packets received by filter
0 packets dropped by kernel

The address I'm using as the ERSPAN destination is the IP address assigned to the interface of the Linux server itself. The Catalyst 4948 that the server is connected to knows nothing of ERSPAN, and simply forwards the ERSPAN traffic to the server.

If you have an IDS that has an IP address that you can ping from the IP address used as the monitor source, and this is

connected to the Catalyst 2960, then the setup looks to be pretty much the same.

The one difference you have is that your Catalyst 2960 is connected to your Nexus 5K via vPC. I don't have a Layer-2 switch connected to my Nexus via vPC, but I do have other servers connected to FEX which has vPC to the Nexus 5K so similar.

Apologies for asking the obvious, but you have done a no shut within the monitor session?

Can you paste the output of a show run monitor and show monitor session from the Nexus.

Regards

Steven Williams · ‎03-23-2013

monitor session is up, but now that you mention it I am sourcing it from a loopback and I am not sure if that is in a "no shut" status or even reachable via my IDS server since it sits in a different vlan.

Steven Williams · ‎03-24-2013

Loopbacks are up. Something weird is that I can only ping one of the two from my IDS device. When I do a tracert, it shows it hits only one of the Nexus 5k's and because there is no routing over the peer link it cant hit the other one, so this is an issue because I need traffic from both nexus devices spanning to the IDS.

Also does my global ip source have to be lookback?

Steve Fuller · ‎03-25-2013

Hi,

There should no be limitations with ERSPAN that require the origin IP address has to be that assigned to a loopback interface.

I don't really understand what your topology is here, but as the Catalyst 2960 is an L2 switch then I assume the Nexus 5K has SVI and is doing the. Can you provide a diagram or some information as to how things are connected. Other than the fact the C2960 is connected via vPC we know very little at this point.

Also the output of a show monitor session so I can see what you're trying to capture.

When you look at any interfaces on the path between the Nexus 5K and the interface your IDS is connected to, do you see any traffic rate increase equivalent to that you see on your ERSPAN source? I'm trying to understand whether the ERSPAN might actually be working but the IDS is not seeing it, it's whether it's getting dropped somewhere along the path.

Clearly this does work as can be seen from what I've provided above so there's something different about your setup.

Regards

Steven Williams · ‎03-25-2013

The VIP for VLAN 105 is 10.170.105.1, I forgot that on the diagram.

I can ping the IDS from the ASA, Nexus 5ks.

I am trunking vlan 105 all the way through.

The loopbacks are 10.170.255.1 and .2.

NEXUS5K001# show monitor session 1
   session 1
---------------
type              : erspan-source
state             : down (Waiting for ARP response)
erspan-id         : 1
vrf-name          : default
destination-ip    : 10.170.105.25
ip-ttl            : 255
ip-dscp           : 0
origin-ip         : 10.170.255.1 (global)
source intf       :
    rx            : Po105
    tx            : Po105
    both          : Po105
source VLANs      :
    rx            :
source VSANs      :
    rx            :

NEXUS5K001#

NEXUS5K002# show monitor session 1

session 1

---------------

type : erspan-source

state : down (Waiting for ARP response)

erspan-id : 1

vrf-name : default

destination-ip : 10.170.105.25

ip-ttl : 255

ip-dscp : 0

origin-ip : 10.170.255.2 (global)

source intf :

rx : Po105

tx : Po105

both : Po105

source VLANs :

rx :

source VSANs :

rx :

NEXUS5K002#

Steve Fuller · ‎03-26-2013

Hi,

The monitor session isn't right here as it shows the state as down (Waiting for ARP response) when it should be showing as up.

ocs5548-1# sh monitor session 1

session 1

---------------

type : erspan-source

state : up

erspan-id : 11

vrf-name : default

destination-ip : 172.17.2.100

ip-ttl : 255

ip-dscp : 0

origin-ip : 192.168.2.133 (global)

source intf :

rx : Eth1/31 Eth1/32

tx : Eth1/31 Eth1/32

both : Eth1/31 Eth1/32

source VLANs :

rx :

If you ping the IDS from each Nexus 5K and then run the show ip arp 10.170.105.25 command do you see a MAC for the IDS IP address? On my setup I see the following:

ocs5548-1# sh ip ro 172.17.2.100

IP Route Table for VRF "default"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

172.17.2.100/32, ubest/mbest: 1/0, attached

*via 172.17.2.100, Vlan172, [2/0], 5w4d, am

ocs5548-1# sh ip arp 172.17.2.100

Flags: * - Adjacencies learnt on non-active FHRP router

+ - Adjacencies synced via CFSoE

# - Adjacencies Throttled for Glean

D - Static Adjacencies attached to down interface

IP ARP Table

Total number of entries: 1

Address Age MAC Address Interface

172.17.2.100 00:07:00 0050.5694.3a34 Vlan172

Can you paste the output of the show ip arp command from each Nexus 5K?

I guess the other question here is what NX-OS version you're running? I'm running 5.1(3)N2(1).

Regards