Cisco Nexus 9000 new Ethanalyzer local interface options

GR33N · ‎08-20-2020

Hi Everyone,

Did somebody notice the new options for ethanalyzer local interfaces?

It now comes with the following:

front-panel
inband
mgmt
port-channel

front-panel is used to physical interface while the port-channel is the port-channel interface itself. I tried to check whether what I can see when I use those options while using a capture-filter "host x.x.x.x" but did not capture anything.

I am looking for documentations and it seems there was nothing wrtiten for it when it comes to what can it really capture or any use case. I was wondering whether anyone from this community has an idea about this.

By the way, I notice them being available for nxos.9.3.4

Thanks!

Christopher Hart · ‎08-20-2020

Hello!

As you noticed, the front-panel and port-channel options were added recently in the NX-OS 9.3(x) minor software release. We've observed a miss in documentation of these new options and are planning on amending the documentation shortly.

To be clear, these new Ethanalyzer options will only capture control plane traffic that is received on a specified front-panel port or port-channel interface. Data plane traffic that ingresses or egresses a specified interface will not be captured with these new options.

For example, if you receive an OSPF Hello packet, Spanning Tree BPDU, LACPDU, etc. on a specific front-panel port (such as Eth1/1) you can use the front-panel option to view that traffic. However, if an ICMP ping from a remote device destined to a host connected to Eth1/1 egresses that interface, that is data plane traffic and will not be observed through an Ethanalyzer control plane packet capture (either with or without these new options.)

If you need to capture data plane traffic, you still have options! All Nexus switches support local SPAN/monitor sessions, which can replicate data plane traffic out of another interface towards a host designated for capturing traffic. Nexus 9000 switches with Cloud Scale ASICs can perform a SPAN-to-CPU session to replicate data plane traffic to the control plane for inspection via Ethanalyzer. Furthermore, Nexus 9000 switches with Cloud Scale ASICs can also perform an ELAM, which will display the forwarding decision made on a specific packet.

I hope this helps - thank you!

-Christopher

GR33N · ‎08-20-2020

Hi Christopher,

Thank you for your clarification, I really got excited as I thought this will be pretty much a tcpdump session that can capture data plane traffic to be used for analyzing tcp/ip performance issues. ;)

I notice that same workaround you mentioned works and behaves the same way when we use the local interface "inband" option, I was wondering whats the main difference or actual use case if I use the port-channel or front-end option?

Thank you!

Sergiu.Daniluk · ‎08-20-2020

Hi @GR33N

I haven't tried it yet, but the advantage of front-end/port-channel vs inbound seems pretty clear. You specify on which interface you want to capture the control plane packets. You still capture the traffic at (nearly) CPU level, but you filter the capture based on front port interface.A good example are ARP packets. You will get a lot of ARP requests from all front ports. But if you know that the ARP request will ingress Eth1/1, you specify the ethanalyzer to capture only ARP packets received on Eth1/1.

Stay safe,

Sergiu

Christopher Hart · ‎08-21-2020

Hello!

@Sergiu.Daniluk's post is correct - the use case for using either the front-panel or port-channel parameters is to assist with filtering control plane traffic received on a specific port. The inband option will show you all control plane traffic received on the supervisor's inband interface, while the front-panel and port-channel parameters will only show you control plane traffic received on the supervisor's inband interface that ingressed the device through a specific front-panel port or port-channel.

Particularly when troubleshooting an issue, there are many circumstances where you may know that you're receiving a specific control plane traffic on a specific port (such as an LACPDU or Spanning Tree BPDU) that you cannot easily filter using display filters or capture filters. The new front-panel or port-channel parameters in conjunction with capture or display filters make this same task trivial.

In case I misunderstood your question and you were asking how the device is able to differentiate control plane traffic received on different interfaces, this is done through a shim header that is added to the packet by the switch's Cloud Scale ASIC. This internal header only exists while the packet is traversing the switch - it is not added to the packet when it exits the switch and is transmitted on the wire. A standard Ethanalyzer control plane packet capture will not display this internal shim header, but the decode-internal option reveals it. An example of this on an inbound OSPF Hello packet is shown below.

N9K# ethanalyzer local interface inband decode-internal display-filter ospf limit-captured-frames 0
<snip>
Capturing on inband
Frame 37 (142 bytes on wire, 142 bytes captured)
CPU-inbound Insieme TOR Single-Encap
    First 8 bytes: 0x0000000000000000
    Next 8 bytes: 0x0000000000000000
    Dest MAC: 0x0000000002020202
    Source MAC: 0x0000000004040404
    1000 1000 1000 1001 = TPID: 34953
    0000 0000 0000 0000 = pcp cfi vid: 0
    0000 0000 0000 0000 0000 0000 0000 0000 = First 4 bytes of Blob: 0x00000000
    1111 1111 = SUP Code: 0xff
    0010 0011 = SUP Quene Number: 0x23
    Timestamp: 0x00000000ae58e721
    1111 1011 = START of Frame: 0xfb
    0... .... = Hdr Type: 0x00
    .0.. .... = Ext Hdr: 0x00
    ..01 01.. = Opcode: 0x05
    .... .... .... ..00 0000 1100 0101 .... = Source Index: 0x000000c5
    .... .... .... 0000 0000 0000 00.. .... = Dest Index: 0x00000000
    ..00 0000 01.. .... = Source Chip: 0x0001
    ..01 0000 00.. .... = Source Port: 0x0040
    ..00 0000 00.. .... = Dest Chip: 0x0000
    ..00 0000 00.. .... = Dest Port: 0x0000
    ..00 0000 110. .... = Outer BD: 0x0006
    .... .... ...0 1000 0001 1100 1... .... = BD: 0x00001039
    .0.. .... = Traceroute: 0x00
    ..0. .... = Don't Learn: 0x00
    ...0 .... = SPAN: 0x00
    .... 0... = Alter if possible: 0x00
    .... .0.. = IP TTL Bypass: 0x00
    .... ..0. = Source is Tunnel: 0x00
    .... ...0 = Dest is Tunnel: 0x00
    0... .... = L2 Tunnel: 0x00
    .0.. .... = SUP Tx: 0x00
    ..00 000. = SUP Code: 0x00
    .... ...0 000. .... = COS De: 0x0000
    ...0 000. = TClass: 0x00
    .... ...0 = src_is_peer: 0x00
    0001 0000 = Packet Hash: 0x10
Ethernet II, Src: e8:65:49:94:8c:3f (e8:65:49:94:8c:3f), Dst: 01:00:5e:00:00:05 (01:00:5e:00:00:05)
Internet Protocol, Src: 10.100.0.30 (10.100.0.30), Dst: 224.0.0.5 (224.0.0.5)
Open Shortest Path First

Partway through this internal header, you'll notice a "Source Index" field that has a hexadecimal value of 0xc5. In decimal, this has a value of 197. This value corresponds with the internal VIF (Virtual InterFace) number assigned to physical interfaces on Nexus 9000 Cloud Scale ASICs. In other words, this value dictates through which interface this particular packet entered the device.

We can confirm which interface maps to a VIF of 197 through the show interface hardware-mappings command, as shown from the same box in my lab below:

N9K# show interface hardware-mappings 
<snip>
Legends:
       SMod  - Source Mod. 0 is N/A
       Unit  - Unit on which port resides. N/A for port channels
       HPort - Hardware Port Number or Hardware Trunk Id:
       HName - Hardware port name. None means N/A
       FPort - Fabric facing port number. 255 means N/A
       NPort - Front panel port number
       VPort - Virtual Port Number. -1 means N/A
       Slice - Slice Number. N/A for BCM systems
       SPort - Port Number wrt Slice. N/A for BCM systems
       SrcId - Source Id Number. N/A for BCM systems
       MacIdx - Mac index. N/A for BCM systems
       MacSubPort - Mac sub port. N/A for BCM systems

-------------------------------------------------------------------------------------------------------
Name       Ifindex  Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF  Block BlkSrcID
-------------------------------------------------------------------------------------------------------
Eth1/48    1a005e00 1    0    71    255   188   -1    0     71    142   17    6     189  1     70      
Eth1/49    1a006000 1    0    60    255   192   -1    0     60    120   15    0     193  1     48      
Eth1/50    1a006200 1    0    64    255   196   -1    0     64    128   16    0     197  1     56      
Eth1/51    1a006400 1    0    28    255   200   -1    0     28    56    7     0     201  0     56

This tells us that this particular OSPF Hello packet entered the switch via interface Ethernet1/50. This makes logical sense, as in my lab, Ethernet1/50 is in the 10.100.0.28/30 subnet and is connected to an upstream OSPF router owning an IP address of 10.100.0.30 (the source IP address of the OSPF Hello packet that was received.)

N9K# show running-config interface Ethernet1/50
<snip>
interface Ethernet1/50
  mtu 9216
  no ip redirects
  ip address 10.100.0.29/30
  no ipv6 redirects
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode
  no shutdown

To tie this back to the new front-panel and port-channel parameters in Ethanalyzer - Ethanalyzer is able to use the source index value within the internal shim header added by the Cloud Scale ASIC of control plane packets to filter display control plane packets received on a specific front-panel or port-channel interface.

I hope this helps - Thank you!

-Christopher