cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Using Ethanalyzer on Nexus platform for control-plane and data-plane traffic analysis

74587
Views
50
Helpful
4
Comments

 

Introduction

Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code. Ethanalyzer is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic. This document covers configuration of Ethanalyzer, examples of its implementation and Ethanalyzer usage together with ACLs "log" option to sniff data plane traffic.

 

Configuration

To configure Ethanalyzer, use the following commands:

Command

Purpose

ethanalyzer local interface

Captures packets sent or received   by the supervisor and provides detailed protocol information.

ethanalyzer local interface inband

Captures packets sent or received   by the supervisor and provides detailed protocol information in the inband   and outband interfaces.

ethanalyzer local interface mgmt

Captures packets sent or received   by the supervisor and provides detailed protocol information in the   management interfaces.

ethanalyzer local interface   {inband | mgmt} brief

Captures packets sent or received   by the supervisor and provides a summary of protocol information.

ethanalyzer local interface   {inband | mgmt} limit-captured-frames

Limits the number of frames to   capture.

ethanalyzer local interface   {inband | mgmt} limit-frame-size

Limits the length of the frame to   capture.

ethanalyzer local interface   {inband | mgmt} capture-filter

Filters the types of packets to   capture.

ethanalyzer local interface   {inband | mgmt} display-filter

Filters the types of captured   packets to display.

ethanalyzer local interface {inband   | mgmt} decode-internal

Decodes the internal frame header   for Cisco NX-OS.

Note Do   not use this option if you plan to analyze the data using Wireshark instead   of Ethanalyzer.

ethanalyzer local interface   {inband | mgmt} write

Saves the captured data to a file.

ethanalyzer local read

Opens the captured data file and   analyzes it.

 

Ethanalyzer does not capture data traffic that Cisco NX-OS forwards in the hardware but you can use ACLs with log option as a workaround (see the corresponding paragraph below).

Ethanalyzer uses the same capture filter syntax as tcpdump and uses the Wireshark display filter syntax.

See the Wireshark weekly tips for helpful hints on using the tool.

 

Filtering

Two types of filters are supported:

1. capture-filter: standard tcmdump capture filter syntax

tcp/udp port <>, src/dst portrange <>, src/dst port <>, src/dst host <>, src/dst net <>, less/greater <>, vlan etc …

2. display-filter: standard wireshark display filter syntax

ip.src==x.x.x.x,  frame.len > x, http.request.uri == http://www.wireshark.org/

 

Capture output

Captured packet data can be written to different locations for easy debugging:

Nexus7000# ethanalyzer local read ?
   bootflash:, logflash:, slot0:, usb1:, usb2:, volatile:

 

1. Write the ethanalyzer ouput to a file on bootflash:

Nexus7000# ethanalyzer local interface inband write bootflash:sniff_capture

2. Read the ethanalyzer ouput from a file on bootflash: (You can also add a “pipe” to the end):

Nexus7000# ethanalyzer local read bootflash:sniff_capture

 

Capture output

File logging limitations and enhancements

The file size is set to 10485. As a result even if you have enough space on flash, you will drop packets if you exceed this size.

 

Switch# ethanalyzer local interface inband  capture-filter " ip proto 89" limit-captured-frames 0 write logflash:ospf.pcap
Capturing on inband
81342
2560 packets  dropped

 

ACLs and Ethanalyzer for Data Plane Sampling

The Ethanalyzer captures only traffic on CPU, so seems as unsuitable solution for the data plane traffic analysis. However, this limitation can be avoided with a use of ACL logging to sample specific packets from data plane.

 

Nexus 7000 ACLs features:

  •   Nexus 7000 ACLs enforced by hardware forwarding engine at wire rate;
  •   Sizeable pool of hardware resources available for classification -64K CL TCAM entries;
  •   ACLs can match based on granular IP- or MAC-based parameters,including L4 protocol and port numbers, packet length, etc.

When we use ACLs and the “log” keyword, access control entries (ACEs) with log keyword cause system to punt a copy of matching packets to supervisor CPU. Key point is that original traffic forwarded or dropped in hardware with no performance penalty. Note that punted copies subjected to hardware rate limiter, forwarding engine hardware enforces rate to avoid saturating inband interface/CPU.

hardware rate-limit access-list-log command adjusts rate (100 pps by default).

 

"log" keyword can be used both with permit or deny ACEs and system syslogs punted packets:

 

  n7010# 2008 Sep 15 19:21:00 n7010 %$ VDC-1 %$ %ACLLOG-6-ACLLOG_FLOW_INTERVAL: Source IP: 10.1.1.2, Destination IP: 10.1.2.2, Source Port: 1024, Destination Port: 40000, Source Interface: Ethernet1/1, protocol: "TCP"(6),  Hit-count = 6224

 

Full Packet Analysis

 

1. Define ACL entry with logging to match traffic of interest

 

ip access-list acl-cap
  permit tcp 10.1.1.3/32 10.1.2.2/32 eq 5000 log
  permit ip any any

 

2. Attach ACL to interface

 

interface e1/1
  ip access-group acl-cap in

 

3. Define ethanalyzer capture and/or display filter to capture just the subject traffic

 

ethanalyzer local interface inband capture-filter “tcp port 5000”

 

All packets matching log ACEs are punted (with rate limiting) – use capture/display filters to see only a subset of traffic matching log ACEs.

 

4. View captured traffic on-switch, or copy to PC/workstation for GUI analysis

 

Example – Brief Decode On-Switch

 

  n7010# ethanalyzer local interface inband brief capture-filter "tcp port 5000" limit-cap 3
   Capturing on eth0
   2008-09-16 18:36:08.315736     10.1.1.3 -> 10.1.2.2     TCP 5000 > 5000 [ACK] Seq=0 Ack=0 Win=4096 Len=454
   2008-09-16 18:36:08.988747     10.1.1.3 -> 10.1.2.2     TCP [TCP Retransmission] 5000 > 5000 [ACK] Seq=0 Ack=0 Win=4096 Len=454
   2008-09-16 18:36:09.873360     10.1.1.3 -> 10.1.2.2     TCP [TCP Retransmission] 5000 > 5000 [ACK] Seq=0 Ack=0 Win=4096 Len=454
   3 packets captured
   n7010#

 

Example – Full Decode On-Switch

 

 

  n7010# ethanalyzer local interface inband capture-filter "tcp port 5000" limit-captured-frames 1 | no-more
   Capturing on eth0
   Frame 1 (508 bytes on wire, 508 bytes captured)
     Arrival Time: Sep 16, 2008 18:33:12.138957000
     [Time delta from previous captured frame: 1221615192.138957000 seconds]
     [Time delta from previous displayed frame: 1221615192.138957000 seconds]
     [Time since reference or first frame: 1221615192.138957000 seconds]
     Frame Number: 1
     Frame Length: 508 bytes
     Capture Length: 508 bytes
     [Protocols in frame: eth:ip:tcp:data]
   Ethernet II, Src: 00:1b:54:c2:40:41 (00:1b:54:c2:40:41), Dst: 00:10:94:00:00:02 (00:10:94:00:00:02)
   <…>
   Internet Protocol, Src: 10.1.1.3 (10.1.1.3), Dst: 10.1.2.2 (10.1.2.2)
     Version: 4
     Header length: 20 bytes
     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
   <…>
     Source: 10.1.1.3 (10.1.1.3)
     Destination: 10.1.2.2 (10.1.2.2)
   Transmission Control Protocol, Src Port: 5000 (5000), Dst Port: 5000 (5000), Seq: 0, Ack: 0, Len: 454
     Source port: 5000 (5000)
     Destination port: 5000 (5000)
<…>

 

Example – Write Data to File

 

  n7010# ethanalyzer local interface inband capture-filter "tcp port 5000" limit-captured-frames 50 write bootflash:test.cap   //Use “write” option to send captured frames to flash file instead of print onscreen.
   Capturing on eth0
   50
   n7010# dir test.cap
       26224     Sep 16 18:40:08 2008  test.cap
   Usage for bootflash://sup-local
    1138716672 bytes used
     671182848  bytes free
    1809899520 bytes total
   n7010# copy bootflash: scp: //Copy cap file off-box to a PC/workstation to view the file contents in GUI version of wireshark.
   Enter source filename: test.cap
   Enter vrf (If no input, current vrf 'default' is considered): management
   Enter hostname for the scp server: dove
   Enter username: tstevens
   Password: ***
   test.cap                                  100%   26KB  25.6KB/s   00:00
   n7010#

 

Filter Examples

1. This example shows captured data (limited to four packets) on the management interface:

 

switch(config)# ethanalyzer local interface mgmt brief limit-captured-frames 4
Capturing on eth1

2008-02-18 13:21:21.841182 172.28.230.2 -> 224.0.0.2 HSRP Hello (state Standy)

2008-02-18 13:21:21.842190 10.86.249.17 -> 172.28.231.193 TCP 4261 > telnet [AC] Seq=0 Ack=0 Win=64475 Len=0

2008-02-18 13:21:21.843039 172.28.231.193 -> 10.86.249.17 TELNET Telnet Data ..

2008-02-18 13:21:21.850463 00:13:5f:1c:ee:80 -> ab:00:00:02:00:00 0x6002 DEC DN

Remote Console

4 packets captured

 


2. This example shows detailed captured data for one HSRP packet:

 

switch(config)# ethanalyzer local interface mgmt capture-filter "udp port 1985"
limit-captured-frames 1

Capturing on eth1

Frame 1 (62 bytes on wire, 62 bytes captured)

Arrival Time: Feb 18, 2008 13:29:19.961280000

[Time delta from previous captured frame: 1203341359.961280000 seconds]

[Time delta from previous displayed frame: 1203341359.961280000 seconds]

[Time since reference or first frame: 1203341359.961280000 seconds]

Frame Number: 1

Frame Length: 62 bytes

Capture Length: 62 bytes

[Frame is marked: False]

[Protocols in frame: eth:ip:udp:hsrp]

 

Ethernet II, Src: 00:00:0c:07:ac:01 (00:00:0c:07:ac:01), Dst: 01:00:5e:00:00:02

(01:00:5e:00:00:02)

Destination: 01:00:5e:00:00:02 (01:00:5e:00:00:02)

Address: 01:00:5e:00:00:02 (01:00:5e:00:00:02)

.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)

.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

Source: 00:00:0c:07:ac:01 (00:00:0c:07:ac:01)

Address: 00:00:0c:07:ac:01 (00:00:0c:07:ac:01)

 

.... ...0 .... .... .... .... = IG bit: Individual address (unicast)

.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

 

Type: IP (0x0800)

Internet Protocol, Src: 172.28.230.3 (172.28.230.3), Dst: 224.0.0.2 (224.0.0.2)

Version: 4

Header length: 20 bytes

Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)

1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)

.... ..0. = ECN-Capable Transport (ECT): 0

.... ...0 = ECN-CE: 0

 

Total Length: 48

Identification: 0x0000 (0)

Flags: 0x00

0... = Reserved bit: Not set

.0.. = Don't fragment: Not set

..0. = More fragments: Not set

Fragment offset: 0

Time to live: 1

Protocol: UDP (0x11)

Header checksum: 0x46db [correct]

[Good: True]

[Bad : False]

 

Source: 172.28.230.3 (172.28.230.3)

Destination: 224.0.0.2 (224.0.0.2)

User Datagram Protocol, Src Port: 1985 (1985), Dst Port: 1985 (1985)

Source port: 1985 (1985)

Destination port: 1985 (1985)

Length: 28

Checksum: 0x8ab9 [correct]

[Good Checksum: True]

[Bad Checksum: False]

 

Cisco Hot Standby Router Protocol

Version: 0

Op Code: Hello (0)

State: Active (16)

Hellotime: Default (3)

Holdtime: Default (10)

Priority: 105

Group: 1

Reserved: 0Authentication Data: Default (cisco)

Virtual IP Address: 172.28.230.1 (172.28.230.1)

 

1 packets captured


3. This example uses a display filter to show only those HSRP packets that have an active HSRP state:

 

switch(config)# ethanalyzer local interface mgmt brief display-filter "hsrp.state==Active" limit-captured-frames 2
Capturing on eth1

2008-02-18 14:35:41.443118 172.28.230.3 -> 224.0.0.2 HSRP Hello (state Active)

2008-02-18 14:35:44.326892 172.28.230.3 -> 224.0.0.2 HSRP Hello (state Active)

2 packets captured

 

4. Other filter examples:

 

ethanalyzer local interface mgmt capture-filter “dst host 172.16.185.1”
ethanalyzer local interface inband capture-filter “stp”
ethanalyzer local interface inband decode-internal capture-filter “stp”
ethanalyzer local interface inband capture-filter “stp” limit-frame-size 64
ethanalyzer local interface inband capture-filter “icmp and host 10.10.10.1” limit-captured-frames 1000 write bootflash:icmp

Comments
Community Member

Quite fresh and informative.

This will help a lot in troubleshooting network and understanding NX-OS. Though its command line is new and not know but this doc will definitely help me setting up capture for Nexus devices...

Ivan, Thanks for creating such doc...

Beginner

I found this document very useful.

Thanks Ivan for creating this.

Very nice write up!

Beginner
How do I turn it off? ethanalyzer local interface inbound-low display-filter ip.addr==IP_OF_SCP_SERVER write boothflash:inbound_low.pcap ethanalyzer local interface inbound-hi display-filter ip.addr==IP_OF_SCP_SERVER write boothflash:inbound_hi.pcap
CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards