Troubleshooting with a NETDR capture on a sup720/6500

Adam Casella · ‎03-09-2011

Troubleshooting with a NETDR capture on a sup720/6500

A netdr capture is preformed on the MSFC CPU controller. This is the closest location you can capture a packet on the MSFC in order to determine why traffic is being punted to the SP OR RP CPU on the MSFC. With a Sup720 or Sup32 it allows one to capture packets on the RP or SP inband. The netdr command can be used to capture both Tx and Rx packets in the software-switching path.

Cat6500#debug netdr capture ?

acl (11) Capture packets matching an acl

and-filter (3) Apply filters in an and function: all must match

continuous (1) Capture packets continuously: cyclic overwrite

destination-ip-address (10) Capture all packets matching ip dst address

dstindex (7) Capture all packets matching destination index

ethertype (8) Capture all packets matching ethertype

interface (4) Capture packets related to this interface

or-filter (3) Apply filters in an or function: only one must match

rx (2) Capture incoming packets only

source-ip-address (9) Capture all packets matching ip src address

srcindex (6) Capture all packets matching source index

tx (2) Capture outgoing packets only

vlan (5) Capture packets matching this vlan number

<cr>

OPTIONS:

· Using the continuous option, the switch will capture packets on the RP-inband continuously fill the entire capture buffer (4096 packets) and then start to overwrite the buffer in a FIFO fashion.

· The tx and rx options will capture packets coming from the MSFC and going to the MSFC respectively.

· The and-filter and the or-filter specify that an and or an or will be applied respectively to all of the options that follow. For example, if you use the syntax below, then both option #1 and option #2 must match for the packet to be captured. Similarly, if the or-filter is used either option #1 or option #2 or both must match for the packet to be captuered.

o debug netdr and-filter option#1 option#2

· The interface option is used to capture packets to or from the specified interface. The interface can be either an SVI or a L3 interface on the switch.

· The vlan option is used to capture all packets in the specified VLAN. The VLAN specified can also be one of the internal VLANs associated with a L3 interface.

· The srcindex and dstindex options are used to capture all packets matching the source ltl and destination ltl indices respectively. Note that the interface option above only allows the capture of packets to or from a L3 interface (SVI or physical). Using the srcindex or dstindex options allows the capture of Tx or Rx packets on a given L2 interface. The srcindex and dstindex options work with either L2 or L3 interface indices.

· The ethertype option allows the capture of all packets matching the specified ethertype.

· The source-ip-address and destination-ip-address options allow the capture of all packets matching the specified source or destination IP address respectively.

Example of Netdr and Interpreting the data:

Below is an example of capturing traffic destined to 10.100.101.10 sourced from 10.10.10.2 going to the RP CPU:

6500-2#debug netdr cap rx and-filter source-ip-address 10.10.10.2 destination-ip-address 10.100.101.10

6500-2#sh netdr cap

A total of 4096 packets have been captured

The capture buffer wrapped 0 times

Total capture capacity: 4096 packets

------- dump of incoming inband packet -------

interface Vl10, routine mistral_process_rx_packet_inlin, timestamp 00:00:11

dbus info: src_vlan 0xA(10), src_indx 0xC0(192), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

10020400 000A0000 00C00000 40080000 00060468 0E000040 00000000 03800000

mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xA(10)

destmac 00.15.C7.26.FB.80, srcmac 00.00.01.00.06.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 46, identifier 0

df 0, mf 0, fo 0, ttl 100, src 10.10.10.2, dst 10.100.101.10

tcp src 0, dst 0, seq 0, ack 0, win 0 off 5 checksum 0x265C

Red = Ingress Vlan of traffic

Blue = Layer 3 interface traffic is coming from

Green = Ethertype and SRC/DST MAC addresses

Purple = IP Header

Orange = SRC index (source of ingress traffic).

Dark Red = Dest Index (where traffic is being sent).

Aqua = Flood bit

You can track down the physical port where this traffic is sourced from by tracking down the mac-address (which is the last device to own this traffic). This can be done in the following manner:

6500-2#show mac-address address 0000.0100.0600

Legend: * - primary entry

age - seconds since last seen

n/a - not available

vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------

Active Supervisor:

* 10 0000.0100.0600 dynamic Yes 0 Gi4/1

You can also look at ingress location of this traffic by taking the source index/ltl index.

The LTL (local target logic) is used by the sup720 forwarding engine in order to distinguish locations to forward a packet. These indexes could be pointed towards a specific port or to an internal location on the switch. In the example below you will use this index to determine ingress port of the traffic above.

Below is an example of how to determine the source of a packet based on the source LTL index of 0xC0 that is seen in the capture above.

6500-2#remote command switch test mcast ltl index c0

index 0xC0 contain ports 4/1

We can see this index is used for port 4/1. We now know this traffic is coming from port 4/1, which was also confirmed based on the port where the MAC is learned.

Determining the source of this traffic via LTL can be very useful when determining the source of the traffic during bridging loop, if multiple types of traffic are being flooded/sent to the CPU, but sent from the same port. You can analyze the netdr information and look to see if it is coming from the same source index.

You can also determine where the traffic is being sent looking at the destination index. In this case we can see that the traffic is being sent to the destination index of 0x380, which is the RP CPU:

6500-2#rem com sw test mcast ltl ind 380

index 0x380 contain ports 2/R

2/R = module/Router

6500-2#show module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

2 2 Supervisor Engine 720 (Active) WS-SUP720-BASE SAD074205FP

4 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1107GS80

In the above case we can see that module 2 is the supervisor with R indicating this is the RP CPU.

If the packet that is captured has the flood bit set in the netdr capture, this traffic is being flooded to all ports within the vlan. In order to see what ports are included within this index you will need to change the destination index to seen in the netdr to account for this bit being set. The change that will be performed is to add 8 to the beginning of the destination index (which means the first bit in the LTL is set). The following is an example of an ARP request broadcast frame, which is flooded to the CPU and to all ports in a vlan. I will also show how you can interpret this data.

6500-2#sh netdr cap
A total of 1 packets have been captured
The capture buffer wrapped 0 times
Total capture capacity: 4096 packets

------- dump of incoming inband packet -------
interface Vl10, routine mistral_process_rx_packet_inlin, timestamp 00:00:03
dbus info: src_vlan 0xA(10), src_indx 0xC0(192), len 0x40(64)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400A(16394)
38020400 000A0000 00C00000 40080000 E0000060 8E0FFBFC 00000008 400A0000
mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0xA(10)
destmac FF.FF.FF.FF.FF.FF, srcmac 00.0C.29.21.DE.6E, protocol 0806
layer 3 data: 00010800 06040001 000C2921 DE6EAC10 3E800000 00000000
AC103E02 FFFF0000 00000000 00000000 00000000 00000000
00000000 00000000 0000FFFF

In the above example the destination index is set to 0x400A and the flood bit is set to 1, both are sent to red. In this case in order to see where this packet would be sent we need to add 8 to first value in the index. For example, the index 0x400A would be changed 0xC00A, which signifies that the flood bit is set when looking up this index. Also, If an index of 0xC12 would become 0x8C12, when the index is only 3 Hex values.

With the two cases above this packet would be flooded in vlan 10 (A = 10) and vlan 3090 (c12= 3090). The rest of the Index after the flood bit is a reprsentation of the Vlan.

We can look at this information from the example netdr above with the following command:

6500-2#remote command switch test mcast ltl index C00A

index 0xC00A contain ports 2/R,, 4/1

If we do not adjust this index when the flood bit is set the index will be listed as “empty”, which can be seen below:

6500-2#remote command switch test mcast ltl index 400A

index 0x400A contain ports * empty *

If you need still assistance with interpreting this information, please open Cisco TAC case.