cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2890
Views
30
Helpful
14
Replies

Nexus 7000 using Data Center Ethernet with Fabricpath not enabled

kjcraigkl
Level 1
Level 1

My Nexus 7000 seem to be encapsulating outgoing packets in FabricPath frames while the feature is not enabled. None of my VLANs are set up as "FP" and none of my other Nexus devices are using it either. 

 

Through a sniffer I set up (using a monitor port), I can see the the vast majority of packets are encapsulated and have the ethertype of 0x8903 which seems to indicate FabricPath. It is using this on edge ports as well, to devices that certainly couldn't understand this encapsulation:

 

DCE Frames.JPGThat screen capture is from WireShark. This makes no sense to me, and makes packet inspection through sniffing impossible. Any help would be appreciated.

1 Accepted Solution

Accepted Solutions

Andrea Testino
Cisco Employee
Cisco Employee

Hi there,

 

Just curious - Are the interfaces involved in this packet capture part of an F3-series module? I've seen this behavior before with that particular ASIC where once SPAN'd we see the additional DCE header even in non-FP networks.

 

We documented this (albeit not clearlyhere - Starting in 6.2(6a), the Fabricpath header is preserved unconditionally. In release 6.2(10) and later, the FP header is preserved by default although a knob was introduced to change this behavior. Using the switchport monitor exclude header command to remove the FP or VLAN tag header for a specified SPAN destination in a VDC or the system default switchport monitor exclude header command to remove the FP or VLAN tag header for all destinations ports in the VDC.

 

The condition is applicable even if Fabricpath is not enabled. 

 

Hope that helps!

 

 

- Andrea, CCIE #56739 R&S

View solution in original post

14 Replies 14

Peter Paluch
Cisco Employee
Cisco Employee

Hello,

If these are truly FP frames then you need to enable the FabricPath dissector in Wireshark to have them properly dissected.

Go to Analyze -> Enabled Protocols, then write "fabricpath" into the search line, and make sure that both "CFP" and "fp_eth" are selected, then click OK.

Would you actually mind sharing the PCAP file with us here? I'd like to have a look.

Best regards,
Peter

Yea, so once I enabled "Cisco FabricPath over Ethernet" option in that menu, I can see the inside of those packets now. Thanks for that information.

 

 

I've attached part of the pcap file.

 

 

 

Hello,

Very interesting indeed - looks like whichever switch is producing those frames, it fills in Switch ID of 0, Sub-Switch ID of 0, and the LID of two ports where very likely those sources are connected - 0x171, and 0x40e. The FTAG is 0 which is unexpected.

Please let me ask you first: How did you first come to this suspicion or observation? I understand that you have configured a SPAN and you are seeing these frames. However, did you start this investigation because on an issue that preceded this finding? In particular, did you have any outage or reachability issue, or perhaps did you run Wireshark directly on an end host (without a SPAN session) and saw those FP frames coming? Can you confirm that you can receive these FP frames on an end host along with normal traffic, without a SPAN session?

The trick is going to be to identify the switch sourcing this frames. I cannot say which is the exact switch that generates them. However, if these frames are received by ordinary switches, they will likely be interpreted as normal Ethernet frames sourced from a specific MAC address 0200.0000.<LID> - so one of the options to trace down the true source of these frames could be to check the MAC address tables of your switches for any sign of these MAC addresses (starting with 0200.0000) and follow up the ports up to the switch that does not contain any of such MAC addresses in their switching tables - chances are this one is the source itself.

Once you believe you are on the switch that seems to be sourcing these FP frames, you can do an additional quick test: Take the LID above, and try the following two commands:

 

show system internal pixm info ltl 0x171
show system internal pixm info ltl 0x40e

The output of these commands should indicate an interface that is "sensible" in terms of where the actual encapsulated packets are coming from. For example, the OSPF speaker 16.2.16.238 should be truly reachable through the interface identified by the LID/LTL 0x40e; the OSPF speaker 16.11.19.162 should be reachable through the interface with LID/LTL 0x171.

Once we know which is the switch sourcing these frames, we can start digging in more deeply.

Best regards,
Peter

I stumbled upon this while using our sniffing setup (switchport monitor to a host with wireshark) to troubleshoot an issue that was eventually determined to be a STP issue. As of right now, I'm not entirely convinced that this is actually causing any problems, but I'm still hesitant to come to that conclusion.

 

Right now, we have RoCE traffic running from a host on the Nexus 7000, through a Juniper switch, to a Nexus 5000 switch. The traffic is running fine (according to our applications).

 

When I sniff traffic transmitted from the 7000's uplink port to a Juniper switch (in route to the Nexus 5000) I see packets with the 0x8903 ethertype. However, when I sniff traffic coming into the Nexus 5000 switch's uplink from the Juniper switch, I see no such ethertype. So either the Juniper is stripping off those headers, or the Nexus 5000 is and isn't mirroring that out to my sniffer.

 

Nexus 7000 --> Juniper --> Nexus 5000

 

Interestingly, when traffic is flowing in the other direction (Nexus 5000 to Nexus 7000) the RoCE traffic is received on the 7000's uplink without 0x8903, but is being sent out to the host's port with 0x8903.

 

Seeing as the traffic seems to be running without issue, either the Nexus 7000 is lying to me about what it is sending to the host, or the host can handle those packets. I'll need to check with our developers on that second possibility. Either way, I'd like to know why I'm seeing this from the 7000, this seems to indicate that the 7000 is the culprit. 

 

I've gone ahead and run the commands you mentioned:

 

Looking at our Nexus 7000 switch's MAC address table, I don't see any MAC addresses starting with 0200.0000, so I did those commands on that switch:

 

Cisco1# show system internal pixm info ltl 0x171

Member info
------------------
Type            LTL
---------------------------------
PHY_PORT       Eth3/10         
FLOOD_W_FPOE   0x8001

 

 

That port is connected to a mainframe network card. Only thing special about that card is that it might be used with IPsec.

 

 

Cisco1# show system internal pixm info ltl 0x40e

PC_TYPE    PORT    LTL      RES_ID       LTL_FLAG      CB_FLAG    PC_NODE_FLAG   MEMB_CNT 
------------------------------------------------------------------------------
Normal    Po2   0x040e    0x16000001   0x00000000   0x00000002   0x00000000   2 
RBH Mode: NON-MODULO
Member rbh rbh_cnt
Eth1/1   0x000000f0   0x04
Eth1/2   0x0000000f   0x04

CBL Check States: Ingress: Enabled; Egress: Enabled

VLAN| BD| BD-St              | CBL St & Direction:
--------------------------------------------------
999 | 0x8b | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
1 | 0x1 | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
2 | 0x19 | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
4 | 0x1a | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
5 | 0x1b | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
10 | 0x1c | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
15 | 0x1d | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
20 | 0x1e | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
22 | 0x1f | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
24 | 0x20 | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
25 | 0x21 | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
26 | 0x22 | INCLUDE_IF_IN_BD  | FORWARDING (Both) 
**Output excluded**

Member info
------------------
Type LTL
---------------------------------
PORT_CHANNEL Po2
FLOOD_W_FPOE 0x8001
FLOOD_W_FPOE 0x8019
FLOOD_W_FPOE 0x801a
FLOOD_W_FPOE 0x801b
FLOOD_W_FPOE 0x801c
FLOOD_W_FPOE 0x801d
FLOOD_W_FPOE 0x801e
FLOOD_W_FPOE 0x801f
FLOOD_W_FPOE 0x8020
** Output Excluded **

This is the uplink (mentioned previously) to our distribution layer switch (the juniper). It definately leads to other OSPF speakers. Let me know if you need more of that output, it seems to continue for all of the VLANs in use on the switch.

 

Let me know if anything I wrote was confusing.

Nevermind, it seems it was a SPAN issue. Thanks for the help Peter.

Sorry, my earlier responses (and creating this thread) were from an old account that I've stopped using. Sorry if that introduces any confusion.

Andrea Testino
Cisco Employee
Cisco Employee

Hi there,

 

Just curious - Are the interfaces involved in this packet capture part of an F3-series module? I've seen this behavior before with that particular ASIC where once SPAN'd we see the additional DCE header even in non-FP networks.

 

We documented this (albeit not clearlyhere - Starting in 6.2(6a), the Fabricpath header is preserved unconditionally. In release 6.2(10) and later, the FP header is preserved by default although a knob was introduced to change this behavior. Using the switchport monitor exclude header command to remove the FP or VLAN tag header for a specified SPAN destination in a VDC or the system default switchport monitor exclude header command to remove the FP or VLAN tag header for all destinations ports in the VDC.

 

The condition is applicable even if Fabricpath is not enabled. 

 

Hope that helps!

 

 

- Andrea, CCIE #56739 R&S

Yea, the modules I have been sniffing from are F3-series.

 

Is this saying that Fabricpath headers are added even if Fabricpath isn't enabled on the switch? Because that seems to be more of what I'm seeing here.

Hi there,

 

Yep! That is exactly correct - The DCE (FP) header will be present on F3-series even when FP isn't enabled ONLY with regards to SPAN sessions.

 

Hope that helps!

- Andrea, CCIE #56739 R&S

Right, that seems to have solved it. I'm happy it is a SPAN issue rather than something larger with the switch.

 

However, I don't seem to be able to see the 802.1Q tag now (or even before when the header was added on). This is an important bit of troubleshooting information...

Unfortunately the command will strip both the FP and the VLAN header off - Moreover, even if you hadn't disabled it, its possible the NIC on the other side strips the dot1q header on ingress - We run into that quite often unfortunately when troubleshooting.

- Andrea, CCIE #56739 R&S

Ah, well that's quite unfortunate, thankfully we have another sniffing setup we can use that doesn't rely on SPAN, but it isn't ideal.

 

Either way, thanks for your help in this issue.

Ah, that's likely what we're seeing here then, though, you said it strips off Fabricpath AND VLAN tag headers? Is there any way to keep the VLAN tag headers? That's an important piece of information I'd like to keep in my traces. Either way I'll try out that command to see if that solves this issue.

Can't find this command on a Nexus 5000. Is there a way to get rid of the FP header on a monitor port?
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco