03-02-2018 12:05 PM - edited 03-08-2019 02:06 PM
My Nexus 7000 seem to be encapsulating outgoing packets in FabricPath frames while the feature is not enabled. None of my VLANs are set up as "FP" and none of my other Nexus devices are using it either.
Through a sniffer I set up (using a monitor port), I can see the the vast majority of packets are encapsulated and have the ethertype of 0x8903 which seems to indicate FabricPath. It is using this on edge ports as well, to devices that certainly couldn't understand this encapsulation:
That screen capture is from WireShark. This makes no sense to me, and makes packet inspection through sniffing impossible. Any help would be appreciated.
Solved! Go to Solution.
03-04-2018 05:10 AM - edited 03-04-2018 05:25 AM
Hi there,
Just curious - Are the interfaces involved in this packet capture part of an F3-series module? I've seen this behavior before with that particular ASIC where once SPAN'd we see the additional DCE header even in non-FP networks.
We documented this (albeit not clearly) here - Starting in 6.2(6a), the Fabricpath header is preserved unconditionally. In release 6.2(10) and later, the FP header is preserved by default although a knob was introduced to change this behavior. Using the switchport monitor exclude header command to remove the FP or VLAN tag header for a specified SPAN destination in a VDC or the system default switchport monitor exclude header command to remove the FP or VLAN tag header for all destinations ports in the VDC.
The condition is applicable even if Fabricpath is not enabled.
Hope that helps!
03-02-2018 12:58 PM
Hello,
If these are truly FP frames then you need to enable the FabricPath dissector in Wireshark to have them properly dissected.
Go to Analyze -> Enabled Protocols, then write "fabricpath" into the search line, and make sure that both "CFP" and "fp_eth" are selected, then click OK.
Would you actually mind sharing the PCAP file with us here? I'd like to have a look.
Best regards,
Peter
03-02-2018 01:22 PM
03-03-2018 05:48 AM
Hello,
Very interesting indeed - looks like whichever switch is producing those frames, it fills in Switch ID of 0, Sub-Switch ID of 0, and the LID of two ports where very likely those sources are connected - 0x171, and 0x40e. The FTAG is 0 which is unexpected.
Please let me ask you first: How did you first come to this suspicion or observation? I understand that you have configured a SPAN and you are seeing these frames. However, did you start this investigation because on an issue that preceded this finding? In particular, did you have any outage or reachability issue, or perhaps did you run Wireshark directly on an end host (without a SPAN session) and saw those FP frames coming? Can you confirm that you can receive these FP frames on an end host along with normal traffic, without a SPAN session?
The trick is going to be to identify the switch sourcing this frames. I cannot say which is the exact switch that generates them. However, if these frames are received by ordinary switches, they will likely be interpreted as normal Ethernet frames sourced from a specific MAC address 0200.0000.<LID> - so one of the options to trace down the true source of these frames could be to check the MAC address tables of your switches for any sign of these MAC addresses (starting with 0200.0000) and follow up the ports up to the switch that does not contain any of such MAC addresses in their switching tables - chances are this one is the source itself.
Once you believe you are on the switch that seems to be sourcing these FP frames, you can do an additional quick test: Take the LID above, and try the following two commands:
show system internal pixm info ltl 0x171 show system internal pixm info ltl 0x40e
The output of these commands should indicate an interface that is "sensible" in terms of where the actual encapsulated packets are coming from. For example, the OSPF speaker 16.2.16.238 should be truly reachable through the interface identified by the LID/LTL 0x40e; the OSPF speaker 16.11.19.162 should be reachable through the interface with LID/LTL 0x171.
Once we know which is the switch sourcing these frames, we can start digging in more deeply.
Best regards,
Peter
03-05-2018 09:25 AM
I stumbled upon this while using our sniffing setup (switchport monitor to a host with wireshark) to troubleshoot an issue that was eventually determined to be a STP issue. As of right now, I'm not entirely convinced that this is actually causing any problems, but I'm still hesitant to come to that conclusion.
Right now, we have RoCE traffic running from a host on the Nexus 7000, through a Juniper switch, to a Nexus 5000 switch. The traffic is running fine (according to our applications).
When I sniff traffic transmitted from the 7000's uplink port to a Juniper switch (in route to the Nexus 5000) I see packets with the 0x8903 ethertype. However, when I sniff traffic coming into the Nexus 5000 switch's uplink from the Juniper switch, I see no such ethertype. So either the Juniper is stripping off those headers, or the Nexus 5000 is and isn't mirroring that out to my sniffer.
Nexus 7000 --> Juniper --> Nexus 5000
Interestingly, when traffic is flowing in the other direction (Nexus 5000 to Nexus 7000) the RoCE traffic is received on the 7000's uplink without 0x8903, but is being sent out to the host's port with 0x8903.
Seeing as the traffic seems to be running without issue, either the Nexus 7000 is lying to me about what it is sending to the host, or the host can handle those packets. I'll need to check with our developers on that second possibility. Either way, I'd like to know why I'm seeing this from the 7000, this seems to indicate that the 7000 is the culprit.
I've gone ahead and run the commands you mentioned:
Looking at our Nexus 7000 switch's MAC address table, I don't see any MAC addresses starting with 0200.0000, so I did those commands on that switch:
Cisco1# show system internal pixm info ltl 0x171 Member info ------------------ Type LTL --------------------------------- PHY_PORT Eth3/10 FLOOD_W_FPOE 0x8001
That port is connected to a mainframe network card. Only thing special about that card is that it might be used with IPsec.
Cisco1# show system internal pixm info ltl 0x40e PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG PC_NODE_FLAG MEMB_CNT ------------------------------------------------------------------------------ Normal Po2 0x040e 0x16000001 0x00000000 0x00000002 0x00000000 2 RBH Mode: NON-MODULO Member rbh rbh_cnt Eth1/1 0x000000f0 0x04 Eth1/2 0x0000000f 0x04 CBL Check States: Ingress: Enabled; Egress: Enabled VLAN| BD| BD-St | CBL St & Direction: -------------------------------------------------- 999 | 0x8b | INCLUDE_IF_IN_BD | FORWARDING (Both) 1 | 0x1 | INCLUDE_IF_IN_BD | FORWARDING (Both) 2 | 0x19 | INCLUDE_IF_IN_BD | FORWARDING (Both) 4 | 0x1a | INCLUDE_IF_IN_BD | FORWARDING (Both) 5 | 0x1b | INCLUDE_IF_IN_BD | FORWARDING (Both) 10 | 0x1c | INCLUDE_IF_IN_BD | FORWARDING (Both) 15 | 0x1d | INCLUDE_IF_IN_BD | FORWARDING (Both) 20 | 0x1e | INCLUDE_IF_IN_BD | FORWARDING (Both) 22 | 0x1f | INCLUDE_IF_IN_BD | FORWARDING (Both) 24 | 0x20 | INCLUDE_IF_IN_BD | FORWARDING (Both) 25 | 0x21 | INCLUDE_IF_IN_BD | FORWARDING (Both) 26 | 0x22 | INCLUDE_IF_IN_BD | FORWARDING (Both)
**Output excluded**
Member info
------------------
Type LTL
---------------------------------
PORT_CHANNEL Po2
FLOOD_W_FPOE 0x8001
FLOOD_W_FPOE 0x8019
FLOOD_W_FPOE 0x801a
FLOOD_W_FPOE 0x801b
FLOOD_W_FPOE 0x801c
FLOOD_W_FPOE 0x801d
FLOOD_W_FPOE 0x801e
FLOOD_W_FPOE 0x801f
FLOOD_W_FPOE 0x8020
** Output Excluded **
This is the uplink (mentioned previously) to our distribution layer switch (the juniper). It definately leads to other OSPF speakers. Let me know if you need more of that output, it seems to continue for all of the VLANs in use on the switch.
Let me know if anything I wrote was confusing.
03-05-2018 10:34 AM
Nevermind, it seems it was a SPAN issue. Thanks for the help Peter.
03-05-2018 10:04 AM
Sorry, my earlier responses (and creating this thread) were from an old account that I've stopped using. Sorry if that introduces any confusion.
03-04-2018 05:10 AM - edited 03-04-2018 05:25 AM
Hi there,
Just curious - Are the interfaces involved in this packet capture part of an F3-series module? I've seen this behavior before with that particular ASIC where once SPAN'd we see the additional DCE header even in non-FP networks.
We documented this (albeit not clearly) here - Starting in 6.2(6a), the Fabricpath header is preserved unconditionally. In release 6.2(10) and later, the FP header is preserved by default although a knob was introduced to change this behavior. Using the switchport monitor exclude header command to remove the FP or VLAN tag header for a specified SPAN destination in a VDC or the system default switchport monitor exclude header command to remove the FP or VLAN tag header for all destinations ports in the VDC.
The condition is applicable even if Fabricpath is not enabled.
Hope that helps!
03-05-2018 10:03 AM
Yea, the modules I have been sniffing from are F3-series.
Is this saying that Fabricpath headers are added even if Fabricpath isn't enabled on the switch? Because that seems to be more of what I'm seeing here.
03-05-2018 10:06 AM
Hi there,
Yep! That is exactly correct - The DCE (FP) header will be present on F3-series even when FP isn't enabled ONLY with regards to SPAN sessions.
Hope that helps!
03-05-2018 10:33 AM
Right, that seems to have solved it. I'm happy it is a SPAN issue rather than something larger with the switch.
However, I don't seem to be able to see the 802.1Q tag now (or even before when the header was added on). This is an important bit of troubleshooting information...
03-05-2018 10:35 AM
Unfortunately the command will strip both the FP and the VLAN header off - Moreover, even if you hadn't disabled it, its possible the NIC on the other side strips the dot1q header on ingress - We run into that quite often unfortunately when troubleshooting.
03-05-2018 10:47 AM
Ah, well that's quite unfortunate, thankfully we have another sniffing setup we can use that doesn't rely on SPAN, but it isn't ideal.
Either way, thanks for your help in this issue.
03-05-2018 10:23 AM
Ah, that's likely what we're seeing here then, though, you said it strips off Fabricpath AND VLAN tag headers? Is there any way to keep the VLAN tag headers? That's an important piece of information I'd like to keep in my traces. Either way I'll try out that command to see if that solves this issue.
04-14-2020 04:15 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide