Local SPAN on Nexus 5548 + FEX

[Kevin Dorrell]

I am having some problems with local SPAN on Nexus 5548.  I have a pair of Nexus 5548 running 5.2(1)N1(5) , mostly used in a vPC configuration with several FEX chassis.  The FEXes are dual-homed, so the vPC is extended down to the FEX.

I am trying to use the local SPAN feature to monitor some traffic.  My SPAN destination monitor switchport is on the central chassis (as required) and is connected to a port on a Solaris workstation running snoop.  When the monitor source port is on the 5548 chassis, everything works as it should, and I get a nice pcap file of captured traffic.  I notice that the interface TX counters on the SPAN destination port are counting up in time to the traffic I am capturing.

My trouble starts when I try to monitor a FEX port.  What is really strange is that the TX counters in the SPAN destination port are still counting up in time to the traffic that I should be capturing, but my capture machine does not see any packets.  If I shut the monitor, the TX counters stop counting, and if I no shut it they resume counting.  If I source from a very quiet FEX port and I ping the host on that port, I can even see the TX counters on the SPAN destination counting each ping (x2).  But apparently no packets come out of the monitor port.

Does anyone know of any bug or restriction relating to this.  I have tried a search, but came up with nothing.

Thanks in advance

Kevin Dorrell

Luxembourg

[Kevin Dorrell]

Bump!

Does nobody have any experience with SPAN from a FEX port?  If not, I guess I have to open a TAC case.

Kevin Dorrell

Luxembourg

[Kevin Dorrell]

Since nobody here seems to know, I think I had better write down what I have found out, and see if anyone can explain it.

I found that if I use a PC with Wireshark to capture the SPAN traffic, then it works, but if I use Solaris with snoop then I see nothing.  You might say that indicates my Solaris with snoop is broken.  But it is not.  If I take the SPAN destination port out of monitor mode and put it on a VLAN, then the Solaris snoop captures the background traffic of the VLAN quite happily.  So there is something about SPAN traffic that snoop does not like, but I cannot see it. The packets I captured on the Wireshark look perfectly normal ... no unusual encapsulations or anything like that.  So I am at loss to explain it.

One interesting point that came up during the discussion is that if you have a dual-homed FEX, and you set up a local SPAN from a FEX port with destination on one of your central chassis (as is required), then maybe you only see half of the traffic on the FEX port.  Maybe you only see that traffic that would pass through the chassis where you are monitoring.  That is, the monitored traffic is not re-converged via the vPC link to reach the destination port.  Can anyone confirm or deny that?

Come on experts ... one of you must have done this in the past. ;-)

Kevin Dorrell

Luxembourg

[timchampion · Accepted solution]

Kevin I was facing the same issue and found this:

http://cciedatacentre.blogspot.co.uk/2013/01/faq-port-mirroring-span-on-nexus-2000.html

Seems kind of obvious now. I guess the solution is for your traffic analyser to have a connection to each 5K and to configure an LSPAN session on each 5k. My only concern is that packets may appear out-of-order in the capture due to differences in the replication latency of each 5k.

[Kevin Dorrell]

Thanks for the response Tim, and I'm sorry I took so long to acknowledge it.

It looks like LSPAN is the way to go, but I think Nexus 5500 cannot act as an LSPAN receiver, only as an LSPAN source.  Maybe this has changed in a later release.  I shall be getting Nexus 7700 soon, then those problems will be over (and no doubt be replaced by some different challenges!)

Thanks again.

Kevin Dorrell
Luxembourgv