Hi,
TBH, I think I'm actually documenting a bug, but I hope someone cam either
- tell me I'm wrong and why, or
- actually do something about fixing the bug
Here's my picture
I have two PCs on the same subnet, one being attached via a VPC to a L2 switch - but the VPC link is down (as in shutdown at both ends).
I want to demonstrate that with the leg of the VPC being unavailable, traffic is then forced to go via the spine, so I set up a Fabric Span with the destination monitor PC attached to Leaf2201, and defined port 1/49 on each leaf to SPAN traffic in both directions for the relevant VPC.
TEST#1
I issue a single ping from the PC on the far right (10.218.12.10) to it's same-subnet neighbour 10.218.12.200 which is only reachable via the VPC orphan port on leaf 2201
WHAT I EXPECTED TO SEE ON THE MONITOR PC
I expected to see on the monitor PC
- An ICMP ping ECHO being SPANned from port 1/49 on leaf 2202 on its way to leaf 2201
- An ICMP ping ECHO being SPANned from port 1/49 on leaf 2201 as it arrived
- An ICMP ping REPLY being SPANned from port 1/49 on leaf 2201 on its way to leaf 2202
- An ICMP ping REPLY being SPANned from port 1/49 on leaf 2202 as it arrived
In other words, I expect to see TWO copies of the ping ECHO and TWO copies of the ping REPLY
WHAT I ACTUALLY SAW ON THE MONITOR PC
The monitor PC showed only packets 2,3 and 4.
THE OUTGOING PACKET FROM 1/49 ON LEAF 2202 was NOT spanned! As you can see below, only ONE copy of the ping ECHO was spanned, but both copied of the ping REPLY.
TEST#2
I also tested pinging in the reverse direction. From 10.218.12.200 (the orphaned PC) to 10.218.12.10.
WHAT I EXPECTED TO SEE ON THE MONITOR PC
I expected to see on the monitor PC
- An ICMP ping ECHO being SPANned from port 1/49 on leaf 2201 on its way to leaf 2202
- An ICMP ping ECHO being SPANned from port 1/49 on leaf 2202 as it arrived
- An ICMP ping REPLY being SPANned from port 1/49 on leaf 2202 on its way to leaf 2201
- An ICMP ping REPLY being SPANned from port 1/49 on leaf 2201 as it arrived
Like last time, I expected to see TWO copies of the ping ECHO and TWO copies of the ping REPLY
WHAT I ACTUALLY SAW ON THE MONITOR PC
The monitor PC showed only packets 1,2 and 4.
AGAIN THE OUTGOING PACKET FROM 1/49 ON LEAF 2202 was NOT spanned! As you can see below, the incoming ICMO ECHO on Leaf2202 was SPANned, but NOT the outgoing REPLY.
My Theory
I believe that when a switch participates in a VPC, it doe NOT SPAN outgoing packets when the VPC is broken. <Edit>I haven't tried in the reverse direction, but </Edit> given that Test#1 and Test#2 above BOTH fail ONLY on leaf 2202, it is something to do with the VPC
<edit>I have now tried the complete reversal, shutting down the other leg of the VPC and forcing traffic between Leaf2201 and 2202 via the Spine. In this variation, it is Leaf2201 that refuses to SPAN forwarded packets, validating my hunch about the VPC. </edit>
FYI - If I reconfigure the topology to remove the VPC and just have a regular Access Port connection, then all four ICMP packets are SPANned as expected.
FYI - I have demonstrated this in the past (earlier versions of ACI) with out a problem.
For more detail, watch this. [Please mentally edit each occurrence of when I say "2201" but mean "2202" - of course, SOMETIMES I actually do mean "2201". You'll work it out.]
Technical details:
apic1# show version
Role Pod Node Name Version
---------- ---------- ---------- ------------------------ --------------------
controller 1 1 apic1 5.2(3g)
spine 1 2101 Spine2101 n9000-15.2(3g)
leaf 1 2201 Leaf2201 n9000-15.2(3g)
leaf 1 2202 Leaf2202 n9000-15.2(3g)
apic1# fabric 2101,2201-2202 show version | grep -A1 Hardware
Hardware
cisco N9K-C93180YC-FX ("supervisor")
--
Hardware
cisco N9K-C93180YC-FX ("supervisor")
--
Hardware
cisco N9K-C9332C ("supervisor")
