ACI Spine and Leaf connectivity via Netscout Passive TAPs (and Nexus Data broker as traffic receiver)

[kishen32]

Hi all, looking for advice on an issue that we facing for traffic visibility by connecting spine and leaf via 40G BiDi taps.

This is the brief description of the issue.

We are connecting ACI spine and leaf via NetScout tap as in attached picture.

Spine and leaf connection up when the cables are connected to the network slots of the taps.

Once the monitor slots are cabled to Nexus data broker, we expect traffic to be sent to the data broker.

Once the link is enabled manually in the data broker ports (which are currently shut), we see it’s not coming up and also the links at spine and leaf goes down.

If I shut the data broker ports, the spine and leaf ports comes up. I am not sure about this behavior.

As a testing, we removed NetScout tap and plug a leaf link directly to the data broker ports, they are up.

Its just via the NetScout tap the links are not coming up at the data broker. We already installed NDB inside the data broker chassis and through GUI, I see everything seems to be fine.

I am not sure what is the issue from switching side via the Data Broker, hence need some guidance. We can't find a working sample for this anywhere, however a few points to state as below.

1. Spine and leaf has not been discovered or registered yet via the APIC contollers, we just testing if all cabling is good at this point.

2. Spine and leaf are on ACI OS, where spine is 9504 model while leaf are 93180FX.

3. Data broker is using NXOS with embedded NDB installed. All the basic configurations on data broker is done (spanning-tree bpdufilter enable on ports, no spanning tree on all vlans, spanning tree as mst and ports blocking multicast and unicast traffic with operational mode tap-aggregation)

[Sergiu.Daniluk]

Hi @kishen32 

To me, it sounds something related to autonegotiation. have you tried configuring speed manually to 40G on NDB N9K?

Apart of the problem, why would you want to add the tap inline between spine-leaf? Note that the traffic inside the fabric is iVXLAN encapsulated. Also if you have for example Intersite L3out there are options to enable CloudSec between VTEPs, so your monitoring will not work.

I think it would make more sense to connect your NDB directly to Leaf front ports and span all desired traffic to it. There is even a integration ACI - NDB: https://www.cisco.com/c/en/us/td/docs/net_mgmt/xnc/nexus_data_broker/deploy_config/3-x/b_Nexus_Data_Broker_Configuration_Guide_38/b_Nexus_Data_Broker_Configuration_Guide_38_chapter_0111.html

Stay safe,

Sergiu

[Robert Burns]

I would recommend Access SPAN instead of Taps. Reason is this gives best visibility and allows you to capture traffic that is switched within a leaf as well as that goes across the fabric.  What exactly are you trying to accomplish with the TAP that can't be done with SPAN?

Robert

[kishen32]

Thanks for the reply guys. The idea of the taps is due to security requirement that needs to have visibility of east-west traffic on all leaf uplinks, hence the need for this arise. I do agree, more visibility is attained by a span directly from leaf to the NDB. Anyway, on closer inspection, i think we might have an qsfp compatibility issue. What we receive from Cisco is Cisco QSFP-40/100-SR BD while on closer look, i think for NDB it requires the QSFP-40G-BD-RXQ SFP, which only receives traffic and will not transmit back any traffic to the taps. I will look for spare RX QSFP to test and validate my findings. Keep you guys posted.