ACI SPAN configuration detailed explanation needed

[andrii.oliinyk]

Hi Folks

i'm quite new to ACI & need quick support on better understanding of SPAN feature on fabric leaf.

i've read about configuration under Tenant>Policy>Troubleshooting>SPAN but not quite clear about destination group where we have to point point source IP.

I understand source IP as IP on leaf from where leaf will send captured data to the destination IP (wireshark or whatever else ERSPAN capable collector). m i correct? what r the applicable options with source IP then? & does ACI really leverage ERSPAN mechanism for this transfer?

thanks in advance

[RedNectar · Accepted solution]

Hi @andrii.oliinyk ,

Stolen from a lab guide used by Housley Communications (but written by me) - links are in my bio

It assumes your tenant is called TenantX, your tenant has an Application Profile called 2Tier_AP and two EPGs - WebServers_EPG and AppServers_EPG.

AppServerBM is one of the AppServers and has an IP of 10.111.11.10 and will be the destination for your SPAN traffic - you will run Wireshark on this PC

Setup Tenant SPAN

• In your tenant, navigate to TenantX > Polices > Troubleshooting > SPAN > SPAN Destination Groups >+ Create SPAN Destination Group

Name:                           AppServerBM_SPAN.DstGrp
Destination EPG:          TenantX       2Tier_AP         AppServers_EPG
SPAN Version:               Version 2
Destination IP:              10.111.11.10
                                      Your AppServerBM
Source IP/prefix:           1.0.0.0/16

Note: 

• The Source IP/prefix is used to identify the source device creating the ERSPAN packets, therefore it is a good idea to use an uncommon network ID – like 1.0.0.0. 
  
  By specifying a prefix rather than a single IP address, ACI will fill add the node ID to the prefix given so you can distinguish which node is sending the packets (especially handy when capturing from multiple nodes simultaneously). So, in the configuration above, if the switch has a node ID of 1201 the source IP of the ERSPAN packets will be set to 1.0.4.177 (because 1201=4*256+177) . For leaf 2201, this will 1.0.8.153.

• Create a SPAN Source Group. In your tenant, navigate to Polices > Troubleshooting > SPAN > SPAN Source Groups >+ Create SPAN Source Group

Name:                           EPGsToApp_SPAN.SrcGrp
Destination Group:       AppServerBM_SPAN.DstGrp

Click [+]  adjacent Create Sources
Name:                           WebServers_SPAN.Src
Direction:                      Both
Source EPG:                  WebServers_EPG

Click Submit

• Connect to the console of your AppServerBM

A problem you are going to have is that when the GRE (ERSPAN) packets start arriving, your Linux monitoring PC is going to send ICMP Type 3 Code 2 (Destination Protocol Unreachable) packets back to either 1.0.4.177 or 1.0.8.153 because your host has no process listening for protocol 47.

You will use two different strategies to overcome this:

1. when capturing traffic, capture only IP protocol 47
2. reject traffic directed to the 1.0.0.0/8 network from your PC by adding a reject route to the routing table.  This strategy stops your spurious ICMP Destination Protocol Unreachables ruining your capture.

• Open a terminal (command) window on your monitoring Linux PC - AppServerBM
• Use the Linux sudo route add -net 1.0.0.0/8 reject command as shown below to add a reject route for packets destined to the 1.0.0.0 network, then check your entry using the route command.

user@appserverbm:~$ sudo route add -net 1.0.0.0/8 reject

• From to the console of your AppServerBM host, navigate to the start button and select Internet > Wireshark

• The order of this step is important.
  1. First click on the eth0 interface,
  2. then enter a capture filter of ip proto 47 and src net 1.0.0.0/16 and start capture on Wireshark by double-clicking the eth0 interface.

That should be enough to get you started.  However, there are some Wireshark tweaks you might want to know about.

Appendix: How to make Wireshark interpret UDP port 48879 as VXLAN 

If you notice that you have captured a collection of UDP packets with destination port 48879 then this indicates that have collected packets with iVXLAN encapsulation, and Wireshark must be directed how to interpret these packets.

• Select one of the UDP packets in the output and expand the UDP header to expose the destination port of 48879.  Right-click on a that UDP destination port and select Decode As…
• Select UDP port 48879 and configure Wireshark to interpret this as VXLAN and click OK

Your Wireshark output should now reveal the iVXLAN headers and the encapsulated IP packets.

Fixup task for 1st Gen ACI Switches

If you notice that Wireshark does not decode the encapsulated payload, complete the following steps.

• In Wireshark, navigate to Edit > Preferences.
• In Wireshark Preferences, expand Protocols and locate ERSPAN
• You can type-to-locate the protocol
• Toggle the option to FORCE to decode fake ERSPAN frame and click OK.

You should now be able to interpret your Wireshark capture.

[andrii.oliinyk]

thanks for clear navigation. just couple of Qs:

what r cons & pros of configuring SPAN under Tenant vs doing this under the Fabric (Access policies)?

we can apply different filters on the SPAN Source Group & under SPAN Source. does filter applied under SPAN Source overrides filter applied in SPAN Source Group?

[RedNectar]

Hi @andrii.oliinyk ,

There are three places you can configure SPAN,

1. Tenant SPAN: In your Tenant - as described above, where you can capture traffic within or between EPGs
2. Fabric SPAN: Under Fabric > Fabric Policies.
   • This SPAN enables to capture traffic travelling between the Leaves and the Spines - great for looking at how iVXLAN encapsulation works
3. Access SPAN: Under Fabric > Access Polices OR under Operations 
   • This SPAN is ideal for tracing traffic between two endpoints (via Operations), and easiest to setup (in fact this MAY be a better option than described above for your situation, but you asked about Tenant SPAN, so I described that)
   • The beauty of setting up SPAN under Operations is that you don't need a dedicated capture PC. Just create a session for the type of SPAN (eg Endpoint to Endpoint), set the targets (they have to have been discovered, so must exist in the station table) and hit Submit.  This reveals the impossibly wide Visibility & Troubleshooting page
   • On this page, click the SPAN option, select APIC as the ERSPAN destination and click the Play ► icon

Sidenote: When you do this, if open another tab in your browser and navigate to Fabric > Access Policies > Policies > Troubleshooting > SPAN, you will see that ACI has created temporary Source, Target and Filters for you

• When you are done, click the Stop ◼︎ icon and (this is the coolest part) you can then click on the pcap file to download it to your local PC, where you can open it in Wireshark

You will see a lot of duplicated packets, (oops - I should have specified 1.0.0.0/16, not 1.0.0.0 as the source IP Prefix so I could filter my Wireshark to show just packets captured for ONE of the leaves...) but that can be the case in all SPANs.

[andrii.oliinyk]

Thanks man for best details ever. Could u pls shad on "we can apply different filters on the SPAN Source Group & under SPAN Source. does filter applied under SPAN Source overrides filter applied in SPAN Source Group" Q?

i have such a configuration implemented by currently unreachable person under Fabric>AccessPolicy>T/S>SPAN

[Doug Byrd]

Wow, thanks! I did not know the visibility and tshoot tool allowed you to download a pcap!  I got a new wrinkle in my brain