ACI ARP deluge!

[RedNectar]

ACI has long been touted as a system where ARP requests are minimised, but I did some testing recently when exploring the ARP Gleaning feature (see BRKACI-3101 if you don't know what ARP Gleaning is) and found that in the case of ARP requests for non-existent IPs, ACI multiplies the number of ARPs sent by up to a factor of 4, plus add additional ARP carrying VLAN tags that could leak onto customer networks and potentially cause havoc.

Here is my setup:

I have four servers (Web Servers) in the same Bridge Domain and the same EPG.  ARP flooding is Disabled on the BD, and the Subnet IP of the BD of 192.168.92.1 serves as the default gateway for each host.

1. Web Server VM1 is a VM on an ESXi host attached to Leaf101 (192.168.92.11)
2. Web Server VM2 is a VM on an ESXi host attached to Leaf102 (192.168.92.12)
3. Web Server BM is a Bare Metal Host directly attached to Leaf102 (192.168.92.10)
• Scenario #1: Attached as Access (802.1P)
• Scenario #2: Attached as Access (Untagged)
• Scenario #3: Attached as Trunk
4. Web Server L2EP is a host attached to a switch that connects via a VPC to both Leaf101 and Leaf102 (192.168.92.200)

• ACI is running version 3.2(1m)
• Leaf switches are N9K-C9396PX

A picture always helps.ARP Gleaning Test Setup [Edited thanks to KELLEYD]

Now, as I said, this was a little experiment to demonstrate how ARP Gleaning worked, so I sent a single ping from each of the four workstations above to a non-existent IP (192.168.92.99).  In each case, this single ping generated 3 ARP requests from the originating host (Lubuntu) and I ran Wireshark captures on all four workstations.  I've included my raw results below.

Findings were exactly the same for all three scenarios listed above (where the Bare Metal Host was attached first as Access (802.1P), then as Access (Untagged) then as Trunk)

• When an ARP request for an unknown IP address is sent by a workstation, ACI generates ARP requests for that unknown IP sourced from the ACI gateway address (192.168.92.1)
1. For hosts attached via VPC, 4 ARP requests are received for every ARP request sent. Multiplication factor=4
2. For directly attached hosts and VMs, one ARP request is received from the Gateway IP for every ARP request sent.
3. For directly attached hosts or switches, an additional ARP request is received from the original host but encapsulated with a VLAN tag that represents the Canonical VLAN of the port where the host/switch is connected.
• This result is very disturbing. It means that if say a port was configured for VLAN 2092, but the canonical VLAN for that port was 45, then the device/switch attached to that port would receive an ARP broadcast not just on VLAN 2092, but on VLAN 45 as well.  If an attached switch already had a VLAN 45 representing some other group of servers, then traffic would be leaking from VLAN 2092 to VLAN 45, and if that VLAN 45 happened to have a device matching the unknown IP address, it could be subjected to a deluge of ARP requests.
• Based on this result, I suspect that ARP requests encapsulated with an 802.1Q tag matching the canonical VLAN are also sent on to the ESXi hosts and the VPCs, but I did not have time to investigate this.

These results beg a few questions:

• Why does ACI send ANY frames encapsulated in the canonical VLAN outside the fabric?
• How do I stop ACI from sending the potentially harmful ARP frames – my results prove that turning off ARP flooding on the Bridge Domain is not sufficient
• Why does ACI send four ARP Gleaning frames on a VPC link for every ARP request sent by a host?
• Why does ACI send ARP Gleaning packest back out the interface the original ARP was received on?

Does anyone have any answers?

Here are my raw results

Scenario #1: The BM Host attached as Access (802.1P) 

Test#1: Send 3xARPs for an unknown IP (on the same subnet) from the VM attached to Leaf 101

Result:

VM Attached to Leaf 101 – the one sending the ARPs sees:

• 3xARP requests from the itself (192.168.92.11) PLUS
• 3xARP requests from the default GW IP (ARP Gleaning at work)

VM Attached to Leaf 102 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work)

BMH Attached to Leaf 102 sees:

• 3xARP requests from the default GW IP PLUS
• 3xARP requests from the source VM (192.16892.11) - indicating ARPs are flooded even with ARP flooding turned off
  • Curiously, the ARPs from the source VM are tagged with VLAN ID 45, which is the canonical VLAN ID for the BMH attached to leaf 102.

Host receiving traffic via VPC Attached to Leaf101+102 sees:

• 12xARP requests from the default GW IP (ARP Gleaning on steroids)

Test#2: Send 3xARPs for an unknown IP (on the same subnet) from the VM attached to Leaf 102

Result:

VM VM Attached to Leaf 102 – the one sending the ARPs sees:

• 3xARP requests from the itself (192.168.92.12) PLUS
• 3xARP requests from the default GW IP (ARP Gleaning at work)

VM Attached to Leaf 101 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work)

BMH Attached to Leaf 102 sees:

• 3xARP requests from the default GW IP PLUS
• 3xARP requests from the source VM (192.168.92.12) - indicating ARPs are flooded, and again these ARPs carry the canonical VLAN ID tag

Host receiving traffic via VPC Attached to Leaf101+102 sees

• 12xARP requests from the default GW IP (ARP Gleaning on steroids)

Test#3: Send 3xARPs for an unknown IP (on the same subnet) from the BMH attached to Leaf 102

Result:

BMH Attached to Leaf 102 – the one sending the ARPs sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work) PLUS
• 6xARP requests from itself (192.168.92.10):
  • 3XARPs in untagged format
  • 3xARPs from source host (itself) encapsulated with canonical VLAN tag

VM Attached to Leaf 101 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work)

VM Attached to Leaf 102 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work)

Host receiving traffic via VPC Attached to Leaf101+102 sees

• 12xARP requests from the default GW IP (ARP Gleaning on steroids) 

Test#4: Send 3xARPs for an unknown IP (on the same subnet) from the BMH attached to a switch via a VPC on Leaf101 and Leaf 102

Result:

Host attached via switch via VPC attached to Leaf101+102

• 3xARP requests from itself (192.168.92.200) PLUS
• 12xARP requests from the default GW IP (ARP Gleaning on steroids)

BMH Attached to Leaf 102 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work) PLUS
• 3xARP requests from the source host (192.168.92.200) encapsulated in canonical VLAN

VM Attached to Leaf 101 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work)

VM Attached to Leaf 102 sees:

• 3xARP requests from the default GW IP (ARP Gleaning at work)

Scenario #2: The BM Host attached as Access (Untagged) 

The results were exactly the same as 802.1P encapsulation

Scenario #3: The BM Host attached as Trunk 

The results were exactly the same with the following exceptions:

• The BM Host recieved the ARP Gleaning requests encapulated with 802.1Q vlan tag of 2092
  • (The spurious ARP requests encapulated witht he canonical VLAN still persisted)
• When the BM attached host send its untagged ARP requests, they were dropped (as expected) by the Leaf switch which was now configured as a Trunk port.

Screendumps for Test#1

Test#1 Capture from VM attached to Leaf101Test#1 Capture from VM attached to Leaf102Test#1 Capture from BMH Host (802.1P) attached to Leaf102show vlan extended
Test#1 Capture from host attached to switch attached via VPC 

Screendumps for Test#2

Test#2 Capture from VM attached to Leaf102Test#2 Capture from VM attached to Leaf101Test#2 Capture from BMH Host (802.1P) attached to Leaf102Test#2 Capture from host attached to switch attached via VPC

Screendumps for Test#3 

 Test#3 Capture from BMH Host (802.1P) attached to Leaf102Test#3 Capture from VM attached to Leaf101Test#3 Capture from VM attached to Leaf102Test#3 Capture from host attached to switch attached via VPC

Screendumps for Test#4

Test#4 Capture from host attached to switch attached via VPCTest#4 Capture from BMH Host (802.1P) attached to Leaf102Test#4 Capture from VM attached to Leaf101Test#4 Capture from VM attached to Leaf102

[KELLEYD]

Wow! First off, great post, as always. It caught my attention, as I have had a todo for some time to get some packet captures to gain a little more insight into the ARP gleaning process before I start making any recommendations one way or another re: ARP flooding. Basically, to make sure that I understood it before others start asking about it. :)

At first I was thinking that you had a specific corner case, but you identified some behaviors that simply shouldn't be. The kicker at the end was most disturbing, re: the additional cvid-encapsulated ARP request.

My gut tells me that it might be platform-specific. In particular, re: forwarding behavior in 3.2 on the first-generation leaf switches vs. second-generation leaf switches. I'm pretty sure it was the Orlando 2018 BRKACI-3545 where said behavior changes and differences were noted.

I'm curious, do you have a second-gen leaf switch you could use, but leave everything else the same? And / or maybe try this with an older release? 3.0 maybe?

My lab is running 3.0(2k). I have a mix of first- and second-gen leaf switches. I have a 9372PX pair, a 9372PX-E pair, and a 93180YC-EX pair. Unfortunately, I just don't have the time during the week, and this is not a good weekend for me to try to get deep into the muck. I can probably duplicate your setup maybe some time next weekend, though.

Edit:  Fixed a typo of my own; Fixed my uh-oh:  I'm running 3.0(2k) not (1k).

[KELLEYD · Accepted solution]

I am running 3.0(2k) not 3.0(1k).  Sorry about that.

Anyway, I tried to duplicate your problem in my lab today, but as it turns out, someone put the 1st-gen leaf switches that were in the lab into production!  So I was unable to test the 1st-gen theory.  For now, anyway.

Good news, I suppose:  Running 3.0(2k), using 93180YC-EX switches, I did NOT see the same behavior out to a directly-attached host.  I tried in all three modes (trunk, dot1p, and untagged).  I also cleared endpoints after each change, just to be sure.  In my case, I used encap vlan-500.  The cvid on the leaf switch was not 500.  I forgot exactly what it was, but what's important is that it wasn't 500.

In all three situations, I only saw the ARP requests from the default gateway being broadcast, as expected.  I did not see any unicast ARP messages, nor did I see anything from any address other than the gateway.

When trunked, I saw the above frames tagged with the expected VID:  500.  I did not see anything tagged with the cvid, nor did I see anything untagged.

In 802.1p and in untagged modes, I saw only untagged frames.  Nothing unexpected.

So again, I suppose that is good news.  I would like to test on this version with a 1st-gen switch, but I may not be able to do so.

We're staring down the barrel at a fabric upgrade and were asked to upgrade the lab to 3.2.  Hopefully that will happen next week.  Once that is completed, I will repeat the test.  Again, on the 93180YC-EX switches at least, and will let you know what I see.

[RedNectar]

Thanks for testing on newer hardware! It certainly is good news!

[KELLEYD]

Just to follow up, I just did the same test with 3.2(2o).  Again, using 93180YC-EX switches.  Same results for me as on 3.0(2k).  I was really starting to suspect the version more than anything else, but maybe it is, in fact, something with 1st gen vs 2nd gen.

Still trying to get my hands on a 1st gen switch.  I really want to try to duplicate your results.

[nileshgore25]

Hello Chris,

Can you please help me understand the ARP requests for unknown IP.

• When ACI Leaf switch receives an ARP request, while ARP flooding is disabled,
  • It will do L3 forwarding for the target IP address inside of the ARP frame (assuming remote endpoint IP is already known on the ingress leaf)
  • it will send the ARP request to the SPINE proxy (in case the remote endpoint IP is not known on the ingress leaf)
• What will happen in case even Spine does not know about the target IP address ?
  1. will it generate ARP request for the target IP address sourced from the BD gateway address ?
  2. will this be broadcast throughout the BD, using the multicast sourced at Spine ?

apart from the ARP query, i have another one for unknown unicast.

• What happens when SPINE receives an unknown unicast from ingress leaf switch (while Hardware proxy is enabled) ?

Regards

Nilesh

[nileshgore25]

Hello Chris,

I have got the answers from the Cisco Live presentation and video. However you expert feedback is welcomed 

BRKACI-3101.ACI.Under.the.Hood.-How.Your.Configuration.is.Deployed

Your blog on ACI arp gleaning is also very good to understand the need of AP flooding as well as the behavior of SPINE in case it misses the sender target IP. 

https://rednectar.net/2018/08/13/aci-arp-gleaning/

Thanks 

Nilesh

[RedNectar]

Hi Nilesh,

Did you get all the answers you need?  That BRKACI-3101 presentation is my all time favourite!

Let me know if you need any more help.

[nileshgore25]

Hello Chris,

Yes, all my queries related to ACI fabric forwarding are now answered.

I will now proceed to study advanced ACI topics like - service insertion followed programability

Priority is to complete the topics on DC lab syllabus

Thank You !

Nilesh