How the L3 routed traffic to an unkown IP in ACI is treated?

[kumarH]

I found two different theory to handle l3 traffic for unknown IP in ACI one is stated below which is from White Paper 739989 and in other cisco live document I have seen that this will be treated via spine proxy method.

"For (L3) routed traffic to an unknown IP, the ACI leaf will generate an ARP request from its BD SVI (pervasive gateway) IP toward the unknown IP in order to detect and learn the unknown IP. Only the leaf with BD SVI IP for the unknown IP subnet will generate an ARP request. This behavior is originally triggered by the spine when the spine couldn’t find the unknown IP, even in the COOP database. This behavior is called silent host detection, or ARP gleaning. This behavior for (L3) routed traffic happens regardless of configuration, such as L2 Unknown Unicast or ARP flooding (mentioned below), as long as the traffic is routed to an unknown IP"

Is it something which I am misunderstanding? As if the unknow l3 traffic is spine proxed then no arp will be generated via ingress leaf.

[RedNectar]

Hi @kumarH ,

Both methods are the same, perhaps explained in different ways.  There's a couple of links to explore ARP Gleaning at the end of my reply.

SO here's the packet-prosessing logic as I interpret the white paper you referred to, and also the BRKACI-3101 slide presentation- I've left out all the bits to do with policy to concentrate on just the routing aspects.

• PRE-REQUESTITE: for ACI to be able to find unknown IPs, the target IP must be on a subnet defined by an ACI Bridge Domain gateway address, or a subnet defined by an EPG gateway address. Also, Unicast Routing must be enabled on the relevant Bridge Domains

1. A frame arrives at a leaf.
2. The leaf determines that it has to route the packet or switch it based on the destination MAC address.
3. Case #1: Switched packet (Destination MAC != Leaf MAC
   1. IF the destination MAC is known,
      1. THEN the Leaf sends it to the destination
      2. ELSE the Leaf sends it to the Spine Proxy
         1. IF the Spine knows the MAC,
            1. *THEN it forwards it to the leaf that has that MAC-Endpoint
            2. ELSE the spine looks at the destination IP.
               1. IF the dest IP Subnet exists in the appropriate ACI VRF,
                  1. THEN the spine sends a special ARP Gleaning request to all Leaves that have IP subnet active (ie, has hosts on that EPG). Or as the white paper describes it: "the leaf with BD SVI IP for the unknown IP subnet will generate an ARP request."
                     1. when the leaves get the ARP Gleaning request, they will generate an ARP request for the unknown IP sourced from the Bridge Domain Gateway IP address. [Now that is weird! But the idea is that the ARP request will stimulate the silent host into replying, and when it replies, the leaf will lear the MAC/IP, report this to the spine, and next time the spine receives a packet to that destination, the path marked * above will be followed]
                  2. ELSE the frame is dropped
4. Case #2: Routed packet (Destination MAC = Leaf MAC
   1. IF the destination IP is known,
      1. THEN the Leaf sends it to the destination
      2. ELSE the Leaf sends it to the Spine Proxy
         1. IF the Spine knows the IP,
            1. THEN it forwards it to the leaf that has that IP-Endpoint
            2. ELSE the spine looks at the destination IP.
               1. IF the dest IP Subnet exists in the appropriate ACI VRF,
                  1. THEN the spine sends an ARP Gleaning request to all Leaves that have IP subnet active (ie, has hosts on that EPG). Or as the white paper describes it: "the leaf with BD SVI IP for the unknown IP subnet will generate an ARP request."
                     1. when the leaves get the ARP Gleaning request, they will generate an ARP request for the unknown IP sourced from the Bridge Domain Gateway IP address. [This is exacly how a normal router would work, so no suprises here]
                  2. ELSE the frame is dropped

I hope this makes it clearer. You'll see that the process is similar for L2 and L3, so long as the pre-requesites are met.

Now - there is a problem though.

Once a host has been learned, and it moves to another leaf but does not send any packets, traffic will continue to be forwared to the old leaf. Once it sends a packet though, all will be good.  Now, if the host is a VM, and you vMotion the VM, then the new Hypervisor will automatically generate a RARP or GARP request using the VM's MAC address to ensure L2 tables are updated globally, and that should avoid the problem.  None the less, CIsco DOES provide an ARP Flooding option is yo uare worried about this case, and to cover the cases where Unicast routing is not enabled.

I hope this helps

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

ARP Gleaning and packet flow Links

Firstly, a link to the one you referred to, which is an AWESOME paper: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2019/pdf/BRKACI-3101.pdf

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9336pq-aci-spine-switch/118930-technote-aci-00.html

https://community.cisco.com/t5/application-centric/aci-arp-deluge/td-p/3677900

https://rednectar.net/2018/08/13/aci-arp-gleaning/

[kumarH]

Thank you for your reply and I understand the above process mentioned by you. whereas I have only confusion with the statement given in the whitepaper as I mentioned in my query.

Let me again repeat my questions.

1. is it true that ingress leaf will generate the ARP for unknown IP?

2. If ARP glean is done my spine then what will be the source IP and mac in the outer and inner header?

[RedNectar]

Hi @kumarH ,

1. is it true that ingress leaf will generate the ARP for unknown IP?

There are two answers to this question. 1. No. And 2. Yes

No. When a packet for an unknown destination IP arrives, it is sent to the Spine. The ingress leaf does not generate an ARP.

Yes. If the Spine does not know the destination, but the IP is in the range of a known subnet for the relevant VRF, the spine sends the ARP glean packet to all leaves that have that Bridge Domain - including the ingress leaf (assuming the ingress leaf has some EPs on that Bridge Domain).  Each Leaf that receives the ARP glean request generates an ARP packet with a source IP of the SVI.

2. If ARP glean is done my spine then what will be the source IP and mac in the outer and inner header?

---

Tne ARP glean process involves two entirely different packets. Firstly, the Spine sends an ARP glean request to the Leaves. And then each leaf generates an ARP packet, with the source IP of the SVI and a Source MAC of 00:0C:0C:0C:0C:0C.  You can see this in the packet captures in the https://rednectar.net/2018/08/13/aci-arp-gleaning/ blog post.  The packet that the spine sends to the leaf is a proprietary packet with a special Ethertype.  I have some packet captures somewhere, but can't find them at the moment.

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---