Any-cast forwarding Mechanism in ACI

[TangoAlfa]

Scenario: Source Leaf does-not know the destination(spine Proxy).

Host-A (IP-A and MAC-A) connected to Leaf01 wants to reach Host-B (IP-B and MAC-B) is connected to Leaf02 but Host-A and Leaf01 does not know  where is Host-B as its not present in their GST. Accordingly the leaf01 sends the packet to spine proxy/Anycast TEP for destination lookup as spine hold COOP DB/spine proxy DB and this packet goes through infra network.

Now my question is here.

1. So the leaf01 send the packet to spine(spine-proxy/any-cast TEP) without VXLAN tunnel? 

2. Once packet reach to destination Host-B and return traffic goes from Host-B to Host-A directly(without spine proxy as GST build by leaf02)  in that return traffic also not use the VXLAN tunnel? or in return traffic use the VXLAN tunnel as leaf02 has the GST?

3. if it use additional iVxLAN encapsulation then how that work.

[RedNectar]

Hi @TangoAlfa ,

Firstly, you have talked about MAC-A and MAC-B and IP-A and IP-B - so I have no idea if you are referring to Layer-2 communication between MAC-A and MAC-B (meaning IP-A and IP-B are on the same subnet) or Layer-3 communication - with IP-A and IP-B on different subnets.

So, to keep my explanation simple, I'll assume IP-A and IP-B are on the same subnet so we are talking about L2 communication.

---

1. So the leaf01 send the packet to spine(spine-proxy/any-cast TEP) without VXLAN tunnel? 

No. The packet will be encapsulated in VXLAN (iVXLAN header) with the destination VTEP (IP address) of the Layer 2 Proxy Anycast address

2. Once packet reach to destination Host-B and return traffic goes from Host-B to Host-A directly(without spine proxy as GST build by leaf02)  in that return traffic also not use the VXLAN tunnel? or in return traffic use the VXLAN tunnel as leaf02 has the GST?

When the spine Proxy, passes the original packet onto Leaf02, it does something quite strange - it sends the iVXLAN encapsulated packet to the VTEP of Leaf02 using the Source (VTEP) IP of Leaf01. Under the normal rules of TCP/IP, it SHOULD send it with one of its own IP addresses - so in fact it spoofs the source IP of the iVXLAN encaksualted packet.

The importance of this step is that when the iVXLAN encapsulated packet arrives at Leaf02, Leaf02 can see that the inner Source MAC address (MAC-A) came from the Source VTEP of Leaf01, and saves this information in its local table (GST)

So - back to your Q2 - the return packet will get iVXLAN encapsulation with a source IP of the VTEP of Leaf02, and destination IP of the VTEP of Leaf01

3. if it use additional iVxLAN encapsulation then how that work.

---

I'm not sure what you mean by "additional iVXLAN encapsulation"

For the L3 story, I suggest you take a look at this video I did for someone else

You can find a deeper dive into Layer 2 in my video that I did on this answer

[TangoAlfa]

Hi Chris, 

Thanks for your reply.

Hope you are doing well.

First of all its my bad that I have not mention L2 or L3. And second thing is that your video is excellent and I also having the same concept in terms of LST, GST stuffs.

Actually I wanted to understand the flow between leaf and spine within infra vrf in the Scenario of Source Leaf does-not know the destination(spine Proxy).

What I know "Each leaf and spine switch having TEP address to talk to each other within infra VRF. For leaf and Spine the TEP address use to establish the VXLAN tunnel but the spine dont use TEP address for forwarding spine use Anycast TEP address for forwarding."

Now my doubt is that Is TEP address and Anycast TEP address are two different interface?  If anycast TEP play the role for forwarding then Anycast TEP also having a VXLAN tunnel with other leaf switches as all these forwarding happen with VXLAN Encapsulation with iVXLAN header. Then is there two different VXLAN tunnel ? Hop not. I am bit confused with two different TEP(TEP and Any-cast TEP) 

And if it is L2 and L3 in both case the anycast TEP is require right? based on the scenario.

[RedNectar · Accepted solution]

Hi @TangoAlfa ,

[I've just realised I never posted this answer that I wrote 2 weeks ago - sorry]

I think you are "over" thinking the concept of tunnels in VXLAN.

VXLAN is just like VLAN - you don't usually talk about VLAN "tunnels" unless it is a specific use case.

I believe that a lot of confusion exists around VXLAN because the original use-case for VXLAN was to provide a path between two (or a few) endpoints.

But unlike VLANs, where we typically assign IP addresses naturally grouped together by IP subnetting, with VXLAN, the tradition is more along the lines of assigning /32 IP addresses to loopbacks and using routing to be able to go anywhere.  In the case of ACI, that routing is provided by the ISIS routing protocols, so to understand ACI VXLAN behaviour, you'll need to dig into ISIS a bit.

But first, I want to take a closer look at your quote: "Each leaf and spine switch having TEP address to talk to each other within infra VRF. For leaf and Spine the TEP address use to establish the VXLAN tunnel but the spine dont use TEP address for forwarding spine use Anycast TEP address for forwarding."

1.  Each leaf and spine switch having TEP address to talk to each other within infra VRF.
   • Correct
2. For leaf and Spine the TEP address use to establish the VXLAN tunnel
   • Correct
3. but the spine dont use TEP address for forwarding
   • Well, they can, but mostly TEP-to-TEP traffic is between Leaf TEPs
4. spine use Anycast TEP address for forwarding.
   • Oh - this used to confuse me too, until I finally got an answer from a CIsco developer at Cisco Live
   • No the spine doesn't use the Anycast address for forwarding - not as a source.  When traffic is sent to the Anycast address and the Spine proxy-forwards it, it leaves the original leaf TEP as the source TEP

[TangoAlfa]

Hi Chrish

"spine use Anycast TEP address for forwarding"  I just heard this in Cisco live(BRKACI-3545) in very first of spine proxy explanation that might create confussion for me. I might wrongly understand the meaning. Anyway you make me clear thanks

[Sergiu.Daniluk]

Here is a diagram I like a lot when it comes to forwarding in ACI:

Take care,

Sergiu

[TangoAlfa]

HiSergiu,

Thanks for sharing

I also gone through with this. I have mention my exact doubt on Chris reply.