ACI forwarding tables

[AirBorn]

Hi experts

I'm a bit confused about forwarding tables used in ACI

1/ The Endpoint table contains EP MAC and IP /32, does this table used for L2 forwarding only, or it is

     also used for L3 forwarding as well ?

2/ In Cisco doc, they says that ACI uses RIB (non /32), Are RIB table and VRF table the same ?

3/ Does VRF table contains EPs IP /32 ?

Regards

[RedNectar]

Hi @AirBorn ,

ACI forwarding tables are more standard than you probably think.

If a frame from an endpoint is sent to the MAC address of the ACI Leaf, it is routed. More on that later

If the frame has any other destination MAC address, the leaf will do a L2 lookup to find the destination MAC. (And if not found, send it to the Anycast address in the Spines for unknown L2 - or flood or drop if so configured).  Assuming the Leaf knows the location of the Destination MAC (local or another leaf), it will determine the EPG of the destination, and determine if a contract is necessary to forward the frame - if so, it will need to look at the L3/4 contents of the packet to determine if a contract exists.

Back to routed packets - ie a frame from an endpoint is sent to the MAC address of the ACI Leaf.

Like a normal router, the leaf will look at the destination IP and determine if a route exists for the destination in the VRF of the source EP. If not (i.e. not even a default route to an external router) it will drop the packet.

But for most packets, a route will exist.

Unlike a normal router, the leaf doesn't just look for the next hop. It now looks at the destination IP and does a /32 lookup to determine the EPG of the destination, which may be the same EPG as the source, in which case it will forward the packet (locally or VXLAN encapsulated and sent to another leaf TEP). Assuming default options for the EPGs of course.

If the /32 lookup finds a match, the leaf will also determine the destiation EPG and apply policy - and forward the packet to the TEP of the destination leaf. 

If the /32 IP lookup fails, (remember the packet has already passed the route lookup test) the leaf now needs to determine if the destination IP is a local route - i.e. learned from within the source's VRF or leaked from another VRF, or remote - i.e. learned from a L3 Out.

If the destination is a local route, the leaf will send it to the Proxy with the SP (Source Policy) bit in the VXLAN header set to indicate that policy has not been applied on this packet.  If the spine knows the destiantion, it will forward the packet and the policy/contract will be applied at the egress leaf.

If the destiantion is a remote route, it will determine the destination L3 EPG (I know there is no such thing as a L3 EPG, but I hope you get the idea) based on the destination IP address and apply policy*. Assuming the packet is allowed, it will be forwarded to the nearest external router like any normal router. This router may be locally attached, or if attached to a remote leaf, the packet will be VXLAN encapsualted and forwared to the TEP of the nearest boarder leaf.

---

* Prior to v1.02 or thereabout, policy for packets forwarded externally was always applied at the boarder leaf. This caused major TCAM issues for boarder leafs, and so the default was changed in that version. However, there is still an option in the configuation of a VRF to Apply Policy on Egress to set it back to the old TCAM thirsty way if you wish.

---

I hope this helps. I didn't answer your questions specifically, so let me know if you want me to.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[AirBorn]

Hi Chris,

Thanks for your answer

So, if i understood well

 1/ If it is a L2 packet, the lookup is done on the Endpoint table for the destination mac address

 2/ If it is a L3 packet, the lookup is done on the VRF table for the destination /32 IP address

How the VRF is populated with /32 IP addresses, does this happens with Endpoint Learning, with a leaf putting

mac&IP /32 in the Endpoint table and IP /32 in the VRF table ?

Regards

[RedNectar · Accepted solution]

---

@AirBorn wrote:

Hi Chris,

Thanks for your answer

So, if i understood well

 1/ If it is a L2 packet, the lookup is done on the Endpoint table for the destination mac address

Correct.  If the destination is local on the same VLAN it basically gets switched like a normal switch.

 2/ If it is a L3 packet, the lookup is done on the VRF table for the destination /32 IP address

More or less. If the destination IP is remote (not a local route in the VRF) then exit leaf is determined as per normal routing, but the policy applied is based on the contract with the Network (or Subnets) defined in the L3 Out.

How the VRF is populated with /32 IP addresses, does this happens with Endpoint Learning, with a leaf putting

mac&IP /32 in the Endpoint table and IP /32 in the VRF table ?

There are two things going on here. One is about determining the way to forward the packet/frame, the other about applying policy.

The VRF itself doesn't fill up with /32 addresses - a fabric <leaf_id> show ip route vrf <tenant_name>:<vrf_name> will show that.

So now it comes down to the difference between local routes and those learned from an external source via a L3Out

If the destination has a match and matches a local route, then a /32 lookup is done for the EP in the local and global station tables [it may do the /32 lookup first - in the end it doesn't matter which order] and from that lookup the destination EPG is determined and policy applied.  If the destination is local and no /32 match found - it's sent to the proxy.

If the destination has a match and matches a remote route, then no need for a /32 lookup - instead policy is applied based on the contacts applied to those "subnets" defined in the Networks section of the L3 Out.  Not so granular as /32s but much more scalable.

Regards

---

I hope this helps

---

[abjain]

Can u please clarify what's /32 and non /32 ip, /128 & non /128 address in ACI's tables ? 

[RedNectar]

Hi @abjain  ,

Firstly, this thread is dead. The question is answered. If you have a NEW question, start a NEW thread.

Meanwhile I'll think about how to answer your question

[TangoAlfa]

Is this question started in new thread? if yes . Please share the link of that thread as its ralavent to this thread and I am exited to read RedNectar Answer for great explanation.