Unknown Unicast in ACI Fabric

[moyeonlee]

Hi Experts,

I have a question about ACI Fabric.

Unknown unicast policy under Bridge Domain has two modes, HW proxy(default) or Flooding. If I choose Flooding How does a leaf treat that unknown packet? does a leaf send that packet to all spines? Normally there is no unknown unicast forwarding in L3 routing. Every Leaf in ACI Fabric is operated on L3 so that makes me confused. 

And in what cases should I enable the ARP flooding under BD?

Thanks in advance.

Paul

[Tomas de Leon · Accepted solution]

Bridge Domain
The bridge domain can be compared to a giant distributed switch. Cisco ACI preserves the Layer 2 forwarding semantics even if the traffic is routed on the fabric. The TTL is not decremented for Layer 2 traffic, and the MAC addresses of the source and destination endpoints are preserved.

Hardware Proxy
By default, Layer 2 unknown unicast traffic is sent to the spine proxy. This behavior is controlled by the hardware proxy option associated with a bridge domain: if the destination is not known, send the packet to the spine proxy; if the spine proxy also does not know the address, discard the packet (default mode).

The advantage of the hardware proxy mode is that no flooding occurs in the fabric. The potential disadvantage is that the fabric has to learn all the endpoint addresses.

With Cisco ACI, however, this is not a concern for virtual and physical servers that are part of the fabric: the database is built for scalability to millions of endpoints. However, if the fabric had to learn all the IP addresses coming from the Internet, it would clearly not scale.

Flooding Mode
Alternatively, you can enable flooding mode: if the destination MAC address is not known, flood in the bridge domain. By default, ARP traffic is not flooded but sent to the destination endpoint. By enabling ARP flooding, ARP traffic is also flooded.

This mode of operation is equivalent to that of a regular Layer 2 switch, except that in Cisco ACI this traffic is transported in the fabric as a Layer 3 frame with all the benefits of Layer 2 multipathing, fast convergence, and so on.

Hardware proxy and unknown unicast and ARP flooding are two opposite modes of operation. With hardware proxy disabled and without unicast and ARP flooding, Layer 2 switching would not work.

The advantage of disabling hardware-based proxy and using flooding for unknown hosts and ARP is that the fabric does not need to learn millions of source IP addresses coming from a given port.

Check out:

Cisco Application Centric Infrastructure Design Guide
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-731960.html

Thank you for using the ACI Cisco Support Community!

[moyeonlee]

Hi All,

Thank you so much for your reply.

Actually there are some silent servers(Linux or Unix-based OS) in the network. They are all dedicated hosts only receiving packets and pre-programmed with default GW MAC address by an application. They don't expose their MAC to any Leaves and also any Spines don't know about it. In that case, any packets sent to those hosts will be discarded at Spines. How can I solve that issue when ACI Fabric adopt those kinds of hosts from the legacy network?

Thanks in advance.

Paul

[Vivek Ganapathi]

Hello All,

Firstly sorry for re-opening a very old thread. But thought relevant to discuss on this one.

In the standard VxLAN implementations with MP-BGP EVPN, the case of unknown unicast is cut down to a large extent in the way the endpoint information is handled. As per my understanding, as soon as the endpoint is connected to a leaf node, the endpoint information is captured by the node (using ARP, CDP information etc.) & sent by the leaf node as a BGP update to all other leaf nodes in the fabric. There are few cases where in the endpoint is a "Silent" host & leaf nodes would be unaware of the endpoint information & hence multicast is being used to figure out these type of endpoints.

But with ACI the BUM traffic handling seems to have changed . Am I correct in my understanding :

1) MP-BGP EVPN is just used to advertise the external routing information into the fabric?

2) Unknown Unicast traffic handling is performed by the Spine switch (in the event we are using Hardware Proxy feature.

IMHO, the standard Non-ACI implementation of VxLAN seems to be much better in handling the unknown unicast than ACI. As per my understanding the only optimization occurring here is on leaf node as they don't need to learn the MAC address of hundred thousand endpoints (only the local endpoints to itself). But it's shifting the MAC learning process to Spine node. Overall, I'm wondering what could have been the decision to make this change.

Regards

Vivek

[bchintak]

Hi Vivek,

Please check ARP gleaning mechanism in ACI for detecting silent hosts 

[dpita · Accepted solution]

A good use case for enabling ARP flooding would be when the Default Gateway resides outside of the ACI Fabric. This non-optimal configuration will require ARP Flooding enabled on the BD. 

[moyeonlee]

Hi Dpita,

Thank you so much for your reply.

How can I verify it is correct or not? There is only L2 External network integrated with ACI Fabric. if every end point has the default GW provided by pervasive SVI of ACI Leaves they can't go to the Ext-L2_Net. So the only solution for that situation is change default GW from the pervasive to External GW of the Ext_L2_Net. Is that correct solution or design? Esp, a customer don't want to have a Ext_L3_Net to solve that issue.

Thanks in advance.

Paul

[cortizm]

Hi Paul,

Was Looking your question.

We've a similar stage or behavior with linux Silent host, and sometimes the server loss connection with the Default gateway and of course service goes down.

We've Flooding Enable and Also Arp floogin enable

Finally you solved this issue? 

how can we change or help these silent host?

[RedNectar]

This post is already answered. If the answer didn't answer your question, ask a new one (with reference to this one if necessary)

[arsalva]

so for a Migration point of view we need ARP flooding when we are doing static binding to EPGs,

1EPG to 1 BD and the BD with  arp flooding? since the legacy network is still on production, we are connecting to 2 cat 6500 and the gateways are still over there, at one point there will be a migration to ACI, turn off gateway on legacy, turn on gateway on ACI.

is this correct?