ACI decision making question

[mmacdonald70]

I had a recent outage and I'm trying to figure out what happened.

We have an ACI network that connects to an NX-OS network with both L2-out and L3-out.  We have the following servers (diagram attached):

Server A (192.168.3.100) - In ACI, part of BD-A with its default gateway in the ACI

Server B (192.168.2.100) - In ACI, part of BD-B, the vlan is extended out to the NX-OS network with its default gateway in the NX-OS network

Server C (192.168.2.101) - In NX-OS.  Part of the same vlan as Server B

I have 2 questions about this:

Question 1:

Since Server B and C are in the same vlan/BD, I assume that they just normally but if Server A wants to talk to Server C, I assume that the following is true.  Please correct me if not.

- Server A checks its routing and arp table and sends the packet with a dest ip of 192.168.2.101 and a dest mac of its default gateway (in NX-OS)

- If Server C is known as an EP in BD-B, ACI will send it directly out the L2 out

- If Server C is not known as an EP in BD-BC, aci will route it through the L3 out

In both cases, Server C's response would be send to its default gateway and routed through the L3-out.

Question 2:

The outage happened when server C had an HA failover and had a new mac address (and didn`t send a GARP).  It was corrected when I cleared the EP.  Is there some knob or switch in ACI to make this work?

[Jason Williams · Accepted solution]

For the first inquiry about reaching server-c from a different subnet, we would need more information. 

Is BD-B an L2 or L3 bridge domain? (unicast routing disabled or enabled)

If the gateway sits outside of the fabric, then it's best practice to have unicast routing disabled on the BD. 

For Q2: 

Lets assume the BD is L3 (routing enabled). If IP-A moves from mac a to mac b and the device does not send a GARP, then there is likely no chance of the BD updating the IP/MAC mapping. If the device does send a GARP packet, but ACI did not update its MAC/IP info then you may have GARP based learning disabled on the BD. 

To enable this you would first need to enable ARP Flooding on the BD main settings tab. Once it's enabled, then go to the L3 configuration of the BD and you should see a new checkbox for GARP based detection. Enable this and that fabric will learn or update endpoints from GARP packets. 

If the BD were L2, then we don't care about IP to MAC learning. Once the IP moves to from mac a to mac b then ACI just needs to received traffic from mac b and it'll know where that endpoint exists. No need to worry about IP learning. 

Jason

[mmacdonald70]

Awesome!  Thank you so much.  Not only was the answer helpful in answering my problem, but I learned something new.

The BD is indeed a L3 bridge domain.  As best practise says that this should be L2, I will do this and I'm pretty sure that this will solve my issue.

As a followup, the system in question doesn't send GARP on failover.  I will research as well but it looks to me that changing the BD to a L2 domain will remove the requirement for GARP based detection.