Hey Cisco,
APIC versions are 5.2(3g)
Leaves + Spines versions are 15.2(3g)
Here is the test:
I have 2 Servers connected by vpc (this is just what I had to work with) to 2 different pairs of leaves.
Each server has an LACP bond associated with an EPG and BD that I tested the configuration changes I am about to describe here. Both servers are obviously configured with an IP on the same subnet.
Before each test scenario I turned off the bond interface and waited until the server's MAC was gone from the Endpoints in BD tab in the APIC and then I ran the next test. I configured a custom endpoint retention policy for the BD that remained the same at each test scenario.
Endpoint Retention Policy:
Hold Interval: 5
Bounce Entry Aging: 180
Local Endpoint Aging Interval: 150
Remote Endpoint Aging Interval: 120
Move Frequency: 256
L3 Unicast Routing is enabled throughout all tests.
Test Scenario 1:
L2 Unknown Unicast=Hardware Proxy
Arp Flooding=Disabled
No BD Subnet configured
Ping between 2 servers doesn't work.
This makes sense, when server A pings server B server A doesn't know server B's MAC address so it sends an ARP message. leaf A receives the ARP request, since ARP flooding is disabled the leaf tries to send the ARP via unicast to server B's address. leaf A doesn't know IP B so it sends the packet to the spine proxy. The spine proxy doesn't know the IP of server B and tries to flood an ARP message to the BD (ARP Gleaning) since there is no SVI (no BD subnet configured) in the BD the packet is discarded, ARP doesn't work hence ping doesn't work.
Can you confirm I described the process correctly?
Test Scenario 2:
L2 Unknown Unicast=Hardware Proxy
Arp Flooding=Disabled
BD Subnet is configured
IP Data-Plane Learning=Enabled
Ping between 2 servers works.
Again this makes sense, the same process happens as test scenario 1 but this time the BD has an SVI and ARP Gleaning works. When Server A sent an ARP request for server B thanks to data plane learning ACI learned Server A's IP and when Server B sent a unicast ARP reply back to server A the spine proxy has a record of server A's endpoint.
Am I correct by assuming that is why this time ping works?
Test Scenario 3:
L2 Unknown Unicast=Hardware Proxy
Arp Flooding=Enabled
Ping between 2 servers works.
Again this makes sense, the same process happens as test scenario 1 but this time leaf A isn't trying to perform a unicast ARP message rather it floods the ARP in the BD and as a result server B responds.
Again I would love confirmation.
Test Scenario 4:
L2 Unknown Unicast=Hardware Proxy
Arp Flooding=Disabled
BD Subnet is configured
IP Data-Plane Learning=Disabled
Ping between 2 servers works.
This doesn't make sense to me. The documentation I listed at the top explicitly states that when IP Data-Plane learning is disabled L2 Unknown Unicast needs to be set to Flood and ARP flooding must be enabled. Remember what I mentioned in Test Scenario 2 the reason that Server B's ARP reply unicast knows how to reach Server A is because the Spine Proxy was able to learn Server A's endpoint thanks to data plane learning when Server A originated its ARP.
If anyone has an explanation to this I would love to hear.
