Hi Chris,   I'm keen to hear from Cisco on this one as well. In the mean time I can offer some speculation.   In brownfield deployments we often don't know if there are devices present that rely on ARP flooding updates for their MAC to IP bindings. It's safer to set the BD to ARP flood to ensure that all ARP gets flooded to every endpoint in the BD rather than having ARP optimised. Maybe there were TAC cases where people had migrated and had issues post-migration so Cisco have changed the default to be the safer option for migrations?   I also noticed another change in 4.2 where VMM bindings in the GUI are defaulting to On-Demand/Pre-Provision rather than On-Demand/Immediate. ... View more

It looks like this is supported from 4.2.1 and later. It requires you to set up a separate L3Out in each site and use Inter-Site L3Out support which was introduce in 4.2.   See here, page 115 through to 139 : https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKACI-2125.pdf   I'm not sure on the encapsulation scope in the L3 Outs of each site so the firewalls see each other in the segment that they peer to ACI on. I'd try setting it from local to VRF, but I haven't seen anything stating that it will work across sites. ... View more

You can't have a BD subnet in a Layer 3 Out a subnet is either configured as a BD subnet or it is configured on a Layer 3 Out, never both at the same time. The fabric has different endpoint learning semantics for the different types of networks so it can't mix them.   If you don't use the load balancer IP address 10.0.100.253 for anything, the simplest solution would be to use a different subnet for transit between ACI and the load balancer and configure this as an SVI in the new Layer 3 Out. Route the VIP subnet via this L3 Out and you are done.   You also have the option to put in /32 routes for the VIPs in the EPG with the load balancer IP as the next-hop. This works only with /32 routes though. ... View more

Yes you can have mixed models in a cluster. Do it one APIC at a time. The cluster scale will remain at the lowest member so you won’t get the higher scale until all three are replaced.  ... View more

Yes you can vMotion between pods in a Multi-Pod fabric.  ... View more

There are some features that are not supported but not many and as releases go by it is getting closer.    However I would not recommend using FEX in a new install. You will have very limited  TCAM available if you try to push multiple top of rack switches into a single parent leaf switch. You will get much better scale out of the network by using Nexus 9K leaf switches and not FEX.  ... View more

I have tested this now. My ingress leaves were second generation and the egress (border) leaves were first generation. My ingress leaf was a Remote Leaf, which may also change things.   The behaviour I saw was that the endpoint on the ingress leaf was learnt at the border when the traffic was to an L3 Out destination.   Here is the ingress leaf route: Rleaf202# show ip route vrf common:Lab 10.0.0.1/32, ubest/mbest: 2/0 *via 10.220.240.32%overlay-1, [200/41], 01w03d, bgp-65000, internal, tag 65000 *via 10.220.240.35%overlay-1, [200/41], 01w03d, bgp-65000, internal, tag 65000   These are the TEP IPs of the two border leaves:   Leaf101# show ip int brief vrf overlay-1 | egrep "10.220.240.3[2|5]" lo0 10.220.240.32/32 protocol-up/link-up/admin-up Leaf102# show ip int brief vrf overlay-1 | egrep "10.220.240.3[2|5]" lo0 10.220.240.35/32 protocol-up/link-up/admin-up   I pinged from a switch that was acting as a client in the fabric:   Client_SW# ping 10.0.0.1 source 10.33.33.33 PING 10.0.0.1 (10.0.0.1) from 10.33.33.33: 56 data bytes 64 bytes from 10.0.0.1: icmp_seq=0 ttl=251 time=0.877 ms 64 bytes from 10.0.0.1: icmp_seq=1 ttl=251 time=0.628 ms 64 bytes from 10.0.0.1: icmp_seq=2 ttl=251 time=0.641 ms 64 bytes from 10.0.0.1: icmp_seq=3 ttl=251 time=1.793 ms 64 bytes from 10.0.0.1: icmp_seq=4 ttl=251 time=0.605 ms --- 10.0.0.1 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.605/0.908/1.793 ms IP address: 10.220.240.35, IP subnet: 10.220.240.35/32   Now we check the endpoint learning:   Rleaf202# show endpoint ip 10.33.33.33 Legend: s - arp H - vtep V - vpc-attached p - peer-aged R - peer-attached-rl B - bounce S - static M - span D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr L - local E - shared-service +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ 4 vlan-30 00b7.6666.7777 L eth1/52 common:Lab vlan-30 10.33.33.33 L eth1/52   The leaf attached to the client machine has learned the IP.   One of the border leaves has learned the IP.   Leaf101# show endpoint ip 10.33.33.33 Legend: s - arp H - vtep V - vpc-attached p - peer-aged R - peer-attached-rl B - bounce S - static M - span D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr L - local E - shared-service +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ common:Lab 10.33.33.33 tunnel18   So with generation 2 ingress and generation 1 egress the L3Out traffic does trigger a learn.   ... View more

So after reading the whitepaper I think you're correct. Traffic from endpoints on the ingress leaf does not trigger and endpoint learn at the border leaf (egress) when the traffic is to an L3 Out and the enforcement is set to ingress. Often the border leaves are also compute leaves in smaller fabrics, so the border leaves will have endpoint learns present due to endpoint to endpoint traffic flows. ... View more

This might be an optimization to stop L3 Out data flows from keeping endpoints alive at the borders. I'd have to test it to see the behaviour with second gen gear. Where is the reference you've included documented? ... View more

The border still learns remote IP endpoints from the ingress leaves (edit: when the traffic is to an endpoint attached to the border and not to the L3 Out).   The prefix to EPG mapping is programmed on the ingress leaves so policy enforcement can happen as soon as the packet enters the fabric. With the mode set to egress the prefixes are only programmed on the border leaves and enforcement happens at the border.    There is also an option to stop IP endpoint learns on the borders altogether (Diasable Remote EP Learn) but this is a separate setting.  ... View more