VXLAN L2VNI ARP propagation issue - N9K's

[Newbee]

Hi all - I'm looking for some VXLAN advice to fix a strange layer 2 unknown ARP issue I have between some (not all) hosts on the same L2VNI/VLAN which are spread across N9K VTEP's. 

Example topology is HOST-A is connected in v10 (L2VNI 10010) on 9k-VTEP-A and HOST-B is connected to 9K-VTEP-B again same VLAN and L2VNI.  There is no routing involved and ARP suppression is enabled on the L2VNI.

My issue is that after about 40 minutes of hosts being idle and not communicating to anything the BGP EVPN table flushes entries on both VTEPs and won’t repopulate them unless both hosts initiate communication with each other (or something else on the network) thus beginning the ARP process and creation of EVPN packets all over again (meaning there's no L2 connectivity across VTEP's when 1 host tries to ping another).

I thought that if HOST-A needs to arp out for HOST-B (which was silent) the arp would be encapsulated and flooded out using multicast to all VTEPs and Host-B would respond like regular Ethernet (just encapsulated) - which is not happening. Multicast looks correct and all the L2VNI info and NVE peers are okay too.

Anyone seen anything like this or could steer me in the right direction - I'm clearly missing something with either the BUM traffic or multicast somewhere but not sure (other than raising it to TAC)

Thanks for anyones help or experiences.

[MHM Cisco World]

Both nexus use vpc or not only vxlan?

[Newbee]

All the Nexus 9k’s are VPC pairs with teamed NICs (so HOST-A connected to 9K-VTEP-A which is a VPC pair & HOST-B to 9K-VTEP-B which is another VPC pair)

Thanks

[MHM Cisco World]

OK, 
this is silent Host issue I think,
you config anycast GW in each Nexus.
there are two VLAN for each host in each nexus
there is SVI for each VLAN.
for VXLAN with silent host the nexus can not flood arp to all leaf without know that this leaf have this subnet

with route type 5 you can make any cast flood arp to all VTEP "leaf" have this subnet and hence detect the silent host.

[Newbee]

Thanks for the response - so if I am understanding you correctly you’re saying that every L2VNI must have an SVI with the IP/subnet created on all VTEPs so the VTEPs know which subnets sit behind them (for silent hosts)?

So passing the L2VNIs (with SVIs but no IP’s) will cause this error - which is how I have done it (VLAN/VNI/SVI without IP)?

[MHM Cisco World]

Yes, you need IP address for SVI and this will advertise through BGP and make all VTEP know that this VTEP have this subnet. 
so when there is silent host "one that is not send ARP or there is no mac address in table" will known by other VTEP.

[Newbee]

Thanks for taking the time to respond.

I wrongly thought that any VTEPs receiving BUM traffic within a L2VNI would simply forward to all member ports within the same L2VNI without needing to have an IP address on the SVI for longest prefix matching and silent host discovery.

I will give it a try tomorrow and let you know the outcome.

[MHM Cisco World]

friend are both Host-A & Host-B same or different subnet ?
i thought they are in different subnet.

[MHM Cisco World]

if the Host-A & Host-B is same subnet
you are right the VTEP will flood ARP for same VNI, 
are you use ingress replication OR multicast ?  

[Newbee]

Same subnet and using multicast pim bidir for the BUM replication....which is why I didn’t think an IP address on the anycast GW SVI was required.  
ARP requests to the silent host work outside of VXLAN (so not a host problem) - waking the silent hosts up with ARP only fails when 2 hosts (same L2VNI and subnet) are on different VTEPs and trying to ARP.

[MHM Cisco World · Accepted solution]

[MHM Cisco World]

are you sure you enter 
ip pim spare-mode 

under both the loopback of VTEP and under the P2P between the VTEP and Spine ?

[MHM Cisco World]

also please share the config of one vPC nexus VTEP ?

[Newbee]

Thanks for the screenshots of the suggested multicast configuration - I have configured “ip pim-sparse mode” on the physical point to point links between spine and leafs but not on the underlay loopbacks.

That’s the most likely cause of the problem however it would be good to know where to apply the config too (just the underlay loopback used as the unnumbered interfaces and not SVIs or VTEP loopbacks or VPC links?)

Also I’ll need to find out what impact applying “ip pim sparse-mode” to the underlay loops will have (hoping no disruption to unicast traffic)

Again thanks - you have really helped pinpoint where I may have gone wrong.