BUM loop in VXLAN multisite CML

[exness-noc]

We have vxlan multisite topology in CML. When we create l2vni between site and start ping from client in site 1 to client in site 2 we get 100% CPU LOAD on BGW because on arp loop.

APR request goes from the 1st site bgw  to the 2nd site bgw, then the same ARP request + 14 bytes goes from the 2nd site bgw towards the 1st bgw, then APR request goes from the 1st site bgw  to the 2nd site bgw plus another 14 bytes, etc. The frame increases in size each time and at some point begins to fragment. Is this expected behavior for a virtual nexus 9k in the role of spine + bgw?

[MHM Cisco World]

 Can I see config of spine and leaf 

MHM

[exness-noc]

Configuration of the BGW, which is shown in the picture CML_1.jpg on the right is in the attachment.  The configuration of the BGW of the 2nd site differs only in the ASN.
I also noticed the following, that when the BGW receives an ARP request from the leaf (fabric-Link), when forwarding this packet to the DCI, it changes the original ARP packet (adds a 14-byte to trailer field in ethernet frame).
I repeated the topology in EVE-NG and got the same result. I don't understand, is this some kind of limitation of the multi-site topology in a virtual environment or I don't conceptually understand why this happens.

[MHM Cisco World]

Are you sure there is no conflict in vtep LO IP between two sites?

MHM

[exness-noc]

Yes.

ams-bgw-1# sh nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
...
nve1 10.250.137.2 Up CP 00:00:47 5201.c528.1b08  -- PIP_VTEP BGW 2nd site
nve1 10.250.137.129 Up CP 00:00:17 0200.0afa.8981 -- VIP_VTEP BGW 2nd site

ld-bgw-2# sh nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 10.250.129.1 Up CP 00:00:49 5217.1a98.1b08    -- PIP_VTEP BGW 1st site
nve1 10.250.129.129 Up CP 00:00:49 0200.0afa.8181   -- VIP_VTEP BGW 1st site
...

[MHM Cisco World]

You use nexus so I hope I direct you correctly 

Now for multisite you need to use 

Fabric site-id xxx

This will use in type6 of evpn which use for BUM between sites

Try use this command and check 

MHM

[exness-noc]

My NXOS software image file is: bootflash:///nxos64-cs.10.4.2.F.bin, in what mode is this command "Fabric site-id xxx" entered. This is the first time I hear about it.

[MHM Cisco World]

evpn multisite border-gateway 64601

This site-id are yoh use same in both GW of sites?

MHM 

[exness-noc]

Yes, it is different on both sides.

[MHM Cisco World]

show bgp l2vpn evpn route-type 6

Share this from both GW NSK of both sites

Thanks 

MHM

[exness-noc]

I don't have route-type 6 on BGW, because I use ingress replication between sites.

[MHM Cisco World]

Forget that 

show nve internal bgp evpn multisite flooding <<- share this 

Also run DCI-track (I dont think it prevent loop but let see it effect in our case)

Also the link between two site config it as L3 port 

MHM

[exness-noc]

I don't have this command in NXOS: show nve internal bgp evpn multisite flooding

About tracking - it is:

ams-bgw-1# sh nve multisite fabric-links
Interface State
--------- -----
Ethernet1/1 Up
ams-bgw-1# sh nve multisite dci-links
Interface State
--------- -----
Ethernet1/6 Up

[MHM Cisco World]

evpn multisite dci-tracking <<- this need 

I will check split horizon command it hidden so add it and press enter 

MHM

[exness-noc]

evpn multisite border-gateway 64601
delay-restore time 30
df-election time 3.0
split-horizon per-site

Result is same. There is loop of arp request and +14 byte every time to trailer in ethernet frame.