problems with loop in simple L2 nx9k cluster

[tiwang]

hi out there - I have a problem i simply cannot spot where it is - we have 2 x 4 nx9k switches in 2 datacenters with a dark fiber in between. It is a simple L2 setup with 2 core switches with 2 distribution switches at each location - and the 2 x 2 coreswitches is intercannected with a set of fibers between each each coreswitch as a dci - see drawing:

my problem is that when i open vpc 45 and 46 - marked red - on the dktondac-coresw1-2 i get a loop - and i can simply not spot where - the mac addresses are "jumping" between po53 and po46 - like this which brings the whole cluster down:

2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 3456.fe2c.86b7 in vlan 901 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0014.5ecd.9c2a in vlan 200 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0014.5ecd.9fb4 in vlan 200 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 00ee.abab.5f4e in vlan 901 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 7018.a7b1.0e3c in vlan 901 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5689.5178 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.566a.93e4 in vlan 3939 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5664.c855 in vlan 3939 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.566e.6b88 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5668.e9c9 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5666.aee5 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5663.39ce in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.566e.e501 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5663.a3a5 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.5667.4e07 in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 1a5b.4c0b.a3d6 in vlan 901 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0050.566e.5aac in vlan 3939 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 00ee.abab.5f5e in vlan 813 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 00ee.abab.5f5e in vlan 116 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac 0000.0c9f.f332 in vlan 818 has moved from Po46 to Po53
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac cc03.d907.fdf9 in vlan 821 has moved from Po53 to Po46
2025 Oct 21 15:32:10 DKTONDAC-CORESW01-1 %L2FM-4-L2FM_MAC_MOVE2: Mac e0cb.bc07.fdf9 in vlan 821 has moved from Po53 to Po46

any suggestions?

[Mark Elsen]

  - @tiwang     As a first reaction from those message it looks like your topology is not loop free
                       and or  you have a network loop. You could start by finding out what is the origin port
                       of those mac addresses and the switch they are on. Then for instance you can isolate
                       that switch from the network and check if these messages then stop

  M.

[Pavel Tarakanov · Accepted solution]

Check for "peer-switch" on vpc domains (should be configured on roots only) and in general confirm status of links from STP perspective (either 49 or 50 should be in blocking state, if I got your topology correctly).

Do you have any special reason to separate links between VPCs on one site to two port-channels? You can bundle them in one VPC and will not have any issues.

[tiwang]

thanks this was in fact also what we have realized - the simpler topology looks like this instead - and i expect that this will run error free: