MAC Move After IOS Upgrade

[DJay11]

 Hi Everyone, 

We have upgraded our Cisco 7706 (Core Switch L3) to version 8.4(10), all our HP Access switches (L2) management IP (vlan 1) is now inaccessible. The Cisco 7706 is the gateway for Vlan 1. All Cisco Access Switch are up. Problem with HP is the management only and switch works fine with no disruption with users. Upon checking logs show on the Cisco core switch below. 

2024 Sep 28 00:39:11 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac f01d.2db6.0b8c in vlan 1 has moved from Po624 to Po104
2024 Sep 28 00:39:14 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac f01d.2db6.0b8c in vlan 1 has moved from Po104 to Po624
2024 Sep 28 00:39:16 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac f01d.2db6.0b8c in vlan 1 has moved from Po624 to Po104
2024 Sep 28 00:39:16 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac 003c.104f.4847 in vlan 1 has moved from Po624 to Po109
2024 Sep 28 00:39:17 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac 0000.5e00.0101 in vlan 1 has moved from Po624 to Po104
2024 Sep 28 00:39:17 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac 00fd.450a.ff40 in vlan 1 has moved from Po113 to Po624
2024 Sep 28 00:39:17 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac 9418.82b4.9740 in vlan 1 has moved from Po109 to Po624
2024 Sep 28 00:39:17 CAMPUS_VDC2 %L2FM-4-L2FM_MAC_MOVE: Mac e007.1bb6.26c0 in vlan 1 has moved from Po108 to Po624

How do I resolve this? We have around 25 HP devices that are currently down (Mgmt). 

[DJay11]

[MHM Cisco World]

Thanks for your topolgy'

I first do show cdp neighbor 

In SW check if the cdp match your topolgy 

L2 loop happened when there is 

1- two or more link between SW and bad stp config 

2- Port channel  issue 

Also if you remember which of these SW was root before upgrade' and is it same after upgrading?

MHM

[DJay11]

I'll check with the topology. Just wondering why it occurs after the IOS upgrade? And only HP devices are affected. Haven't check the root bridge for vlan 1 but after the upgrade, it was the Cisco Core Switch 2. Tried to transfer it to Cisco Core Switch 1 but problem still exist.  

[MHM Cisco World]

Moslty is stp issue 

If you can run 

Show spanning tree 

Check if you see any error message in link connect Core to HP SW

MHM

[DJay11]

Issue has escalated that affected our access to our data center but was already resolved after removing vlan 1 in the data center switches. Management vlan in the campus network are now accessible. 

We have vlan 1 on campus network that has 2 different subnet. 

Vlan 1 - 172.22.1.0/24
Gateway: Core Switch 

Vlan 1 - 172.25.125.1/0 (used by a stand alone access point)
Gateway: Firewall

The firewall is directly connected to our transport VDC, so traffic is from campus VDC - Transport VDC- Firewall. Vlan 1 should only be in the campus network. We remove the vlan 1 in Transport VDC. has this been the culprit? 

[DJay11]

updated diagram:

[Giuseppe Larosa]

Hello @DJay11 ,

from what you have described and from the additional info that you have provided there may be two issues here:

- the log Messages about MAC move points to L2 possible issues (STP interoperability between Cisco and HP switches or the presence of multi homed servers that are doing some form of NIC teaming as suggested by @David Ruess ).

The addtional information that you have provided is about Layer 3 and IP subnets: there are  two different VLAN 1 in two different VDC defined on the core Nexus 7706 switches.

And also that in campus networks  there are two IP subnets in VLAN 1.

>> The firewall is directly connected to our transport VDC, so traffic is from campus VDC - Transport VDC- Firewall. Vlan 1 should only be in the campus network. We remove the vlan 1 in Transport VDC. has this been the culprit?

Vlan 1 - 172.22.1.0/24
Gateway: Core Switch

Vlan 1 - 172.25.125.1/0 (used by a stand alone access point)
Gateway: Firewall

From your last network diagram that is very clear we see the two VLAN 1 one  in VDC campus and the the one defined in VDC transport that has the Gateway on the FW.

It is a good idea in this case to remove VLAN 1 on the trunk links between VDC campus and VDC transport.

Just to understand this change solved the L3 issue:

>> Management vlan in the campus network are now accessible. 

But do you still see the MAC address move events in show logging ?

if so there is still something to investigate at Layer 2.

Identify the involved port-channels po624, po108, po109, po113.

what switches are behind each of them ? Are they a standalone Cisco switch , a Nexus VPC pair , an HP standalone switch  or an HP switch stack ?

Follow one MAC address to find out where it is learned now at access layer.

identify OUI vendor.

For STP:

check what type of STP is running on the HP switches. It may be Rapid STP 802.1W single instance and interaction with Cisco Rapid PVST happens exactly only in Vlan1.

Check the STP root for VLAN 1 on HP switch and on Cisco side. They should agree on the root bridge Bridge ID.

Hope to help

Giuseppe

[DJay11]

Thank you for this information. No more mac flapping logs seen on the campus core after removing the vlan 1 from trunks port going to transport vdc. We also have changed the vlan assignment of the stand alone AP to vlan 125.  Sharing also the recommendation of our vendor to avoid these incident to happen again. 

[Giuseppe Larosa]

Hello @DJay11 ,

thanks for your feedback so removing  VLAN 1 from L2 trunks between VDC campus and VDC transport solved your issue.

Actually, this is the right design choice to do.

Hope to help

Giuseppe

[MHM Cisco World]

OK what is STP mode run in Core and HP ?

MHM

[DJay11]

HP - MSTP
Cisco Access - RSTP
Cisco Core - RSTP

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/lan-switching/multiple-instance-stp-mistp-8021s/116464-configure-pvst-00.html

this link and what you use as workaround explain something here, 
vlan1 & vlan2 and above priority is key here to solve issue 

PVST simulation runs seamlessly with two critical rules:

• If the root bridge for CIST is within a non-MST region, the spanning-tree priority of VLANs 2 and above within that domain must be better (lesser) than that of VLAN 1.
  
  
• If the root bridge for CIST is within a MST region, VLANs 2 and above defined in the non-MST domains must have their spanning-tree priorities worse (greater) than that of the CIST root.

[DJay11]

Thank you for the help. 

[DJay11]

We have 2 issues encountered after the IOS upgrade. 

1. Vlan 1 - resolved

2. Mac move between FW and in our Core & WAN VDC in data center. 

Recommendation of TAC is to configure peer-switch on our VDCs. Peer-switch is only configured in Transport VDC.