Invalid MAC address endlessly flapping between two ports

[Razvan Craciun]

Hi guys,

I've hit an issue that I can't get to the end of, and thought that you might be able to help...

I have an invalid MAC address that is flapping and continously looping between in aVLAN of one of our remote sites LAN. The mac address is

6000.86dd.6000 and I can't find it on any access port of any switch in my network, with no idea of what to generate. I mention that I have rapid-pvst enabled on all switches and that the STP topology is stable, with the proper switches blocking the proper ports.

The core stack CPU is at 90% CPU with the below process being the main resource drain:

69    290298741597995435         18 13.25% 10.89% 10.59%   0 HLFM address lea

The log buffer is full of these messages, but only on the core switch, as the other access switches see that mac on the uplink ports only, and no flapping is detected.

Dec 20 19:38:39 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi1/0/25 and port Gi2/0/26

Dec 20 19:38:54 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi1/0/25 and port Gi2/0/26

Dec 20 19:39:09 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi1/0/25 and port Gi2/0/26

Dec 20 19:39:24 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi2/0/26 and port Gi1/0/25

The core switch is a 2 3750 stack and all the access switches are 2960S stacks (3-4 switches per stack).

For a better understanding of this issue, I have attached a network diagram and some command outputs.

I already tried clearing the cam tables simlutaneously, but with no effect (the siwtches forward frames a lot faster than me sending the clear commands from the ssh sessions).

I would appreciate any idea for solving this issue.

[gert00002]

Hi Razvan,

I seen this often on misconfigured ports. The trunk forward the packets and on the other side they are received as "not trunk packets", and thus the same mac seem to enter the Vlan at 2 places.

It can also be produced by creating a monitor session on one switch and send the packet to an edge port on another switch, that will cause massive duplicate packets. But I don't expect you to have that setup in a production network.

Gert

[Razvan Craciun]

Hi Gert,

You would be right on that one, I don't have this scenario on this network. Would you guys know where could this MAC originate from? It doesn't seem to be a legitimate MAC address to me...

[_dansmith_]

I know this is old, but I'd say it's a 'bond' which has either gone wrong or one of the NICs is faulty.

[paul driver]

Hello

This output suggests a stp.loop


Start by checking.your topology diagram then begining from the stp root switch progess to each switch which has a trunk interconnect

Evenually you should see a port that should be in a blocking state causing the loop

Sh interface.trunk
Sh spanning-tree vlan 75

Res
Paul

Sent from Cisco Technical Support Android App

[engineerangelo]

Hi, we are currently experiencing this exact same problem too. Anyone knows the solution to this?

[engineerangelo]

Answer: There is a bug on the 3750X IOS. The IOS used was 15.0(2)SE4. We upgraded it to SE7.

The bug states that these malformed IPv6 packets are being forwarded by blocking ports. That's why the switches suddenly has a high CPU.

[engineerangelo]

Hello,

It happened again, but this time with a different Invalid MAC address 48:22:86:dd:60:00. We are engaged with Cisco TAC but they are currently unable to find this MAC yet.

This just appears on the logs of the core switch with it appearing on the uplinks. From the access switches, it doesn't point to an access port.

The odd thing about this is the MAC address is an invalid one.

[Tom Vanhout]

Do you happen to have a wireshark capture of these packets?
 

[ridejio]

Hi,

Have you solved the issue? I need your help.

[paul driver]

Hello

This output suggests a stp.loop


Start by checking.your topology diagram then begining from the stp root switch progess to each switch which has a trunk interconnect

Evenually you should see a port that should be in a blocking state causing the loop

Sh interface.trunk
Sh spanning-tree vlan 75

Res
Paul

Sent from Cisco Technical Support Android App

[Razvan Craciun]

Hi pdriver,

I do not understand how a port in a blocking state can cause the loop. The Blocking state is supposed to break the loop, right? I have initially attached the output of the "Sh spanning-tree vlan 75" command. STP topology seems to have converged well and no loops seem to exist on the VLAN 75 STP topology.

The blocked ports for this vlan are on 29stk2 (Gi2/0/49) and 30stk2(Gi2/0/49):

---------------------------------------------------------------------------------------------------------------------------------------------------

mnla_29stk2#sh spanning-tree vlan 75

VLAN0075

  Spanning tree enabled protocol rstp

  Root ID    Priority    24651

             Address     8cb6.4f76.5780

             Cost        4

             Port        49 (GigabitEthernet1/0/49)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32843  (priority 32768 sys-id-ext 75)

             Address     b862.1fed.7d80

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi3/0/2             Desg FWD 19        128.110  P2p Edge

Gi3/0/47            Desg FWD 4         128.155  P2p

Gi3/0/48            Desg FWD 4         128.156  P2p

Gi1/0/17            Desg FWD 19        128.17   P2p Edge

Gi1/0/49            Root FWD 4         128.49   P2p

Gi2/0/37            Desg FWD 19        128.91   P2p Edge

Gi2/0/49            Altn BLK 4         128.103  P2p

---------------------------------------------------------------------------------------------------------------------------------------------------

mnla_30stk2#sh spanning-tree vlan 75

VLAN0075

  Spanning tree enabled protocol rstp

  Root ID    Priority    24651

             Address     8cb6.4f76.5780

             Cost        4

             Port        49 (GigabitEthernet1/0/49)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32843  (priority 32768 sys-id-ext 75)

             Address     b862.1fe3.9a00

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi3/0/31            Desg FWD 4         128.139  P2p

Gi3/0/48            Desg FWD 4         128.156  P2p

Gi1/0/49            Root FWD 4         128.49   P2p

Gi2/0/49            Altn BLK 4         128.103  P2p

---------------------------------------------------------------------------------------------------------------------------------------------------

[paul driver]

Hello

I am sorry you misunderstood my last post -when I stated " Evenually you should see a port that should be in a blocking state causing the loop "


I was saying a port that should be in a blocking state is currently forwarding causing the loop.

Most of the times this is caused by a misconfiguration of a access port and attaching a unwarranted switch/hub to the network introducing a loop

Have you also checked for any span sessions or the not so nice stp bpdufilter command applied to any access ports that now have a switch/or hub attached?

Res
Paul

Sent from Cisco Technical Support iPad App

[Razvan Craciun]

Yes Paul, soory for the missunderstanding. The loop is only contained in VLAN 75.

There is only one span session that is involving other VLANs, not vlan 75 (and it is configured on the core switch only). It is not configured for any of the trunk ports.

mnla_core_1#sh run | inc monitor session

monitor session 1 source vlan 50 - 65

monitor session 1 destination interface Gi1/0/24

monitor session 1 destination interface Gi2/0/14

monitor session 2 source vlan 50 - 65

monitor session 2 destination remote vlan 505

mnla_core_1#sh cdp ne Gi1/0/24

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

mnla_core_1#sh cdp ne Gi2/0/14

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

mnla_core_1#sh run int Gi1/0/24

Building configuration...

Current configuration : 122 bytes

!

interface GigabitEthernet1/0/24

description orecx-monitor-new

switchport access vlan 10

speed 1000

duplex full

end

mnla_core_1#sh run int Gi2/0/14

Building configuration...

Current configuration : 118 bytes

!

interface GigabitEthernet2/0/14

description orexc-monitor

switchport access vlan 10

speed 1000

duplex full

end

[Haihua Rong]

Hello Razvan,

Please attach a show logg from 29stk2  and 30stk2

Haihua