Mac flap errors on stacked 3750G

Elton Babcock · ‎02-28-2013

Hey guys,

I currently have a stack of 3750G switches with a cross stack etherchannel connected to a 3750E stack in the distribution layer.

The 3750G stack is running 15.0-2SE IP Base. I began noticing the Mac flap issue when I was Re-enabling dot1x on the switchports. As first I thought dot1x maybe have contributed to this so I removed it from the stack temporarily.

I am still seeing Mac flap logs usually when a machine reboots or re-connects back to the network. These are all wired desktops.

I started looking at the etherchannel configuration which is using LACP. The 3750E stack looks fine with all ports in mode active. On the problematic 3750G stack I noticed 3 of the ports in the etherchannel set to mode active and 1 port set to mode passive. The port shows as bundled but I can't imagine this is ok.
The only other difference on this one port is there is no mls qos commands like on the other 3 interfaces in the port channel.

I mainly need to know what the Mac flapping is about. Whether it is a bug or related to the current etherchannel configuration.

Thanks in advance,

Elton

Sent from Cisco Technical Support iPhone App

Elton Babcock · ‎03-01-2013

Anyone have anything on this? Please help.

Thanks, Elton

Sent from Cisco Technical Support iPhone App

Leo Laohoo · ‎03-01-2013

MAC Address flapping, particularly on a stack switches or a chassis, means that the downstream device's NIC teaming configuration isn't doing right.

Can you post the output to the command "sh etherchannel summary"?

Wilson Bonilla · ‎03-01-2013

Mac flapping means that a Mac address is being learnt across two different physical interfaces simultaneously, this most likely is a STP, layer 2 loop, Do show log in your switch and check the interfaces de flapping is coming from, make sure those interfaces are properly configure and verify the spanning tree topology.

By the way check who is the manufacturer of the Mac address flapping that would help to understand what's is the end device having issues.

Regards
Wilson B

Sent from Cisco Technical Support iPhone App

Elton Babcock · ‎03-03-2013

It really seems to be random ports that are flapping and at random times. The flaps show ip in the log are not consistant with any behavior on the switch. Here is a portion of the log when no one was in the office over the weekend.

412825: Mar 2 20:30:40: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi2/0/9

and port Gi3/0/48

412826: Mar 2 20:31:08: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25

and port Gi3/0/48

412827: Mar 2 20:31:41: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi3/0/14

and port Gi3/0/48

412828: Mar 2 20:32:04: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/19

and port Gi3/0/48

412829: Mar 2 20:32:29: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25

and port Gi3/0/48

412830: Mar 2 20:32:45: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25

and port Gi3/0/48

412831: Mar 2 20:33:17: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi3/0/14

and port Gi3/0/48

412832: Mar 2 20:33:24: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/10

and port Gi3/0/48

412833: Mar 2 20:33:53: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25

and port Gi3/0/48

412834: Mar 2 20:34:05: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25

and port Gi3/0/48

412835: Mar 2 20:36:08: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/17

and port Gi3/0/48

412836: Mar 2 20:36:29: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi2/0/9

and port Gi3/0/48

412837: Mar 2 20:36:59: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi3/0/2

and port Gi3/0/48

412838: Mar 2 20:37:06: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi2/0/13

and port Gi3/0/48

412839: Mar 2 22:18:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to dow

n

43N-3750-ST1#show etherchannel summary
Flags: D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 1
Number of aggregators: 1

Group Port-channel Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/49(P) Gi1/0/50(P) Gi2/0/49(P)
                                 Gi2/0/50(P)

412825: Mar 2 20:30:40: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi2/0/9
and port Gi3/0/48
412826: Mar 2 20:31:08: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25
and port Gi3/0/48
412827: Mar 2 20:31:41: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi3/0/14
and port Gi3/0/48
412828: Mar 2 20:32:04: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/19
and port Gi3/0/48
412829: Mar 2 20:32:29: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25
and port Gi3/0/48
412830: Mar 2 20:32:45: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25
and port Gi3/0/48
412831: Mar 2 20:33:17: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi3/0/14
and port Gi3/0/48
412832: Mar 2 20:33:24: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/10
and port Gi3/0/48
412833: Mar 2 20:33:53: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25
and port Gi3/0/48
412834: Mar 2 20:34:05: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/25
and port Gi3/0/48
412835: Mar 2 20:36:08: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi1/0/17
and port Gi3/0/48
412836: Mar 2 20:36:29: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi2/0/9
and port Gi3/0/48
412837: Mar 2 20:36:59: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi3/0/2
and port Gi3/0/48
412838: Mar 2 20:37:06: %SW_MATM-4-MACFLAP_NOTIF: Host 6431.508a.f10f in vlan 66 is flapping between port Gi2/0/13
and port Gi3/0/48
412839: Mar 2 22:18:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to dow
n

As you can see multiple ports are flapping. On this switch it is limited to mostly desktop machines. It appears it is only one MAC in the log but I do see multiple MACs at various times during the day.

Here is an output of the "show etherchannel summary" on the 3750G end.

43N-3750-ST1#show etherchannel summary
Flags: D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 1
Number of aggregators: 1

Group Port-channel Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/49(P) Gi1/0/50(P) Gi2/0/49(P)
                                 Gi2/0/50(P)

Here is the etherchannel summary from the other end on a 3750E switch.

43N-3750E#show etherchannel 11 summary
Flags: D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 11
Number of aggregators: 11

Group Port-channel Protocol    Ports
------+-------------+-----------+-----------------------------------------------
11     Po11(SU)        LACP      Gi1/0/21(P) Gi1/0/22(P) Gi2/0/21(P)
                                 Gi2/0/22(P)

One of the interfaces on the etherchannel on the 3750G switch is configured from the rest and I have a feeling this may be causing the issue but I cant be sure.

interface GigabitEthernet1/0/50

description C3-43N

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,16,66,90,91,98,99,166,901

switchport mode trunk

srr-queue bandwidth share 1 30 35 5

priority-queue out

mls qos trust dscp

auto qos trust dscp

channel-group 1 mode active

spanning-tree portfast disable

interface GigabitEthernet2/0/50

description C3-43N

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,16,66,90,91,98,99,166,901

switchport mode trunk

channel-group 1 mode passive

spanning-tree portfast disable

Otherwise I am stummped on this issue. I need to eventually re-apply dot1x configuration to this switch but I need to get the mac flapping issue under control first. Also note that there is no connectivity issues on this switch at all.

Thanks, Elton

smogra · ‎03-04-2013

Trace the MAC back to its source. Look for:

- A link flapping on a upstream switch, causing spanning tree TCNs and Spanning Tree reconvergence.

- A unidirectional link on an upstream switch causing the loop.

- A hub or switch connected to a portfast enabled access port where this mac is learned. Shut this port down and see if this breaks the loop.

Regards,

Sweta

Wilson Bonilla · ‎03-04-2013

Hello

6431.508a.f10f belogs to a HP device.

What is connected to Gi3/0/48 and Gi1/0/25?

What is the configuration of Gi3/0/48 and Gi1/0/25?

What is vlan 66?

Regards!

Wilson B.

Elton Babcock · ‎03-04-2013

That MAC address is actually an HP laptop.

Connected to both gi3/0/48 and Gi1/0/25 are some dumb switches with a few desktops/servers in our servers and help desk area.

Here is the configuration of both:

interface GigabitEthernet1/0/25

description EE ITtech

switchport access vlan 66

switchport mode access

no cdp enable

spanning-tree portfast

interface GigabitEthernet3/0/48

description SYSLAN HUB

switchport access vlan 66

switchport mode access

spanning-tree portfast

Note that even though it says HUB it is really a switch. I have verified that there are no physical loops on these two switches.

As for VLAN 66 this is one of our user VLANs.

Thanks, Elton

Elton Babcock · ‎03-22-2013

Hey guys,

I would like to re-work on this issue as I am still having issues with mac flapping on this one particular switch stack. I have confirmed there are no physical loops on this switch stack.

I mostly if not all times see the mac flaps in relation to when a machine reboots and their link status goes up and down a few times.

I have verified that our etherchannels are configured correctly on both ends and everything appears fine.

If anyone else has any ideas I would love to hear them. I am now beginning to wonder if this is a bug on this version of code (15.0(2)SE) for the 3750G switch. We are only running this code on this switch stack for testing. From web searches it doesn't appear that this is an issue anywhere else.

Please let me know if you have any more suggestions.

Thanks, Elton

Elton Babcock · ‎06-20-2013

I just wanted to update everyone on the mac flapping issue that I was having a while back. I finally have it resolved and it actually had nothing related to the Cisco equipment.

This issue was related to Microsoft SCCM Wake up Proxy. See the links below to better explain what the wake up proxy is.

http://technet.microsoft.com/en-us/library/dd8eb74e-3490-446e-b328-e67f3e85c779#BKMK_PlanToWakeClients

https://supportforums.cisco.com/thread/2220560

wgurney · ‎07-09-2013

Hello Elton,

Thanks for posting this resolution! Had this problem on a customer network this weekend and was chasing a loop. Logs going crazy. Thankfully came across your post.

The customer disabled the wake up proxy using SCCM on the call centre PCs. Problem gone! Appreciated!

Sent from Cisco Technical Support iPhone App

Elton Babcock · ‎07-09-2013

Great to hear my post helped. I have been helped so many times on here it is nice to hear my post helped someone.

Leave it to Microsoft to create a product that causes instability in the network. Apparently they forgot that layer 1-3 comes first.

Sent from Cisco Technical Support iPhone App