Please help me solve this - mac flapping log entries

lisacoody · ‎02-18-2016

I have an issue that is making me crazy. The situation is occurring in a student dormitory setting and the students are complaining about slow network. The college I work for does not provide wifi in the dormitory buildings so the students bring their own wifi routers. Each apartment has one Ethernet connection. On VLAN 208 I am seeing MACFLAP errors. Everything I read says to track down the mac address listed in the error and you'll find the problem. I tracked the mac and it's the VLAN interface on the core switch (0022.5593.b7c4). The errors are not constant - maybe 20-30 times a day on .136 and 2-3 times a day on .135. There are flapping errors on switch .133 in building 3a because this switch connects 3b to the core.

To try solving the problem, I opened a ticket with Cisco TAC. The engineer enabled portfast and bpdu guard to calm excessive topology changes. We waited a couple of days to see if there were any further TCN's (there aren't) so the engineer told me there's nothing wrong with the switches or their configuration and he couldn't help me any further. I'm still seeing mac flapping on two switches in one building and I'm not sure how to stop it.

To determine what type device is connected on the ports that are flapping, I enter the command "sh mac address-table int gi0/xx" and no mac address shows up. If I bounce the port then I can get the mac. All three devices are Netgear routers.

0/23 = a063.9173.4b27

0/13 = dcef.09d3.5cc0

0/38 = a063.9177.eb4d

Log entries from .136 in building 3b

Feb 18 01:25:54.470: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/50 and port Gi0/38
Feb 18 11:46:49.412: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/49 and port Gi0/13
Feb 18 12:21:54.900: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/49 and port Gi0/13
Feb 18 12:22:41.683: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/49 and port Gi0/13
Feb 18 12:26:36.431: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/49 and port Gi0/13
Feb 18 12:28:51.446: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/49 and port Gi0/13
Feb 18 12:36:14.156: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/49 and port Gi0/13

Log entries from .135 in building 3b

Feb 13 16:50:01.383: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 02:32:01.080: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 12:41:36.445: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 13:20:40.835: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 13:25:52.832: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 13:31:30.692: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 13:42:18.494: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 15 13:47:15.803: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 16 07:32:27.947: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 16 16:14:45.849: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49
Feb 18 01:25:54.657: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/23 and port Gi0/49

Log entries from .133 in building 3a:

Feb 16 12:59:47.673: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 16 20:49:21.316: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 16 22:28:53.308: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 16 23:52:22.910: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 17 06:53:19.753: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 17 14:33:08.168: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 17 18:51:59.170: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 17 19:09:54.011: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 17 21:28:17.838: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 17 22:16:38.835: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49
Feb 18 01:25:54.512: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.5593.b7c4 in vlan 208 is flapping between port Gi0/52 and port Gi0/49

Has anyone seen this before? Any assistance is greatly appreciated!

Lisa

Jon Marshall · ‎02-18-2016

Lisa

You can get mac flaps with wireless clients moving from one AP to another but the mac in your messages is the SVI on the core switch.

Simply put the routers are somehow creating a loop in your network which would also account for the slowness at times.

The switch in 3a is seeing the the mac address of the SVI on gi0/49 which is what it should be doing and is recording that mac address to that port.

Somehow in 3b the routers are looping that packet back so that 3a now sees the same mac address coming in on gi0/52 which it shouldn't and that is why you see those messages.

When you see those messages you need to have a look at the routers and see exactly how they have been configured because they are creating a loop.

Jon

lisacoody · ‎02-19-2016

Thanks for your input, Jon. I appreciate you taking the time to respond to my query.

What you said makes sense. I'll take a look at the students' equipment and see if what I can find out.

Lisa

lisacoody · ‎02-19-2016

I was able to look at one of the students' equipment. Her computer and router were wired and configured correctly. I copied the contents of her router's syslog and found some interesting info there, but I don't understand what it means yet.

She has what looks like a brand new laptop with a trial version of antivirus software. The antivirus definitions are out of date so I suspect her laptop might be infected. Could her laptop be the cause of the mac flapping problem?

I spanned her port got a wireshark capture file. I'm not sure if it is acceptable to upload it to this forum.

Thanks in advance!

Lisa

Jon Marshall · ‎02-19-2016

I am not a security expert but that does not look good so I would get her laptop up to date as quickly as possible.

However there are other users accessing the device by the looks of it so it may not be her.

How exactly does this setup work ie. if it is a router where are they getting their IPs from and how do you control that and how does it relate to the vlan you connect the routers to ?

Is internet connectivity centralised ?

The log may be the cause or it may be something else, difficult to say without knowing more about your network setup.

Yes you can upload packet captures here.

Jon

lisacoody · ‎02-19-2016

Thanks, Jon. The core switch in the diagram (building 1) hands out DHCP addresses. That switch connects to an ASA, then out to the Internet.

I strongly suggested to the student that she install antivirus software on her computer ASAP (I gave her a link to Avast).

I tried several times to upload the packet capture (zipped) and it failed each time.