Re: MAC-FLAP causing High CPU

Claire1642 · ‎09-06-2018

I think there is an issue with a server configuration that I am seeing from my switch.

Im seeing constant MAC flap notifications in the log, and I think it might have caused outage symptoms last week. We rebooted the switch and it came back up straight away, but in hindsight I checked the crash-info files and there was nothing created to suggest an outage to the switch. The "outage" has happened twice now, so I really need some help to track down the issue.

Can anyone advise what I should check? I pasted a portion of the logs I see below:

Sep 6 14:19:43.654 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:19:53.679 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:19:54.093 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:20:09.230 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:20:10.011 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/37 and port Gi3/0/45
Sep 6 14:20:24.076 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:20:28.125 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:20:32.530 BST: %SW_MATM-4-MACFLAP_NOTIF: Host 4083.de32.846b in vlan 102 is flapping between port Te2/1/4 and port Te1/1/3
Sep 6 14:20:40.132 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:20:45.287 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:20:53.561 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:20:54.125 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:21:05.099 BST: %SW_MATM-4-MACFLAP_NOTIF: Host 4083.de32.846b in vlan 102 is flapping between port Te1/1/3 and port Te2/1/4
Sep 6 14:21:08.209 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:21:09.724 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:21:24.264 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:21:28.092 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:21:40.076 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:21:46.093 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:21:53.266 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:21:54.059 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:22:09.212 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:22:10.041 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:22:20.638 BST: %SW_MATM-4-MACFLAP_NOTIF: Host 4083.de32.de9d in vlan 102 is flapping between port Te1/1/3 and port Te2/1/4
Sep 6 14:22:24.030 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:22:28.126 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:22:40.012 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/37 and port Gi3/0/45
Sep 6 14:22:48.234 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:22:54.027 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:22:54.485 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:23:08.172 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:23:09.367 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:23:24.052 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:23:29.468 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:23:40.184 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:23:43.457 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:23:54.018 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:23:55.187 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:24:09.780 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:24:10.017 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:24:24.023 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:24:28.127 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi3/0/43 and port Gi4/0/43
Sep 6 14:24:40.047 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:24:53.380 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.495d in vlan 102 is flapping between port Gi4/0/43 and port Gi3/0/43
Sep 6 14:24:54.092 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/45 and port Gi3/0/37
Sep 6 14:24:59.405 BST: %SW_MATM-4-MACFLAP_NOTIF: Host 4083.de32.846b in vlan 102 is flapping between port Te2/1/4 and port Te1/1/3

Thanks so much!

Leo Laohoo · ‎09-06-2018

Track down where the two wired clients are connected to and shut down the port.

Claire1642 · ‎09-06-2018

I cant do that as the servers are providing critical services. There is not high CPU at the moment, but I suspect at the time of the reported "outage" there was, and I think it might be due to the mac flap.
I say "outage" because I see no evidence that the switch went down at all, but the reported symptom was that it did.

Do you think there could be some error with the server config?

andresfr · ‎09-06-2018

Hello Clare,

Are all the ports showing in the logs trunk ports? Can you share the configuration of some of those?

Make sure you're not having spanning-tree portfast on any of the trunk ports.

You can track those host by using the following command:

show mac add add bc30.5bd7.46cb <--- for example

Then, if you're having a switch connected to the interfaces showing in the MAC flapping notification, connect to that switch and run the same command. Repeat until you find that the output reports and access port instead of a trunk port. You might need to run the command a couple of times on each switch just to make sure that you're not getting a different interface everytime.

Regards,

Claire1642 · ‎09-06-2018

Thanks for your reply,

The ports are access ports and are connected directly to what I assume is a host with 2 connections. If I use one of the log notifications as an example

Sep 6 15:43:40.107 BST: %SW_MATM-4-MACFLAP_NOTIF: Host bc30.5bd7.46cb in vlan 102 is flapping between port Gi3/0/37 and port Gi3/0/45

interface GigabitEthernet3/0/37
description ### UKRP02 Port2 ###
switchport access vlan 102
switchport mode access
device-tracking attach-policy IPDT_MAX_10
et-analytics enable
spanning-tree portfast
spanning-tree bpduguard enable
end

interface GigabitEthernet3/0/45
description ### UKRP02 Port1 ###
switchport access vlan 102
switchport mode access
device-tracking attach-policy IPDT_MAX_10
et-analytics enable
spanning-tree portfast
spanning-tree bpduguard enable
end

Ive never seen a server report the same mac address on two ports though? Usually it would be a different MAC for each one.

andresfr · ‎09-07-2018

Are you teaming those two NIC cards/Ports on the server? does the server support STP BPDUs and/or etherchannel?

One option would be to configure those two interfaces on the switch as a port-channel if the server supports that (LACP or without negotiation - mode on). The other option would be, if the server supports dot1q encapsulation and STP, to configure those interfaces on the switch as trunk interfaces without portfast and bpdu-guard so the switch can block one of the ports. However, since you're already having BPDU guard, I don't think the server could be sending BPDUs because that would cause the interfaces to go into err-disabled and that's not happening, right?

Regards,

Leo Laohoo · ‎09-07-2018

@Claire1642 wrote:

I cant do that as the servers are providing critical services.

Track down those MAC addresses that are coming up on the logs. Are they going to a single, physical server with multiple links to a switch or stack?

IF they are, are the NICs configured for teaming or etherchannels?