MAC-FLAP

[switched switch]

Hi All,

Had an issue today and needs some assistant.
Topology is connected like follows.

SWITCH1 -> SWITCH2 -> SWITCH3 -> SWITCH4 gi 1/0/24 -> IP DEVICE

IP DEVICE was firmware upgraded today, and following upgrade, the mgmt interface failed to come alive of IP DEVICE (http and ping etc).
In SWITCH4 logs, I saw some MAC-FLAP logs between 1/0/24 and the uplink from SWITCH4 to SWITCH3.
What I noticed was the MAC address for IP DEVICE wasn't being learnt on gi 1/0/24 on SWITCH4 as I expected. I did a mac trace to find the location of it, and found SWITCH2 had it pointing to SWITCH1, yet on SWITCH1, the MAC address wasnt assigned to any port and the mac address table didnt contain it.

I tried to clear the mac on every switch device but it immediately reappeared (using command clear mac address-table dynamic address H:H:H).
As it was only mgmt affected, I left IP DEVICE disconnected for about an hour but the MAC address didnt time out and remained in every switch (pointing direction to SWITCH1)

I fixed it by restarting SWITCH1, and immediately it started working again.

What caused this?
How could I have resolved this without restarting switches? (production environment).
What other steps can I carry out to troubleshoot further?

[Mark Malone · Accepted solution]

Hi

I would have cleared the arp for that mac too at layer 3 then cleared the mac at layer 2 , if it still persisted after that I would have tried statically fix it to the port rather than rebooting as Prod switch , sounds like the layer 2 mac sync wasn't working for whatever reason between the switches was stuck,  it could have been some software issue , you could try replicate it but may be more trouble than its worth now since you rebooted it , these switches aren't looped back at all together through redundant stp links ? are the timer defaults for the mac table the same on each switch , they haven't been altered and there is definitely no static macs set for that ip device anywhere on the local lan , if not I would think something went bit screwy alright and the reboot cleared the issue

[switched switch]

Hi mark,

sorry for the late response and thanks for the reply.

i did clear the ip arp at the router as well but not at the same time as doing the switch, maybe 20 mins later or so. Not sure if I should have done them immediately after each other but I wouldn't think so.

thanks for the comment re statically setting it, do switches override dynamic entries with static mac adresses? Ie it it was learning dynamic via one interface and static via another nterface where will it send the traffic?

the switches are looped back via another path although this path wasn't seen by any l2 devices. Re timers, by default are all the same? I haven't before set different timers for mac caching.

its through incidents like these that make me feel like a less than adequate engineer especially with a boss looking over your shoulder!!

[Mark Malone]

Hi

i did clear the ip arp at the router as well but not at the same time as doing the switch, maybe 20 mins later or so. Not sure if I should have done them immediately after each other but I wouldn't think so.

that should of been good enough once the entry was removed at l3 boundary whichi presume is the router not the switch

thanks for the comment re statically setting it, do switches override dynamic entries with static mac adresses? Ie it it was learning dynamic via one interface and static via another nterface where will it send the traffic?

it should override it from memory , easily checked on a standard port like a phone , set the static mac to match it for the port and check the table it should not have a dynamic entry anymore but I cant say for certain in the issue with you had as it sounds like there was some form of software issue there

look things break all the time even without engineers interaction but actually less with Cisco devices in my opinion but there is no escaping faults unfortunately

what is switch1s current software version and is it the same as the other LAN switches ?