I'm at a loss to explain this one. We just lost power to one of the switches in our Catalyst 3750 stack (of course it had to be the master) and another took over as master. From this stack, we have a two port etherchannel to a stack of Brocade VDX switches. When the new master took over (switch 7 in this case), I noticed in the logs a handful of MAC addresses constantly MAC-flapping. The thing is that they were MAC-flapping between the port-channel and one of the physical interfaces of that port-channel. It was only happening on one of the port-channel interfaces, not both and it was on the switch that became the master...
Mar 24 16:36:38.342: %SW_MATM-4-MACFLAP_NOTIF: Host 14fe.b5d1.3bc0 in vlan 112 is flapping between port Gi7/0/48 and port Po2
Mar 24 16:36:40.515: %SW_MATM-4-MACFLAP_NOTIF: Host 8c7c.ff6e.3f00 in vlan 119 is flapping between port Gi7/0/48 and port Po2
Mar 24 16:36:40.699: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5dbe.0c05 in vlan 495 is flapping between port Gi7/0/48 and port Po2
Mar 24 16:36:48.165: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5dbe.1a5a in vlan 670 is flapping between port Gi7/0/48 and port Po2
Mar 24 16:36:50.640: %SW_MATM-4-MACFLAP_NOTIF: Host e41f.13d9.9df9 in vlan 111 is flapping between port Gi7/0/48 and port Po2
Mar 24 16:36:51.202: %SW_MATM-4-MACFLAP_NOTIF: Host 000a.f750.28a0 in vlan 112 is flapping between port Po2 and port Gi7/0/48
Mar 24 16:36:51.202: %SW_MATM-4-MACFLAP_NOTIF: Host 000a.f750.28a0 in vlan 115 is flapping between port Po2 and port Gi7/0/48
Mar 24 16:36:52.234: %SW_MATM-4-MACFLAP_NOTIF: Host 8c7c.ff5c.6301 in vlan 115 is flapping between port Po2 and port Gi7/0/48
Mar 24 16:37:03.986: %SW_MATM-4-MACFLAP_NOTIF: Host 8c7c.ff6e.3f00 in vlan 119 is flapping between port Gi7/0/48 and port Po2
Mar 24 16:37:04.238: %SW_MATM-4-MACFLAP_NOTIF: Host 14fe.b5d7.d08e in vlan 112 is flapping between port Po2 and port Gi7/0/48
Mar 24 16:37:04.632: %SW_MATM-4-MACFLAP_NOTIF: Host 001d.d8b7.1ca1 in vlan 566 is flapping between port Gi7/0/48 and port Po2
#show ether 2 detail
Group state = L2
Ports: 2 Maxports = 8
Port-channels: 1 Max Port-channels = 1
Protocol: -
Minimum Links: 0
Ports in the group:
-------------------
Port: Gi7/0/48
------------
Port state = Up Mstr In-Bndl
Channel group = 2 Mode = On Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 0 Load = 0x00 Protocol = -
Age of the port in the current state: 369d:04h:28m:41s
Port: Gi8/0/47
------------
Port state = Up Mstr In-Bndl
Channel group = 2 Mode = On Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 0 Load = 0x00 Protocol = -
Age of the port in the current state: 8d:08h:17m:35s
Port-channels in the group:
---------------------------
Port-channel: Po2
------------
Age of the Port-channel = 394d:09h:41m:43s
Logical slot/port = 10/2 Number of ports = 2
GC = 0x00000000 HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = -
Port security = Disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Gi7/0/48 On 0
0 00 Gi8/0/47 On 0
Time since last port bundled: 8d:08h:17m:48s Gi8/0/47
Time since last port Un-bundled: 8d:08h:22m:56s Gi8/0/47
This was causing some of our hosts to alternate between being reachable and unreachable. I decided to shutdown gi7/0/48 which stopped the MAC flapping and resolved connectivity, but what I noticed in the logs when I did this was the following internal error:-
Mar 24 16:49:30.352: %LINK-5-CHANGED: Interface GigabitEthernet7/0/48, changed state to administratively down
Mar 24 16:49:29.724: %PM-3-INTERNALERROR: Port Manager Internal Software Error (*prev != NULL: ../switch/pm/pm_port_sm.c: 3344: pm_del_port_from_agport) (N16FujitsuSwitch.com.au-1)
-Traceback= 1906284 19589FC 1959050 1730250 1732358 16B6284 17B2548 17B275C 1B8C2EC 1B82D78 (N16FujitsuSwitch.com.au-1)
Mar 24 16:49:29.724: %PM-3-INTERNALERROR: Port Manager Internal Software Error (ag->nextInBundle == NULL: ../switch/pm/pm_port_sm.c: 3476: pm_port_un_bundle_remote_port) (N16FujitsuSwitch.com.au-1)
-Traceback= 1906284 1959088 1730250 1732358 16B6284 17B2548 17B275C 1B8C2EC 1B82D78 (N16FujitsuSwitch.com.au-1)
To me, it looks like gi7/0/48 both was and wasn't a member of the port-channel which was causing the issues seen. I re-enabled the interface and it joined the port-channel correctly and continued operating without any issues - no more MAC flapping seen.
I've done some searching for the above internal error but can only find a vague description of what the error is about and suggests doing a show tech-support and getting Cisco to analyse, but since these switches are EOL, we can no longer get support for these switches. I'm happy that everything is working again but I'm wondering if there's something I could be doing to avoid this happening again (other than stopping techs from knocking power cables out of switches).
Incidentally, software version on the switches is 12.2(53)SE.
Thanks,
Mark J.
