Ghost in the wire? MACFLAPPING

[Garrison Botts]

I've got an issue that even TAC is having trouble finding the answer to:


I have a customer with a small campus.  At the main agg site, there is a stack of 3650s which also does their routing. At the smaller buildings, there are individual 3650s tied in via gig fiber (single uplink).  They have a helpdesk in the main site also on a 3650 that is connected via cat6 copper.  There are about 20 vlans with vlan 995 being the VoIP vlan.  PCs are connected through the phone PC ports.. 

Issue:

One of the buildings on the campus (call it Bldg A) is having voice issues where the phones just resets by itself. Looking into this, we see logs showing:

(Campus Agg switch)

%SW_MATM-4-MACFLAP_NOTIF: Host 0007.7d42.ccae in vlan 995 is flapping between port Gi1/0/9  (Uplink to Helpdesk Switch) and port Te4/1/4 (Uplink to smaller bldg)

(Bldg A switch)

%SW_MATM-4-MACFLAP_NOTIF: Host 0007.7d42.ccae in vlan 995 is flapping between port Gi1/0/6 (User Station) and port Gi1/1/1 (Uplink to Agg Switch)


Helpdesk  switch shows nothing in the logs as far as MACFLAPs

When I go to Bldg A switch, I see this during that time (it goes back to 1/0/6 for both when good) :

nX-C-CHA-2#sh mac add add 0007.7d42.ccae
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
407 0007.7d42.ccae DYNAMIC Gi1/0/6
995 0007.7d42.ccae DYNAMIC Gi1/1/1
Total Mac Addresses for this criterion: 2

Vlan Mac Address Type Ports
---- ----------- -------- -----
407 0007.7d42.ccae DYNAMIC Gi1/0/6
995 0007.7d42.ccae DYNAMIC Gi1/0/6
Total Mac Addresses for this criterion: 2

Unplugging the Helpdesk switch - MACFLAPs go away.  Plugging the uplink in without ANY other ports connected - MACFLAP begins again.. 

Things we've tried:

Packet captures on Helpdesk Switch - Cisco has and saw nothing.

Down rev'ing the Helpdesk Switch. 
Clearing arp/ Rebooting ALL switches.
Replacing Helpdesk switch - this is a brand new switch stack and Bldg A switch less than a year old as well.

Did a complete site walk to ensure there were no hubs/desk switches connected. 
Spanning-tree shows the proper root ports to the 995 vlan bridge  and all other phones seem to be working fine. 

Thoughts? 

[chrihussey]

Just some things to look at:

1- Are you manually pruning and allowing only the necessary VLANs on the trunk links?

2- Is the native VLAN on the trunk links set correct on all sides?

3 - Is it just one, some or all phones at Bldg A?

4- Is there the possibility of a phone at building A with both ports (LAN and PC) plugged into the switch?

5- Would it be possible to provide some of the configs of the help Desk, Bldg A and Agg site?

Thanks

[Garrison Botts]

1) Yes.. I have tried manually pruning as well as having it open. Cisco TAC and I have tried everything. Nothing seems to explain it.

2) Yes... The customer has left the native vlan default. 

3)  There are 45 phones in Bldg A at least.. There are only 3 that are having this macflap issue. It's truly a crazy thing. All different models of phones too..

4) Our site walk of Bldg A and the Helpdesk area verified that everything is wired properly. This just one day started happening. No reason.. I've rebooted the Callmanagers too..  The only thing I haven't done is looked into the built in Wireless Lan Controllers on the switches. I was thinking "maybe" they are creating some type of bridge. The only problem with that is this is just happening on the voice vlan.   Very crazy stuff...

5) I can upload the configs after I sanitize.. But Cisco TAC spent 20 hrs looking and saw nothing.. Gimme a few to do so...