Mac Address Flapping Tripping Port Security

[Seth B]

Starting on Monday, 7/11 @ 5:12 PM, our network has been seeing a sudden widespread storm of port security alerts across dozens of sites and ranging across all ages and models of Cisco devices - 3548, 3550, 2960S, 3560, and new 9300. Traditionally, we would normally only see 1 or 2 per week as someone might manually move a device. I'm looking for some insight on how to hunt down the root cause of this activity.

Port Security is traditionally enabled as such:
switchport port-security
switchport port-security maximum (insert number here)
switchport port-security aging type inactivity
switchport port-security aging time 2

What we are seeing in syslog activity is mac address roaming, or flapping. Mac address on interface 1 may roam to interface 7, and then 15, and then 38. However, after removing port security and clearing the cam table, the MAC address will show on it's original interface and be back on #1. This seems to be impacting computers only - we have not seen any mac addresses for phones, credit card machines, or printers included in this. 

Things we have performed so far:
*clearing port all and bouncing interfaces - does not resolve
*removing port security, clearing mac table, and re-applying - does not resolve
*removing port security and leaving off config - bypasses problem for stability. However, we do not see any mac bouncing in logs tripping STP, we see no loops in LLDP, nor do we see any loops in CDP
*physically went to site to search for rogue devices - none found
*performed a wireshark but didn't see any odd results
*performed IOS upgrades on some of the equipment and reloaded - activity continued
*reached out to our security office to inquire about active scans run - there are several active scans every week, and one of the scans lined up with our activity, but did not hit any of the subnets in question. Vendor is Qualys
*introduced IP device tracking into the port-security config and enabled port-security debugging to physically see MACs aging, being released, and moving. Activity all looked normal, but by going by syslogs, its as if machines are physically being moved even though they are not.

The only next steps I can think of is 1) spanning one of the switches and performing another wireshark  2) having one of the department IT reimaging a handful of machines to see if the MACs on those devices still roam around.

Any advice is greatly appreciated.

[MHM Cisco World]

connect AP to SW make this issue. 

[Seth B]

There are no APs in play here. These are hardwired computers. There are no physical loops or dual homed APs. We did find a site that used a lot of splitters/hubs and we had to power cycle them to clear their cache and allow them to let go of MAC addresses, but that site was a unique situation. We have others that don't have any splitters/hubs at all and it is just a computer hard wired directly to the jack - no inline phones or anything else.

[MHM Cisco World]

if you interconnect two SW, the mac is learn from Trunk and it already learn from the port the PC connect to it this lead port security violation since the MAC now is learn from different port.
so check the trunk interconnect SW.

[Seth B]

Also no problems there. None of the flapping is between network devices. The end user mac addresses are only bouncing within ports of the same switch. We have dozens of sites which only have a single Cisco 9300 and we see bouncing on those, and then we have dozens of sites that have downstream 2960s for example, and we may only see bouncing on switch #2 but not the others at that site.  None of the mac addresses problems are going across the trunks.

At first I suspected Qualys security scans as we have an inventory scan and a Cisco rail scan performed on Mondays. The Cisco scan looks for any interface with an IP address assigned to it and tries to log into the device and run a number of commands all within 2 minutes, and then logs out. However, we have cross referenced subnets and the Cisco scan only seems to hit the 10.254.x.x VPN trunks from PE to CE, but is not actually scanning any user VLANs or SVIs, nor hitting any switch management 10.253.x.x subnets. I think the scans are now out of question, but still on the back of my mind. 

Suspect #2 was our SCCM office pushing out patches or Windows updates, but they reported none performed on Monday and were pushed out on Tuesday instead. I'd still like to have a couple of machines reimaged to base image to test it out.

Suspect #3 = rogue device. I havn't seen any activity in packet sniffing, and the security office hasn't caught anything in their software either.

I know there is no way for port-security to break across dozens and dozens (if not hundreds) of sites all at the same time without a reason. Any sort of hack or threat that you can think of that could cause these symptoms?

[W-ALI]

I configured it as below with no issue , also I added configuration of storm-control & bpduguard to avoid looping

switchport port-security maximum 3
switchport port-security maximum 1 vlan access
switchport port-security maximum 1 vlan voice
switchport port-security violation restrict
switchport port-security mac-address sticky
storm-control broadcast level 10.00
storm-control action shutdown
spanning-tree portfast
spanning-tree bpduguard enable

[Seth B]

We also keep STP, STP bpduguard, and STP root guard applied on all interfaces. Nothing trips if port security is removed. One thing we don't have is storm-control, but I've cleared counters at several sites and monitored broadcast as well as hardware CPU/memory levels to confirm everything looked normal.

[MHM Cisco World]

Hi, friend

I understand that there are many Site connect via L2VPN, 
but let me explain here one important thing,
there are two plane control and Data Plane.
You run STP BPDU guard , Root Guard and this work in control plane and depend on send receive BPDU...
Here you run STP but the L2VPN is not pass BPDU between Site, this make Control Plane never detect L2 LOOP,
then the Data Plane with storm-control detect the LOOP and also Port-Security.
SO 
you have L2 LOOP and you need to check connect to SP to allow BPDU OR solve the L2 LOOP by config single L2 domain in each Site.

[SinghRaminder]

What are the ports connected to that you are seeing the Mac Address Flapping ?

[Seth B]

It may change from site to site as our specific layout for voip sites are to daisy chain PCs from the phones. But the two sites I have specifically been troubleshooting do not have voip and the computers plug directly into the network jacks. The ports being tripped are the access ports for these computers. Sometimes I even see the MAC address flip security on it's own port (which I've never seen before). If I go to check the reason why it err-disabled, it shows "due to MAC xxx.xxx on interface x/x" but the computer was sitting there the entire time. Almost like it fell asleep and then woke up and port security didn't like it.

I've even raised the maximum number of devices allowed to 5 on each interface at my test site just in case some MACs are roaming, it would still provide sufficient space.

[SinghRaminder]

can you please post the log message here :