Port security maximum keeps allocating MAC addresses to ports

[AMACOMX]

We've configured port security in a 3650 switch using switchport port-security maximum 2, just a day after the configuration people started to state there is no network 

investigating with show interfaces status showed error-disabled for some ports 

checking show mac address-table interface gigabitEthernet x/x/x on the disabled ports gave a 2 mac addresses one of them is the intended machine and the other is not, say the intened is FF:F1 and then non intended is AA:AA

checking the mac address table on the other ports with the error-disabled status also showed the intended mac for the machine say FF:F2 but also has the second mac AA:AA

*each disabled port showed the the right machine and the AA:AA mac 

for curiosity we changed config to switchport port-security maximum 3, and shutdown then no shutdown, this locked the ports again and shows the intended mac FF:F1 and the unintended AA:A1 and another unintended AA:A2

we added the maximum to 5 and we get a variety of nice fake/unintended mac addresses, the environment is large and its hard to check if the unintended mac is a real machine with an issue 

also arp -a on the computers doesn't show the fake mac address

the issue is with stacked pairs of catalyst 3650 switches

the environment have citrix VDI running on HP thin-clients, printers and normal computers

the port security config is 

per port:

switchport port-security
switchport port-security maximum 2

global config:

errdisable recovery cause bpduguard
errdisable recovery cause link-flap
errdisable recovery cause psecure-violation
errdisable recovery cause mac-limit
errdisable recovery interval 1800

the version of the IOS running is:

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24PD 03.03.05SE cat3k_caa-universalk9 INSTALL

the question is, how do i troubleshoot this issue?

[paul driver]

Hello

What device is connected to that port(s) - The OUI suggests an HP device

[AMACOMX]

a HP computer but hte NIC is realtech lan card with another mac address, same goes to the rest of the ports

[Mark Malone]

Hi

that software is 5 years old should be updated to something newer and more stable, could be buggy behavior but i would check the OUI of these fake MACs what exactly it belongs too the vendor may give an idea where there coming from , the fact VDI is running may be the cause or if virtual systems are been ran on the PCs themselves so the 2 macs would be increased significantly

https://www.wireshark.org/tools/oui-lookup.html

[AMACOMX]

we agree about the old IOS, the OUI of the mac stated an HP card but we dont have HP nics in the network, ports with vdi machines are set to 2 max mac adresses which is sufficient i suppose 

[Mark Malone]

The MACs have to be coming from somewhere on the network , PS is doing what its setup to do but its seeing more than the allocated MACs on the port`, should be easy enough to trace a couple of them to confirm

[Giuseppe Larosa]

Hello AMACOMX,

to see the same MAC address on several ports is unlikely (unless virtual machine are cloned including their MAC address).

Also when you increase the MAC limit additional MAC addresses are seen.

The IOS XE version is quite old and I agree with Mark you need to upgrade the IOS XE  as first step.

You need also to verify how many MAC addresses per port are used with VDI if any virtual machine is running on the clients you may need more MACs allowed per port.

Hope to help

Giuseppe

[AMACOMX]

guess i'll start with discussing the IOS upgrade with the decision makers, and about the VDI i think 2-3 mac adresses are ok, but when we raised max to 5 it was populated with mac addresses. another point taken while we were investigating, when we removed port security and restarted the switch, each port have one (the right) mac address, this behavior happens only when the port security is configured 

[moe52689]

we have changed the entire switch to  WS-C2960X-48LPD-L  software version 15.0(2a)EX5 image name C2960X-UNIVERSALK9-M

the problem still persists with 5 mac addresses as maximum we found fake macs over ports and they got caught in error-disabled state

[Giuseppe Larosa]

Hello moe52689,

the initial switches were C3650?

you have moved to a stack of C2960 with a different IOS version and you still have the same issue?

Hope to help

Giuseppe

[moe52689]

The switch is a single it's not a stacked switch, but yes we changed the switch from C3650 to C2960X and we are still facing the same issue

same configuration as @AMACOMX have posted before 

Bare in mind that i have port fast and BPDU guard Configured on that switch per port

[Giuseppe Larosa]

Hello,

can you setup a SPAN session with source a port with this configuration to see if these "fake" MAC addresses are real  or not ?

I am not sure that is possible to use a port with port security as a source port for a SPAN session but I would try.

If you can demonstrate that those MAC addresses are not seen on wire you can say there is an issue on the switch side.

Hope to help

Giuseppe

[AMACOMX]

we are currently searching more into the problem and so far there are other posts with issues appealingly like our's, we are reading more into the span session to understand it more

[Georg Pauwen]

Hello,

on a side note, you might be undergoing some sort of MAC and/or ARP spoofing attack. You could try and enable DAI (Dynamic ARP Inspection) on your switches, and check if the behavior changes. Below is a sample config:

2960#conf t

2960(config)#ip arp inspection vlan x

2960(config)#interface range FastEthernet0/1 - 24

2960(config-if-range)#ip arp inspection trust

[AMACOMX]

we currently updated a switch's IOS to 3.6.10E, and renabled port security, so far 2 days the behavior didn't happen, but we didnt put the port security on the ports with citrix thinclients and printers yet. we are adding them phase by phase currently

so thank you all about recommending the IOS update... we will update you again when we add the port security on all ports and again when we test on another switch stack