Phantom MAC popping port security

[Karl Jacobsen]

I work at a college and we set up port security on all our switches on our network. We have all 3000 series switchs (mostly 3750 and 3560) connected by fiber to each building from our core (also 3750G). We run port security mainly to keep endusers from from moving their IP phones from office to office or building to building without the IT department knowing (911 concerns). All the switches are running IOS 12.2 and the standard is set up to accept 2 sitckied MAC address (one for the computer and one for the phone) and to shutdown if the MAC detected doesn't match the stickied MAC or if the same MAC shows up on two different ports.

The problem I'm seeing is an end user comes to work in the morning and turns on their machine. The link is established and the switch confirms the MAC address on the port. Recently, the port has been popping because it detected an unexpected MAC on the port. According to the log the MAC that is picked up belongs to a desktop on a switch in another building on campus. In fact, the desktop in question has never been connected to the switch with the error. From the little I understand about port security, it doesn't check across switches, correct? This has happened several times in the last couple of months; we usually see one a week. It has happened to different endusers in different buildings but always when the machine is first turned on and always the same offending MAC address. I don't believe this to be a case of malicious MAC address spoofing. If I perform a shut/no shut on the port the error clears. I've tried clearing the MAC address table on the switch but, so far, no luck.

Any ideas? Is it an IOS version issue?

[andysainsbury]

Can you post a config snippet of the interface with the issue as this would help diagnose the problem.

I setup mac/port based security like this a couple of years back and had exactly the same problem, which seems unrelated but actually is down to the way the switch is aging out the mac addresses.

We decided not to go for the 'sticky' method as it can cause issues when you save the config with write mem, if the host mac address changes, for example the user replaced the desktop with a laptop, the switch is stuck with the old mac.  This was going to be too much admin for us.

Also the aging issue was causing a pc being plugged into a totally different switch on the same router to cause an error, because the mac hadn't been removed from the CAM table of the switch with port security.

We now opt for any 3 mac addresses at one time on an interface:

interface FastEthernet0/5

switchport access vlan 111

switchport mode access

switchport port-security maximum 3

switchport port-security

switchport port-security aging time 20

switchport port-security violation restrict

switchport port-security aging static

no cdp enable

spanning-tree portfast

spanning-tree guard root

!

Be careful of mini hubs/switches!

If a mini hub is connected to a switch port, the mac addresses never get aged out, as the interface line protocol never goes down.

So if you build a pc off a mini hub in an office then deploy it out in a room, the mac is still active on the first switch with the mini hub, as it never gets flushed it causes a port security error.

Have a go with my config and see what happens.

Cheers,

Andy

[Karl Jacobsen]

Hi Andy,

     Thanks for the heads up. Here's a typical switchport config.... Usually the max number is set to 2, this one happens to be set to 3.

interface FastEthernet0/11
description KPCT205KB
switchport access vlan 124
switchport mode access
switchport voice vlan 132
switchport port-security maximum 3
switchport port-security
switchport port-security mac-address sticky
switchport port-security mac-address sticky 00e0.b8cc.44ff vlan access
switchport port-security mac-address sticky 0015.6278.801c vlan voice
srr-queue bandwidth share 10 10 60 20
srr-queue bandwidth shape 10 0 0 0
mls qos trust device cisco-phone
mls qos trust cos
auto qos voip cisco-phone
spanning-tree portfast
end

To be clear, the offending MAC has never been attached to this switch. The issues seem to happen to both laptop and desktop alike with no particular pattern.

[andrewswanson]

hello
    very odd sounding problem. this is a long shot, but there is a cdp enhancement called second port disconnect (or host movement detection) that allows cisco phones to pass on the status (up/down) of their access ports to the switch. this can affect port security:

"Upon receiving this notification, the switch can clear the security record for the PC."

and

"Upon receiving a host presence TLV notification of a link down on the IP phone's data port, port security removes from the address table all static, sticky, and dynamically learned MAC addresses. The removed addresses are added again only when the addresses are learned dynamically or configured."

page 12 of this doc shows supported phones/firmware

http://cisco.biz/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6638/config_guide_c17-605524.pdf

ios release notes (3750/3560) where enhancement was introduced

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps9703/product_bulletin_c25-523873_ps5012_Products_Bulletin.html

certainly doesn't explain the same mac causing violations on your switchports. did this behaviour start after an ios or cucm upgrade?

hth
andy

[Karl Jacobsen]

Interesting. The problem started about 6 to 8 months ago and happens maybe 2 to 3 times a month. It's pretty infrequent. We are running mostly 7940 and 7941 phones. I've seen this happen on both model number phones and both models are running the current SCCP firmware. It definately was not something that came about right after an upgrade of CUCM or the switch IOS.

[gchialliacn]

I apologize from bringing this post back after 4 years, but I'm running into a similar problem.

Have you ever found the root cause?

Tahnks

[Karl Jacobsen]

Unfortunately no. The problems eventually disappeared.

[Fvicente]

i got same issue

and i don't fidn where the problem come from

[Karl Jacobsen]

On a side note, the new nic card came in today for the machine that is supposed to be causing the ports to pop... It shouldn't be the culprit but I'll try replacing it.

[Karl Jacobsen]

Alright, interesting new development.... I swapped out the nic card in the machine that is supposed to be popping these ports. I swapped it out on March 1st and today the issue came back. I checked the log on the switch and the mac address that is reporting is another desktop compter in another building seperate from the popped port and the previous phantom culperit. GRRRRRRR!

What information is shared between switches? Is there any way the MAC address' stickied on a switch can be shared with another switch in a different building, but on the same vlan?