Port Security violation, please help

radoslav-drabik · ‎12-28-2010

Hi,

I've got problem with port security on port Fast4/4. There is currently Cisco IP phone 7961 connected and nothing else. I still get PSECURE_VIOLATION. What can cause the problem? Please help. Thank you.

Here are my logs and configuration:

#show logging
Dec 27 10:21:05.631 CET: %PM-4-ERR_DISABLE: psecure-violation error detected on Fa4/4, putting Fa4/4 in err-disable state
Dec 27 10:21:05.639 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 27 10:24:05.646 CET: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa4/4
Dec 27 13:14:51.073 CET: %PM-4-ERR_DISABLE: psecure-violation error detected on Fa4/4, putting Fa4/4 in err-disable state
Dec 27 13:14:51.077 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 27 13:17:51.072 CET: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa4/4
Dec 27 14:32:39.083 CET: %PM-4-ERR_DISABLE: psecure-violation error detected on Fa4/4, putting Fa4/4 in err-disable state
Dec 27 14:32:39.087 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 27 14:35:39.081 CET: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa4/4
Dec 27 15:16:59.369 CET: %PM-4-ERR_DISABLE: psecure-violation error detected on Fa4/4, putting Fa4/4 in err-disable state
Dec 27 15:16:59.373 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 27 15:19:59.356 CET: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa4/4

#show mac address-table interface fasTEthernet 4/4
Unicast Entries
vlan mac address type protocols port
-------+---------------+--------+---------------------+--------------------
412 0023.339c.e1cf static ip,ipx,assigned,other FastEthernet4/4

Multicast Entries
vlan    mac address     type    ports
-------+---------------+-------+--------------------------------------------
112    ffff.ffff.ffff   system Fa4/1,Fa4/2,Fa4/3,Fa4/4,Fa4/6,Fa4/9,Fa4/10
                                Fa4/11,Fa4/12,Fa4/14,Fa4/15,Fa4/17,Fa4/18
                                Fa4/33,Fa4/35,Fa4/40,Fa4/41,Fa4/43,Fa4/44
                                Fa4/45,Fa4/46,Fa4/47,Fa4/48,Fa5/48,Gi1/1
                                Gi1/2,Switch
412    ffff.ffff.ffff   system Fa4/1,Fa4/2,Fa4/3,Fa4/4,Fa4/6,Fa4/9,Fa4/10
                                Fa4/11,Fa4/12,Fa4/14,Fa4/15,Fa4/17,Fa4/18
                                Fa4/33,Fa4/35,Fa4/40,Fa4/41,Fa4/43,Fa4/44
                                Fa4/45,Fa4/46,Fa4/47,Fa4/48,Fa5/48,Gi1/1
                                Gi1/2,Switch

#show port-security interface fastEthernet 4/4
Port Security              : Enabled
Port Status                : Secure-up
Violation Mode             : Shutdown
Aging Time                 : 1 mins
Aging Type                 : Inactivity
SecureStatic Address Aging : Disabled
Maximum MAC Addresses      : 3
Total MAC Addresses        : 1
Configured MAC Addresses   : 0
Sticky MAC Addresses       : 0
Last Source Address:Vlan   : 0023.339c.e1cf:412
Security Violation Count   : 0

#interface FastEthernet4/4
switchport access vlan 112
switchport mode access
switchport voice vlan 412
switchport port-security maximum 3
switchport port-security
switchport port-security aging time 1
switchport port-security aging type inactivity
no logging event link-status
load-interval 60
qos vlan-based
no snmp trap link-status
tx-queue 3
priority high
ip dhcp snooping limit rate 10
end

#show cdp neighbors fastEthernet 4/4
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID Local Intrfce Holdtme Capability Platform Port ID
SEP0023339CE1CF Fas 4/4 168 H P M IP Phone Port 1

#show power inline fastEthernet 4/4
Available:3700(w) Used:685(w) Remaining:3015(w)

Interface Admin Oper Power(Watts) Device Class
From PS To Device
--------- ------ ---------- ---------- ---------- ------------------- -----

Fa4/4 auto on 7.1 6.3 IP Phone 7961 2

Interface AdminPowerMax AdminConsumption
(Watts) (Watts)
---------- --------------- --------------------

Fa4/4 15.4 15.4

#show ver
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASEK9-M), Version 12.2(50)SG6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Dec-09 23:12 by prod_rel_team
Image text-base: 0x10000000, data-base: 0x11C3225C

ROM: 12.2(31r)SGA1
Dagobah Revision 226, Swamp Revision 34

cza-ua-12300a uptime is 34 weeks, 16 hours, 53 minutes
System returned to ROM by reload
System restarted at 21:28:42 CEST Mon May 3 2010
System image file is "bootflash:cat4500-ipbasek9-mz.122-50.SG6.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C4506 (MPC8245) processor (revision 10) with 262144K bytes of memory.
Processor board ID FOX1222GVRS
MPC8245 CPU at 266Mhz, Supervisor II+
Last reset from Reload
6 Virtual Ethernet interfaces
192 FastEthernet interfaces
2 Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

Mahesh Gohil · ‎12-28-2010

Hello,

According to the config this should not happen. You have allowed 3 vlan and you are learning only one so it should not cause

violation. Moreover the the aging time is also one minute.

I was searching some bug for ios you are running with. Only one bug which cause this kind of message is CSCsq34665

but the condition mentioned is different (I mean according to bug if you have configured dot1q , but in your case it is access port)

However the workaound shown is to disable LLDP on phone, you can try with this or finally you can log case with cisco or you can

try connecting phone with some other port and look for the behavior

Regards

Mahesh

rtjensen4 · ‎12-28-2010

I use Mitel phones (YUCK!) on my 4506s with SUP II+ and 12.2(53) SG1, and I have the same port-security setup you do. The only difference is that my aging time is longer. I know the Mitel phones stick themselves on BOTH vlans for a little bit as it boots up. Do you have a PC connected to the phone as well? Perhaps this is what's causing your problems.

I would try to raise the max allowed MACs up to maybe 5 and bring the port up. Try to do show port-security interface fastEthernet 4/4 a few times as it boots to see how many MACs the port sees. Do you have other 7961s in use that are working fine with this port-security setup.

radoslav-drabik · ‎12-28-2010

I have got 2x 48-ports blade with IP phones 7961 and all of them works fine with these settings. Just that one has this problem. I will try to plug this Ip phone to another port and when the issue persist I will replace it.

Is there anything I can do about it?

Leo Laohoo · ‎12-29-2010

Is there anything I can do about it?

I've never seen this setup happen on a Cisco IP phone (and I have LOTS!). So (please humour me) if it's only a phone connected to the port (and nothing else), can you please remove "switchport access vlan 112"?

Peter Paluch · ‎12-29-2010

Rado,

One more question: is it - by any means - possible that the MAC address 0023.339c.e1cf is already learned on another secure port? A port security violation occurs also if a secure port receives a frame whose sender's MAC is already learned on a different secure port.

Best regards,

Peter

ohassairi · ‎12-28-2010

port security violation occurs not only when you exceed the number of allowed mac

addresses but also if your device is causing an IP conflict!

so make sure the attached phone has not an already used IP address!

Peter Paluch · ‎12-28-2010

Hi,

Are you sure about the IP address conflict? The port security is fundamentally concerned with so-called secure MAC addresses and it should not be related to IP addressing. Can you kindly provide any documentation that would support your hypothesis? Thank you!

Best regards,

Peter

radoslav-drabik · ‎12-29-2010

Hi,

I've tried to change Violation mode to "Restrict" and today I have got this:

#show port-security interface fastEthernet 4/4
Port Security              : Enabled
Port Status                : Secure-up
Violation Mode             : Restrict
Aging Time                 : 1 mins
Aging Type                 : Inactivity
SecureStatic Address Aging : Disabled
Maximum MAC Addresses      : 3
Total MAC Addresses        : 1
Configured MAC Addresses   : 0
Sticky MAC Addresses       : 0
Last Source Address:Vlan   : 0023.339c.e1cf:412
Security Violation Count   : 3

#show logging

Dec 27 14:32:39.087 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 27 14:35:39.081 CET: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa4/4
Dec 27 15:16:59.369 CET: %PM-4-ERR_DISABLE: psecure-violation error detected on Fa4/4, putting Fa4/4 in err-disable state
Dec 27 15:16:59.373 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 27 15:19:59.356 CET: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa4/4
Dec 29 04:01:19.048 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.

On Thursday, I am going to plug this phone to another port and will see. If the problem persist I will replace that phone. I will let you know the result.

Thank you all for you responses.

Peter Paluch · ‎12-29-2010

Rado,

This does not make any sense If you read the output carefully it says that your port was err-disabled although you have set your reaction mode to Restrict instead of Shutdown! Now, a Restrict mode simply throws away "illegitimate" frames and logs them but it shall not move the port to the err-disabled state. How come, then, that your port was still err-disabled by port security?

To me, it sounds more like an IOS bug than a phone's problem. Are you able to perform an IOS upgrade? Replacing the phone may not prove helpful - after all, the phone cannot do anything special to trip the port security protection - it certainly does not "migrate" between VLANs wildly and it dilligently uses its own MAC address only.

Best regards,

Peter

radoslav-drabik · ‎12-29-2010

Peter,

I set the "restrict" mode yesterday 28th. The last log is from 29th and it didn't went to err-disable status. Besides of that I am using this IOS for plenty of switches with IP phones connected to them.

Can port security have anything with IP DHCP snooping?? Here is my output from "show logging". The last one was ommited just for Fast4/4.

Dec 28 16:39:36.567 CET: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPREQUEST, chaddr: 0022.fa42.d314, MAC sa: 001c.25a0.71e0
Dec 28 16:42:26.558 CET: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPREQUEST, chaddr: 0022.fa42.d314, MAC sa: 001c.25a0.71e0
Dec 28 16:43:52.556 CET: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPREQUEST, chaddr: 0022.fa42.d314, MAC sa: 001c.25a0.71e0
Dec 28 16:44:33.558 CET: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPREQUEST, chaddr: 0022.fa42.d314, MAC sa: 001c.25a0.71e0
Dec 29 04:01:19.048 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 0023.339c.e1cf on port FastEthernet4/4.
Dec 29 08:58:25.867 CET: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPRELEASE, chaddr: 0021.5c43.fb8f, MAC sa: 0021.8658.39f5
Dec 29 09:04:28.384 CET: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPREQUEST, chaddr: 0026.c629.41e4, MAC sa: 0022.6818.8204

ohassairi · ‎12-29-2010

yes i am sure about ip conflict. i had this problem in my company.

When there is an IP address conflict, the PC that generated the conflict will send out a gratuitous ARP, with the other PC's mac-address, just like ARP poisoning.

the PC is doing it to correct a problem it generated.

for example: PC1 has an IP address of 1.1.1.1 and a mac-address of a.a.a.a, he is downloading, surfing the internet etc. PC2 comes online with the IP address of 1.1.1.1 and a mac-address of b.b.b.b. PC2 then sends an ARP "Who has 1.1.1.1 tell 1.1.1.1 b.b.b.b".

When that happens, your routers ARP table can be poisoned, once PC1 replies with "I have 1.1.1.1" PC2 corrects any potential communication problems by sending a GARP with PC1's IP and mac-address.

So the switch will learn a.a.a.a in two ports: PC1 port and PC2 port.

According to configuration guide, this will trigger the portsecurity.

In fact: in configuration guide we can read:

It is a security violation when one of these situations occurs:

•The maximum number of secure MAC addresses have been added to the address table, and a station whose MAC address is not in the address table attempts to access the interface.

•An address learned or configured on one secure interface is seen on another secure interface in the same VLAN.

In our case we have the second scenario.

Peter Paluch · ‎12-30-2010

Hello,

Thank you for your explanation. However, I have to say I do not agree with it.

You are correct about quoting the Configuration Guide about port security violations. It has to be stressed that the point "An address learned or configured on one secure interface is seen on another secure interface in the same VLAN" is concerned with MAC addresses only, not with IP addresses. This is just to stress an important concept.

Your explanation of the Gratuitous ARP process (do not call it GARP, as this acronym refers to the Generic Attribute Registration Protocol - a completely different issue) is correct only up to the point of both PC1 and PC2 verifying whether anyone else on the segment has the same IP address by sending an ARP Request asking for the same IP address. However, all frames sent by PC1 will be sourced from MAC_1 (its own MAC address) and all frames sent by PC2 will be sourced from MAC_2. It is illegal for PC2 to ever send frames sourced from MAC_1 and vice versa. The contents of ARP messages, both requests and replies, are completely irrelevant to the issue - the switch learns MAC addresses from Ethernet headers only, and does not care about the frame payload. Thus also the port security is concerned only with source MAC address seen in the Ethernet header, not with MAC addresses seen in ARP message body. Sure, the situation would be different if we spoke about Dynamic ARP Inspection or about IP Source Guard but in this case, we are discussing solely the port security feature itself. All this means that it is impossible, even with an IP address conflict, that the same MAC address is learned on two different secure ports.

Moreover, I am not familiar with the "back off" sequence you have described regarding PC2 - that makes it "correct" the IP/ARP mapping by sending a Gratuitous ARP Reply stating back that the MAC_1 is the rightful owner of the IP address. I have not encountered any similar behavior (what was the operating system in use?).

Best regards,

Peter

ohassairi · ‎01-01-2011

hi peter

i agree with you that in theory (in books) IP conflict is not related to portsecurity but it seems to be an indirect cause of this mac address table conflict.

we are using cisco3750 as access switches and we are configuring port security in all access ports to allow only 2 mac addresses by port. the violation action was to shutdown the port.

in that time errdisable recovery was not configured

since we are not using dhcp, some times we fall in IP conflict and i was surprised that both ports go shutdown due to port security configuration (as seen form log).

in the begining, i also didn't find any relation between IP conflict and port security until i found the explanaition i pasted in ciscoforum (https://supportforums.cisco.com/message/629251#629251)

i reporoduced the problem and i captured traffic by sniffer in both computers (windows xp professional) and it was like it was discribed !

Peter Paluch · ‎01-04-2011

Hi,

Thank you for responding. I think we both agree on these points:

A switch learns MAC addresses from Ethernet frame headers, never from the frame payload
A computer never sends Ethernet frames sourced from a different MAC address than its own
The port security reacts only to MAC addresses learned in the CAM and seen in frame headers

These facts inevitably lead to the conclusion that if the port security disabled the port in the case of IP address conflict, the reason must have been different.

Is it possible that your switches were running Dynamic ARP Inspection (DAI) or IP Source Guard (IPSG)? I would definitely have to verify your observations in my lab and try to reproduce the behavior you have described. One thing is for sure: the Gratuitous ARP that "corrects" the wrong mapping is definitely specific for a particular operating system. I can say with 100% certainty that GNU/Linux does not behave this way.

Best regards,

Peter