cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1618
Views
1
Helpful
16
Replies

Cisco 9200 err-disable AUTHMGR-5-SECURITY_VIOLATION

117222400
Level 1
Level 1

Hi Expert,

Recently, one port on our 9200 switch went to err-disable state and the status LEDs are all off. 

The port configuration is as below: it is connected to a desktop PC

117222400_0-1728434470316.png

The logs is as below, it looks before it goes to err-disable state, the port up and down for many times. seems the user was rebooting the machine

117222400_0-1728441925852.png

 

during error, checked the controller:

117222400_3-1728434665278.png

117222400_4-1728434718331.png

117222400_5-1728434751371.png

117222400_7-1728434848657.png

 

The machine authentication in ISE is not responded:

117222400_0-1728436119355.png

 

I just found this link seems the same issue, but not sure why the pc's MAC changed?

https://github.com/inverse-inc/packetfence/issues/1588

It occurred the second time, and can be resolved by shutdown/no shutdown to reset the port.

But we still need to find out the root cause. Any ideas about it?

Thanks 

 

 

 

 

16 Replies 16

Leo Laohoo
Hall of Fame
Hall of Fame

Remove all the config off that interface and see how many MAC addresses are coming out of that port.  

A PC hosting VM is one suspect for having two (or more) MAC addresses.  Another is laptop dock because the dock itself has one MAC address and the laptop has another.

Thanks very much for your reply, the user is using a desktop and doesn't need a dock, I think he won't touch the network cable when it is working. It seems occured when he shut down/reboot the desktop. 

I set up a remote PS session to his desktop and only 2 adaptors found, no such MAC 0848.8905.3e52 found

117222400
Level 1
Level 1

 

Just found another port the same error at several days ago ( we moved the same desktop to the new port), but the MAC address is not the same

117222400_0-1728442761557.png

 

 

Also I checked the port-security, it seems not enabled, 

117222400_0-1728452578474.png

but the MAC address learnt from this port is showing "STATIC"

117222400_1-1728452591306.png

I assume the "sticky" is still on? or it is because below: authentication violation default is "shutdown"?

117222400_2-1728452781832.png

 

 

 

I need to see log detail 

Share it here 

MHM

"Date","Time","Facility","Level","Host Name","Message Text"
"2024-10-09","17:22:43","Local7","Notice","10.2.9.5","37464: Oct 9 17:22:43.162: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-09","17:22:43","Local7","Error","10.2.9.5","37463: Oct 9 17:22:42.162: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-09","17:22:39","Local7","Error","10.2.9.5","37462: Oct 9 17:22:38.936: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-09","17:22:38","Local7","Notice","10.2.9.5","37461: Oct 9 17:22:37.932: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-09","17:22:33","Local7","Notice","10.2.9.5","37460: Oct 9 17:22:33.894: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-09","17:22:33","Local7","Error","10.2.9.5","37459: Oct 9 17:22:32.893: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-09","17:22:28","Local7","Error","10.2.9.5","37458: Oct 9 17:22:27.552: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-09","17:22:27","Local7","Notice","10.2.9.5","37457: Oct 9 17:22:26.557: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-09","09:55:45","Local7","Notice","10.2.9.5","36587: Oct 9 09:55:44.726: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-09","09:55:44","Local7","Error","10.2.9.5","36586: Oct 9 09:55:43.724: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-09","09:55:36","Local7","Error","10.2.9.5","36583: Oct 9 09:55:35.900: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-09","09:55:33","Local7","Notice","10.2.9.5","36581: Oct 9 09:55:32.588: %LINK-5-CHANGED: Interface GigabitEthernet8/0/41, changed state to administratively down"
"2024-10-09","09:55:28","Local7","Notice","10.2.9.5","36579: Oct 9 09:55:27.970: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:interface GigabitEthernet8/0/41 "
"2024-10-08","16:46:47","Local7","Notice","10.2.9.5","35580: Oct 8 16:46:46.048: %AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface GigabitEthernet8/0/41, new MAC address (0848.8905.3e52) is seen.AuditSessionID Unassigned"
"2024-10-08","16:46:46","Local7","Warning","10.2.9.5","35579: Oct 8 16:46:46.031: %PM-4-ERR_DISABLE: security-violation error detected on Gi8/0/41, putting Gi8/0/41 in err-disable state"
"2024-10-08","16:46:43","Local7","Error","10.2.9.5","35578: Oct 8 16:46:42.796: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","16:46:42","Local7","Notice","10.2.9.5","35577: Oct 8 16:46:41.794: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","16:46:38","Local7","Notice","10.2.9.5","35576: Oct 8 16:46:38.830: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-08","16:46:38","Local7","Error","10.2.9.5","35575: Oct 8 16:46:37.829: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-08","16:46:31","Local7","Error","10.2.9.5","35574: Oct 8 16:46:31.625: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","16:46:31","Local7","Notice","10.2.9.5","35573: Oct 8 16:46:30.627: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","09:39:51","Local7","Notice","10.2.9.5","35107: Oct 8 09:39:51.062: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-08","09:39:51","Local7","Error","10.2.9.5","35106: Oct 8 09:39:50.060: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-08","09:39:47","Local7","Error","10.2.9.5","35105: Oct 8 09:39:46.725: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","09:39:46","Local7","Notice","10.2.9.5","35104: Oct 8 09:39:45.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","09:39:38","Local7","Notice","10.2.9.5","35103: Oct 8 09:39:38.727: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-08","09:39:38","Local7","Error","10.2.9.5","35102: Oct 8 09:39:37.725: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-08","09:39:32","Local7","Error","10.2.9.5","35101: Oct 8 09:39:31.529: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-08","09:39:31","Local7","Notice","10.2.9.5","35100: Oct 8 09:39:30.572: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","17:11:08","Local7","Notice","10.2.9.5","32778: Oct 4 17:11:07.805: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","17:11:07","Local7","Error","10.2.9.5","32777: Oct 4 17:11:06.808: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","17:11:04","Local7","Error","10.2.9.5","32776: Oct 4 17:11:03.592: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","17:11:03","Local7","Notice","10.2.9.5","32775: Oct 4 17:11:02.589: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","17:10:58","Local7","Notice","10.2.9.5","32773: Oct 4 17:10:58.714: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","17:10:58","Local7","Error","10.2.9.5","32772: Oct 4 17:10:57.714: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","17:10:52","Local7","Error","10.2.9.5","32771: Oct 4 17:10:52.396: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","17:10:52","Local7","Notice","10.2.9.5","32770: Oct 4 17:10:51.397: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","14:31:50","Local7","Notice","10.2.9.5","32693: Oct 4 14:31:50.942: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","14:31:50","Local7","Error","10.2.9.5","32692: Oct 4 14:31:49.940: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","14:31:47","Local7","Error","10.2.9.5","32691: Oct 4 14:31:46.717: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","14:31:46","Local7","Notice","10.2.9.5","32690: Oct 4 14:31:45.712: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","14:31:41","Local7","Notice","10.2.9.5","32689: Oct 4 14:31:41.986: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","14:31:41","Local7","Error","10.2.9.5","32688: Oct 4 14:31:40.985: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-04","14:31:36","Local7","Error","10.2.9.5","32687: Oct 4 14:31:35.513: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-04","14:31:35","Local7","Notice","10.2.9.5","32686: Oct 4 14:31:34.509: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-03","16:49:23","Local7","Notice","10.2.9.5","31747: Oct 3 16:49:22.984: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-03","16:49:23","Local7","Error","10.2.9.5","31746: Oct 3 16:49:21.981: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to up"
"2024-10-03","16:49:19","Local7","Error","10.2.9.5","31745: Oct 3 16:49:18.744: %LINK-3-UPDOWN: Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-03","16:49:18","Local7","Notice","10.2.9.5","31744: Oct 3 16:49:17.740: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to down"
"2024-10-03","16:49:10","Local7","Notice","10.2.9.5","31743: Oct 3 16:49:10.859: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/0/41, changed state to up"

it stuck at 16:46 after the user left the office and then next day the admin run shutdown/no shutdown to resolve the issue.

there Steps in detail explain the error in ISE share it 

ise+live+log+radius+dtls.png

117222400_0-1728514284619.png

Steps
11001 Received RADIUS Access-Request
11017 RADIUS created a new session
15049 Evaluating Policy Group
15008 Evaluating Service Selection Policy
15048 Queried PIP - Normalised Radius.RadiusFlowType
15048 Queried PIP - DEVICE.Device Type
11507 Extracted EAP-Response/Identity
12500 Prepared EAP-Request proposing EAP-TLS with challenge
12625 Valid EAP-Key-Name attribute received
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12502 Extracted EAP-Response containing EAP-TLS challenge-response and accepting EAP-TLS as negotiated
12800 Extracted first TLS record; TLS handshake started
12545 Client requested EAP-TLS session ticket
12542 The EAP-TLS session ticket received from supplicant while the stateless session resume is disabled. Performing full authentication
12805 Extracted TLS ClientHello message
12806 Prepared TLS ServerHello message
12807 Prepared TLS Certificate message
12808 Prepared TLS ServerKeyExchange message
12809 Prepared TLS CertificateRequest message
12810 Prepared TLS ServerDone message
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
11001 Received RADIUS Access-Request
11018 RADIUS is re-using an existing session
12504 Extracted EAP-Response containing EAP-TLS challenge-response
12505 Prepared EAP-Request with another EAP-TLS challenge
11006 Returned RADIUS Access-Challenge
12935 Supplicant stopped responding to ISE during EAP-TLS certificate exchange (step latency=120000 ms Step latency=120000 ms)
61025 Open secure connection with TLS peer
5411 Supplicant stopped responding to ISE

 

 

&&Windows GPO

Your windows GPO for wireless and wired network is also important. Open your 802.1x GPO and navigate to Security Settings -> Wired Network (802.3) Policies -> Network Profile -> IEEE 802.1X Settings.

Check the following settings and set them accordingly:

Setting Value
Maximum Authentication Failures 5
Maximum EAPOL-Start Messages Sent 5
Held Period (seconds) 1
Start Period (seconds) 5
Authentication Period (seconds) 30

The values are suggestions to start with. They may need to be adjusted to fit your environment.

&&Client Firewall and Antivirus

Your clients firewall or antivirus could be the problem, especially if you are using authentication timeouts in your ISE policies. For testing purposes you can temporarily disable your firewall or antivirus and check if the problem persists.

Hi 

Thanks very much for your reply. Only this desktop PC has the issue and the others which are in the same domain with same settings don't have the issue. Also, it seems another new MAC triggered the security violation, which means not the failures. I've checked no other devices like ip phones, the cable connected to the PC directly and the other end is connected to data point under the desk. I've changed all the cables and changed another data point, to see if it happened again.

Arne Bier
VIP
VIP

The port security err-disable should only have kicked in because there was a violation in the number of MAC addresses seen in the DATA domain - and this restriction only applies to multi-domain
I am fairly sure that multi-auth is the default (hidden from show run) in IOS - but you could try to confiure it manually

conf t
interface gig 8/0/41
access-session host-mode multi-auth
end

In multi-auth mode the switch will allow multiple MAC addresses in the DATA domain, and every MAC address is subject to RADIUS auth.  It doesn't explain WHY you get spurious MAC addresses on the switch, but I have never found multi-domain to be a reliable config method, simply because some phones can have the nasty ide effect of first landing their MAC in the DATA domain , before the flip over to the VOICE domain. If, at this time a PC is connected to the back of the phone, then you have 2 MAC addresses in the DATA domain, albeit for a split second - but this is enough to err-disable the interface.

Hi 

Thanks very much for your reply.

 

It happened again today, with another new MAC address been seen. That's very strange, I can't find the new MAC coming from where...

117222400_0-1728608316340.png

We don't have IP phones or HUB or deck for the PC.

Maybe we can set it to  authentication violation restrict or replace, because we use Windows 10 machine certificate and user certificate for authentication, we don't mind what MAC address it is.

 

117222400
Level 1
Level 1

I tested again, the port doesn't have any "sticky" feature, as I used 2 different usb network adaptor to connect to the same port. They have different MAC addresses and both got DHCP IP and working properly. 

Just suspect where is the new MAC coming from, only 2 adaptors on the PC, one is the normal one, another is the Zscaler adaptor, none of them have the above detected "new MAC is seen.." that's very odd.

117222400
Level 1
Level 1

Today morning the issue happened again. The same issue same desktop, and we can see that there are some up and downs yesterday afternoon, the last state at yesterday is UP, and suddenly it went to err-disable at today morning, other new MAC address appeared. The user is working from home and RDP to the desktop. no other devices at all.. where is the MAC address from..

117222400_0-1728623818057.png

 

 

 

I found this bug: https://bst.cisco.com/quickview/bug/CSCus17694 but our version is:

Cisco IOS XE Software, Version 17.09.04a
Cisco IOS Software [Cupertino], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 17.9.4a, RELEASE SOFTWARE (fc3)

 

is it possible to verify whether the  memory is leaky or not , when it happened or after I fixed it?

A work around might be setup the err-disable recovery to 5 seconds?