I'm working on an established site with 4 4500 switches installed. All are being used as access switches. All are supporting IP phones authenticating via mab. All have an unauthenticated fallback configuration. It's multi-domain auth. As best I can tell, when the site was migrated to ISE in late March of this year, *one* of the switches started behaving very erratically with regard to dot1x. Previous auth was configured on ACS - and was presumably working fine (I don't have any contrary records).
The main sign of trouble (beyond the end-user complaints) are 'security violation' logs that indicate multi-domain auth may not be operating correctly. Initially I found the following thread that was a very similar configuration, with very similar symptoms:
https://community.cisco.com/t5/switching/dot1x-issue-intermittent-connectivity-in-multidomain-mode-avaya/td-p/3356308
Where my situation deviates is the extraordinary (to me anyway) logs which show that the security violation on specific end-user ports - UPS NIC and workstation ports, no ip phone - is triggered when Cisco WAP's connected to OTHER ports (that have NO dot1x configuration) are seen as attempting to authenticate on ports they have no physical connection to. I initially tried to blame physical layer problems at the site - rogue switches/hubs, patch panel problems, etc... as triggering these messages. But after an hour long webex which established no environmental issues were affecting the infrastructure and there was a twin (platform and configuration) right next to the affected switch that was having no issues and generating no logs, I have to suspect that something funky is going on with the "bad" switch.
The show version:
------------------ show version ------------------
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500es8-UNIVERSALK9-M), Version 03.06.07.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Wed 12-Jul-17 21:28 by prod_rel_team
Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: 15.1(1r)SG5
hostname uptime is 9 weeks, 5 days, 5 hours, 19 minutes
Uptime for this control processor is 9 weeks, 5 days, 5 hours, 21 minutes
System returned to ROM by power-on
System restarted at 05:47:55 UTC Sun Apr 14 2019
System image file is "bootflash:/cat4500es8-universalk9.SPA.03.06.07.E.152-2.E7.bin"
The "typical" port configuration of all 4500's at the site:
interface GigabitEthernet2/47
description Dynamic_Auth_Port (This port actually goes to the UPS NIC, and doesn't need to be doing dot1x
switchport mode access but it should be able to accommodate a simple host without issues!!!!)
switchport voice vlan 281
authentication event fail action authorize vlan 801
authentication event no-response action authorize vlan 801
authentication host-mode multi-domain
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication violation replace
mab
no snmp trap link-status
dot1x pae authenticator
dot1x timeout quiet-period 10
dot1x timeout tx-period 15
dot1x timeout supp-timeout 31
dot1x max-req 3
storm-control broadcast level 1.00
storm-control action shutdown
storm-control action trap
The log messages representative of the issue:
Jun 25 10:29:16.697 UTC: %DOT1X-5-FAIL: Authentication failed for client (0013.c602.adc5) on Interface Gi2/47 AuditSessionID 0AABAF01000083CE73CC99DC
Jun 25 10:29:41.291 UTC: %DOT1X-5-FAIL: Authentication failed for client (2829.8606.af32) on Interface Gi2/46 AuditSessionID 0AABAF01000083CF73CCFBA4
Jun 25 10:31:21.682 UTC: %DOT1X-5-FAIL: Authentication failed for client (0004.f29e.0cea) on Interface Gi1/40 AuditSessionID 0AABAF0100008337713A3348
Jun 25 10:41:57.423 UTC: %AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface GigabitEthernet2/47, new MAC address (0004.f2a3.1b8d) is seen.AuditSessionID Unassigned
Jun 25 10:41:57.423 UTC: %AUTHMGR-5-MACREPLACE: MAC address (2829.8606.af9d) on Interface GigabitEthernet2/47 is replaced by MAC (0004.f2a3.1b8d) AuditSessionID 0AABAF01000083B673825628
Jun 25 10:41:57.435 UTC: %AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface GigabitEthernet2/47, new MAC address (00a7.42a2.d9fe) is seen.AuditSessionID Unassigned
Jun 25 10:41:57.435 UTC: %AUTHMGR-5-MACREPLACE: MAC address (0004.f2a3.1b8d) on Interface GigabitEthernet2/47 is replaced by MAC (00a7.42a2.d9fe) AuditSessionID 0AABAF01000083D173D8EA84
Jun 25 10:41:57.610 UTC: %AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface GigabitEthernet2/47, new MAC address (706e.6d59.d531) is seen.AuditSessionID Unassigned
Jun 25 10:41:57.610 UTC: %AUTHMGR-5-MACREPLACE: MAC address (00a7.42a2.d9fe) on Interface GigabitEthernet2/47 is replaced by MAC (706e.6d59.d531) AuditSessionID 0AABAF01000083D273D8EA8C
Jun 25 10:41:58.116 UTC: %MAB-5-FAIL: Authentication failed for client (706e.6d59.d531) on Interface Gi2/47 AuditSessionID 0AABAF01000083D373D8EB3C
Jun 25 10:42:44.194 UTC: %DOT1X-5-FAIL: Authentication failed for client (706e.6d59.d531) on Interface Gi2/47 AuditSessionID 0AABAF01000083D373D8EB3C
These are just a couple examples - one of those is an IP phone on port 1/45. Another is an access point on 1/18. When this happens, not only does the UPS on 2/47 log connection flaps and become unavailable, but the ip phones go offline as well. *Interestingly*, the WAPs - which have no dot1x configuration on the ports - never get disconnected or log link flaps.
interface GigabitEthernet1/18
description WAP
switchport trunk native vlan 101
switchport mode trunk
no snmp trap link-status
I can not figure out how to describe this behavior in simple, easily searchable terms, and I what descriptions I *have* tried have yielded nothing really close. Searching on the log messages revealed the forum result I posted, but beyond that, the dot1x issues I have reviewed are very specific.
Has anyone ever seen dozens of devices connected to other physical ports on a switch attempt to authenticate through totally unconnected dot1x ports? 2/47 is not the only connected port - it's happening on 2/46 and a couple other ports, but 2/47 is by far the most popular for this sort of thing. I'm not even sure if this will have any relationship to the biggest problem which is that this switch is experiencing random connection loss on any port very close to the auth timer expiry, but not always - it is causing dropped calls and stalled file transfers and widespread frustration for virtually every user connected to this switch. I think the supervisor is just having some sort of brain issue, but I was hoping the bizarre dot1x behavior could help narrow down exactly what. Devices connecting to non-dot1x (WAPs, up/downlinks) ports never lose connectivity - even when they are implicated as the device attempting to authenticate. Very strange!
Thanks for reading this.
