RADIUS-Auth not working after FW-Upgrade to 4.1.6.64 (Catalyst 1300)

[bm231]

Hello,

i'm using C1300-24P-4X-Switches with 802.1X-Authentication against NPS on a Windows Server 2022-Server.
After updating the Cisco firmware from 4.1.4.1 to 4.1.6.64 the Radius-Authentication with computer certificates isn't working anymore; in the NPS-Log, i'm getting this error:

Reason Code: 16 Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect

I'm sure, that the NPS-Server is working as expected, because our WiFi is also using 802.1X against the same NPS-Server without any problem. Also, only the firmware update of the switches changed since yesterday.

If a user could't be authenticated, it gets a guest vlan assigned. Thats actual the case for all users/ports.

I checked the Release-Notes from Cisco (Release Note for Cisco Catalyst 1200 and 1300 Series Switches Firmware Version 4.0.0.91 - 4.1.6.54); there's a info, that some Radius-behaviour has changed. As i understand, the "Message-Authenticator attribute" was added and is disabled by default. I've enabled and disabled it for testing without any success.

Has anybody an idea, what's the cause for this problem? I don't find any errors in the configuration

This is the running configuration; i've replaced sensitive data with "xxx":

v4.1.6.54 / RLSB4.1.6_951_410_024
CLI v1.0
file SSD indicator encrypted
@
ssd-control-start
ssd config
ssd file passphrase control unrestricted
no ssd file integrity control
ssd-control-end xxx
!
!
unit-type-control-start
unit-type unit 1 network gi uplink te
unit-type unit 2 network gi uplink te
unit-type unit 3 network gi uplink te
unit-type unit 4 network gi uplink te
unit-type unit 5 network gi uplink te
unit-type unit 6 network gi uplink te
unit-type unit 7 network gi uplink te
unit-type unit 8 network gi uplink te
unit-type-control-end
!
port jumbo-frame
vlan database
vlan xxx
exit
voice vlan oui-table add 0001e3 Siemens_AG_phone
voice vlan oui-table add 00036b Cisco_phone
voice vlan oui-table add 00096e Avaya
voice vlan oui-table add 000fe2 H3C_Aolynk
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone
voice vlan oui-table add 00e075 Polycom/Veritel_phone
voice vlan oui-table add 00e0bb 3Com_phone
dot1x system-auth-control
dot1x mac-auth radius
bonjour interface range vlan 1
hostname xxx
line console
exec-timeout 30
exit
encrypted radius-server host xxx key xxx priority 1
encrypted radius-server host xxx key xxx priority 2
aaa accounting dot1x start-stop group radius
username admin password encrypted xxx privilege 15
ip ssh server
ip ssh logging enable
ip ssh password-auth
ip http timeout-policy 0
clock timezone CET +1
sntp server xxx poll
sntp server xxx poll
no sntp server pool.ntp.org
no sntp server time-a.timefreq.bldrdoc.gov
no sntp server time-b.timefreq.bldrdoc.gov
no sntp server time-c.timefreq.bldrdoc.gov
no sntp server time-pnp.cisco.com
ip domain name xxx
ip name-server  xxx xxx
no pnp enable
cbd probe enable
cbd address xxx
encrypted cbd key id xxx secret xxx
cbd connection enable
!
interface GigabitEthernet1/0/1
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C12
!
interface GigabitEthernet1/0/2
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description "C5"
!
interface GigabitEthernet1/0/3
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B12
!
interface GigabitEthernet1/0/4
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B13
!
interface GigabitEthernet1/0/5
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B6
!
interface GigabitEthernet1/0/6
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B7
!
interface GigabitEthernet1/0/7
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C7
!
interface GigabitEthernet1/0/8
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C11
!
interface GigabitEthernet1/0/9
 dot1x radius-attributes vlan static
 description "C8"
!
interface GigabitEthernet1/0/10
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/11
 description "C6"
!
interface GigabitEthernet1/0/12
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/13
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B24
!
interface GigabitEthernet1/0/14
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B19
!
interface GigabitEthernet1/0/15
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description D13
!
interface GigabitEthernet1/0/16
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description B18
!
interface GigabitEthernet1/0/17
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description D14
!
interface GigabitEthernet1/0/18
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/19
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/20
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/21
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/22
 dot1x radius-attributes vlan static
 description "B22"
 switchport access vlan xxx
!
interface GigabitEthernet1/0/23
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet1/0/24
 description xxx
 switchport mode trunk
 switchport general allowed vlan add xxx tagged
 switchport trunk allowed vlan xxx
!
interface TenGigabitEthernet1/0/1
 description xxx
 switchport mode trunk
 switchport trunk allowed vlan xxx
!
interface TenGigabitEthernet1/0/2
 switchport mode trunk
!
interface GigabitEthernet2/0/1
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C1
!
interface GigabitEthernet2/0/2
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description D6
!
interface GigabitEthernet2/0/3
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description D7
!
interface GigabitEthernet2/0/4
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description D9
!
interface GigabitEthernet2/0/5
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/6
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description D8
!
interface GigabitEthernet2/0/7
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/8
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/9
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/10
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/11
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/12
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/13
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C15
!
interface GigabitEthernet2/0/14
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C16
!
interface GigabitEthernet2/0/15
 dot1x radius-attributes vlan static
 description "C24"
!
interface GigabitEthernet2/0/16
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C23
!
interface GigabitEthernet2/0/17
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C22
!
interface GigabitEthernet2/0/18
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description C21
!
interface GigabitEthernet2/0/19
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/20
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
!
interface GigabitEthernet2/0/21
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x radius-attributes vlan static
 switchport access vlan 11
!
interface GigabitEthernet2/0/22
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x radius-attributes vlan static
 switchport access vlan 11
!
interface GigabitEthernet2/0/23
 dot1x guest-vlan enable
 dot1x reauthentication
 dot1x authentication 802.1x mac
 dot1x radius-attributes vlan static
 dot1x port-control auto
 description "E4"
!
interface GigabitEthernet2/0/24
 dot1x radius-attributes vlan static
 description xxx
 switchport mode trunk
 switchport general allowed vlan add xxx tagged
 switchport trunk allowed vlan xxx
!
interface TenGigabitEthernet2/0/1
 description xxx
 switchport mode trunk
 switchport trunk allowed vlan xxx
!
interface TenGigabitEthernet2/0/2
 switchport mode trunk
 switchport trunk allowed vlan xxx
 switchport trunk allowed vlan add xxx
 switchport trunk allowed vlan add xxx
!
exit
interface bluetooth 0
 shutdown
!
ip default-gateway xxx

Edit 1:

I'm not sure, but... the complete NPS-Log-Entry for the deny:

Der Netzwerkrichtlinienserver hat einem Benutzer den Zugriff verweigert.

Wenden Sie sich an den Administrator des Netzwerkrichtlinienservers, um weitere Informationen zu erhalten.

Benutzer:
	Sicherheits-ID:			NULL SID
	Kontoname:			xxxxxf5f441
	Kontodomäne:			xxx
	Vollqualifizierter Kontoname:	xxx\xxxxxf5f441

Clientcomputer:
	Sicherheits-ID:			NULL SID
	Kontoname:			-
	Vollqualifizierter Kontoname:	-
	ID der Empfangsstation:		xxx
	ID der Anrufstation:		xxx

NAS:
	NAS-IPv4-Adresse:		xxx
	NAS-IPv6-Adresse:		-
	NAS-ID:			-
	NAS-Porttyp:			Ethernet
	NAS-Port:			69

RADIUS-Client:
	Clientanzeigename:		xxx
	Client-IP-Adresse:			xxx

Authentifizierungsdetails:
	Name der Verbindungsanforderungsrichtlinie:	802.1X Ethernet
	Netzwerkrichtlinienname:		-
	Authentifizierungsanbieter:		Windows
	Authentifizierungsserver:		Domain controller
	Authentifizierungstyp:		PAP
	EAP-Typ:			-
	Kontositzungs-ID:		3035303030303532
	Protokollierungsergebnisse:			Die Kontoinformationen wurden in die lokale Protokolldatei geschrieben.
	Ursachencode:			16
	Ursache:				Authentifizierungsfehler aufgrund der Nichtübereinstimmung von Benutzeranmeldeinformationen. Der angegebene Benutzername ist keinem vorhandenen Benutzerkonto zugeordnet, oder das Kennwort war falsch.

This looks like a error message regarding a MAC-based authentication. If i look into the older logs with success-messages, they contain the Hostname of the client, which tried to authenticate with 802.1X:

Der Netzwerkrichtlinienserver hat einem Benutzer den Zugriff gewährt.

Benutzer:
	Sicherheits-ID:			xxx\HOSTNAME$
	Kontoname:			host/hostname.domain.tld
	Kontodomäne:			Domain
	Vollqualifizierter Kontoname:	domain\HOSTNAME$

I already disabled the MAC-based authentication on my Test-Networkport; but i sill getting errors that look like the first one above.

Edit 2: Here's the error-message logged by the switch for the failed authentication:

%SEC-W-SUPPLICANTUNAUTHORIZED: MAC xxx was rejected on port gi2/0/17 due to wrong user name or password in Radius server

Edit 3: I'm very consufed. I switched back to firmware 4.1.4.1 and applied a configuration backup, which was taken with 4.1.4.1: I still can't authenticate with 802.1X. I've checked the windows updates on client/server; no updates we're installed in the time range, i've updated the switches. Also, i've rebootet our primary Radius-Server (DC) without success. Any suggestions?

[MHM Cisco World]

Are this issue solved?

MHM

[bm231]

Hello MHM,

no, the problem still exists. I don't have further ideas, what's the cause for this issue.
On the last weekend, i've resettet a Stack of four Catalyst 1300-Switches, with 4.1.6.64 already installed and configured them from scratch.

The 802.1X is still not working (with the described error messages). I'm absolute not sure, what i should do next. The NPS-Servers are working still fine with our Ruckus-WLAN and 802.1X.

I know, that this problem seems curious and the switches and the firmware update should not be the cause, because if i switch back to 4.1.4.1 and the old config file, it's still not working. But i didn't change other settings in NPS, DCs, Roles, GPOs, etc.

I also checked things like the CA Certificate expiration, that the clients have valid computer certificates, matched the running configuration for 802.1X (GPOs, NPS-Settings, PSKs, etc.) against our documentation - but they are all correct. That 802.1X is working with the WiFi-Environment shows, that there can't be a gross mistake.

Do you have any ideas, how i could troubleshoot this issue further?

Thanks in advance,
Bastian