I have a TAC open already, but I figured I'd touch base and see if anyone had any extra thoughts (I'm hoping I'm just being dumb and missing something).
So odd scenario, middle of a new gear deployment (changing out Catalyst 4500/3850/2960X gear for Catalyst 9k gear). Most gear was received running 16.12.4. Devices (mix of 9300 and 9400 for wired access; no wireless) were authenticating to ISE 3.0 Patch 2 just fine for MAB (haven't made full transition to dot1x yet) against a MAC Whitelist. "Show authentication session" would return pretty results and tag everything in the DATA domain (single-host mode as we don't use VOIP). Upgraded my 9300 and 9400 switches to 17.3.3 and my 9400s no longer authenticate devices to ISE via RADIUS and "show authentication session" tags them as UNKNOWN domain (I also see no entries under RADIUS Live Logs on ISE). The 9300 switches authenticate just fine. TACACS (Device admin on ISE) performs just fine and aaa test for RADIUS is successful every time (obviously with correct credentials). Since the 9400s are not authenticating to ISE via RADIUS, dACLs are not being pushed effectively (this can be bypassed with a "ip permit any any" on the default ACL, but obviously that is not acceptable for the long term). Debugging RADIUS is showing the that a couple attributes are missing on the 9400 switches:
9300 (can authenticate at 17.3.3)
RADIUS: Framed-IP-Address [8] 6 <IP address of device>
RADIUS: User-Name [1] 19 "<mac-address of device>"
RADIUS: Cisco AVpair [1] 43 "audit-session-id=<session ID for device>"
RADIUS: Cisco AVpair [1] 12 "method=mab"
truncated...
9400 (can not authenticate at 17.3.3)
RADIUS: Cisco AVpair [1] 43 "audit-session-id=<session ID for device>"
RADIUS: Cisco AVpair [1] 12 "method=mab"
truncated...
For configuration references, see below.
AAA commands (ignore the privilege levels, they're a pre-TACACS holdover)
aaa new-model
aaa group server tacacs+ <TACACS GROUP>
server name <PRIMARY ISE NODE>
server name <SECONDARY ISE NODE>
ip vrf forwarding <MANAGEMENT VRF>
aaa group server radius <RADIUS GROUP>
server name <PRIMARY ISE NODE>
server name <SECONDARY ISE NODE>
ip vrf forwarding <MANAGEMENT VRF>
!
aaa authentication login console local
aaa authentication login vty local
aaa authentication login ISE_TACACS_LIST group <TACACS GROUP> local
aaa authentication enable default group <TACACS GROUP> enable
aaa authentication dot1x default group <RADIUS GROUP>
aaa authorization config-commands
aaa authorization exec default local
aaa authorization exec vty local
aaa authorization exec <TACACS LIST> group <TACACS GROUP> local
aaa authorization commands 0 <TACACS LIST> group <TACACS GROUP> local
aaa authorization commands 2 <TACACS LIST> group <TACACS GROUP> local
aaa authorization commands 5 <TACACS LIST> group <TACACS GROUP> local
aaa authorization commands 15 <TACACS LIST> group <TACACS GROUP> local
aaa authorization network default group <RADIUS GROUP>
aaa accounting update periodic 5
aaa accounting auth-proxy default start-stop group <RADIUS GROUP>
aaa accounting dot1x default start-stop group <RADIUS GROUP>
aaa accounting exec <TACACS LIST> start-stop group <TACACS GROUP>
aaa accounting commands 2 <TACACS LIST> start-stop group <TACACS GROUP>
aaa accounting commands 5 <TACACS LIST> start-stop group <TACACS GROUP>
aaa accounting commands 15 <TACACS LIST> start-stop group <TACACS GROUP>
!
truncated...
!
authentication mac-move permit
dot1x system-auth-control
!
ip route vrf <MANAGEMENT VRF> 0.0.0.0 0.0.0.0 <MANAGEMENT VRF GATEWAY>
ip tacacs source-interface GigabitEthernet0/0 vrf <MANAGEMENT VRF>
ip radius source-interface GigabitEthernet0/0 vrf <MANAGEMENT VRF>
!
truncated...
!
ip access-list extended <DEFAULT ACL>
permit ip any any //This is the bypass since no dACL is being pushed
permit udp any eq bootpc any eq bootps
permit udp any any eq domain
permit icmp any any
permit ip any host <PRIMARY RADIUS SERVER IP>
permit ip any host <SECONDARY RADIUS SERVER IP>
deny ip any any
!
truncated...
!
tacacs server <PRIMARY TACACS SERVER>
address ipv4 <PRIMARY TACACS SERVER IP>
key <KEY>
tacacs server <SECONDARY TACACS SERVER>
address ipv4 <SECONDARY TACACS SERVER IP>
key <KEY>
!
radius-server attribute 6 on-for-login-auth
radius-server attribute 6 support-multiple
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server attribute 31 mac format ietf upper-case
radius-server attribute 31 send nas-port-detail
radius-server dead-criteria tries 3
radius-server deadtime 30
!
radius server <PRIMARY RADIUS SERVER>
address ipv4 <PRIMARY RADIUS SERVER IP> auth-port <RADIUS AUTH PORT> acct-port <RADIUS ACCT PORT>
key <KEY>
radius server <SECONDARY RADIUS SERVER>
address ipv4 <SECONDARY RADIUS SERVER IP> auth-port <RADIUS AUTH PORT> acct-port <RADIUS ACCT PORT>
key <KEY>
** NOTE: ISE serves both RADIUS and TACACS, so the IPs are actually the same.(Primary TACACS = Primary RADIUS; Secondary TACACS = Secondary RADIUS)
Interface config (generic config applied through macro)
switchport mode access
switchport access vlan <VLAN>
switchport nonegotiate
switchport block unicast
device-tracking
ip access-group <DEFAULT ACL> in
authentication event fail action next-method
authentication event server dead action reinitialize vlan <VLAN>
authentication open
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity server
authentication violation restrict
mab
snmp trap mac-notification change added
snmp trap mac-notification change removed
dot1x pae authenticator
storm-control broadcast level 20.00
spanning-tree guard root
ip verify source tracking
no shutdown
** Disclaimer: Realize that some commands are in place for transitional purposes.
As everything was working perfectly fine prior to the 17.3.3 upgrade and it appears that no commands have been depreciated (verified against baseline config), I'm at a loss. I have multiple 9300 switches at 17.3.3 and they're working fine, but every 9400 I have at 17.3.3 (5 in total), will not authenticate devices (they did previously at 16.12.4). RADIUS Live logs on the ISE servers will show the successful login from the "test aaa group <RADIUS GROUP> <USERNAME> <PASSWORD> legacy", so I know I can hit the RADIUS server properly. I'm either missing something obvious, or 17.3.3 may have an issue with my config and RADIUS.
Appreciate any thoughts.
