ISE Showing successful dot1x, switches swapping for wired APs

[jbulloch]

Good morning/evening/time of day,

In our current environment our "back end" team and our "wireless" team recently converted our AP's from MAB authentication to 8x. The majority of our AP's converted over fine and are functioning without fail. These AP's that fail show correct dot1x in ISE but on switch will show constantly flipping between dot1x which will fail, then to mab. I can't find a difference. We can allow MAB on the ise profile and it will auth but the goal is to keep them to dot1x. 

I can run tcpdump from ise and use the radius t/s tool in the diagnostics tools and see the requests hitting ISE. I can also run dot1x debug from the switch and i receive this (ntp time has since been corrected) so it appears to be talking to the server.  Running an aaa test from the switches in question with a functional account is successful. A device tracking/sensor profile is also feeding ISE the APs LLDP information for policy sorting. 

Am i missing something simple here or could someone suggest further troubleshooting i may be overlooking?

Mar 24 19:43:24.002 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] Posting EAPOL_EAP for 0x3E000FDE
Mar 24 19:43:24.002 edt: dot1x_auth_bend Gi0/2: during state auth_bend_request, got event 6(eapolEap)
Mar 24 19:43:24.002 edt: @@@ dot1x_auth_bend Gi0/2: auth_bend_request -> auth_bend_response
Mar 24 19:43:24.002 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] 0x3E000FDE:entering response state
Mar 24 19:43:24.002 edt: dot1x-ev:[2462.cecb.04fe, Gi0/2] Response sent to the server from 0x3E000FDE
Mar 24 19:43:24.002 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] 0x3E000FDE:request response action
Mar 24 19:43:24.023 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] Posting EAP_REQ for 0x3E000FDE
Mar 24 19:43:24.023 edt: dot1x_auth_bend Gi0/2: during state auth_bend_response, got event 7(eapReq)
Mar 24 19:43:24.023 edt: @@@ dot1x_auth_bend Gi0/2: auth_bend_response -> auth_bend_request
Mar 24 19:43:24.023 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] 0x3E000FDE:exiting response state
Mar 24 19:43:24.023 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] 0x3E000FDE:entering request state
Mar 24 19:43:24.023 edt: dot1x-ev:[2462.cecb.04fe, Gi0/2] Sending EAPOL packet
Mar 24 19:43:24.023 edt: dot1x-registry:registry:dot1x_ether_macaddr called
Mar 24 19:43:24.023 edt: dot1x-ev:[2462.cecb.04fe, Gi0/2] Sending out EAPOL packet to MAC 2462.cecb.04fe
Mar 24 19:43:24.027 edt: dot1x-packet:EAPOL pak Tx - Ver: 0x3 type: 0x0
Mar 24 19:43:24.027 edt: dot1x-packet: length: 0x001E
Mar 24 19:43:24.027 edt: dot1x-packet:EAP code: 0x1 id: 0x4E length: 0x001E
Mar 24 19:43:24.027 edt: dot1x-packet: type: 0x37
Mar 24 19:43:24.027 edt: dot1x-packet:[2462.cecb.04fe, Gi0/2] EAPOL packet sent to client 0x3E000FDE
Mar 24 19:43:24.027 edt: dot1x-sm:[2462.cecb.04fe, Gi0/2] 0x3E000FDE:response request action
Mar 24 19:43:24.041 edt: dot1x-packet:[2462.cecb.04fe, Gi0/2] Queuing an EAPOL pkt on Authenticator Q
Mar 24 19:43:24.041 edt: dot1x-packet:EAPOL pak rx - Ver: 0x1 type: 0x0

The switch is confiqured as followed:

Interface:

switchport access vlan 253
switchport mode access
switchport block unicast
authentication host-mode multi-domain
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication violation restrict
mab
dot1x pae authenticator
dot1x timeout tx-period 10
storm-control broadcast level bps 20m
storm-control unicast level bps 62m
spanning-tree portfast edge
spanning-tree bpduguard enable
spanning-tree guard root

Global config:

auth mac-move permit

dot1x system auth-control

Triple AAA:

aaa new-model

aaa group server tacacs+ ISE_TACACS
server name ISE-PSN-1
server name ISE-PSN-2
ip tacacs source-interface Vlan2100
!
aaa group server radius ISE_RADIUS
server name ISE-PSN-1
server name ISE-PSN-2
ip radius source-interface Vlan2100
!
aaa authentication login TAC_AUTHEN group ISE_TACACS local
aaa authentication enable default group ISE_TACACS enable
aaa authentication dot1x default group ISE_RADIUS
aaa authorization console
aaa authorization config-commands
aaa authorization exec TAC_AUTHOR group ISE_TACACS local if-authenticated
aaa authorization commands 15 TAC_AUTHOR group ISE_TACACS if-authenticated
aaa authorization network default group ISE_RADIUS
aaa accounting update newinfo
aaa accounting auth-proxy default start-stop group ISE_RADIUS
aaa accounting dot1x default start-stop group ISE_RADIUS
aaa accounting exec TAC_ACCT start-stop broadcast group ISE_TACACS
aaa accounting commands 15 TAC_ACCT start-stop broadcast group ISE_TACACS

aaa server radius dynamic-author
client x.x.10.13 server-key 7 054A422A22594C2C1D07051D5A5E577E
client x.x.10.12 server-key 7 041A4F230C344E6B0D1B171843595F50
client x.x.10.11 server-key 7 100F4D3C0602102E08063824757A6061
client x.x.10.10 server-key 7 054A422A22594C2C1D07051D5A5E577E

radius config:

radius-server attribute 6 on-for-login-auth
radius-server attribute 6 support-multiple
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server dead-criteria time 10 tries 3
radius-server deadtime 15
!
radius server ISE-PSN-1
address ipv4 x.x.10.12 auth-port 1645 acct-port 1646
key 7 1456562E0F11280E202A213A73415442
!
radius server ISE-PSN-2
address ipv4 x.x.10.13 auth-port 1812 acct-port 1813
key 7 054A422A22594C2C1D07051D5A5E577E

[MHM Cisco World]

You need to auth the AP not wifi client via SW?

This AP is local mode or flex mode?

MHM

[jbulloch]

Good morning,

Thanks for the response. 

The AP's are in a local mode and connect via to a WLC (aruba). We are in this case authing the AP's via ISE before they handle the usual tunnel / exit from the controller of the vlans and the controller handles the clients. This has been working fine for us in MAB auth for ages, and is working fine for us where the AP's are working with 8x. 

[Arne Bier]

As @MHM Cisco World correctly asked, is the WAP configured to be in local mode or Flex - looking at your switch config, the access mode would indicate that you are not in Flexconnect, and your WAPs are creating a CAPWAP tunnel back to the WLC. Hence, access VLAN is all you need. That would be correct in that case.

The command

authentication host-mode multi-domain

is strictly speaking only for switchports that need one DATA MAC address, and one VOICE MAC address - check what MAC addresses your're learning on that switch interface

show mac address int gig0/2

and if you only see the MAC address of the Cisco WAP, then you can change the host-mode to 

authentication host-mode single-host

But that's not the issue here. The issue is why the WAP is failing 802.1X and you can get that reason in ISE, in Live Logs details. Show us an example of what ISE is reporting when 802.1X fails.  Perhaps those WAPs don't trust the ISE EAP certificate, or the 802.1X supplicant has not been provisioned correctly. In that case, the WAP is doing the right thing, it fails-back to non-802.1X mode and tries again, and again ... until you fix the issue.

[jbulloch]

Hi arnie, 

Thanks for the response.  I deploy auth host-mode multi-domain in my template for user facing ports to support VOIP users. I've never had an issue with it being left on AP ports as there's only one device and no tag/untag going on, so for ease of configuration    it remains. I can confirm that changing it to either of cisco's three options makes no difference, and only the AP's mac is visible on the port.

When i check in ISE endpoints, it reports 8x auth.

The live logs seem to report similar, I've blanked out corporate naming convention per policy. 

Overview

Event	5200 Authentication succeeded
Username	<our username>
Endpoint Id	24:62:CE:CB:04:FE
Endpoint Profile	Aruba-AP-515_3560CX
Authentication Policy	WIRED_NETWORK >> EAP-PEAP
Authorization Policy	WIRED_NETWORK >> 0066_Aruba_AP_802.1x_EAP-PEAP
Authorization Result	0066_Aruba_AP

Authentication Details

Source Timestamp	2024-02-01 08:25:17.522
Received Timestamp	2024-02-01 08:25:17.522
Policy Server	ise-psn-1
Event	5200 Authentication succeeded
Username	<Our Username>
Endpoint Id	24:62:CE:CB:04:FE
Calling Station Id	24-62-CE-CB-04-FE
Endpoint Profile	Aruba-AP-515_3560CX
IPv4 Address	x.x.x.x
Authentication Identity Store	<OUR AD>
Identity Group	Aruba-AP-515_3560CX
Audit Session Id	9D8D00EF00000602B583F598
Authentication Method	dot1x
Authentication Protocol	PEAP (EAP-MSCHAPv2)

However, switch is still flapping between the two.

[MHM Cisco World]

Ok the AP dont run any Radio now and you only auth the AP.

Can i see from SW

Show authentication session interface x details 

Thanks 

MHM

[jbulloch]

Sure thing,

here it is authed over to mab. The dot1x fails over almost immediately. 

Interface:  GigabitEthernet0/2
          MAC Address:  2462.cecb.04fe
         IPv6 Address:  Unknown
         IPv4 Address:  x.x.66.187
            User-Name:  24-62-CE-CB-04-FE
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-domain
     Oper control dir:  both
      Session timeout:  1800s (server), Remaining: 1760s
       Timeout action:  Reauthenticate
      Restart timeout:  N/A
Periodic Acct timeout:  N/A
       Session Uptime:  71s
    Common Session ID:  9D8D00EF0000061CB5CCF94F
      Acct Session ID:  0x000012E9
               Handle:  0x8D00022A
       Current Policy:  POLICY_Gi0/2

Local Policies:
        Service Template: DEFAULT_LINKSEC_POLICY_SHOULD_SECURE (priority 150)
      Security Policy:  Should Secure
      Security Status:  Link Unsecure

Server Policies:
           Vlan Group:  Vlan: 66
              ACS ACL:  xACSACLx-IP-Aruba_AP_MAB_3560-63c6bae1

Method status list: 
      Method            State

      dot1x              Stopped
      mab                Authc Success

[MHM Cisco World]

No need friend 

You mention that you change the AP from MAB to 802.1x but as I see the auth success is for MAB and 802.1x is stopped.

MHM

[MHM Cisco World]

Common Session ID:  <<- there number appear here in show authentication session check it in ISE live logs.

MHM

[jbulloch]

Yes, that is my point of confusion. The ISE live logs (and end point details) report dot1x, but the switch is constantly trying to do dot1x and flapping between dot1x and mab. Of course, this means tunnel to controller is never built. 

Has this is only occurring with a handful of our APs, i am starting to think it is a issue with the particular AP config. However person in charge of this insists the APs are all built from the same template so i am doing my due diligence to say it is not the switches. 

[MHM Cisco World]

I dont think it issue of AP is issue of ISE the wire conditional you use for auth is same and hence the MAB is auth and then 802.1x 
that my guess 
but you can sure by look to common session ID, if it same and the auth-policy is for PEAP then OK if the common session ID is different then the ISE is auth the AP via MAB. 
MHM

[jbulloch]

For instance i just moved over to my SSH workstation across the room, and same AP is now:

  Interface:  GigabitEthernet0/2
          MAC Address:  2462.cecb.04fe
         IPv6 Address:  Unknown
         IPv4 Address:  Unknown
            User-Name:  1nnaruba
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-domain
     Oper control dir:  both
      Session timeout:  3600s (server), Remaining: 3200s
       Timeout action:  Reauthenticate
      Restart timeout:  N/A
Periodic Acct timeout:  N/A
       Session Uptime:  402s
    Common Session ID:  9D8D00EF0000061CB5CCF94F
      Acct Session ID:  0x0000132B
               Handle:  0x8D00022A
       Current Policy:  POLICY_Gi0/2

Local Policies:
        Service Template: DEFAULT_LINKSEC_POLICY_SHOULD_SECURE (priority 150)
      Security Policy:  Should Secure
      Security Status:  Link Unsecure

Server Policies:
           Vlan Group:  Vlan: 155
            SGT Value:  16

Method status list: 
      Method            State

      dot1x              Authc Success

[MHM Cisco World]

and the VLAN is change form 66 to 155 (is that OK) if not this can keyfactor to check the issue in ISE 
and the dot1x is authc success perfect
MHM

[jbulloch]

Hi, i'am not sure on the vlan. I don't have ability to see the actual rules built out in ISE. 

However the session ID was the same, and when i punched it into live logs i got back both the eap and mab auth back to back. I guess i need to go have a word with our network defense guys about whatever policies they are running.

[jbulloch]

In additon, here is the authentication log. I've had to remove some coporate information here as well.

Other Attributes ConfigVersionId 210 DestinationPort 1645 Protocol Radius NAS-Port 50102 Framed-MTU 1500 State 37CPMSessionID=9D8D00EF00000602B583F598;38SessionID=ise-psn-1/494795943/10900087; NetworkDeviceProfileId b0699505-3150-4215-a80e-6753d45bf56c IsThirdPartyDeviceFlow false AcsSessionID ise-psn-1/494795943/10900087 SelectedAuthenticationIdentityStores <OUR ID Store> SelectedAuthenticationIdentityStores Internal Users AuthenticationStatus AuthenticationPassed IdentityPolicyMatchedRule EAP-PEAP AuthorizationPolicyMatchedRule 0066_Aruba_AP_802.1x_EAP-PEAP EndPointMACAddress 24-62-CE-CB-04-FE EapChainingResult No chaining ISEPolicySetName WIRED_NETWORK IdentitySelectionMatchedRule EAP-PEAP AD-Error-Details Domain trust is one-way AD-User-Resolved-Identities <our ad id> AD-User-Candidate-Identities <our ad id> TotalAuthenLatency 109 ClientLatency 91 AD-User-Resolved-DNs  <ad info> AD-User-DNS-Domain  <ad info> AD-Groups-Names <ad group> AD-User-NetBios-Name <ad user> TLSCipher DHE-RSA-AES256-SHA256 TLSVersion TLSv1.2 DTLSSupport Unknown HostIdentityGroup Endpoint Identity Groups:Profiled:Aruba-AP-515_3560CX Network Device Profile Cisco Location  <loc string> Device Type Device Type#All Device Types#Switch IPSEC IPSEC#Is IPSEC Device#No Name Endpoint Identity Groups:Profiled:Aruba-AP-515_3560CX ExternalGroups S-1-5-21-2039752546-350242969-1258846019-513 IdentityAccessRestricted false RADIUS Username  <user name> Device IP Address x.x.x.x CPMSessionID 9D8D00EF00000602B583F598 Called-Station-ID 04:6C:9D:F8:5D:02 CiscoAVPair <av pair> Result Class CACS:9D8D00EF00000602B583F598:ise-psn-1/494795943/10900087 Session-Timeout 1800 seconds Termination-Action RADIUS-Request Tunnel-Type (tag=1) VLAN Tunnel-Medium-Type (tag=1) 802 Tunnel-Private-Group-ID (tag=1) 66 EAP-Key-Name 19:00:00:1d:10:14:d7:7d:26:78:9f:e6:2f:32:09:e5:51:41:4c:c8:b3:d8:e6:bb:db:91:45:5f:70:70:18:30:38:68:d0:a8:6d:94:cd:7d:a3:55:9f:1d:58:33:ba:6d:48:87:2a:b0:83:7c:97:d9:21:7c:e9:f2:5b:fa:5c:9f:0d cisco-av-pair ACS:CiscoSecure-Defined-ACL=#ACSACL#-IP-Aruba_AP_MAB-63c6ba1a MS-MPPE-Send-Key **** MS-MPPE-Recv-Key **** LicenseTypes Essential license consumed. Session Events 2024-02-01 08:25:17.522 Authentication succeeded

Steps 11001 Received RADIUS Access-Request - NNSY_PROD_AD 11017 RADIUS created a new session - nnsy.sy 15049 Evaluating Policy Group - NNSY_PROD_AD 15008 Evaluating Service Selection Policy - nnsy.sy 15048 Queried PIP - NNSY_PROD_AD 11507 Extracted EAP-Response/Identity - NNSY_PROD_AD (2 times) 12756 Prepared EAP-Request proposing TEAP with challenge 12625 Valid EAP-Key-Name attribute received 11006 Returned RADIUS Access-Challenge 11001 Received RADIUS Access-Request 11018 RADIUS is re-using an existing session 12301 Extracted EAP-Response/NAK requesting to use PEAP instead 12300 Prepared EAP-Request proposing PEAP with challenge 12625 Valid EAP-Key-Name attribute received 11006 Returned RADIUS Access-Challenge 11001 Received RADIUS Access-Request 11018 RADIUS is re-using an existing session 12302 Extracted EAP-Response containing PEAP challenge-response and accepting PEAP as negotiated 12318 Successfully negotiated PEAP version 0 12800 Extracted first TLS record; TLS handshake started 12805 Extracted TLS ClientHello message 12806 Prepared TLS ServerHello message 12801 Prepared TLS ChangeCipherSpec message 12802 Prepared TLS Finished message 12305 Prepared EAP-Request with another PEAP challenge 11006 Returned RADIUS Access-Challenge 11001 Received RADIUS Access-Request 11018 RADIUS is re-using an existing session 12304 Extracted EAP-Response containing PEAP challenge-response 12318 Successfully negotiated PEAP version 0 12802 Prepared TLS Finished message 12803 Extracted TLS ChangeCipherSpec message 12804 Extracted TLS Finished message 12816 TLS handshake succeeded 12311 PEAP session resumed successfully 15041 Evaluating Identity Policy 15048 Queried PIP - Network Access.EapTunnel 22072 Selected identity source sequence - PEAP 15013 Selected Identity Source - NNSY_PROD_AD 24432 Looking up user in Active Directory - NNSY_PROD_AD 24325 Resolving identity - 1nnaruba@nnsy.sy 24313 Search for matching accounts at join point - nnsy.sy 24359 Incoming identity was not rewritten - 1nnaruba@nnsy.sy 24319 Single matching account found in forest - sy 24323 Identity resolution detected single matching account 24355 LDAP fetch succeeded - nnsy.sy 24458 Not all Active Directory attributes are retrieved successfully - NNSY_PROD_AD 24100 Some of the expected attributes are not found on the subject record. The default values, if configured, will be used for these attributes - NNSY_PROD_AD 22037 Authentication Passed 12312 PEAP fast-reconnect - skipping inner method 12305 Prepared EAP-Request with another PEAP challenge 11006 Returned RADIUS Access-Challenge 11001 Received RADIUS Access-Request 11018 RADIUS is re-using an existing session 12304 Extracted EAP-Response containing PEAP challenge-response 24715 ISE has not confirmed locally previous successful machine authentication for user in Active Directory 15036 Evaluating Authorization Policy 24209 Looking up Endpoint in Internal Endpoints IDStore - 1nnaruba 24211 Found Endpoint in Internal Endpoints IDStore 15048 Queried PIP - Session.ANCPolicy 24432 Looking up user in Active Directory - 1nnaruba 24355 LDAP fetch succeeded 24416 User's Groups retrieval from Active Directory succeeded 24355 LDAP fetch succeeded 24458 Not all Active Directory attributes are retrieved successfully 24100 Some of the expected attributes are not found on the subject record. The default values, if configured, will be used for these attributes 15048 Queried PIP - NNSY_PROD_AD.ExternalGroups 15048 Queried PIP - Network Access.UserName 15048 Queried PIP - Network Access.AuthenticationStatus 15016 Selected Authorization Profile - 0066_Aruba_AP 11022 Added the dACL specified in the Authorization Profile 22081 Max sessions policy passed 22080 New accounting session created in Session cache 12306 PEAP authentication succeeded 11503 Prepared EAP-Success 11002 Returned RADIUS Access-Accept