EAP-FAST failed authentication because password mismatch

[Arne Bier]

Hello

I am slowly going insane because ISE is reporting that I have the wrong password for my locally defined Identity that I use during an EAP-FAST authentication.

• ISE 2.4 patch 5
• My authenticator is a Cisco 3750X switch running 15.2(2)E8
• The Cisco WAP is a 4800 and it is attached to the switch.  The switch port is configured for 802.1X
• I can do MAB authentications successfully.

My intention is to make the AP authenticate to ISE using EAP-FAST (username: appadmin3 password: Sup3rS3cret  )

The AP is registered to a WLC on version 8.8.100.0 - I have configured the AP via the WLC GUI as well as CLI.  Each time I use a different username to prove whether it has accepted my input data. 

I run debugs on the 3750 switch and also tcpdump on the ISE PSN.  I can see the correct user name each time. I entered the Radius shared secret in Wireshark to validate the Message Authenticator.  It all checks out clean.

The EAP-FAST tunnel is setup.  ISE finds the username each time too, but every time it tells me the password is wrong.

So far I have tried remediation

1. Removed the switch aaa config and rebuilt from scratch (to prove radius shared secret is correct). Set same shared secret on ISE
2. Manual config on the AP:   LAB-4800-01#capwap ap dot1x username appadmin2 password Sup3rS3cret
3. Extreme measures: restart application ise :-)
4. Tried EAP-PEAP on the AP instead
5. Tried a 2600 series AP on older code.
6. Rebooted AP.  shut interface,  clear auth sessions ... all the usual tricks  

And to top it off, I have seen this working well in another customer using ISE 2.4 patch 5 - albeit with a different set of APs.

I wish I could decode the password that the AP is sending. It is not present in the Radius authentication because it's an inner EAP method.

An annoying thing about ISE is also that if the user is not defined in ISE then the LiveLogs don't tell you what the attempted User-Name was (e.g. if I tried to auth with appadmin3 but that user didn't exist in ISE) - the LiveLogs report some useless message stating that username is "INVALID" -  and the user "INVALID" could not be found.  That's useless for troubleshooting purposes.

Steps

 	11001	Received RADIUS Access-Request
 	11017	RADIUS created a new session
 	15049	Evaluating Policy Group
 	15008	Evaluating Service Selection Policy
 	15048	Queried PIP - Normalised Radius.RadiusFlowType
 	11507	Extracted EAP-Response/Identity
 	12500	Prepared EAP-Request proposing EAP-TLS with challenge
 	12625	Valid EAP-Key-Name attribute received
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12101	Extracted EAP-Response/NAK requesting to use EAP-FAST instead
 	12100	Prepared EAP-Request proposing EAP-FAST with challenge
 	12625	Valid EAP-Key-Name attribute received
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12102	Extracted EAP-Response containing EAP-FAST challenge-response and accepting EAP-FAST as negotiated
 	12800	Extracted first TLS record; TLS handshake started
 	12805	Extracted TLS ClientHello message
 	12806	Prepared TLS ServerHello message
 	12808	Prepared TLS ServerKeyExchange message
 	12810	Prepared TLS ServerDone message
 	12811	Extracted TLS Certificate message containing client certificate
 	12105	Prepared EAP-Request with another EAP-FAST challenge
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12104	Extracted EAP-Response containing EAP-FAST challenge-response
 	12812	Extracted TLS ClientKeyExchange message
 	12813	Extracted TLS CertificateVerify message
 	12804	Extracted TLS Finished message
 	12801	Prepared TLS ChangeCipherSpec message
 	12802	Prepared TLS Finished message
 	12816	TLS handshake succeeded
 	12131	EAP-FAST built anonymous tunnel for purpose of PAC provisioning
 	12105	Prepared EAP-Request with another EAP-FAST challenge
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12104	Extracted EAP-Response containing EAP-FAST challenge-response
 	12125	EAP-FAST inner method started
 	11521	Prepared EAP-Request/Identity for inner EAP method
 	12105	Prepared EAP-Request with another EAP-FAST challenge
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12104	Extracted EAP-Response containing EAP-FAST challenge-response
 	11522	Extracted EAP-Response/Identity for inner EAP method
 	11806	Prepared EAP-Request for inner method proposing EAP-MSCHAP with challenge
 	12105	Prepared EAP-Request with another EAP-FAST challenge
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12104	Extracted EAP-Response containing EAP-FAST challenge-response
 	11808	Extracted EAP-Response containing EAP-MSCHAP challenge-response for inner method and accepting EAP-MSCHAP as negotiated
 	15041	Evaluating Identity Policy
 	15013	Selected Identity Source - Internal Users
 	24210	Looking up User in Internal Users IDStore - appadmin3
 	24212	Found User in Internal Users IDStore
 	22063	Wrong password
 	22057	The advanced option that is configured for a failed authentication request is used
 	22061	The 'Reject' advanced option is configured in case of a failed authentication request
 	11823	EAP-MSCHAP authentication attempt failed
 	12105	Prepared EAP-Request with another EAP-FAST challenge
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request
 	11018	RADIUS is re-using an existing session
 	12104	Extracted EAP-Response containing EAP-FAST challenge-response
 	11810	Extracted EAP-Response for inner method containing MSCHAP challenge-response
 	11815	Inner EAP-MSCHAP authentication failed
 	11520	Prepared EAP-Failure for inner EAP method
 	12117	EAP-FAST inner method finished with failure
 	22028	Authentication failed and the advanced options are ignored
 	12965	Sent EAP Result TLV indicating failure
 	12105	Prepared EAP-Request with another EAP-FAST challenge
 	11006	Returned RADIUS Access-Challenge
 	11001	Received RADIUS Access-Request (step latency=31183  ms Step latency=31183 ms)
 	11018	RADIUS is re-using an existing session
 	12104	Extracted EAP-Response containing EAP-FAST challenge-response
 	12814	Prepared TLS Alert message
 	12852	Cryptographic processing of received buffer failed
 	61025	Open secure connection with TLS peer
 	11504	Prepared EAP-Failure
 	11003	Returned RADIUS Access-Reject

user

[paul]

Arne,

I can't help on the EAP-FAST issue, but I agree on the INVALID username and in my mind that is a bug.  I don't mind if ISE sets the Network Access:Username to INVALID because this is set from the authentication process, but changing the RADIUS username which is the value sent from the NAD to INVALID is just wrong.  ISE should be reporting exactly what the NAD sent in the details without manipulation.

[Damien Miller]

I provided feedback through my CSE to return the functionality around "INVALID" usernames.  Also agree, completely useless when you are trying to troubleshoot, and disabling it for thirty minutes after the fact doesn't fix the previous attempts.  Often by the time we are engaged to look at an issue, the user isn't actively authenticating.  Trying to unmask for thirty minutes just frustrates me.  I think the reason they gave for this behavior change was to mask potential passwords entered accidentally in the username fields. 

I just want to ability to disable masking for good, returning it back to how it was.  

One thing that came to mind, but sounds like you ruled out anyways with different accounts.  I have had invalid passwords like this when there is a matching machine account with the same name as a user account in AD.  User actually matched on machine, and spits back an invalid password.  

[Arne Bier]

Just to chime in to what @Damien Miller  is referring to.  I had no idea this feature existed. yes it's intentional and you can suppress the INVALID thing for 30 min - I had no idea

[paul]

Yeah, I didn't know that existed either.  Learned something new today so I guess I can take the rest of the day off.  :)

Like I said I don't mind the masking for Network Access:Username and if that is the field that is populated in the Live Logs and RADIUS authentication reports that is fine.  99% of the people are only going to be looking at that data. But when I drill into the details to troubleshoot the RADIUS username should show the real username sent by the NAD. 

[Arne Bier · Accepted solution]

Quick update. I noticed that my customer’s Root and Issuing CA cert was installed without the checkbox “use for client auth”. I noticed this when our EAP-TLS was failing (and the error message was “ISE does not recognise the client CA chain” - it took me a while to figure out how that could be). I then retested the Cisco 2600 AP EAP-FAST and now it works. It doesn’t make much sense but I am glad. Weird thing is that when I unchecked the Client checkbox and retested the AP auth, it still worked. I was unable to reproduce the issue. 

I will mark this as resolved. Main lesson learned out of this is to never forget to check that Client checkbox for the entire PKI CA chain. Seems that any TLS tunnel establishment needs at least one complete CA chain enabled for client auth. Sort of makes sense.