EAP-TLS to Azure ISE is failing but not with an ISE node in the DC/LAB

[Scott Fella]

I have a four node ISE 3.2 cluster in Azure, two PAN+MNT and two separate PSN's.  My plan was to migrate off our old ISE 2.2 in our DC eventually.  Currently I have migrated all our TACACS+ to our Azure ISE cluster with no issues.  I started to test EAP-PEAP and EAP-TLS and pointing a test controller to the Azure PSN's. EAP-PEAP is working, but EAP-TLS is not.  I do have a TAC case opened and they are still looking at data. 

I spun up an ISE 3.2 VM in our lab and joined that to the Azure cluster.  When I point the controller WLAN to this node, EAP-TLS works fine.  I also have standalone ISE 3.1 and a ISE 3.2 in the lab and EAP-TLS works fine, eliminating certificates and configuration as all the nodes have the same policies (did a restore to keep things consistent).  

What we have, is all our sites even our lab is connected to our DC via DMVPN.  Our connection to Azure VNET is via an IPSec tunnel from our DC router to our Azure Virtual Gateway.  Our MTU on that tunnel is set for 1438, but Azure support mentioned that default on their VPN is 1400.  Would this cause any issues?  My next step would be after the holidays is to setup a new Virtual Gateway and tunnel with MTU of 1400 or lower and stand up a new ISE node joined to that cluster to test with.

Errors we see are the following:

5400,5411,5440

[marce1000]

 - FYI : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/218197-configure-ise-3-2-eap-tls-with-azure-act.html#anc9

 M.

[Scott Fella]

Thanks @marce1000.  I did follow a few guides I found and then thing is that EAP-TLS is working with a PSN joined to the cluster in Azure but is located in the lab.  My issue is when I point the controller WLAN to the PSN in Azure, auth fails.  We are also not looking up in Azure AD.  We are using copies of Preloaded_Certificate_Profile that will reference the CN or the SAN in the policies.  

My Deployment is like the foollowing:

PANMNT01 (Azure)

PANMNT02 (Azure)

PSN01 (Azure) - Failing on EAP-TLS

PSN02 (Azure)- Failing on EAP-TLS

PSN03 (Lab but joined to the cluster)- Working on EAP-TLS

[thomas]

You stated the error codes but did not provide any of the actual details that tell you Why it failed which is in the FailureReason field.

5411 and 5440 errors are usually associated with certificate trust issues - your supplicant does not trust the ISE PSN certificate(s). Check you supplicant configuration and ISE certificates on those PSNs.

[Scott Fella]

@thomas Here are some screen shots of the various errors we see when the controller wlan is pointing to the Azure PSN's.  The thing is, I have another PSN that is not in Azure that is joined to the same cluster and that works fine.  I have the full chain imported in ISE and also making sure that the team that owns the device has the full chain of ISE in the trusted cert store.  Initially for the Windows 11, they only had the root CA as that has been working fine on all the old and test ISE cluster that I have.  We do have an mtu mismatch from our DC router tunnel to Azure VPN which we will have to address in January.

Majority of the Failure reasons are:

5440 Endpoint abandoned EAP session and started new

12935 Supplicant stopped responding to ISE during EAP-TLS certificate exchange

[Greg Gibbs]

If there is an MTU mismatch, that is likely the culprit. EAP-TLS is very sensitive to fragmentation and the generic errors you are seeing are common when the supplicant drops the handshake due to fragmentation of the certificate chain. I would suggest doing a packet capture on the client to see what the EAP communication looks like to determine if you are getting part of the chain and then the session drops.
If you have the ability, you could also try dropping the MTU on the access switch temporarily to prevent the MTU mismatch along the path and see if that resolves the issue.

[Scott Fella]

Thanks, I will talk to my peer to see if they want to drop the mtu on the switch where the controller is connected to. This way we can also test again to see if this might help temporarily until my peers can schedule a change after the holidays on the tunnel.

[Scott Fella]

Tested today and set the mtu on the access switch interface vlan to 1300 and then 1000 and EAP-TLS was still failing.  I will test more after the holidays.

[Scott Fella · Accepted solution]

Just wanted to post my solution, at least part.  I had to open a ticket with Azure VPN team to enable-udp-fragment-reordering.
https://learn.microsoft.com/en-us/answers/questions/996062/azure-drops-my-udp-fragmentated-packets-when-they

The is done on a subscription level and you choose which virtual gateway this flag needs to be enabled on.  Keep in mind that any new virtual gateways will not have this flag and you must open another ticket to get this enabled.

Once the engineer enabled this flag, EAP-TLS started working. Our packet captures showed that EAP-TLS was getting fragmented, but Azure drops these since they are out of order.  We still had a mismatch on the tunnel mtu, ours is at 1500 and Azure is at 1400, but that is something that needs to be cleaned up.  

So hopefully this helps if you run into this issue in Azure.

[wifievangelist]

@Scott FellaIn my opinion, this is not really a solved solution. If Cisco provided a way to control the size of the RADIUS packet sent to ISE, we wouldn't have to worry about fragmentation in the first place. Cisco did provide this for RADIUS access-responses sent from ISE (using the framed MTU on the WLC) but not for packets in the other direction. They still don’t take the cloud seriously, and it is very disappointing.

[Scott Fella]

@wifievangelist I think this would not be an issue if MS didn't drop udp fragmentation.  The issue is that radius is not getting to ISE, and would be an issue with other vendors also. It is what it is and that was a fix from MS.

[wifievangelist]

My answer is that this is a cisco forum not a microsoft one. I am here to see what cisco can do not Microsoft. And I suggested what cisco could have done, what is your thought on that?

[Greg Gibbs]

@wifievangelist , also see a related discussion here:
https://community.cisco.com/t5/network-access-control/azure-packet-fragmentation/td-p/5205223

There are multiple levels of fragmentation involved and one of the problems is that the Windows native supplicant uses large EAP messages (1470 bytes), which forces the IP fragmentation. This is a hardcoded setting which cannot be changed.
The result of the fragmentation is that the last packet is smaller, leading to a faster transmit, and therefore received out-of-sequence.

See EAP Fragmentation Implementations and Behavior

Cisco has no control over the behaviour of the Windows native supplicant.

This issue is not present in other public cloud environments like AWS.

[wifievangelist]

it doesnt make sense to me. Radius work based on access request and response. Lets say if the last rdius packet was smaller but, I wont send that until I get radius access reponse for a previous one anyways. So, that is not the case. It might be the case that the larget packets itself got fragmented along the path and due to platform limitations azure started dropping the out of order.

Other thing is even if the native supplicant sends this 1470 byte eap frame (when I was checking it was bigger than that), radius as a protocol can still engineer and fragment the eap frames from client machines while putting adding it as an AVP so that one radius packet will not exceed the certain number.

It is a simple question, if ISE when sends its certificate not resulting in fragmantation then why WLC sending will result? Because WLC cannot make it a smaller byte size when engineer the radius packet and cisco has to just say that it is a platform limitation just like MS acked their limitation.