topic Re: Azure Cloud deployment - fragmentation issues in Network Access Control

Azure Cloud deployment - fragmentation issues

Jagermeister — Tue, 21 Jan 2025 13:32:56 GMT

Hi all,

I have my ISE cluster deployed in Microsoft Azure and I have spent for over a week to troubleshoot a certain issue that I'm having. Unfortunately it seems that Azure is causing this and I'm wondering what everyone does to mitigate this.

The issue:

Access for 802.1X supplicants is very unreliable, sometimes it works but other times it fails 802.1X and the supplicants do not want to authenticate anymore. The 802.1X method is PEAP with EAP-TLS as inner-method. Checking the logs on the PAN are showing the following in these scenarios:

Failure Reason: 5440 Endpoint abandoned EAP Session and started new
Failure Reason 5411 Supplicant stopped responding to ISE

I can reproduce this behavior by simply cycling the switch port a few times, it just 'breaks' again.

After taking several pcaps and analyzing them, it seems that the traffic is fragmenting sometimes, which might make sense since I'm trying to achieve certificate based authentication. I've read multiple sources and it seems that Microsoft Azure is dropping out of order UDP traffic and that this can cause the two failure messages that I listed above.

I've decided to test this and deployed a PSN node outside of Microsoft Azure and joined this node to my cluster. All clients that I point towards PSN authenticate perfectly fine. The PAN's alert reports also only report that the dropped radius packets are from my Azure PSN and NOT from the locally deployed PSN.

Now I have read about the following options;

- Get Microsoft to enable 'allow out-of-order fragments' option

- Pin the subscription to ensure all instances within that subscription are deployed on hardware generation 7

The thing is that Microsoft seem to require that the Azure subscription needs to be empty. I think this is quite a PITA, since almost no one has an empty subscription and creating a new one is not always an option.

I'm wondering if something can be done outside of Azure to improve this situation. Would adjusting the MTU of the Cisco ISE PSN itself help for example?

Re: Azure Cloud deployment - fragmentation issues

Mark Elsen — Tue, 21 Jan 2025 17:32:15 GMT

- FYI : https://community.cisco.com/t5/network-access-control/eap-tls-to-azure-ise-is-failing-but-not-with-an-ise-node-in-the/m-p/4783440#M580104

Re: Azure Cloud deployment - fragmentation issues

Greg Gibbs — Wed, 22 Jan 2025 04:31:26 GMT

There are multiple layers of fragmentation at play here. The main culprit is the large EAP messages used by the Windows supplicant. Changing the MTU on the PSNs will not make a difference for that.

See the discussion here for more details - https://community.cisco.com/t5/network-access-control/azure-packet-fragmentation/td-p/5205223

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Wed, 22 Jan 2025 10:36:04 GMT

Thanks for your answer,

So, fragmentation using a Windows supplicant is unavoidable if I understand correctly? I'm currently trying to get in touch with Microsoft support to see what they can do for me.

Another idea that I had is to to use Radius over TLS (Radsec), unfortunately it seems that ISE only supports Radius over DTLS, which is probably not solving this since its also UDP. I guess I can perform another test by deploying a PSN in one of the two regions that have the fix applied already and see if it works properly.

Do other public cloud providers also drop out of order UDP in your experience or is this just Azure?

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Wed, 22 Jan 2025 10:39:31 GMT

Thanks,

Seems that I cannot do much without Microsoft enabling that flag. Quite a bummer since its very inconvenient that the requirement is that the Azure subscription needs to be empty.

Re: Azure Cloud deployment - fragmentation issues

Greg Gibbs — Wed, 22 Jan 2025 23:07:04 GMT

I worked with a large global resources company to migrate their ISE cluster from on-prem to AWS, which does not have this issue. As far as I know, MS is the only supported cloud provider that has this issue.

For customers I speak with that are planning the move to public cloud and have multi-cloud environments, I strongly recommend deploying ISE in AWS instead of Azure for this reason.

Re: Azure Cloud deployment - fragmentation issues

danielecappelletti — Fri, 24 Jan 2025 09:14:18 GMT

We also have this problem... We read on this Microsoft Q&A link below, that the paramater enable-udp-fragment-reordering is only possible to enable on a new subscription without any resources deployed on it: https://learn.microsoft.com/en-us/answers/questions/996062/azure-drops-my-udp-fragmentated-packets-when-they

Could you update this thread if you find a way to do this?

Thanks

Re: Azure Cloud deployment - fragmentation issues

PSM — Fri, 24 Jan 2025 14:59:30 GMT

@danielecappelletti Microsoft can only do that only some limited number of SKUs and not all. They can't do it for ISE machines SKUs

Re: Azure Cloud deployment - fragmentation issues

ccieexpert — Sat, 25 Jan 2025 08:45:45 GMT

If you have a few NADs, then you could build a IPSEC tunnel all the way from a NAD to ISE as shown below:

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/222720-configure-ipsec-tunnel-between-cisco-wlc.html

this is with WLC, but switches like 9300 also support ipsec tunnels.. When you use a ipsec tunnel, the fragmented packets are encapsulated in the tunnel , so Azure doesnt see it (altleast that is what i think ) as the tunnel terminates on ISE. But this may not be scalable for 100s of NADs...

Also, you say it works sometimes and fails intermittenly.. i havent looked into the details of the problem of out of order. . Have you actually taken packet captures on the supplicant/NAD and also at Azure to see if they are truly out of order ? How do you connect to Azure ? do you have multiple paths which is causing out of order or is it because of the fragmentation process which on a device, which may cause a smaller fragment (compared to a larger) getting send first (and thus causing out of order ? it would be good to understand where it is going out of order and see if we can mitigate that.

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Mon, 27 Jan 2025 14:51:43 GMT

Thanks for your reply,

Today I have made some other captures but I'm starting to get a bit lost to be honest. I have made two pcaps; one on the office SD-WAN appliance (LAN interface), and one on the ISE PSN.

SD-WAN:

PSN:

taking radius session ID 153 ( identification frame 2e7a) for example

LAN SD-WAN:

ISE:

I think this suggests that the PSN in azure actually receives the UDP traffic and the Access-Challenge that is sent back seems also to be received on the LAN interface of the SD-WAN appliance. So, on the ISE logging the clients keeps failing with the following reason codes:

Failure Reason: 5440 Endpoint abandoned EAP Session and started new
Failure Reason 5411 Supplicant stopped responding to ISE

In the pcap's I do not see any access-accept message until many attempts have been made:

These attempts are all from the same supplicant, logs show over a 50 failed.

Any idea?

Re: Azure Cloud deployment - fragmentation issues

Scott Fella — Mon, 27 Jan 2025 18:21:39 GMT

You need to open a ticket with Azure engineering to enable that flag. I worked at MS and when I moved ISE to Azure, I had to have Azure engineering enable udp fragmentation. It's pretty quick and its not impacting, but they have to enable it on a virtual gateway. I posted this a while back when I got things working.

https://community.cisco.com/t5/network-access-control/eap-tls-to-azure-ise-is-failing-but-not-with-an-ise-node-in-the/td-p/4739038

** Update **

Would adjusting the MTU of the Cisco ISE PSN itself help for example? No... I tried that also:)

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Mon, 27 Jan 2025 18:31:28 GMT

Hi Scott,

Thanks, I'm still waiting for Microsoft to respond. So, in my case I'm having an express route to Azure. Are you saying that I need an empty subscription AND that they have to enable it per express route / VPN gateway?

also, when you were having these issues, did it fail every single time or just regularly? In my case it does succeed sometimes but it fails a lot.

Re: Azure Cloud deployment - fragmentation issues

Scott Fella — Mon, 27 Jan 2025 18:48:47 GMT

Well this can only be done on a virtual gateway, in your case the VPN gateway. So if your ISE cube is deployed over multiple virtual gateway's, then you need to have each on touched. We didn't have an empty subscription, we had a few hundred items in that subscription which I was worried about, but once they enabled that flag, took a few seconds, there was no glitch, no tickets and things started to work.

The fragmentation was only on EAP-TLS, PEAP was working fine. I tested this with a VM on-prem vs in Azure so I can at least have a baseline of what was working and packet captures to compare. EAP-TLS failed every time no matter what device we tested with. We tested with PEAP just for this use case, but everything needed to be using EAP-TLS to pass auth in production. We didn't allow PEAP at all. Keep in mind, once you get an engineer, they will need to escalate that to a tier 1 engineer to do the work.

Just reference this to the engineer:

https://learn.microsoft.com/en-us/answers/questions/996062/azure-drops-my-udp-fragmentated-packets-when-they

Re: Azure Cloud deployment - fragmentation issues

Scott Fella — Mon, 27 Jan 2025 18:53:37 GMT

The github link no longer works:) but here is something from their docs:

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-tcpip-performance-tuning#azure-and-fragmentation

Re: Azure Cloud deployment - fragmentation issues

Scott Fella — Mon, 27 Jan 2025 18:55:46 GMT

@Jagermeister You might even want to look at this. I was also testing with the 9800-CL in Azure. This doc has info on ISE in Azure also with packet captures.

https://www.cisco.com/c/en/us/support/docs/troubleshooting/222339-troubleshoot-fragmentation-issues-affec.html

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Tue, 28 Jan 2025 15:57:17 GMT

Thanks,

Today I did a little test and configured my supplicant to use EAP-TLS instead of PEAP-EAP-TLS. Oddly enough it actually works fine with EAP-TLS but PEAP-EAP-TLS is constantly failing. One thing I noticed in a pcap on the client is that the Meraki switch first seems to attempt EAP-TLS, then legacy NAK's are sent and after a requested EAP-PEAP is sent. Probably not really a problem but it might delay the authentication a bit.

I'm still struggling to find 'evidence' in my PCAPs that Azure is actually dropping it. I can confirm that some radius access-requests were transmitted from the client (eg id=18) but on the ISE PSN I see id 17 and 19 but 18 isn't present, meaning that it didn't arrive.

Re: Azure Cloud deployment - fragmentation issues

Scott Fella — Tue, 28 Jan 2025 16:02:04 GMT

They document that they will drop any UDP packets that come in out of order. The reason I think EAP-TLS fialed for me was the size of the certificate we were using. I still think you should have them enable the flag and go from there, the Azure engineer can also take a packet capture on their end to validate that they are dropping the packets, you will not have the ability to do that capture, but they can.

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Tue, 28 Jan 2025 16:53:12 GMT

Yes I guess so. I actually just encountered the same behavior for EAP-TLS again so false alarm I guess :(. Hopefully Microsoft will reply soon to me

Re: Azure Cloud deployment - fragmentation issues

Scott Fella — Tue, 28 Jan 2025 17:39:52 GMT

Keep us posted. Have then take a capture while you test so you have that info. They can verify that its being dropped.

Re: Azure Cloud deployment - fragmentation issues

Jagermeister — Fri, 07 Feb 2025 13:58:08 GMT

little update here - I am in touch with Microsoft now but they claim that they can only enable this flag for traffic that is originating from the internet to a public IP. We are using a Azure Express route so that is not the case and they claim that they CANNOT enable this for a express route. What a nightmare!!

I'm not sure what I can do anymore, I simply do not understand how people are able to use Cisco ISE in combination with a ExpressRoute then. Trying to reduce the payload to avoid fragmentation seems impossible and sending radius traffic over the internet is also not an option for me.