topic Re: Cisco ISE Deployment in Azure - Nightmare experience! in Network Access Control

Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Mon, 28 Oct 2024 15:51:26 GMT

Hi,

Has anyone been able to successfully deploy ISE in Azure using expressroute from on-premise to the cloud.

We have had ISE running in Azure for about 3-4 months now and have noticed a large amount of fragmentation using EAP-TLS.

The Cisco guide suggests a fix has been applied in East Asia and West Central US however it's not been applied to UK South where our VMs are located. We have also raised this with Microsoft support however they cannot tell us what fix this is or when it will be rolled out to our region.

We enquired about the "enable allow out-of-order-fragments" option however they said this could only be applied if the traffic is coming from the internet, not via expressroute or VPN which is obviously not going to work as we wouldn't send radius traffic straight over the internet! Other requirements include deploying VMs in a brand-new empty subscription and deploying to a Dv4 VM, again this is not possible as the VMs are already in use within an existing subscription.

It's incredibly frustrating as Cisco can't seem to provide much info on the workaround and Microsoft are just fobbing us off by saying that the information is from Cisco and not from them!

I'd be grateful if other members on this forum have successfully deployed ISE in Azure with connectivity via ER or VPN and not seen the fragmentation issues when using EAP-TLS.

TIA.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Cristian Matei — Mon, 28 Oct 2024 16:38:17 GMT

Hi,

Welcome to the jungle, it is a well-known challenge. Look here, work with Microsoft and move your VM's on Gen7 HW:

https://www.ciscolive.com/on-demand/on-demand-library.html?zid=pp&search.event=1716482947962001yag9&search=BRKSEC-2039#/session/1717269125663001tXab

Best,

Cristian.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Tue, 29 Oct 2024 10:18:13 GMT

Thanks @Cristian Matei - the errors on that webinar are exactly what we are seeing in Azure. Microsoft just keep fobbing us off with the following:

Customer is receiving out-of-order fragments via an instance level public IP from the internet. ExpressRoute and VPN first party gateways are not supported. IP fragments do not work in load balancing scenarios, so ensure this is a public IP attached directly to a VM.
Customer requires an empty subscription.
Customer wishes to deploy a VM SKU that is compatible with hardware that supports out-of-order fragments. Typically, this is Dv4 and earlier (the newest SKUs such as Dv5 do NOT support this).

Are you aware of anyone who has successfully got Microsoft to fix this issue? It would be good to know who the people are in the Cisco engineering team who have worked on Microsoft with this are so maybe they could shed some light on whats required.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

thomas — Tue, 29 Oct 2024 13:25:09 GMT

Documented at https://cs.co/ise-berg#azure and

Deploy Cisco ISE Natively on Cloud Platforms
- Cisco ISE on Azure Cloud Services
  - Known Limitations of Cisco ISE in Microsoft Azure Cloud Services - includes details about the need for allow out-of-order fragments for UDP

Re: Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Tue, 29 Oct 2024 14:04:18 GMT

thanks @thomas i have also logged a support ticket with Cisco to see if they can provide any information. My account manager at Microsoft has asked me to find out exactly what fix has been applied in East Asia and West Central US as it specifically highlights this in the deployment guide, do you know what this fix is?

Due to this known issue, do one of the following:

Select regions where Azure Cloud has already implemented the fixes: East Asia (eastasia) and West Central US (westcentralus).”

Re: Cisco ISE Deployment in Azure - Nightmare experience!

thomas — Tue, 29 Oct 2024 15:07:23 GMT

This is 100% a Microsoft Azure issue. You will need to ask them. This problem does not exist in other cloud providers.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Tue, 29 Oct 2024 15:12:26 GMT

@thomas I absolutely agree however the documentation from Cisco says it's been resolved in 2 regions but nobody can tell me what the resolution was so I can ask Microsoft to make the same fix in UK South. I have asked senior engineers at Microsoft but they cannot seem to find out what this supposed fix is so I am hoping someone at Cisco can point me in the right direction!

Re: Cisco ISE Deployment in Azure - Nightmare experience!

thomas — Tue, 29 Oct 2024 15:22:06 GMT

From Known Limitations of Cisco ISE in Microsoft Azure Cloud Services :

In Azure, a networking virtual network stack drops out-of-order fragments without forwarding them to the end virtual machine host. This design aims to address the network security vulnerability FragmentSmack, as documented in Azure and fragmentation.

Cisco ISE deployments on Azure typically leverage VPN solutions like Dynamic Multipoint Virtual Private Networks (DMVPN) and Software-Defined Wide Area Networks (SD-WAN), where the IPsec tunnel overheads can cause MTU and fragmentation issues. In such scenarios, Cisco ISE may not receive complete RADIUS packets and an authentication failure occurs without triggering a failure error log.

Due to this known issue, do one of the following:
1. Select regions where Azure Cloud has already implemented the fixes: East Asia (eastasia) and West Central US (westcentralus).
2. Cisco ISE customers should raise an Azure support ticket. Microsoft has agreed to take the following actions:
  1. Pin the subscription to ensure all instances within that subscription are deployed on hardware generation 7.
  2. Enable the "allow out-of-order fragments" option, which allows fragments to pass through to the destination instead of being dropped.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Tue, 29 Oct 2024 15:30:06 GMT

I understand that and have seen that article numerous times but Microsoft are saying they can only enable out of order fragments for VMs with a public IP attached to the NIC, this isn't applicable to us because its internal traffic it doesn't go over the internet.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Cristian Matei — Tue, 29 Oct 2024 16:24:07 GMT

Hi,

Mate, as an engineer I feel your pain; my scenarios were done via Internet, being aware of the challenge. I suggest open a TAC case to get the info you're looking for from Cisco, someone has to know; otherwise, if you're stuck and can't move VM's to a region with the fix, try deployment over Internet and use IPsec tunnels for RADIUS packets or RADIUS over DTLS.

Good luck,

Cristian.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Damon Kalajzich — Mon, 10 Mar 2025 08:14:20 GMT

I found most cisco devices fragment incorrectly, but I managed to work around the fragmentation issue in azure by implementing the following
https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine-33/220568-configure-ise-3-3-native-ipsec-to-secure.html
Essentially this hides the out of order fragments within IPSEC so Azure is none the wiser.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Fri, 14 Mar 2025 08:38:55 GMT

Hi @Damon Kalajzich , thanks for your feedback, how would you achieve this if you're using Cisco Meraki for Wireless 802.1x?

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Damon Kalajzich — Sun, 16 Mar 2025 23:21:16 GMT

Sorry I don't have any experience with Meraki, to work it would require the device (NAD) sending the radius request to have the ability to configure a ipsec tunnel to ISE.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

CitizenGenet — Thu, 20 Mar 2025 11:25:55 GMT

Is anyone else dealing with this problem still? We're currently in the testing phase of an ISE deployment in Azure and I believe we're encountering this exact situation.

However, the Microsoft agreement no longer seems to be in place because our consistent entreaties to their support engineers seem to be in vain. Something may have changed on their side, but they're saying the 'enable udp out of order fragments' solution is no longer supported (this is the response even after account rep involvement).

I don't have to remind anyone here, but it throws a major wrench in your plans when core tenet networking laws, such as udp packet reassembly, are no longer something you can reliably account for when implementing a solution.

A lot of the threads on this topic seem to just end or they're older so I'm wondering if anyone has recently experienced this and has successfully developed a creative workaround?

I think the ipsec tunnel is an interesting idea but it doesn't seem to scale well if you have a lot of NADs and the maintenance, difficulty in troubleshooting, etc. seems to be a major consideration.
We've also been down the radius-dtls path, which seemed promising. However some of our equipment, like Meraki, only implemented Radsec - so that's out the window (at least for our specific situation).
Also, I'm wondering if anyone approached this from the MTU perspective? For example, restricting the MTU at the OS level of client PCs to something like 1200 to avoid fragmentation from the start?

Just trying to come up with some ideas and explore all the contingency plans. I'm interested to see if anyone else is still fighting this and if you're having any success.

Thanks,

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Damon Kalajzich — Thu, 20 Mar 2025 22:39:53 GMT

I have been dealing with it the last few months, and yes it is difficult to get MS to acknowledge this. I got to the point where I was advised that MS could enable this feature for me, but...
It can only been enabled on a empty subscription, They pin any new resources added to the subscription to a specific hardware cluster with the feature enabled. There are other caveats which would mean a complete re-architect of our azure deployment, like you can't use a Azure VPN gateway or Azure Firewall in the path of the traffic to ISE, as these can't be pinned to the specif hardware with the feature enabled.

This is why I have gone down the path of using IPSEC from the NAD to ISE to hide the out of order fragments that cisco devices send from the azure network stack.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Damon Kalajzich — Thu, 20 Mar 2025 22:42:52 GMT

My MS case number is 2501270030001062 if you want to pass that on to your MS engineer so they can see it is still a option on their side, but I do note that my MS engineer did say he believed that the option is going to be phased out over time. As all their new VM SKU's with advanced networking don't support the option.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

InfraISE2020 — Fri, 21 Mar 2025 07:32:19 GMT

I am in the same position as @CitizenGenet as we use Cisco Meraki for wireless and radius-dtls is not supported on Meraki and EAP-DTLS is not supported on Cisco ISE so were are pretty stuck at the moment. I think Cisco and Microsoft are aware of the issue and trying to resolve it but nothing seems close. One of the suggestions was to use FastPath over expressroute (additional cost) but that doesn't work with virtual WANs so thats not an option either.

Re: Cisco ISE Deployment in Azure - Nightmare experience!

CitizenGenet — Fri, 21 Mar 2025 12:03:13 GMT

Thanks for providing that case number, Damon – that’s very helpful but it’s not inspiring reading some of the caveats involved in the answer you received from Microsoft.

Are you having success with your IPSEC idea? Is it working as expected and you’re getting consistent authentications?

Re: Cisco ISE Deployment in Azure - Nightmare experience!

Damon Kalajzich — Mon, 24 Mar 2025 00:58:14 GMT

we have the IPSEC solution working on 5 of our 7 9800 WLC's for some reason we have 2 9800 WLC where the IKE service is not starting, TAC investigating.
There are also some caveats with the ISE solution to be aware of. It requires a new Interface and IP on the ISE node to service the IPSEC and radius connection. G0 can't be used. The traffic selection on what to send via the IPSEC tunnel is done via a static route, so all communication from the ISE node to the NAD will be sent via the tunnel, even if you are hitting ISE via the original interface IP (non ipsec tunnel IP) so it all communication not just radius traffic. so if migrating radius also migrate tacacs and COA etc,

Re: Cisco ISE Deployment in Azure - Nightmare experience!

CitizenGenet — Thu, 31 Jul 2025 08:54:38 GMT

I've been working on this for a few months and just wanted to provide an update on the experience, in case it helps someone else in this position.

Everyone's topology or connection into Azure will be different. In my scenario, there are NVAs deployed in Azure that terminate SDWAN tunnels or from our on-premises environment. Based on other posts related to this topic, this seems to be an uncommon configuration but it creates an opportunity to fix this without Microsoft enabling the allow out-of-order udp fragments flag.

At these NVAs (e.g. routers, firewalls) you can configure packet reassembly that will take the out-of-order udp RADIUS packets (after traversing the internet) and put them back in order before they leave the outgoing interface and traverse the rest of your Azure environment. For Cisco, the command ip virtual-reassembly in/out will accomplish this goal. For other vendors, like Palo Alto, they seem to automatically reassemble fragments.

In regard to Microsoft's enable out-of-order udp fragments flag, that was a longer process. Ultimately, this flag was enabled for one Vnet. We didn't have to empty our subscription or rebuild our servers or anything like that. However, we were told this flag is not supported if you have other Microsoft based objects in your Azure traffic path, like a standard load balancer, firewall, or vHUB connection.