topic If you want send me a in Network Access Control

ISE behind load balancer

bert.lefevre — Mon, 11 Mar 2019 01:37:10 GMT

I have a question regarding ISE profiling servers that are placed behind a load balancer:

If you have a ISE environment where both computers and users are being authenticated, and Machine Access Restriction (MAR) is enabled (so users can only authenticate on a previously authenticated machine), are the ISE servers aware of all succesfull computer authentications handled by the other ISE servers?

For example:

There are 2 ISE appliances (ISE01 and ISE02) behind a load balancer.

A user starts up his computer, and computer authentication is handled by ISE01 (and the authentication is successful). At the moment the user logs in on that computer, the load balancer chooses ISE02 to authenticate the user.

Will ISE02 be aware that the corresponding computer was already succesfully authenticated on ISE01, so that the user is able to log in? Or will it deny the user authentication because it thinks the computer is not (yet) authenticated and Machine Access Restrictions is enabled?

Kind regards,

Bert

ISE behind load balancer

Nicolas Darchis — Thu, 08 Dec 2011 17:05:43 GMT

are the ISE servers aware of all succesfull computer authentications handled by the other ISE servers?

=> No

they are independant servers that just replicate their configuration.

So a user should authenticate always with the same ISE.

Moreover a load balancer kills profiling since profiling requires you to span some traffic to an ISE

Re: ISE behind load balancer

bert.lefevre — Fri, 09 Dec 2011 08:18:02 GMT

Nicolas,

thanks a lot for this explanation. Now I'm at least warned that we shouldn't place them behind a load balancer (although load balancing ISE policy servers is mentioned in the Cisco ISE user guide under section 9 "Setting up Cisco ISE in a distributed Environment").

Actually I don't understand why Cisco didn't implement synchronization of the machine cache for MAR (which is in fact just a cache of the mac-addresses of the authenticated computers) between ISE servers that are in the same node group. Synchronizing a table of mac-adresses isn't a big challenge I assume? Or is there another reason this wasn't implemented?

Implementing this synchronization would be a big improvement if you ask me, as this adds extra redundancy in case one ISE server fails and users try to log on to machines that were already authenticated on that failed ISE.

Kind regards

Re: ISE behind load balancer

Nicolas Darchis — Fri, 09 Dec 2011 08:48:35 GMT

I would guess it's probably a bit more complex than a mac table synchronization. Especially the real time synchronization could take a lot of bandwitdh/cpu I can guess but yes it would make sense as a feature request. I think the feature request list has to be 1km long now for ISE 🙂

I'll check with the developers if it's already on their roadmap or not.

Re: ISE behind load balancer

Nicolas Darchis — Tue, 13 Dec 2011 15:40:24 GMT

After checking, the MAR cache synchronization will be present in ACS 5.4

Logically it should also be included in a future ISE release but no further details.

Re: ISE behind load balancer

bert.lefevre — Wed, 14 Dec 2011 10:12:50 GMT

Nicolas,

I'm glad to hear that the MAR cache sync is already in development for ACS and I hope it will be soon implemented in ISE also. I'll keep an eye on new release notes.

Thanks a lot!

Re: ISE behind load balancer

bert.lefevre — Wed, 14 Dec 2011 10:13:41 GMT

Nicolas,

I'm glad to hear that the MAR cache sync is already in development for ACS and I hope it will be soon implemented in ISE also. I'll keep an eye on new release notes.

Thanks a lot!

ISE behind load balancer

Craig Hyps — Wed, 18 Jan 2012 20:25:01 GMT

>> they are independant servers that just replicate their configuration.

So a user should authenticate always with the same ISE.

Moreover a load balancer kills profiling since profiling requires you to span some traffic to an ISE <<

Not entirely correct. Policy Service nodes are most certainly supported behind a load balancer which is the intention of a node group. This is often the preferred method for high availability and scaling. In addition to supporting load distribution of RADIUS and other requests, members of a node group maintain a heartbeat to determine if a peer member should fail. If so, the Monitoring node is queried to determine if there are any transient sessions which may require clean-up via RADIUS COA to help ensure that an endpoint is left in a defunt auth state. LB functionality will depend on load balancer used. Cisco ACE for example supports stickiness of RADIUS transactions based on source IP, Calling-Station-ID, or Framed-IP-Address.

The impact of LB on profiling or other Policy Service node functions depends on the service/probe in question. For services like client provisioning, posture, and central web auth, https redirection always occurs back to the node which terminated the RADIUS session, so LB is transparent provided direct access is permitted to the real IP for redirected https trnasactions (RADIUS tranasactions would be sent to virtual IP).

Specific to profiling, SNMP Queries can be triggered and will be sent by Policy Service node that received the RADIUS Accounting Start packet (assumes RADIUS probe enabled) or SNMP Trap (assumes SNMP Trap probe enabled). SPAN is only one data collection method used primarily for HTTP or DHCP capture. Methods other than SPAN/RSPAN are available to capture this data, but if used, then it is correct that there is no specific mechansim to move SPANs from one interface to another in case of NIC or node failure. I believe intelligent taps are available that can accomplish this, or else traffic can be mirrored to multiple nodes at the cost of duplicating profile data.

As noted, replication of MAR cache will be added to ACS 5.4, and no, this feature is not altogether trivial due to the number of transactions and updates that must be replicated and kept in sync across each node performing RADIUS services.

/CH

Re: ISE behind load balancer

g-hopkinson — Fri, 27 Jan 2012 09:56:47 GMT

Hi,

Thanks, this is useful information.

In the documentation, it mentions that when using a node group the NAS should have all of the ISE's configured under AAA to allow CoA. Would it be possible to use the VIP address and NAT the ISE's when they instigate a outbound connection from behind the ACE for CoA, or is Radius a bit deeper than that.

Would you configure a node group for a pair of policy nodes on a remote site that were not load balanced, what makes this specific to policy server nodes behind a LB. Assume both policy server nodes where configured in all NAS's on that particular site.

I assume when profiling is carried out all data is replicated to the admin node anyway, this is using DHCP helper, DNS, SNMP. When you start to look at a distributed ISE architechure and using profiling it starts to get messy, potentially a lot of helper, SNMP addresses have to be configured in NAS's.

Thanks.

Hi Craig, We are in the

laposilaszlo — Mon, 29 Jun 2015 07:08:59 GMT

Hi Craig,

We are in the process of migrating our ISE infrastructure from ACE to F5.

We followed your document for the configuration.

All looks ok except EAP-TLS authentication. (PEAP user/computer works fine)

In the document there is nothing special mentioned that needs to be done for TLS.

I think it may be related to fragmentation but not sure.

I can also add here that if we point the NAD's to the PSN directly it works.

The problem is only when we use the VIP.

(PEAP work with the VIP also)

Do you know if something special needs to be done for TLS to work.

Any information or hint is appreciated.

Thanks,

Laszlo

It is not uncommon to see

Craig Hyps — Mon, 29 Jun 2015 14:34:29 GMT

It is not uncommon to see RADIUS load balancing issues with EAP-TLS related to fragmentation. The typical cases are either 1) failure of load balancer to reassemble large RADIUS packets, for example, TLS with larger key sizes, or 2) dropping of fragments by load balancer that are deemed too small. For first case, both Cisco ACE and F5 LTM should accommodate automatic reassembly if using the standard LB mechanism for RADIUS. LTM does not reassemble FastL4 by default, but that protocol is normally not used and guide does not use that profile for RADIUS. If fragments too small, for both ACE and LTM you would need to change the default minimum fragment size to accept the exceptionally small fragment for reassembly. This can serve as a workaround, but recommend find and eliminate the device causing RADIUS packets to be fragmented below reasonable size.

Another common issue in load balancing is failure to understand exact path taken for the entire flow to/from real servers. Often there is a case where ingress packets take one path but responses take another path. This asymmetry often results in packet drops by load balancer or other device in the path.

/CH

Hi laposilaszlo,

pvzcisco07 — Wed, 21 Jun 2017 04:10:31 GMT

Hi laposilaszlo,

Did you end up resolving the issue with EAP-TLS?

I am using a F5 to load balance RADIUS & having the same issue.

I am not sure if I want to alter the fragment size as a work around.

Regards,

Raj

Hi Raj,

laposilaszlo — Wed, 21 Jun 2017 05:10:29 GMT

Hi Raj,

Yes we solved it.

In our case it was the Nexus switch. It has a security feature that discards small UDP packets.

And the last part of the certificate was a small UDP packet that got discarded.

So we disabled this one and all is ok now.

After this we had some problem on the F5 regarding UDP fragments that was solved with an F5 upgrade.

This was a long time ago so this fix sould be in the current releses.

laszlo

Hi Laszlo,

pvzcisco07 — Wed, 21 Jun 2017 05:19:50 GMT

Hi Laszlo,

Thanks for the quick reply!

We run a Nexus core as well. Could you please tell me how to check/disable the feature?

We are running ISE 2.2 & the issue still seems to persists.

We are running version 11.5.3 on the F5.

Regards,

Raj

MTN-GDC-AGG-N7018A-1# show

laposilaszlo — Wed, 21 Jun 2017 09:57:30 GMT

MTN-GDC-AGG-N7018A-1# show hardware forwarding ip verify module 3

IPv4 and v6 IDS Checks Status Packets Failed

-----------------------------+---------+------------------

address source broadcast Enabled 0

address source multicast Enabled 0

address destination zero Enabled 0

address identical Enabled 134

address reserved Enabled 2334940

address class-e Disabled --

checksum Enabled 0

protocol Enabled 0

fragment Enabled 34254

length minimum Enabled 0

length consistent Enabled 0

length maximum max-frag Enabled 0

length maximum udp Disabled --

length maximum max-tcp Enabled 0

tcp flags Disabled --

tcp tiny-frag Enabled 176552

version Enabled 0

-----------------------------+---------+------------------

IPv6 IDS Checks Status Packets Failed

-----------------------------+---------+------------------

length consistent Enabled 0

length maximum max-frag Enabled 0

length maximum udp Disabled --

length maximum max-tcp Enabled 0

tcp tiny-frag Enabled 0

version Enabled 0

Workaround:

Disable packet length check using command

Config t

no hardware ip verify length

Thanks Laszlo,

pvzcisco07 — Thu, 22 Jun 2017 02:40:13 GMT

Thanks Laszlo,

The fragment feature was disabled on my switch, so I enabled it & disabled the minimum length as suggested by you. But the issue still persists

7010-CS01(config)# show hardware forwarding ip verify module 3 | i fra

fragment Enabled 0

7010-CS01# show hardware forwarding ip verify module 3 | i leng
length minimum Disabled --

The failure logs on ISE shows the failure reason as below

5411 Supplicant stopped responding to ISE
Failure Reason	12931 Supplicant stopped responding to ISE after sending it the first EAP-TLS message

As soon as I point it to the PSN instead of the VIP it works straight away.

I have logged a TAC case for the issue.

On a side note, did you have TACACS traffic load balanced through the F5?

I also forgot to mention, our

pvzcisco07 — Thu, 22 Jun 2017 04:47:50 GMT

I also forgot to mention, our deployment is not inline. The PSNs & the F5 are on different networks.

We only balance Radius for

laposilaszlo — Thu, 22 Jun 2017 04:59:28 GMT

We only balance Radius for now.

The best thing you can do is capture traffic, that is what TAC is going to ask you also.

Capture on the client, then on F5 and ISE.(F5 has this functionality from CLI, you can capture incomming and outgoing traffic at the same time)

Then you will see where the packets are missing.

This is what we did and found the problem.

In our deployment the ISE PSN's have their gateway in the F5.

If PEAP is working then 100% that it is a fragmentation problem.

Laszlo

If you want send me a

laposilaszlo — Thu, 22 Jun 2017 05:10:45 GMT

If you want send me a screnshoot of the F5 config for ISE and i can compare it to our.

laszlo

Hi Laszlo,

pvzcisco07 — Fri, 23 Jun 2017 07:56:41 GMT

Hi Laszlo,

In my case the servers are on different L3 domains to the F5. So basically, the f5 is not inline with the servers which is recommended design on the guide. So, is this going to work in my setup? I would have thought it would work at least with SNAT but it doesn't. I have attached the screenshots