ISE 2.4 Licence usage exploding

[derobbacher]

Hello Cisco ISE experts,

we are suffering now the second time from a continous license usage increase

in this important big hospital with around 2500 Radius WLAN Clients:

After crossing our bought 4000 licences I was noticed by an alarm again.

The last time the trouble went up to around 14.000 just 2 days befor xmas.

Leaving me with only 2 days to go to solve the problem before ISE would stop working.

It took me several days and sleepless nights until

Cisco TAC did the following to cure the situation last time:

Perform steps in order:

o             [3]Purge M&T Operational Data

o             [4]Reset M&T Database

Then:

o             [1]Reset M&T Session Database

o             [2]Rebuild M&T Unusable Indexes

In my feeling the ise does not free again licences after WLAN clients disconnect.

Are there any known bugs around this problem ?

And who can advise what to do ?

Is this problem solved meanwhile ?

Please advise what to do.

Kind regards

Wini

[Arne Bier]

do you have RADIUS Accounting enabled on your NAS's?  And if so, is it working (i.e. shared secret correct and reaching ISE etc.)? It's usually the case that ISE cannot manage the sessions because it has no idea about whether or not the client is still active. RADIUS accounting is your only hope.

If RADIUS accounting is enabled and working (and Interim-Updates are set to either 0 (for Cisco gear) or perhaps 3600 seconds) then ISE should be getting some information about those sessions at least on that interval)

Failing that, it might be a bug - I think there have been bugs related to ISE no releasing licenses.

You might also consider whether you have clients that are doing MAC randomization (e.g. newer Android clients) - back in the day it was only Apple that randomized the Wi-Fi MAC address in their probe requests (which NAC doesn't care about)- but Android and Windows 10 can now randomize even once the Wi-Fi station is associated. That plays all sorts of havoc with NAC solutions that rely on the MAC address of an endpoint being deterministic/stable. I don't know how ISE handles that - I have not looked into it.

regards

[Damien Miller]

I'll add something on to this that an astute customer pointed out to me recently. If you set up a WLC, and define the radius servers with the "network user" box checked, then a misconfiguration on the WLAN will send radius accounting starts without even leveraging ISE for authentication. This WLAN was open auth and not 802.1x, but it still sends accounting.  

Ex. This WLC has two servers under radius accounting servers where "network user" is checked making them the default. The second image shows the misconfiguration. By checking the two boxes "authentication servers = enabled" and "accounting servers = enabled", and not specifying radius server 1 or 2 in the list....This sends radius accounting to ISE and also creates stales endpoint entries in the context visibility DB.  

[Arne Bier]

When I see this kind of config in customer networks I usually uncheck the boxes - I have not seen a valid use case for it, unless somebody is counting the number of unauthenticated guests in a RADIUS server somewhere. 

I'd like to see what these records look like. Does ISE only care about accounting records for sessions where ISE was involved in the Authentication? I guess Accounting without the other two initial AA_ 's doesn't make much sense.

I would have thought that ISE would at least log the accounting records (Acct Start) and then close off the record when the Acct Stop came in. In that case there should be no licensing confusion.

[Damien Miller]

It might but I'm not 100% certain on how it behaves with this sort of misconfiguration. It needs some analysis in a lab since in this case the impact on licensing in production was not the focus but rather the massive growth of the endpoint DB. It certainly generates a lot of useless mac/endpoint entries in context visibility if there is high turn over of guests.

[neil.woodhouse]

BTW stale sessions on ISE should be removed after a maximum of 5 days

https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/215419-ise-session-management-and-posture.html#anc7

[yogesh2009]

1) Make sure accounting is set up correctly.You can get accounting report from ISE and compare number of starts to stops 

2) Get the authentication report from ISE and check for stale connections

3) If you use any ssid with mab , once user passed mab auth it will consume base license.This could be an issue if random users attempt to connect to guest ssid .It is perhaps better to put an extra layer of auth on the guest ssid otherwise some random user can cause base license consumption and also deplete your guest dhcp pool.

4)I dont think Mac randomization will play  role here. afaik , mac randomization is used during probing only so I doubt that can be the issue here.

[Arne Bier]

Hi @yogesh2009 

MAC randomization is probably not the cause here but I flagged it because newer Android devices are now allowing the MAC address to be randomized AFTER association - that means, it's no longer used only during probing. Have a quick read of this Android article.

Same thing in Windows 10 - have a read here.

This is already causing some issues with MAB authentication.

The Cisco BU wrote a nice article on it - there is a bit mask in the MAC OUI that indicates whether the MAC address is random - the Cisco employee documented a nice filter that you can use in ISE to deal with these use cases - highly recommend you check it out. 

[yogesh2009]

thanks for this useful info. I will definitely check it out.

[yogesh2009]

I read the article and this could be a  possibility .This is why I do not like guest networks are "technically open" :)

[derobbacher]

Hi Guys,

thank You for Your Feedback.

I'm not convinced, sorry.

Isn't this maybe Cisco Bug CSCvj50257 ?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvj50257

Symptom:
Mismatch in active endpoint counter and Live Sessions

Conditions:
--- ISE 2.3 patch 2 reset and synchronized context visibility
--- ISE 2.4 clean install

Workaround:
None

Last Modified:

Jun 8,2020

Status:

Fixed

For me it Looks like the Problem is fixed (Status Fixed)

But I cannot find information about which ISE Patch or SW-Release fixed this Problem.

Can anyone please point me to the solution for this bug ?

Thank You very much

Kind regards

Wini

[Greg Gibbs]

The bug link you posted shows ISE 2.4 patch 6 in the 'Known Fixed Releases' section and the Release Notes also list that bug ID in the fixes.

https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/release_notes/b_ise_24_rn.html#id_98767

If you have patch 6 or higher installed but the same symptom exists, it is likely not the same root cause as that particular bug. If that's the case, you might need to open a TAC case to investigate in more detail.