ISE Alarm Message: Profiler Queue Size Limit Reached

[zachartl]

Hello,

We've a six appliance ISE deployment. Two PANs ; Two PSNs RADIUS ; Two PSNs TACACS+.

We started receiving this message this morning and they've persisted since. I went to the Support Community and picked up on an older support thread from 2016, that referred to this error message. The recommendation/best practice is to limit the profiler, RADIUS, to ONE/Single PSN. Does this still hold true? We're at version 3.1 P7.

Thank you,

Terry

[Arne Bier]

I can't say I have seen that one before, and I have a few deployments with that exact configuration. I only enable Profiler on the two PSNs that are handling the RADIUS traffic. In most cases I also only enable the following probes:

• RADIUS (where the customer's switches/WLC are Cisco and have Device Sensor enabled)
• AD (for customers that have ISE integrated with AD)
• NMAP (for customers that have a lot of IOT and this helps with profiling)

The other probes are disabled. 

What probes do you have enabled?

[zachartl]

Hi Arne,

These are the Probes we have enabled - SNMP ; DHCP ; RADIUS ; Nmap ; Active Directory. These PSNs are our RADIUS PSNs. Over the weekend we began getting these messages and I'm so far unable to find a root cause. While searching for a related issue I found a Community thread that spoke to one profiler per deployment. I've since enabled the Endpoint Attribute Filter hoping this might help. This PSN is the Primary RADIUS PSN and it's taking a resource beating and the messages above persist as a result. 

[Arne Bier · Accepted solution]

Ok that makes sense. You said "appliances" - are these SNS-36XX or SNS-37XX? In VM deployments, the first question that Cisco TAC would ask is whether you have reserved the CPU and MEM resources. 

Queues build up when the rate of arrival exceeds the ability to process the events - this means that perhaps the PSNs are under spec'd?  it could also be a bug. But let's assume for now it's not.  How many RADIUS requests per second does the PSN get at the busiest time of the day?

Are you able to spread some of the RADIUS load across the other PSNs?  It sounds like you don't have a load balancer, but you could try to manually load balance by re-arranging the Primary and Secondary RADIUS server configs in your largest WLCs or Switch Stacks. Manual Load Balancing

Are you sure you need the DHCP probe? (only needed if you have ip helper statements on your switches - but if these switches are capable of Device Sensor then you should use Device Sensor and disable DHCP probe in ISE)

Are you sure you need the SNMP probe? This is used to poll (investigate) the switch to learn about endpoints. In most cases you don't need this if the switch sends the data in RADIUS.

[zachartl]

They are SNS-36XXs. I've disabled DHCP Probe. I can disable SNMP Probe. 

Can you please tell me where to look to get the RADIUS Request Count (when you get a chance)?

I will look into how the WLCs are managing Device/Device Counts and how they're configured to use the Primary and Secondary RADIUS PSNs.

Thank you! 

[Arne Bier]

You can see this data in the Key Performance Metrics Reports.

Operations > Reports > Diagnostics > Key Performance Metrics

[zachartl]

Way cool,

we hit 8299 RADIUS Requests in an Hour, on one of our RADIUS PSNS and 8448 on the other that our apex, today.

[zachartl]

Good Morning,

I ended up having to access the CIMC of the RADIUS PSN. Within the Server Utilization Screen, CPU and Overall Utilization were just at 100%. I was unable to determine a root cause and so I decided on a Hard Reset via the CIMC. It's been a couple of hours and we appear to be okay. I'd like to know how this came about. We just installed P7 to remediate some CVEs. Not sure what to make of it.

Thank you for your Assistance, it was very Helpful.