ISE Alarm: Warning: Profiler Queue Size Limit Reached

[wpalumbo06]

Anyone know what this error means and more importantly, is it anything to really be concerned about?  We started receiving this today for one of our PSNs and have been getting the alert every five minutes.  There hasn't been any 'known' impacts from this but it's very annoying and the Cisco documentation is a little vague.

Thank you....

[nspasov]

What version and patch # are you running?

[wpalumbo06]

1.2 patch 2

[nspasov]

Hmm, I have had issues with profiling but not since moving to v1.2. What profiling sensors do you have enabled? What is your deployment look like:

1. How many endpoints

2. Types of profiling probes enabled

3. Wired/wireless

4. How many ISE nodes and what personas are enabled on each node

[wpalumbo06]

1. How many endpoints
17,500 active endpoints

2. Types of profiling probes enabled
DHCP
HTTP
RADIUS
NMAP
DNS
SNMP query
SNMP trap

3. Wired/wireless
wired

4. How many ISE nodes and what personas are enabled on each node

16 appliances (2 PAN, 2 MNT and 12 PSNs)  Each node has a single personna.

[nspasov]

OK, so you have a pretty good size deployment so things will have to be looked at carefully. So:

1. Are you running 15.x code on your switches? If yes, do you have IOS Sensor running?

2. SNMP Traps is a bit of a heavy hitter on the profiling probes. Any specific reason(s) you are running that? The reason I ask is because you don't really need that probe if you have the Radius one running

3. Do you have "EndPoint Attribute Filter" enabled?

4. Are you sending all profiling information to all PSNs? And are all PSNs configured with the above mentioned profiling probes?

[wpalumbo06]

1. Are you running 15.x code on your switches? If yes, do you have IOS Sensor running?

Yes - 15.x for 4500s and 3750s.  We use SNMP for 6500s

2. SNMP Traps is a bit of a heavy hitter on the profiling probes. Any specific reason(s) you are running that? The reason I ask is because you don't really need that probe if you have the Radius one running

6500s are using this...but I found it is also enabled on roughly 200 3750s that are configured with device sensor

3. Do you have "EndPoint Attribute Filter" enabled?

Yes.

4. Are you sending all profiling information to all PSNs? And are all PSNs configured with the above mentioned profiling probes?

All PSNs are behind f5 and all PSNs are configured with the same probes.

[nspasov]

Hmm, I am sorry but I will have to ask more questions:

1. You mentioned that your PSNs are behind a load balancer, but are the nodes in a "node group?" If they are not you should place them in a node group. If they are you will need to split them as the max recommended nodes per node group is 10. 

2. If the nodes are indeed in a node group are they all L2 adjacent?

3. If your deployment consists of VMs and not physical appliances, can you confirm that both the adequate CPU/RAM are allocated and reserved for the VMs?

Here are also some recommendations:

1. If possible, move to IOS sensor on all of your switches and disable the probes that will no longer be needed. 

2. If #1 is not possible check the configurations on your NADs and where Device Sensor is configured you should remove, IP Helper and SNMP Query based configs. This will prevent duplicate information from being sent to the PSNs

3. Look to completely eliminate SNMP Traps based configurations for ISE. That probe along with Netflow and the Span probes are pretty heavy hitters

4. Make sure that you are using Device Sensor on your WLCs as well

5. Use the latest patch

6. Get a support case going with Cisco and have them take a look :)

For more info you should take a look at the following Cisco Live Sessions:

BRKSEC-3697 and BRKSEC-3699

Thank you for rating helpful posts!