topic Re: ISE Logs too slow in Network Access Control

ISE Logs too slow

manvik — Fri, 27 May 2022 09:23:35 GMT

ISE logs tooooo slow, it keeps showing "fetching records" for tacacs, radius, alrm logs etc

deployment is a simple one with DC & DR nodes

MNT, ADM, monito are all in one node

Re: ISE Logs too slow

ahollifield — Fri, 27 May 2022 12:26:33 GMT

Version? Are you within the scale limits for a standalone deployment? VM or appliance?

https://www.cisco.com/c/en/us/td/docs/security/ise/performance_and_scalability/b_ise_perf_and_scale.html

Re: ISE Logs too slow

manvik — Fri, 27 May 2022 12:58:34 GMT

ISE 3.1 patch 3

new installation

2 node standalone

concurrent tacacs auth 200 devices. Total 400 devices

Re: ISE Logs too slow

ahollifield — Fri, 27 May 2022 13:12:53 GMT

Do you have the proper VM resources allocated? Also, are you seeing any alarms on the dashboard for disk read/write speeds?

https://www.cisco.com/c/en/us/td/docs/security/ise/3-1/install_guide/b_ise_InstallationGuide31/b_ise_InstallationGuide31_chapter_2.html#hardware-virtual-appliance-req

Re: ISE Logs too slow

manvik — Fri, 27 May 2022 13:24:10 GMT

yes 32 GB RAM and 16 CPU

There was a "queue link error timeout" between DC and DR in the Alarma

Re: ISE Logs too slow

ahollifield — Fri, 27 May 2022 23:46:07 GMT

That is related to the ISE messaging service (how logs are transferred between nodes) and is likely the cause of your issue. Re-generate the ISE Root CA and then the ISE messaging certificate for all nodes.

Re: ISE Logs too slow

Marcelo Morais — Sat, 28 May 2022 01:19:09 GMT

Hi @manvik ,

if my understanding is correct, at Home > Alarms dashboard, you have a Queue Link Error, am I correct?

If the answer is yes, could you please:

1. click the Queue Link Error link

2. take a PrtScr of the new window (generated by the Queue Link Error link)

Regards

Re: ISE Logs too slow

thomas — Sat, 28 May 2022 15:35:54 GMT

Please see to provide the community with enough details to help you in the future.

Re: ISE Logs too slow

manvik — Mon, 06 Jun 2022 11:26:52 GMT

Alarm says; Queue Link Error: Message=From DR ISE to DC ISE; cause=Timeout

I removed DR ISE from the cluster, now there no delay in logs.

Re: ISE Logs too slow

ahollifield — Mon, 06 Jun 2022 12:04:37 GMT

Timeouts typically indicate the ISE messaging port TCP/8671 is not allowed through the firewall.

Re: ISE Logs too slow

Marcelo Morais — Mon, 06 Jun 2022 13:37:08 GMT

Hi @manvik ,

please take a look at the following document: ISE - Queue Link Error.

Hope this helps !!!

Re: ISE Logs too slow

manvik — Thu, 09 Jun 2022 12:31:57 GMT

thank you @ahollifield @Marcelo Morais

the ports are open between both ISE nodes. I removed DR ISE from cluster slowness disappeared.

Again added DR ISE to cluster, now the logs are slow again.

I get the attached error now.

Re: ISE Logs too slow

ahollifield — Thu, 09 Jun 2022 15:08:20 GMT

Unknown CA. You need to re-generate the ISE root CA and then the ISE Messaging certificates on all nodes.

Re: ISE Logs too slow

adamscottmaster2013 — Thu, 09 Jun 2022 15:22:00 GMT

I run into this issue before. To resolve it, I did:

1- Administration --> Logging --> Log settings

Uncheck the box "Use "ISE Messaging Service" for UDP Syslogs delivery to MnT"

I think that will resolve your issue.