topic Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN) in Network Access Control

ISE 2.6 Patch 4 Deferred/Removed

obrien-r — Thu, 27 Feb 2020 22:31:28 GMT

For any like me that was awaiting some of the fixes in Patch 4 and jumped on and downloaded it, please be aware it has been deferred and removed from CCO due to a bug/problem it introduces. Bug is CSCvt18276

According to my CCO notification, the expected replacement - Patch 5 - is due approx 12 Mar

Anyway, hope that information helps someone...

Re: ISE 2.6 Patch 4 Deferred/Removed

Leo Laohoo — Fri, 28 Feb 2020 00:00:44 GMT

@obrien-r wrote:

the expected replacement - Patch 5 - is due approx 12 Mar

12 March 2020? Is there enough time to test if Patch 5 fixes CSCvt18276 and, most importantly, not break anything else?

NOTE: ISE 2.6, patch 4 bring back "old memories" of the very, very short-lived ISE 2.4, patch 2 (ISE 2.4 Patch 2 Release Now Available).

Re: ISE 2.6 Patch 4 Deferred/Removed

obrien-r — Fri, 28 Feb 2020 00:19:01 GMT

I would expect so as there is already a "hotfix" available from the TAC

Re: ISE 2.6 Patch 4 Deferred/Removed

Damien Miller — Fri, 28 Feb 2020 00:27:45 GMT

2.4 patch 4 was deferred, 4 is an unlucky number for ISE.

Re: ISE 2.6 Patch 4 Deferred/Removed

cjkaufman@dmgov.org — Mon, 02 Mar 2020 16:35:18 GMT

Thanks so much for this heads up! We were going to install patch 4 this morning. (had already downloaded it.) Thankfully I saw this bug in my weekly Cisco email...

cjk

PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Leo Laohoo — Thu, 09 Apr 2020 05:33:04 GMT

Heads-up to everyone: We are having tremendous issues with ISE 2.6, patch 6: Extremely high CPU + very high external authentication latency.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Marvin Rhoads — Thu, 09 Apr 2020 15:23:09 GMT

Thanks for sharing @Leo Laohoo . Did you get a BugID from the TAC?

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Leo Laohoo — Fri, 10 Apr 2020 02:07:27 GMT

@Marvin Rhoads wrote:

Did you get a BugID from the TAC?

No, @Marvin Rhoads.

In order to stabilize the cluster, we had to roll back (and remove) patch 6.

I'll pass on the Bug ID when TAC shares it with us.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Ping Zhou — Fri, 10 Apr 2020 03:30:23 GMT

@Leo Laohoo In this case, which version is stable...relatively? I was gonna patch mine, but now I guess I need to wait a little bit.

@Leo Laohoo wrote:
@Marvin Rhoads wrote:
Did you get a BugID from the TAC?
No, @Marvin Rhoads.
In order to stabilize the cluster, we had to roll back (and remove) patch 6.
I'll pass on the Bug ID when TAC shares it with us.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Leo Laohoo — Fri, 10 Apr 2020 03:48:55 GMT

For the time being, ISE 2.6, patch 5.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Ping Zhou — Fri, 10 Apr 2020 03:55:52 GMT

@Leo Laohoo Thanks!

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Leo Laohoo — Thu, 23 Apr 2020 02:37:31 GMT

Update: TAC believes there is a memory leak in the profiler.
~~No Bug ID (yet).~~
14 April 2020

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Leo Laohoo — Thu, 23 Apr 2020 02:39:49 GMT

Update: 23 April 2020
TAC has identified the issue to be CSCvt18276 which has re-emerged in patch 6.
I guess the next step is to wait for TAC to release a fix/patch to patch 6?

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Marvin Rhoads — Thu, 23 Apr 2020 02:47:22 GMT

Sev 1 Catastrophic bug with 88 support cases re-appears!!!

So much for regression testing.

Thanks for sharing the BugID @Leo Laohoo

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Arne Bier — Sun, 26 Apr 2020 11:08:37 GMT

Does this only affect customers who need Guest Portal?

What is the recommended ISE 2.6 patch release (patch 3 or 5?)

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Leo Laohoo — Sun, 26 Apr 2020 11:37:47 GMT

Hi Arne,
ISE 2.6, patch 5.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

jedubois — Tue, 28 Apr 2020 16:08:09 GMT

Folks,

This is not an issue with patch 6. The problem here is that endpoints grew to a very large size because of CSCvt18276 which only impacts patch 4. Since these large endpoints still exist in the database after upgrading to patch 6 and we fixed CSCvt18276 in patch 6, we can now read those very large endpoints into cache. This causes the high resource utilization being seen. The workaround for CSCvt18276 needs to be done once in this case. Restoring from a backup taken prior to applying patch 4, or deleting the endpoint database and allowing re-profiling to occur.

Again, this it not a new issue in patch 6, it is left over specifically for deployments that had installed patch 4, and did not rollback to patch 3 and remediate the issue.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

alex.duckworth — Thu, 30 Apr 2020 02:29:33 GMT

The email advice that went out after patch 4 was pulled was poor.

"--- If you have downloaded patch 4 for ISE 2.6, installed it and not facing the issue:

Please install the hotfix immediately to fix the issue caused by bug CSCvt18276."

There was no instructions to know if an endpoint was corrupted, nor any way to identify if there had been any impact.

So with the advice that the hotfix would "fix" the issue, we assumed it was resolved and would not cause a sev1 to when we installed the next patch. The only two options that were actually available to us at the end of the day were to restore from backup (which was not feasible, primarily due to the elapsed time that had occurred since patch 4 was installed and the number of changes that had occurred since) or wipe the endpoint database, which due to it's size (360,000+ endpoints) could not be done from the GUI.

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

Damien Miller — Fri, 01 May 2020 04:40:33 GMT

A little concerning when the solution was to dump all the endpoints and/or restore a backup. I'm left wondering how many deployments are out there that didn't realize this was/is a problem and still have patch 4+5 installed, or rolled 4 back and installed patch 5 later on, and are now going to install patch 6. Having 89 reported cases on that p4 bug, how many went unreported/unnoticed.

Any way to identify that a deployment could be impacted?

Re: PSA: ISE 2.6, patch 6 (THUMBS DOWN)

hslai — Sat, 02 May 2020 05:26:36 GMT

First of all, CSCvt18276 affects only those deployments with more than one PSN and more than one PSN enabled for profiling. Secondly, only if ISE 2.6 Patch 4 ever installed, which can be identified by ISE admin CLI:

show version history