topic Re: Issues after upgrading ISE 2.6 to patch 8 in Network Access Control

Issues after upgrading ISE 2.6 to patch 8

ryan14 — Fri, 12 Feb 2021 19:14:27 GMT

After upgrading ISE to latest patch in 2.6, my web GUI page will randomly stop working after several days. If you reboot the server, it will be fine for several days, then stop working again. Other services such as auth to the ISE server work fine, just the GUI is the problem. This was not a problem until after we installed the patch. Anyone else have this issue?

Re: Issues after upgrading ISE 2.6 to patch 8

Marcelo Morais — Fri, 12 Feb 2021 19:37:40 GMT

Hi @ryan14

when you said "GUI page will randomly stop working", in other words, what is the status of the Application Server process when you use the following command:

show application status ise

If the State = Initializing the try to use the following command:

application start ise

Hope this helps

Re: Issues after upgrading ISE 2.6 to patch 8

ryan14 — Fri, 12 Feb 2021 20:20:08 GMT

Yeah, so I my application status was stuck in initializing, but then after I tried to start it, something must of crashed ISE because my SSH session locked up.

Database and Application server services stopped running. I ended up going to console, halting services and performing a graceful reboot.

GUI is restored but I'm placing bets the GUI will stop working again since this isn't the first time.

Re: Issues after upgrading ISE 2.6 to patch 8

Damien Miller — Fri, 12 Feb 2021 22:37:56 GMT

The typical cause of the GUI to randomly stop loading is the VMware environment being backed up, or snapshotsbug taking place on the nodes. If an ISE node has a spapshot taken, or the storage is quiesced, then it will immediately crash the applications inside and stop working. The only way to recover from this is to power cycle the VM.

But since you say the authentication continues, then it could be another issue. Is it possible that the authentication is taking place on other nodes not impacted by this? Example, the PAN is unavailable, but other dedicated PSNs or redundant nodes are handling authentication?

Re: Issues after upgrading ISE 2.6 to patch 8

ryan14 — Mon, 15 Feb 2021 13:08:28 GMT

Thanks, I have checked that I have no current snapshots. I have paused Veeam backups to see if that results in any improved behavior with ISE web services. We only have one ISE node.

Re: Issues after upgrading ISE 2.6 to patch 8

Marcelo Morais — Mon, 15 Feb 2021 15:03:50 GMT

Hi @ryan14

are you able to install a fresh version of ISE 2.6 P8 (same hostname/IP Addr) and restore the backup on this fresh Node?

PS.: you can shutdown your VM and recreate another one from scratch.

Have a nice one.

Re: Issues after upgrading ISE 2.6 to patch 8

Damien Miller — Mon, 15 Feb 2021 18:41:15 GMT

Veeam backups require a quiesce of the storage which is the same as a snapshot, it freezes the VM for a moment to create a consistent point in time, often breaking the ISE applications with it. Hopefully pausing the Veeam backups will solve your issue, it's not recommended to backup ISE VM's from an external tool, but instead backup ISE from within the application leveraging the backup to repository capability.

Re: Issues after upgrading ISE 2.6 to patch 8

ryan14 — Mon, 15 Feb 2021 19:49:27 GMT

The "head scratcher" is that we have been backing it up with Veeam for over a year (same backup schedule) and haven't had an issue. The only change has been the patch to ISE.

Re: Issues after upgrading ISE 2.6 to patch 8

Marcelo Morais — Mon, 15 Feb 2021 22:48:06 GMT

Hi @ryan14 ,

take a look at: ISE Installation Guide, search for VM Virtual Machique Requirements.

Note: although it worked on the past, backing up ISE with an External Tool is not Cisco recommendation.

Hope this helps !!!

Re: Issues after upgrading ISE 2.6 to patch 8

Zainal.Aljufri — Tue, 16 Mar 2021 00:23:47 GMT

We are seeing exactly the same issues after upgrading to Patch 8 on 2.6, from a TAC engineer I spoke with it appears that 2.6 is plagued with performance related issues and resource leaks.

Re: Issues after upgrading ISE 2.6 to patch 8

Marcelo Morais — Tue, 16 Mar 2021 01:54:40 GMT

Hi @Zainal.Aljufri ,

what kind of issue you are experiencing? Delay on PAN GUI?

Regards.

Re: Issues after upgrading ISE 2.6 to patch 8

ryan14 — Tue, 16 Mar 2021 11:47:46 GMT

Knock on wood, my problems went away after upgrading to 2.8 patch 3.

Re: Issues after upgrading ISE 2.6 to patch 8

Zainal.Aljufri — Tue, 16 Mar 2021 12:28:48 GMT

Do you mean 2.7 patch 3? The TAC engineer seemed confident 2.7 was the way to go but I don't really want to go 2.7 without some solid evidence it is indeed stable.

Re: Issues after upgrading ISE 2.6 to patch 8

Zainal.Aljufri — Tue, 16 Mar 2021 12:31:34 GMT

Yes we are seeing delay, intermittent not loading of alarms and system summary on home page. Radius live logs also doesnt always load.

Seeing high load average that gets corrected by an application restart but eventually start getting alarms for that.

Have also seen a re-occurence of high authentication latency in Patch 8 that we had several occurences of in Patch 7 resulting in customer impact.

Re: Issues after upgrading ISE 2.6 to patch 8

ryan14 — Tue, 16 Mar 2021 13:01:39 GMT

Yea my bad 2.7 patch 3.

Re: Issues after upgrading ISE 2.6 to patch 8

Marcelo Morais — Tue, 16 Mar 2021 13:26:20 GMT

Hi @ryan14 ,

in other words, you were having issues (delays on PAN GUI, ...) and 2.7 P3 solved them, is that correct?

Any BUG ID that TAC identify that were related to your issues?

Hi @Zainal.Aljufri ,

OK, you have issues on 2.6 P8 ... I am having issues related to delay on PAN GUI on 2.7 P2 ... a possibility is the CSCvt81194 CPU spikes are being observed at policy HitCountCollector., fixed on 3.0, 2.7 P3 and 2.6 P8.

Any BUG ID that TAC identify that were related to your issues?

Regards

Re: Issues after upgrading ISE 2.6 to patch 8

Zainal.Aljufri — Tue, 16 Mar 2021 13:42:58 GMT

Thanks, do update us if you encounter any issues on 2.7 patch 3. How long have you been running this version with no issues?

Re: Issues after upgrading ISE 2.6 to patch 8

ryan14 — Tue, 16 Mar 2021 14:20:28 GMT

To previous question, no TAC did not find any bug ID. I didn't have any choice other than to try and upgrade b/c what did I have to lose? The GUI kept crashing so the server was in a bad state as is. Worse case if it was still problematic after upgrade, then yes I would see building a new server from a base image and importing a backup.

Been running on this for almost 2 weeks. We monitor the web services and haven't had any further alarms of it being not reachable.