Issues after upgrading ISE 2.6 to patch 8

[ryan14]

After upgrading ISE to latest patch in 2.6, my web GUI page will randomly stop working after several days. If you reboot the server, it will be fine for several days, then stop working again. Other services such as auth to the ISE server work fine, just the GUI is the problem. This was not a problem until after we installed the patch. Anyone else have this issue?

[Marcelo Morais]

Hi @ryan14 

 when you said "GUI page will randomly stop working", in other words, what is the status of the Application Server process when you use the following command:

show application status ise

If the State = Initializing the try to use the following command:

application start ise

Hope this helps

[ryan14]

Yeah, so I my application status was stuck in initializing, but then after I tried to start it, something must of crashed ISE because my SSH session locked up.

Database and Application server services stopped running. I ended up going to console, halting services and performing a graceful reboot.

GUI is restored but I'm placing bets the GUI will stop working again since this isn't the first time.

[Damien Miller]

The typical cause of the GUI to randomly stop loading is the VMware environment being backed up, or snapshotsbug taking place on the nodes. If an ISE node has a spapshot taken, or the storage is quiesced, then it will immediately crash the applications inside and stop working. The only way to recover from this is to power cycle the VM. 

But since you say the authentication continues, then it could be another issue. Is it possible that the authentication is taking place on other nodes not impacted by this? Example, the PAN is unavailable, but other dedicated PSNs or redundant nodes are handling authentication? 

[ryan14]

Thanks, I have checked that I have no current snapshots. I have paused Veeam backups to see if that results in any improved behavior with ISE web services. We only have one ISE node.

[Marcelo Morais]

Hi @ryan14 

 are you able to install a fresh version of ISE 2.6 P8 (same hostname/IP Addr) and restore the backup on this fresh Node?

PS.: you can shutdown your VM and recreate another one from scratch.

Have a nice one.

[Damien Miller]

Veeam backups require a quiesce of the storage which is the same as a snapshot, it freezes the VM for a moment to create a consistent point in time, often breaking the ISE applications with it. Hopefully pausing the Veeam backups will solve your issue, it's not recommended to backup ISE VM's from an external tool, but instead backup ISE from within the application leveraging the backup to repository capability. 

[ryan14]

The "head scratcher" is that we have been backing it up with Veeam for over a year (same backup schedule) and haven't had an issue. The only change has been the patch to ISE.

[Marcelo Morais · Accepted solution]

Hi @ryan14 ,

 take a look at: ISE Installation Guide, search for VM Virtual Machique Requirements.

 Note: although it worked on the past, backing up ISE with an External Tool is not Cisco recommendation.

Hope this helps !!!

[Zainal.Aljufri]

We are seeing exactly the same issues after upgrading to Patch 8 on 2.6, from a TAC engineer I spoke with it appears that 2.6 is plagued with performance related issues and resource leaks.

[Marcelo Morais]

Hi @Zainal.Aljufri ,

 what kind of issue you are experiencing? Delay on PAN GUI?

Regards.

[Zainal.Aljufri]

Yes we are seeing delay, intermittent not loading of alarms and system summary on home page. Radius live logs also doesnt always load.

Seeing high load average that gets corrected by an application restart but eventually start getting alarms for that.

Have also seen a re-occurence of high authentication latency in Patch 8 that we had several occurences of in Patch 7 resulting in customer impact.

[ryan14]

Knock on wood, my problems went away after upgrading to 2.8 patch 3.

[Zainal.Aljufri]

Do you mean 2.7 patch 3? The TAC engineer seemed confident 2.7 was the way to go but I don't really want to go 2.7 without some solid evidence it is indeed stable.

[ryan14]

Yea my bad 2.7 patch 3.