topic Re: ISE backup hung in Network Access Control

ISE backup hung

muradhattar — Mon, 11 Mar 2019 06:23:49 GMT

It appears like the backup is hung on the primary ISE 1.3 node.

- I created a repository in the GUI and started a backup to it. The repository was not accessible so the backup got stuck at 75%, it wasn't actually doing anything for the last week.

ISE01# show backup status
%% Configuration backup status
%% ----------------------------
% backup name: pre_2.0_upgrade
% repository: Shared_Network_Drive
% start date: Mon Nov 30 11:35:59 PST 2015
% scheduled: no
% triggered from: Admin web UI
% host: pwise01.dpw.co.la.ca.us
% status: Backup is in progress...
% progress %: 75
% progress message: Moving Backup file to the repository

%% Operation backup status
%% ------------------------
% No data found. Try 'show backup history' or ISE operation audit report

- created an FTP repository in the command line , verified that the ISE server can access it, then started a backup to it.

ISE01#backup configuration_1 repository upgrade ise-config encryption-key plain SOAndSO
% Warning: Waiting up to 1200 seconds for APP_BACKUP to finish...

- if I try stopping the ise application I get database locked

application stop ise

Waiting up to 20 seconds for lock: APP_BACKUP APP_BACKUP to complete
Database is still locked by lock: APP_BACKUP APP_BACKUP. Aborting. Please try it later
% Error: Another ISE DB process (APP_BACKUP APP_BACKUP) is in progress, cannot perform Application Stop at this time

- tried cancelling the job from the GUI I get this message

Have you given a reload via

nspasov — Fri, 15 Jan 2016 01:16:39 GMT

Have you given a reload via the CLI?

Re: Have you given a reload via

andres.picos — Wed, 31 Jul 2019 14:48:40 GMT

Reload is a solution? Really?

Re: Have you given a reload via

Damien Miller — Wed, 31 Jul 2019 15:26:22 GMT

If you have a back up issue of your own it would be better to start a discussion. Reloads are not usually required to clear a stuck backup anymore, but in 1.3 that or TAC were the only options.

Re: Have you given a reload via

andres.picos — Wed, 31 Jul 2019 16:14:55 GMT

I've arrived here searching for a solution in a similar issue in ISE 2.4 p6. and coincidentally I found this post.

I found a bug for ISE 2.4 p6 (CSCvq13817) where a workaround is "Reload the ISE PAN server".

I think it is absurd this kind of solutions that Cisco provides and the large number of Bugs that Cisco platforms have.

Re: Have you given a reload via

Ruud.Hoba — Mon, 23 Sep 2019 14:27:20 GMT

[smirk]

Have you tried turning it off and on again?!?!

[/smirk]

Command:

backup backupname repository repository-name ise-config encryption-key plain hashkey

solves as well.. even in the GUI the progress bar starts showing the progress 🙂

Re: Have you given a reload via

Ruud.Hoba — Sat, 14 Dec 2019 19:25:13 GMT

Oh. By the way. The bug remains, so its very likely the next time your scheduled backup runs, it gets stuck again. And can be solved by the CLI command.
Upgrading is recommended, more recent versions are available...
Note as well that after a reload the backup schedule needs to be reconfigured. The old one is bad and won't work. Haven't tried it but I guess a cron job or similar is also plausible. And when using HPNA for example, a command line script can be configured and scheduled to be ran from the HPNA server to the ISE.

Re: Have you given a reload via

Mat Barry — Mon, 30 Dec 2019 19:22:10 GMT

This worked for me. Bravo, Sir.

Re: ISE backup hung

rene_braun — Mon, 20 Apr 2020 07:26:13 GMT

Bug is still there on ISE 2.6 P3, but the workaround is working fine.

Re: ISE backup hung

Ruud.Hoba — Mon, 20 Apr 2020 13:16:30 GMT

To bad the bug is still there.. But hey, they can't fix all at once..
I haven't been upgrading recently to test other versions/patches.

Good to hear the workaround is still valid.. 🙂

Re: ISE backup hung

Darkmatter — Mon, 27 Apr 2020 09:30:33 GMT

This bug was still present in ISE 2.7 but it's fixed since the install of Patch 1

I can confirm i experienced the same issue but backups are now properly being done on ISE 2.7 P1

Re: ISE backup hung

DMog85286 — Thu, 07 Oct 2021 15:28:48 GMT

This bug just reared its ugly head here on 2.7 Patch 4.

Went to test a sftp backup after creating the repository and the GUI went to displaying Configuration Data Backup 0% .... for over 24 hours with no progress. Went to Stop it and there is a delay** of being able to click Stop; not sure why that is happening...but when the Stop is active, I click Stop and nothing happens. Logged in to CLI and checked backup status; and shows my config "Backup is in progress..."

I used the 'workaround'; performing a manual backup via CLI and even though it made it to 75% and then failed; it did clear up the hung GUI session.

**Does anyone know what causes this delay? Whenever I attempt to work at all in the Backup & Restore tab within the GUI it happens. I attempt to modify a scheduled backup or do a manual one, I get the Page Unresponsive | Wait or Exit Page option window. Using Chrome or Edge doesn't matter.

Re: ISE backup hung

Marcelo Morais — Fri, 08 Oct 2021 01:09:56 GMT

Hi @DMog85286 ,

please use the following command to check for any new info about the issue:

ise/admin# debug backup-restore all

Hope this helps !!!

Re: ISE backup hung

DannyDulin — Wed, 14 Sep 2022 15:36:20 GMT

This bug appeared for us on 2.7 patch 3.

We experience that delay too. I'm not sure what it is but I wonder if it has anything to do with reaching out to the repositories.

Sure would like to find out this answer.