ISE backup stuck and can't restart service or reboot server

[tachyon05]

ISE backup is stuck and has been saying "cancelling" for more than a week.  I disabled scheduled backups several days ago, but I still can't stop and restart the service because the system thinks the backup is still in progress.  In the CLI, the status of backup is also "cancelling".

ise-a/admin# app stop ise

Waiting up to 20 seconds for lock: APP_BACKUP to complete

Database is still locked by lock: APP_BACKUP. Aborting. Please try it later
% Error: Another ISE DB process (APP_BACKUP) is in progress, cannot perform Application Stop at this time

I tried to restart the server, but the below saving process for ADE-OS takes many many hours and never seems to finish.  Any suggestions?

ise-a/admin# reload

WARNING: A backup or restore is currently in progress! Continue with reload? (yes/no) [no] ? yes
Save the current ADE-OS running configuration? (yes/no) [yes] ?
Generating configuration...

[Damien Miller]

If you haven't already tired it, then try entering "application configure ise" on the primary admin node, and select option 24.

[tachyon05]

Thank you for sharing this option to force cancel backup.  However, I think the issue now is ADE-OS saving process never completes, preventing me from reloading the appliance.  Any suggestion on how to go around that? 

As shown in my initial post, I am presented with option to ignore the stuck backup process and proceed with the reload - except it gets stuck on the ADE-OS saving.

[Marcelo Morais]

Hi @tachyon05 

 are you able to do a write memory on the CLI?

Note: remember that you can:

ise/admin# reload
Save the current ADE-OS running configuration? (yes/no) [yes] ? no
Continue with reboot? [y/n] y

Hope this helps !!!

[tachyon05]

When I issue the write memory command, it never seems to complete the execution and I just see the cursor keeps blinking like it is working.  Any other suggestions?

[Marcelo Morais]

Hi @tachyon05 

 as soon as you are able to generate a backup (after @Damien Miller suggestion), try to generate a backup (to save the config) and after that a reload (without the option to save).

Hope this helps !!!

[tachyon05]

Unfortunately, Damien's suggestion is not an option on my version of ISE 2.3.  Thanks

[Greg Gibbs · Accepted solution]

Does the CLI hang when you issue the 'show run' command as well? If so, you might try disabling CDP from the CLI.

There have been some past bugs where CDP was causing the CLI to hang. It's an easy thing to try disabling to see if it helps, just in case.