ISE Backup Fail

[berndmueller1]

Hi all,

Trying do Backup ISE Virtual Appliance via FTP Repository Fails....

%% Configuration backup status
%% ----------------------------
% backup name: XXXXXXXXXXX
% repository: XXXXXXXXXXX
% start date: Tue Jul 02 11:46:37 UTC 2019
% scheduled: no
% triggered from: Admin web UI
% host: XXXXXXXXXXX
% status: Backup is in progress...
% progress %:
% progress message:

Same scene when using FTP, TFPT, SFTP with different targets.

Only way to work again with the ISE is reboot the appliance.

Version 2.4.0.357 Patchlevel 6

Thanks for all answers! 

[Arne Bier]

Hi Bernd

not much we can do here because ISE is a black box for us end users.  It looks like a TAC case to me.  

You can run all the debugs you like but if the cause of the problem is some Linux file that is causing the process hang then only TAC can fix that for you.

I have major qualms with ISE's "config backup" process in general. It's not a config backup at all - it's a garbage collection in my opinion because that monstrosity of a file contains more garbage than useful config data.  

In contrast, I recently upgraded another major vendor's vendors's AAA platform the other day with loads of endpoints and users and config etc - the backup was done via the https GUI and the resultant file could be downloaded via https GUI (hurray!) after the backup was done (took 2 minutes for a config backup).  File was 35MB.   Yes folks.  35MB.  That's around the size I would expect for a bunch of XML and other stuff.  Cisco ISE (if you unpack and boil down the backup file) config is also quite small.  But we don't get to chose what that process tries to cram in there (e.g. Linux system logs that nobody needs).  And the other vendors's config restore took a few minutes - and best of all, the config restore didn't unload all the previous machine's garbage onto it.  It's kind of how I would expect a config restore to work. 

It's just config backup and restore - this simple feature should JUST WORK so we can fret about the real complex stuff.

[hslai]

ISE 2.4 has a new option "Force Backup Cancellation" in ISE admin CLI command "application configure ise":

myISE24/admin# application configure ise

Selection configuration option
...
[22]Generate Heap Dump
[23]Generate Thread Dump
[24]Force Backup Cancellation
[0]Exit

If that's not working, please engage TAC.

One of Arne's points is being track by CSCuq59764

[berndmueller1]

Option 24 does not work either. Has already been tried.

[hslai · Accepted solution]

Please do engage Cisco TAC to troubleshoot. You may SSH to ISE admin CLI and do the following to see some detailed logging:

terminal length 0
show logging system ade/ADE.log tail

Below are some sample logging entries of an ISE CFG backup triggered from ISE admin web UI

2019-07-11T01:33:59.762771+00:00 ise-1 ADEOSJAVAAPI[669]: ADEAUDIT 2010, type=BACKUP, name=BACKUP STARTED, username=admin, cause=A backup has been inititated, adminipaddress=10.1.100.6, interface=GUI, detail=Initiating backup test01 to repository rwFTP
2019-07-11T01:34:01.630452+00:00 ise-1 logger: info:[backup-restore:backup:isecfgbackup.sh] Checking Disk Space...
2019-07-11T01:34:01.632905+00:00 ise-1 logger: info:[backup-restore:backup:isecfgbackup.sh] The data filesystem is 49 % full which is below threshold of 70 , hence continuing backup...
...
2019-07-11T01:34:17.724592+00:00 ise-1 logger: info:[backup-restore:backup:isecfgbackup.sh] Starting DB export backup for SCN 11024045 and host ise-1
...

2019-07-11T01:40:32.386402+00:00 ise-1 logger: info:[backup-restore:backup:isecfgbackup.sh] Export success for sync identifier ...
2019-07-11T01:40:32.429847+00:00 ise-1 logger: info:[backup-restore:backup:isecfgbackup.sh] rolling back undo retention after taking backup
...

2019-07-11T01:43:00.834986+00:00 ise-1 logger: [backup.sh] backup file test01-CFG10-190711-0133.tar.gpg successfully created
2019-07-11T01:43:02.245530+00:00 ise-1 ADEOSJAVAAPI[669]: ADEAUDIT 2011, type=BACKUP, name=BACKUP SUCCESS, username=admin, cause=A backup has completed, adminipaddress=10.1.100.6, interface=GUI, detail=Backup test01-CFG10-190711-0133.tar.gpg to repository rwFTP success