topic Re: ISE Backup Failing in Network Access Control

ISE Backup Failing

csosup — Thu, 09 May 2019 10:31:20 GMT

We have an issue where our configuration backup for ISE will fail most of the time saying either failed to copy to the repository (which is on a Windows server) or it will say back up aborted, the operational backup which happens daily and goes to the same repository works fine every night. Both the ISE appliance and Windows repository server are virtual and on the same host, so i am confused why the config back up will fail both on its scheduled back up and if i do an on-demand backup.

We we first started having this is the file system was full with old backups (before we push them out to the repository) so they were all deleted and as it stands we have the below space free

Internal filesystems:
/ : 60% used ( 102572040 of 180476204)
/dev/shm : 0% used ( 0 of 8167076)
/boot : 7% used ( 29623 of 481764)
/storedconfig : 2% used ( 1583 of 89231)
/tmp : 1% used ( 3312 of 1975372)
all internal filesystems have sufficient free space

We are running version 2.0.0.306 as well if that helps.

All has been fine for over a year and then suddenly its just stopped backing up the config on a regular basis

Re: ISE Backup Failing

Mark Elsen — Thu, 09 May 2019 11:45:35 GMT

- Yes but is there sufficient free space on the Windows repository server too ? For clarification post the exact error as seen when an on-demand backup is tried.

Re: ISE Backup Failing

csosup — Thu, 09 May 2019 12:07:43 GMT

Yes on the Windows server there is 112Gb free as we speak, the other day that was down to 30GB free but that should be more than enough considering the config back up is only 1GB (operational backup that works is only 139MB).

Below is the ext message from last nights backup failure.

Backup Failed

Details :

Backup Failed : Server=*****; Message=Backup failed: copy Daily_backup-CFG10-190508-2100 out to repository ****_sftp failed

Description :

The ISE Backup Operation Failed

Suggested Actions :

Ensure the network connectivity between ISE and Repository, Ensure the credentials used for the repository is correct, Ensure that there is sufficient disk space in the repository, Ensure there is WRITE privileges for the repository user

*** This message is generated by Cisco Identity Services Engine (ISE) ***

Sent By Host : *****

Re: ISE Backup Failing

Mark Elsen — Thu, 09 May 2019 12:55:24 GMT

- Ok, check how much (large) of the designated file has been created at the repository server. If it still at 0 it could indicate some more fundamental problem such as with credentials or another problem. In the same context -> try to examine the sftp server's logfile. See if more info can be found in there.

Re: ISE Backup Failing

csosup — Thu, 09 May 2019 15:30:07 GMT

Thats the weird thing on the server the files for this back up that fails is on the server at just over 1GB, but yet i still get the email come through saying it has failed

I will check the logfile for the SFTP server and come back

Re: ISE Backup Failing

Damien Miller — Thu, 09 May 2019 15:44:05 GMT

Download the backup file and decrypt it locally on your windows machine with something like www.gpg4win.org. If the file decrypts without issues then your backup technically did not fail, ISE just reported that it did.

In the that case that the backup decrypts, I have seen this issues where the FTP server "hangs" while it is writing a large file. When ISE tries to "list" the directory to confirm the file was transferred successfully, the FTP server is busy and doesn't respond. This is common when the FTP server has external storage such as CIFS/NFS, the network is slow, and the temp file to final destination copy is slow. ISE reports this as a failed backup.

Re: ISE Backup Failing

Jason Kunst — Thu, 09 May 2019 16:07:10 GMT

Have you tried using a different method? Non secure ftp?

Also if need critical support reach out to TAC for deep troubleshoting

Re: ISE Backup Failing

csosup — Fri, 10 May 2019 07:47:30 GMT

I have a feeling it could be something to do with the size, as the operational backup works all the time to the same repository buts its only 140MB, where as the config backup is 1GB

I will have a look at decrypting the back up from the repository, i might even try changing the SFTP software on the windows box and even try it on FTP as a test

Re: ISE Backup Failing

csosup — Fri, 10 May 2019 08:42:16 GMT

Thats the other problem, i didnt set this up i have inherited and it appears we no longer have Cisco support for ISE... so hence why i am on here lol

Last nights config backup failed with the below message, but as i said in my previous post i will try some of the suggestions.

Backup Failed

Details :

Backup Failed : Server=*****; Message=DB BACKUP FAILED : . Backup aborted

Description :

The ISE Backup Operation Failed

Suggested Actions :

*** This message is generated by Cisco Identity Services Engine (ISE) ***

Sent By Host : *******

Re: ISE Backup Failing

csosup — Fri, 10 May 2019 08:43:58 GMT

Just thinking is there anyway to shrink the configuration backup in size?

Re: ISE Backup Failing

Mark Elsen — Fri, 10 May 2019 10:21:56 GMT

>I will have a look at decrypting the back up from the repository, i might even try changing the SFTP software on >the windows box and even try it on FTP as a test

- Any status updates on these items ?

Re: ISE Backup Failing

hslai — Fri, 17 May 2019 03:22:57 GMT

Nope.

If your ISE on ISE 2.2 / 2.3 / 2.4, you might run into CSCvi21737.

Also, CSCuq59764 is an enhancement request and not yet addressed.

Re: ISE Backup Failing

howon — Mon, 20 May 2019 01:42:31 GMT

Please open TAC SR if you have not already to find root cause. Please note that since EoL dates were announced for 2.0, some fixes may only be addressed in later versions of ISE:

https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/bulletin-c25-741733.html

Re: ISE Backup Failing

csosup — Mon, 20 May 2019 09:45:35 GMT

Even older.. we are on 2.0.0.306!

Re: ISE Backup Failing

csosup — Mon, 20 May 2019 09:47:13 GMT

Thats the issue we dont have any support for ISE.

I think we might have to bite the bullet and get some now for the first time

Re: ISE Backup Failing

csosup — Mon, 20 May 2019 10:05:25 GMT

Looking at CSCvi21737 it says to delete files in /var/log/journal.

I dont seem to have a /var folder on the root

Re: ISE Backup Failing

Jason Kunst — Mon, 20 May 2019 11:00:29 GMT

That’s likely not exposed unless you have root access.

Also you’re running older version of ise that no longer supported. Recommend start looking at ise 2.2 or 2.4 for long term recommended release

Re: ISE Backup Failing

xtreme — Mon, 24 Jun 2019 15:37:15 GMT

We are facing exactly the same problem ..Running ise2.2 patch 12. Backup fails at the point of copying file to the server. Has this actually been fixed in patch 14? Are we hitting CSCvi21737 ?

Re: ISE Backup Failing

eagle05 — Tue, 18 Feb 2020 21:38:35 GMT

Hello,

I am running version 2.6 patch 1,2,3 and having the same issue it will not finish the backup. It starts it and then drops at 60% or 80%.

Any ideas?

Thank you