Cisco ISE SFTP backup failing

[Kamran Mustafayev]

Hello,

I have Cisco ISE cluster 3.2.0.542 and we brought up SFTP server on some Windows machine 

I can validate SFTP server from ISE, and can start a backup process, but it fails at 75%, here are the logs:

ciscoise01/admin#backup testbackup repository SFTP_BACKUP ise-config encryption-key plain xxxx
Warning: Do not use CTRL+C or close this terminal window until the backup is completed.
% backup in progress: Starting Backup...10% completed
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: testbackup-CFG10-240709-1718.tar.gpg
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Indexing Engine Data...45% completed
% backup in progress: Backing up ISE Logs...50% completed
% backup in progress: Completing ISE Backup Staging...55% completed
% backup in progress: Backing up ADEOS configuration...55% completed
% backup in progress: Moving Backup file to the repository...75% completed
File transfer error

Anyone faced such problem?

Thank you.

[Torbjørn]

Is there enough space on your repository? Can you see anything in the logs on ISE or the SFTP server? 

[Kamran Mustafayev]

I believe its some kind of permission to folder issues, just want to double check with you, here are the logs:

show repository SFTP_BACKUP
C:
E:

ciscoise01/admin#debug transfer 7
ciscoise01/admin#backup testbackup repository SFTP_BACKUP ise-config encryption-key plain xxx
6 [888269]:[info] transfer: cars_xfer.c[333] [system]: sftp dir of repository SFTP_BACKUP requested
6 [888269]:[info] transfer: cars_xfer_util.c[2634] [system]: Server validation successful xx.xx.xx.xx
7 [888269]:[debug] transfer: sftp_handler.c[1215] [system]: Running sftp command: xx.xx.xx.xx xxx *** / ls -l /
6 [888269]:[info] transfer: sftp_handler.c[629] [system]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: xx.xx.xx.xx remote user: xxx command: ls -l /
7 [888269]:[debug] transfer: sftp_handler.c[639] [system]: fd is:5
7 [888270]:[debug] transfer: sftp_handler.c[322] [system]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes xxx@ xx.xx.xx.xx
7 [888269]:[debug] transfer: sftp_handler.c[523] [system]: Found sftp prompt; No more data to read
7 [888269]:[debug] transfer: sftp_handler.c[1074] [system]: sftp parent status 0
7 [888269]:[debug] transfer: cars_xfer_util.c[2613] [system]: ssh_list xfer succeeded
Warning: Do not use CTRL+C or close this terminal window until the backup is completed.
% backup in progress: Starting Backup...10% completed
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: testbackup-CFG10-240710-2144.tar.gpg
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Indexing Engine Data...45% completed
% backup in progress: Backing up ISE Logs...50% completed
% backup in progress: Completing ISE Backup Staging...55% completed
% backup in progress: Backing up ADEOS configuration...55% completed
% backup in progress: Moving Backup file to the repository...75% completed
6 [888269]:[info] transfer: cars_xfer.c[248] [system]: sftp copy out of /opt/backup/backup-testbackup-1720633445/testbackup-CFG10-240710-2144.tar.gpg requested
6 [888269]:[info] transfer: cars_xfer_util.c[2634] [system]: Server validation successful xx.xx.xx.xx
7 [888269]:[debug] transfer: cars_xfer_util.c[598] [system]: copying file to remote server: xx.xx.xx.xx with full path /testbackup-CFG10-240710-2144.tar.gpg
7 [888269]:[debug] transfer: sftp_handler.c[1313] [system]: Running sftp command: xx.xx.xx.xx xxx*** /testbackup-CFG10-240710-2144.tar.gpg put /opt/backup/backup-testbackup-1720633445/testbackup-CFG10-240710-2144.tar.gpg /testbackup-CFG10-240710-2144.tar.gpg
6 [888269]:[info] transfer: sftp_handler.c[629] [system]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 13 remote host: xx.xx.xx.xx remote user: xxx command: put /opt/backup/backup-testbackup-1720633445/testbackup-CFG10-240710-2144.tar.gpg /testbackup-CFG10-240710-2144.tar.gpg
7 [888269]:[debug] transfer: sftp_handler.c[639] [system]: fd is:13
7 [915332]:[debug] transfer: sftp_handler.c[322] [system]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes xxx@xx.xx.xx.xx
> [888269]:[error] transfer: sftp_handler.c[1243] [system]: sftp_copy_callback: sftp copy failed. line:<dest open("/testbackup-CFG10-240710-2144.tar.gpg"): Permission denied
3 [888269]:[error] transfer: sftp_handler.c[934] [system]: sftp_run_parent Error: unable to handle sftp output
7 [888269]:[debug] transfer: sftp_handler.c[1074] [system]: sftp parent status -302
File transfer error
ciscoise01/admin#undebug all

[Kamran Mustafayev]

Yes, there are enough space on repo, just posted the debugs from ISE below fyi

will try to get logs from sftp as well

[Arne Bier]

debugging transfer issues with this command

debug transfer 7

And then run the backup command again - should give some clues.

You say it's validated?  Does a "show repo ..." produce a directory listing?  Does the user account have write permissions?

[Kamran Mustafayev]

I believe its some kind of permission to folder issues, just want to double check with you, here are the logs:

show repository SFTP_BACKUP
C:
E:

ciscoise01/admin#debug transfer 7
ciscoise01/admin#backup testbackup repository SFTP_BACKUP ise-config encryption-key plain xxx
6 [888269]:[info] transfer: cars_xfer.c[333] [system]: sftp dir of repository SFTP_BACKUP requested
6 [888269]:[info] transfer: cars_xfer_util.c[2634] [system]: Server validation successful xx.xx.xx.xx
7 [888269]:[debug] transfer: sftp_handler.c[1215] [system]: Running sftp command: xx.xx.xx.xx xxx *** / ls -l /
6 [888269]:[info] transfer: sftp_handler.c[629] [system]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: xx.xx.xx.xx remote user: xxx command: ls -l /
7 [888269]:[debug] transfer: sftp_handler.c[639] [system]: fd is:5
7 [888270]:[debug] transfer: sftp_handler.c[322] [system]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes xxx@ xx.xx.xx.xx
7 [888269]:[debug] transfer: sftp_handler.c[523] [system]: Found sftp prompt; No more data to read
7 [888269]:[debug] transfer: sftp_handler.c[1074] [system]: sftp parent status 0
7 [888269]:[debug] transfer: cars_xfer_util.c[2613] [system]: ssh_list xfer succeeded
Warning: Do not use CTRL+C or close this terminal window until the backup is completed.
% backup in progress: Starting Backup...10% completed
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: testbackup-CFG10-240710-2144.tar.gpg
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Indexing Engine Data...45% completed
% backup in progress: Backing up ISE Logs...50% completed
% backup in progress: Completing ISE Backup Staging...55% completed
% backup in progress: Backing up ADEOS configuration...55% completed
% backup in progress: Moving Backup file to the repository...75% completed
6 [888269]:[info] transfer: cars_xfer.c[248] [system]: sftp copy out of /opt/backup/backup-testbackup-1720633445/testbackup-CFG10-240710-2144.tar.gpg requested
6 [888269]:[info] transfer: cars_xfer_util.c[2634] [system]: Server validation successful xx.xx.xx.xx
7 [888269]:[debug] transfer: cars_xfer_util.c[598] [system]: copying file to remote server: xx.xx.xx.xx with full path /testbackup-CFG10-240710-2144.tar.gpg
7 [888269]:[debug] transfer: sftp_handler.c[1313] [system]: Running sftp command: xx.xx.xx.xx xxx*** /testbackup-CFG10-240710-2144.tar.gpg put /opt/backup/backup-testbackup-1720633445/testbackup-CFG10-240710-2144.tar.gpg /testbackup-CFG10-240710-2144.tar.gpg
6 [888269]:[info] transfer: sftp_handler.c[629] [system]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 13 remote host: xx.xx.xx.xx remote user: xxx command: put /opt/backup/backup-testbackup-1720633445/testbackup-CFG10-240710-2144.tar.gpg /testbackup-CFG10-240710-2144.tar.gpg
7 [888269]:[debug] transfer: sftp_handler.c[639] [system]: fd is:13
7 [915332]:[debug] transfer: sftp_handler.c[322] [system]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes xxx@xx.xx.xx.xx
> [888269]:[error] transfer: sftp_handler.c[1243] [system]: sftp_copy_callback: sftp copy failed. line:<dest open("/testbackup-CFG10-240710-2144.tar.gpg"): Permission denied
3 [888269]:[error] transfer: sftp_handler.c[934] [system]: sftp_run_parent Error: unable to handle sftp output
7 [888269]:[debug] transfer: sftp_handler.c[1074] [system]: sftp parent status -302
File transfer error
ciscoise01/admin#undebug all

[Torbjørn]

I agree, seems like a permission issue. What SFTP server software are you using?

[Kamran Mustafayev]

Its an OpenSSH on Windows VM

[JPavonM]

A similar SFTP setup is working for me.

Have you added the username that ISE backup is using to the "sshd_config_default" file in Windows' OpenSSH?

#CISCO ISE Backups
Match User domain\BackupISE
ChrootDirectory E:\Backups\ISEBackups

Have you added that username as SFTP user account in Windows?