Cisco ISE BackUp

[Netmart]

Hello,

I am wondering, what the root cause could for failing to send generated backup to remote repository.

Running the back on ISE box itself, all phase can be traced.

And it seems that the configuring is completed, but the transfer is failing - here I was using TFTP.

% backup in progress: Moving Backup file to the repository...75% completed

% Transfer timed out.

 % File transfer error

% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command

% Creating backup with timestamped filename: ConfigBackup-CLI-CFG10-200326-0705.tar.gpg

% backup in progress: Starting Backup...10% completed

% backup in progress: Validating ISE Node Role...15% completed

% backup in progress: Backing up ISE Configuration Data...20% completed

% backup in progress: Backing up ISE Indexing Engine Data...45% completed

% backup in progress: Backing up ISE Logs...50% completed

% backup in progress: Completing ISE Backup Staging...55% completed

% backup in progress: Backing up ADEOS configuration...55% completed

% backup in progress: Moving Backup file to the repository...75% completed

% Transfer timed out.

 % File transfer error

[balaji.bandi]

what ISE version ?

what Model of Remote backup method ? SCP/FTP/SFTP ?

% backup in progress: Moving Backup file to the repository...75% completed

as per this error looks for me far end folder permission (repository) issue to write the files on the backup destination.

You can also run debug on ISE and check what is the error :

# debug backup-restore backup

[Arne Bier]

@Netmart - ISE allows you to configure a repository using various protocols (including tftp) - but what you do with that repository is important - for storing data, you can't use tftp as a protocol - you can use tftp repo for other things that do not involve writing data.

The only supported protocols for an ISE repo that involve storing data are ftp, SFTP, NSF and local disk.

[Netmart]

Thank you very much Arne.

I set up sftp, also tested the privilege access to ISE destination folder by running a SFTP session, downloading a test file with the credentials configured in ISE:

Version:3.1.0.518

repository server-sftp
  url sftp://server-sftp/data/sftp/ISE
  user cisco password hash ******

# backup test-server-sftp repository server-sftp ise-config encryption-key plain *****

% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command

% Creating backup with timestamped filename: test-brutus-sftp-CFG10-240701-0018.tar.gpg

% backup in progress: Starting Backup...10% completed

% backup in progress: Validating ISE Node Role...15% completed

% backup in progress: Backing up ISE Configuration Data...20% completed

% backup in progress: Backing up ISE Indexing Engine Data...45% completed

% backup in progress: Backing up ISE Logs...50% completed

% backup in progress: Completing ISE Backup Staging...55% completed

% backup in progress: Backing up ADEOS configuration...55% completed

% backup in progress: Moving Backup file to the repository...75% completed

% Failure occurred during request

I hope we do  not hit: https://bst.cisco.com/bugsearch/bug/CSCwd63717?rfs=qvlogin

[Arne Bier]

Are you able to view the directory contents of that SFTP repository, from the vantage of the ISE CLI?  Put a simple file in directory /server-sftp/data/sftp/ISE    and then check if you can view the file:

show repo server-sftp

if that doesn't work, then I suspect that you haven't created the crypto host key on the CLI - if your repo URL is

sftp://myserver.com/ then your command would be

crypto host_key add host myserver.com

if your repo URL contains an IP address, then use the IP address in the command above.

A useful debugging command for seeing what ISE is doing when you test those show/backup commands:

debug transfer 7

[Netmart]

Hello Arne, please see output below.

Since this is a production environment, does running "debug transfer 7" has any impact on ISE application services?

sh repository server-sftp

% Error: Repository server-sftp could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).

% Failure occurred during reques

[Arne Bier]

It doesn't break to enable the debugs - just disable them once you're done.

Do you have the plain text password of the username "cisco" ? If so, then log into the ISE Admin GUI, and just overwrite the password for that repo config.  If you have done a config restore, then ISE will complain and force you to overwrite the password (even if the password hasn't changed)

Can you ping the SFTP server?

[Netmart]

Hi Arne,
I am able to ping the sftp server.
I also run from a different linux box sftp commnands to this repository/sftp-server by using same cisco user credentials - no problem.

/admin# show repository server-sftp
6 [520055]:[info] transfer: cars_xfer.c[225] [admin]: sftp dir of repository server-sftp requested
6 [520055]:[info] transfer: cars_xfer_util.c[2297] [admin]: Server validation successful brutus
7 [520055]:[debug] transfer: sftp_handler.c[1095] [admin]: Running sftp command: brutus cisco *** /data/sftp/ISE/ ls -l /data/sftp/ISE/
6 [520055]:[info] transfer: sftp_handler.c[585] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 8 remote host: brutus remote user: cisco command: ls -l /data/sftp/ISE/
7 [520055]:[debug] transfer: sftp_handler.c[594] [admin]: fd is:8
7 [520061]:[debug] transfer: sftp_handler.c[292] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes cisco@brutus
3 [520055]:[error] transfer: sftp_handler.c[365] [admin]: sftp_select Error: timeout!
7 [520055]:[debug] transfer: sftp_handler.c[964] [admin]: sftp parent status -999
% Error: Repository server-sftp could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Failure occurred during request

[Netmart]

...yes, host key has been added:

# crypto host_key add host server-sftp

host key fingerprint added

Operating in CiscoSSL FIPS mode

# Host server-sftp found:line 1

server-sftp  RSA SHA256:*******

[Arne Bier]

It looks like TCP/22 is not allowed (blocked by firewall / ACL) between ISE and that SFTP server. 

Instead of ping, see if you can get a response from doing an SSH from the ISE CLI, to the SFTP server (SSH/SFTP normally default to TCP/22)

[adamscottmaster2013]

There could be several things such as:

#1:  Firewalls/ACL between the ISE and sFTP server,

#2:  iptables on the sFTP server itself,

#3:  /etc/hosts.allow or /etc/hosts/deny on the sFTP server that prevents your ISE server to connect.  Yes, it is there, in addition to the iptables itself,

The best thing to do is to create a dummy sFTP on the ISE with the same hostname/IP address as the actual sFTP server (named it dummy or something like that) and gives it the same username/pw of the sFTP server.  After that, on the command, add the host key like "crypto host_key add host dummy.cisco.com" or "crypto host_key add host X.X.X.X".  Once you confirmed that the key is successfully added, do a "show repository dummy" and you should see a listing of all the file in that directory of the username you specified when creating the "dummy" repository. 

If you can't get the host key added in ISE, it means tcp/22 is being blocked somewhere.  If you're able to successfully add the host key but can not view the repository, it means the sFTP server is likely implementing the /etc/hosts.allow or /etc/hosts.deny (assuming the username and pw is valid).  Remember, tcpdump is your friend....

[Netmart]

Thank you Arne.
Based on TCPdump and manual SSH from ISE box into server of Repository, it seems that port TCP22 is allowed.

[netmart2]

All those restrictions can be be ruled out (!)

I was even able to SSH into server of repository by using same credentials.

I also took a tcpdump and monitored incoming ssh connections:  

Able to confirm the SSH connection between ISE box and repository has been established. However, ISE sends Finish and closes TCP connection without having any data sent.

12:40:10.817610 IP sftp-server.ssh > ise-box.37912: Flags [S.], seq 1378995211, ack 1513259488, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
12:40:10.820623 IP ise-box.37912 > sftp-server.ssh: Flags [.], ack 1, win 229, length 0
12:40:10.820996 IP ise-box.37912 > sftp-server.ssh: Flags [P.], seq 1:35, ack 1, win 229, length 34
12:40:10.821005 IP sftp-server.ssh > ise-box.37912: Flags [.], ack 35, win 58, length 0
12:40:11.026064 IP sftp-server.ssh > ise-box.37912: Flags [P.], seq 1:22, ack 35, win 58, length 21
12:40:11.029028 IP ise-box.37912 > sftp-server.ssh: Flags [.], ack 22, win 229, length 0
12:40:11.029860 IP ise-box.37912 > sftp-server.ssh: Flags [P.], seq 35:579, ack 22, win 229, length 544
12:40:11.029866 IP sftp-server.ssh > ise-box.37912: Flags [.], ack 579, win 62, length 0
12:40:11.032480 IP sftp-server.ssh > ise-box.37912: Flags [P.], seq 22:534, ack 579, win 62, length 512
12:40:11.042211 IP ise-box.37912 > sftp-server.ssh: Flags [P.], seq 579:851, ack 534, win 237, length 272
12:40:11.050075 IP sftp-server.ssh > ise-box.37912: Flags [P.], seq 534:1382, ack 851, win 66, length 848
12:40:11.093316 IP ise-box.37912 > sftp-server.ssh: Flags [.], ack 1382, win 250, length 0
12:41:11.089153 IP ise-box.37912 > sftp-server.ssh: Flags [F.], seq 851, ack 1382, win 250, length 0
12:41:11.095743 IP sftp-server.ssh > ise-box.37912: Flags [F.], seq 1382, ack 852, win 66, length 0
12:41:11.098674 IP ise-box.37912 > sftp-server.ssh: Flags [.], ack 1383, win 250, length 0

I hope we do not hit the following bug, though we do run Service Pack3:

Host:  **

Personas: Administration, Monitoring, Policy Service (SESSION,PROFILER,DEVICE ADMIN)

Role: PRI(A), SEC(M)

System Time: Jul 02 2024 12:24:18 PM******

FIPS Mode: Disabled

Version:3.1.0.518

Patch Information: 3

ISE 3.1 certain SFTP servers stopped working after upgrade to patch 4/5
CSCwd89657  

[netmart2 · Accepted solution]

Eventually, I was able to fix the sftp upload issue by removing the configured from GUI at: 

Admin  > System > Maintenance > Repository

And creating the repository via CLI.

% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command

% Creating backup with timestamped filename: test-sftp-CFG10-240705-1724.tar.gpg

% backup in progress: Starting Backup...10% completed

% backup in progress: Validating ISE Node Role...15% completed

% backup in progress: Backing up ISE Configuration Data...20% completed

% backup in progress: Backing up ISE Indexing Engine Data...45% completed

% backup in progress: Backing up ISE Logs...50% completed

% backup in progress: Completing ISE Backup Staging...55% completed

% backup in progress: Backing up ADEOS configuration...55% completed

% backup in progress: Moving Backup file to the repository...75% completed

% backup in progress: Completing Backup...100% completed