ISE nodes unable to see SFTP repository

[paynewj]

Our ISE deployment consists of (4) nodes - (2) PANs and (2) PSNs - and only 1/4 can access the repository where the Log4J patch file is currently located.

I’ve recreated the repository via the ISE Admin console and the config deployed to all (4) of the nodes, but the only one that's able to connect and see the contents of the repository is our primary PAN. The secondary PAN can't connect, nor can the (2) PSNs in our deployment. 

I validated the repository in the GUI after it was created.

As mentioned, I was able to see the contents of the repo using the show repository command on our primary PAN, but received the following error when running the same command on all other nodes:

show repository ISE_Repo
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).

I've tried manually removing and then re-adding the repository and running the crypto host_key add host <host IP> command, but get the same result.

Any help would be greatly appreciated.

[Arne Bier]

Hello @paynewj 

The crypto command MUST be issued on EVERY ISE node. The sftp config is replicated, but the crypto keys are individual per node. It's a manual once-off chore, but without it, you won't get access to the SFTP server.

regards

Arne

[paynewj]

I appreciate the response, @Arne Bier .

I've issued the command on each of the nodes. I received confirmation that the host key fingerprint was added and that it was opperating in CiscoSSL FIPS mode.

Here's a debug from the one that's working:

ISESERVER/admin# show repo ISE_Repo
6 [16939]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [16939]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [16939]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [16939]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [16942]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
7 [16939]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [16939]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
7 [16939]:[debug] transfer: cars_xfer.c[268] [admin]: freed file list
ise-apply-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz
ise-rollback-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz

Debug from one that isn't:

ISESERVER/admin# show repo ISE_Repo
6 [30438]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [30438]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [30438]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [30438]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [30438]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [30440]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
3 [30438]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [30438]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error

Not sure why it's still saying "host key is not configured, or parsing error" after the crypto command has been entered.

I appreciate you time and assistance with this issue, Arne, and anyone else that's able to help.

[Arne Bier]

I can reproduce your error. It looks like a fault with the crypto host key still.

nac2/admin# debug transfer 7
nac2/admin# show repository ubuntu
6 [11523]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [11523]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [11523]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [11523]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [11523]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [11525]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
3 [11523]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [11523]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
nac2/admin#
nac2/admin# show crypto host_keys
nac2/admin#

Add the key - BUT - make sure it matches how it was defined in the repo - if you're using IP address in the repo, then add the key as an IP address. If using a hostname, then add the key as a hostname. I used an IP address in my repo URL.

nac2/admin#
nac2/admin# crypto host_key add host 10.48.148.10
host key fingerprint added
Operating in CiscoSSL FIPS mode

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4
nac2/admin# show repository ubuntu
6 [12976]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [12976]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [12976]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [12976]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [12976]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [12978]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
7 [12976]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
% Repository is empty
7 [12976]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [12976]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
nac2/admin#

My advice would be to list all the crypto keys, delete them all, and re-add

show crypto host_keys

And to demonstrate the disconnect between IP address and hostname in the crypto processing, here is an example where I add the crypto using the DNS hostname - and then the whole thing doesn't work again.

nac2/admin# nslookup ubuntu
Trying "ubuntu.networks.local"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54919
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ubuntu.networks.local. IN  ANY

;; ANSWER SECTION:
ubuntu.networks.local 3600 IN A    10.48.148.10

Received 67 bytes from 10.48.148.11#53 in 15 ms

nac2/admin# crypto host_key add host ubuntu
host key fingerprint added
Operating in CiscoSSL FIPS mode

# Host ubuntu found: line 1
ubuntu RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4
nac2/admin# debug transfer 7
nac2/admin# show repository ubuntu
6 [32151]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [32151]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [32151]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [32151]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [32151]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [32153]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
3 [32151]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
% Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
7 [32151]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
nac2/admin#

[fitzie]

Is it possible to have two SFTP repositories with the same IP?  I want to have 1 repo for backups and one for patches.  Whenever I try to add the second repo things go south.  This is an air-gapped network, so it's not as easy to work with as my other nets.

[Arne Bier]

Hi @fitzie 

I don't think ISE will let you add two repos with the same IP.  I would do it differently. Have one repo, but create two different accounts - one for backups, and one for patches. The account/user login directory should then point to two different paths.

If you need two separate repos because of disk space constraints, then of course create another repo on the additional host.

[Brian McPhillips]

Hi, Did you ever get this resolved? Experiencing a similiar issue on 3.0 patch 5. Backups were working and now have stopped. The error is saying the repository cannot be accessed.

I deleted the host keys to readd and now they are not showing in show crypto host_keys when i readd them. the command line displays they have been added correctly

[Arne Bier]

Let's see what @paynewj has to say.

In my experience the crypto host_keys always add, even in ISE 3.0 patch 5.

If they don't get added then please validate 100% that the ISE node can reach the SFTP's IP/hostname of the host you're adding (e.g. ping/ssh to the SFTP host from ISE).

What does the "debug transfer 7" tell you when you try to add the host_key?

Do you have any other ISE hosts that are able to add the crypto host_keys to the same SFTP host?

After a config restore, the repository password has to be re-entered in the ISE GUI. This is an expected thing in ISE and is a separate step to the host_keys issue.

[Brian McPhillips · Accepted solution]

Hi Arne,

I randomly noticed the "ssh delete host X.X.X.X" command on ISE whilst troubleshooting and tried that and it fixed the issue. My steps were. 

1.  ssh delete host X.X.X.X

2. crypto host_key delete host X.X.X.X

3. Deleted Repo from GUI

4. Re Add repo from GUI

5. crypto host_key add host X.X.X.X

When i was running crypto host_key add host,  it said the entry was added but didn't display the String output. the SSH command must remove the old entry properly.

[Arne Bier]

I am glad it all worked out. Hopefully it will help someone else in future too

[rishisemwal]

Hello,

1. I reconfigured SFTP after changing the IP address of ISE server. It's not generating the key 

BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode

2. Because of that it is not validating. Please see the error

Repository validation failed due to error - CARS_RM_NOT_FOUND : -200 : Repository not found.

3. I can not delete it to reconfigure.. Please see the error

Unabled to delete repository(s): sftp (used in scheduled/on-demand backups)

[Arne Bier]

Don’t manage your repository configs via CLI. Make changes in the Admin UI because that will propagate the changes to all nodes in the deployment. The only thing you need to do on the CLI is to ensure that the crypto key is configured on every node on which you need to access that repo. 

[rishisemwal]

Thanks for your response, I'm using CLI to add crypto key host only but response is showing as below, not showing[cid:024309ce-f287-4f05-a546-3f823932f22e] encrypted message.

# BEN-ISE/admin# crypto host_key add host 10.155.23.123
host key fingerprint added
Operating in CiscoSSL FIPS mode

#. I used following command to remove old key and reconfigure.


1. ssh delete host X.X.X.X

2. crypto host_key delete host X.X.X.X


Thanks

[Arne Bier]

I can't see the screenshot you tried to embed in the message (I think?).

I'm getting confused - you're able to delete the old crypto keys, yes?

But when you try adding a new crypto key then you get an error?

In my experience the error when adding crypto keys tends to be a network communication error - during this process, ISE tries to reach the remote host to exchange public keys - and if TCP/22 is being blocked (or the remote end fails to establish a response to ISE) then the crypto command will fail.  You can try enabling the debug below, before you issue the the crypto add command

debug transfer 7

[rishisemwal]

Hello,

Please see result below,


[cid:dd3cf8fb-9d99-4ee3-8b8f-c217e5d6071c]
UNMISS-BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode

UNMISS-BEN-ISE/admin# sh repository sftp
3 [9587]:[error] transfer: cars_xfer.c[204] [admin]: couldn't get repository sftp
% Error: Repository sftp could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Repository not found
UNMISS-BEN-ISE/admin# debug transfer 7
UNMISS-BEN-ISE/admin#