cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
9518
Views
27
Helpful
21
Replies

ISE nodes unable to see SFTP repository

paynewj
Level 1
Level 1

Our ISE deployment consists of (4) nodes - (2) PANs and (2) PSNs - and only 1/4 can access the repository where the Log4J patch file is currently located.

I’ve recreated the repository via the ISE Admin console and the config deployed to all (4) of the nodes, but the only one that's able to connect and see the contents of the repository is our primary PAN. The secondary PAN can't connect, nor can the (2) PSNs in our deployment. 

I validated the repository in the GUI after it was created.

As mentioned, I was able to see the contents of the repo using the show repository command on our primary PAN, but received the following error when running the same command on all other nodes:

 

show repository ISE_Repo
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).

 

I've tried manually removing and then re-adding the repository and running the crypto host_key add host <host IP> command, but get the same result.

Any help would be greatly appreciated.

1 Accepted Solution

Accepted Solutions

Hi Arne,

 

I randomly noticed the "ssh delete host X.X.X.X" command on ISE whilst troubleshooting and tried that and it fixed the issue. My steps were. 

 

1.  ssh delete host X.X.X.X

2. crypto host_key delete host X.X.X.X

3. Deleted Repo from GUI

4. Re Add repo from GUI

5. crypto host_key add host X.X.X.X

 

When i was running crypto host_key add host,  it said the entry was added but didn't display the String output. the SSH command must remove the old entry properly.

View solution in original post

21 Replies 21

Arne Bier
VIP
VIP

Hello @paynewj 

 

The crypto command MUST be issued on EVERY ISE node. The sftp config is replicated, but the crypto keys are individual per node. It's a manual once-off chore, but without it, you won't get access to the SFTP server.

 

regards

Arne

I appreciate the response, @Arne Bier .

 

I've issued the command on each of the nodes. I received confirmation that the host key fingerprint was added and that it was opperating in CiscoSSL FIPS mode.

 

Here's a debug from the one that's working:

ISESERVER/admin# show repo ISE_Repo
6 [16939]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [16939]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [16939]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [16939]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [16942]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
7 [16939]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [16939]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
7 [16939]:[debug] transfer: cars_xfer.c[268] [admin]: freed file list
ise-apply-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz
ise-rollback-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz

 

Debug from one that isn't:

ISESERVER/admin# show repo ISE_Repo
6 [30438]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [30438]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [30438]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [30438]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [30438]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [30440]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
3 [30438]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [30438]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error

 

Not sure why it's still saying "host key is not configured, or parsing error" after the crypto command has been entered.

 

I appreciate you time and assistance with this issue, Arne, and anyone else that's able to help.

 

 

I can reproduce your error. It looks like a fault with the crypto host key still.

 

nac2/admin# debug transfer 7
nac2/admin# show repository ubuntu
6 [11523]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [11523]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [11523]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [11523]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [11523]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [11525]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
3 [11523]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [11523]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
nac2/admin#
nac2/admin# show crypto host_keys
nac2/admin#

Add the key - BUT - make sure it matches how it was defined in the repo - if you're using IP address in the repo, then add the key as an IP address. If using a hostname, then add the key as a hostname. I used an IP address in my repo URL.

 

nac2/admin#
nac2/admin# crypto host_key add host 10.48.148.10
host key fingerprint added
Operating in CiscoSSL FIPS mode

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4
nac2/admin# show repository ubuntu
6 [12976]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [12976]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [12976]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [12976]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [12976]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [12978]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
7 [12976]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
% Repository is empty
7 [12976]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [12976]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
nac2/admin#

My advice would be to list all the crypto keys, delete them all, and re-add

show crypto host_keys

And to demonstrate the disconnect between IP address and hostname in the crypto processing, here is an example where I add the crypto using the DNS hostname - and then the whole thing doesn't work again.

 

nac2/admin# nslookup ubuntu
Trying "ubuntu.networks.local"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54919
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ubuntu.networks.local. IN  ANY

;; ANSWER SECTION:
ubuntu.networks.local 3600 IN A    10.48.148.10

Received 67 bytes from 10.48.148.11#53 in 15 ms

nac2/admin# crypto host_key add host ubuntu
host key fingerprint added
Operating in CiscoSSL FIPS mode

# Host ubuntu found: line 1
ubuntu RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4
nac2/admin# debug transfer 7
nac2/admin# show repository ubuntu
6 [32151]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [32151]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [32151]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [32151]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [32151]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [32153]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
3 [32151]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
% Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
7 [32151]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
nac2/admin#

 

Is it possible to have two SFTP repositories with the same IP?  I want to have 1 repo for backups and one for patches.  Whenever I try to add the second repo things go south.  This is an air-gapped network, so it's not as easy to work with as my other nets.

Hi @fitzie 

I don't think ISE will let you add two repos with the same IP.  I would do it differently. Have one repo, but create two different accounts - one for backups, and one for patches. The account/user login directory should then point to two different paths.

If you need two separate repos because of disk space constraints, then of course create another repo on the additional host.

Hi Arne,

I tried the proposed solution and it did not work. I can ping and SSH from my ISE 2.7 patch 7 into the SFTP server (see next), BUT I cannot get the KEY FINGERPRINT as per your example (I also used IP for the SFTP no DNS resolution). I mean, not getting the 2nd part of the CRYPTO HOST KEY ADD HOST "IP" (see also below). I have tried everything and still not working. Any thoughts? I am exploring if the option of adding in the KNOWN Hosts of my ISE the key/fingerprint

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4

 

myISE/admin# show repository myrepo13
6 [17055]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository myrepo13 requested
6 [17055]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.1.1.100
7 [17055]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.1.1.100 testing *** /home/isebck/ ls -l /home/isebck/
6 [17055]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.1.1.100 remote user: testing command: ls -l /home/isebck/
7 [17055]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [17057]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes nsmgmt@10.1.1.100
3 [17055]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
% Error: Repository myrepo13 could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
7 [17055]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
myISE/admin#

NOTHING from the following command output

myISE/admin#
myISE/admin show crypto host_keys
myISE/admin#

As I mentioned, when I followed the procedure of removing everything from CLI / GUI and recreated it, the next command did not provide me with the 2nd part of the output as your example

myISE/admin# 
myISE/admin# crypto host_key add host 10.1.1.100
host key fingerprint added
Operating in CiscoSSL FIPS mode

**************NO RSA SHA 256 fingerprint output***************

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4

myISE/admin#

 

********************SSH from ISE to SFTP Server*******************

my ISE/admin# ssh 10.1.1.100 testing
Operating in CiscoSSL FIPS mode
FIPS mode initialized
The authenticity of host '10.1.1.100 (10.1.1.100)' can't be established.
RSA key fingerprint is SHA256:0uSiedgerDEDGGG2341DDFTRVdfs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.100' (RSA) to the list of known hosts.
testing@10.1.1.100's password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-78-generic x86_64)

System information as of Mon Aug 28 02:01:21 PM EDT 2023

System load: 1.11767578125 Processes: 273
Usage of /: 7.8% of 588.04GB Users logged in: 1
Memory usage: 8% IPv4 address for ens160: 10.1.1.100
Swap usage: 0%

*** System restart required ***
Last login: Mon Aug 28 13:48:56 2023 from 172.16.1.1
testing@mytoolbox:~$ ^C
testing@mytoolbox:~$

 

@ajc 

The issue with the IP address you used in the crypto host_key add host command. 

You must use the IP address of the SFTP server - your example shows:

crypto host_key add host 172.22.4.100

but according to your logs, the IP address of the SFTP server is 10.1.100 - this should work:

crypto host_key add host 10.1.100

 

I corrected the info on my previous post, 10.1.1.100 is the SFTP server IP.

However, looks like ISE only supports old cipher/hostkey algorithms because I installed a free SFTP on a PC and tried using a different ISE running same version 2.7 patch 7, it worked. SEE BELOW.

The initial test that did not work was using an UBUNTU SFTP server with the most recent cipher/hostkey algorithms that ISE does not support at all and are used during the negotiation below. I am proceeding to run more tests and post here the results. 

Also, it is Interesting to notice that this desktop SFTP app I used gives you a DEBUG option that clearly shows what is going on. 

SFTP REPOSITORY.pngCRYPTO HOST KEY ISE.png

UPDATE: We tried everything via Shell on our ISE 2.7 patch 5 version and we were unable to solve this issue so TAC recommended to try ISE 3.2 patch 3 that contains the fix. I am deploying a VM and running test. I will keep you posted.

Hi, Did you ever get this resolved? Experiencing a similiar issue on 3.0 patch 5. Backups were working and now have stopped. The error is saying the repository cannot be accessed.

 

I deleted the host keys to readd and now they are not showing in show crypto host_keys when i readd them. the command line displays they have been added correctly

 

 

Let's see what @paynewj has to say.

In my experience the crypto host_keys always add, even in ISE 3.0 patch 5.

If they don't get added then please validate 100% that the ISE node can reach the SFTP's IP/hostname of the host you're adding (e.g. ping/ssh to the SFTP host from ISE).

What does the "debug transfer 7" tell you when you try to add the host_key?

 

Do you have any other ISE hosts that are able to add the crypto host_keys to the same SFTP host?

 

After a config restore, the repository password has to be re-entered in the ISE GUI. This is an expected thing in ISE and is a separate step to the host_keys issue.

Hi Arne,

 

I randomly noticed the "ssh delete host X.X.X.X" command on ISE whilst troubleshooting and tried that and it fixed the issue. My steps were. 

 

1.  ssh delete host X.X.X.X

2. crypto host_key delete host X.X.X.X

3. Deleted Repo from GUI

4. Re Add repo from GUI

5. crypto host_key add host X.X.X.X

 

When i was running crypto host_key add host,  it said the entry was added but didn't display the String output. the SSH command must remove the old entry properly.

I am glad it all worked out. Hopefully it will help someone else in future too

rishisemwal
Level 1
Level 1

Hello,

 

1. I reconfigured SFTP after changing the IP address of ISE server. It's not generating the key 

 

BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode

 

2. Because of that it is not validating. Please see the error

Repository validation failed due to error - CARS_RM_NOT_FOUND : -200 : Repository not found.

 

3. I can not delete it to reconfigure.. Please see the error

 

Unabled to delete repository(s): sftp (used in scheduled/on-demand backups)