topic Re: ISE nodes unable to see SFTP repository in Network Access Control

ISE nodes unable to see SFTP repository

paynewj — Fri, 17 Dec 2021 00:46:47 GMT

Our ISE deployment consists of (4) nodes - (2) PANs and (2) PSNs - and only 1/4 can access the repository where the Log4J patch file is currently located.

I’ve recreated the repository via the ISE Admin console and the config deployed to all (4) of the nodes, but the only one that's able to connect and see the contents of the repository is our primary PAN. The secondary PAN can't connect, nor can the (2) PSNs in our deployment.

I validated the repository in the GUI after it was created.

As mentioned, I was able to see the contents of the repo using the show repository command on our primary PAN, but received the following error when running the same command on all other nodes:

show repository ISE_Repo
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).

I've tried manually removing and then re-adding the repository and running the crypto host_key add host <host IP> command, but get the same result.

Any help would be greatly appreciated.

Re: ISE nodes unable to see SFTP repository

Arne Bier — Fri, 17 Dec 2021 03:03:46 GMT

Hello @paynewj

The crypto command MUST be issued on EVERY ISE node. The sftp config is replicated, but the crypto keys are individual per node. It's a manual once-off chore, but without it, you won't get access to the SFTP server.

regards

Arne

Re: ISE nodes unable to see SFTP repository

paynewj — Fri, 17 Dec 2021 13:24:51 GMT

I appreciate the response, @Arne Bier .

I've issued the command on each of the nodes. I received confirmation that the host key fingerprint was added and that it was opperating in CiscoSSL FIPS mode.

Here's a debug from the one that's working:

ISESERVER/admin# show repo ISE_Repo
6 [16939]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [16939]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [16939]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [16939]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [16942]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
7 [16939]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [16939]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
7 [16939]:[debug] transfer: cars_xfer.c[268] [admin]: freed file list
ise-apply-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz
ise-rollback-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz

Debug from one that isn't:

ISESERVER/admin# show repo ISE_Repo
6 [30438]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [30438]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [30438]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [30438]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [30438]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [30440]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
3 [30438]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [30438]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error

Not sure why it's still saying "host key is not configured, or parsing error" after the crypto command has been entered.

I appreciate you time and assistance with this issue, Arne, and anyone else that's able to help.

Re: ISE nodes unable to see SFTP repository

Arne Bier — Fri, 17 Dec 2021 20:40:25 GMT

I can reproduce your error. It looks like a fault with the crypto host key still.

nac2/admin# debug transfer 7
nac2/admin# show repository ubuntu
6 [11523]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [11523]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [11523]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [11523]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [11523]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [11525]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
3 [11523]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [11523]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
nac2/admin#
nac2/admin# show crypto host_keys
nac2/admin#

Add the key - BUT - make sure it matches how it was defined in the repo - if you're using IP address in the repo, then add the key as an IP address. If using a hostname, then add the key as a hostname. I used an IP address in my repo URL.

nac2/admin#
nac2/admin# crypto host_key add host 10.48.148.10
host key fingerprint added
Operating in CiscoSSL FIPS mode

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4
nac2/admin# show repository ubuntu
6 [12976]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [12976]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [12976]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [12976]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [12976]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [12978]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
7 [12976]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
% Repository is empty
7 [12976]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [12976]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
nac2/admin#

My advice would be to list all the crypto keys, delete them all, and re-add

show crypto host_keys

And to demonstrate the disconnect between IP address and hostname in the crypto processing, here is an example where I add the crypto using the DNS hostname - and then the whole thing doesn't work again.

nac2/admin# nslookup ubuntu
Trying "ubuntu.networks.local"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54919
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ubuntu.networks.local. IN  ANY

;; ANSWER SECTION:
ubuntu.networks.local 3600 IN A    10.48.148.10

Received 67 bytes from 10.48.148.11#53 in 15 ms

nac2/admin# crypto host_key add host ubuntu
host key fingerprint added
Operating in CiscoSSL FIPS mode

# Host ubuntu found: line 1
ubuntu RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4
nac2/admin# debug transfer 7
nac2/admin# show repository ubuntu
6 [32151]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested
6 [32151]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10
7 [32151]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/
6 [32151]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/
7 [32151]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [32153]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10
3 [32151]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
% Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
7 [32151]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
nac2/admin#

Re: ISE nodes unable to see SFTP repository

Brian McPhillips — Mon, 28 Mar 2022 13:12:23 GMT

Hi, Did you ever get this resolved? Experiencing a similiar issue on 3.0 patch 5. Backups were working and now have stopped. The error is saying the repository cannot be accessed.

I deleted the host keys to readd and now they are not showing in show crypto host_keys when i readd them. the command line displays they have been added correctly

Re: ISE nodes unable to see SFTP repository

Arne Bier — Mon, 28 Mar 2022 20:09:43 GMT

Let's see what @paynewj has to say.

In my experience the crypto host_keys always add, even in ISE 3.0 patch 5.

If they don't get added then please validate 100% that the ISE node can reach the SFTP's IP/hostname of the host you're adding (e.g. ping/ssh to the SFTP host from ISE).

What does the "debug transfer 7" tell you when you try to add the host_key?

Do you have any other ISE hosts that are able to add the crypto host_keys to the same SFTP host?

After a config restore, the repository password has to be re-entered in the ISE GUI. This is an expected thing in ISE and is a separate step to the host_keys issue.

Re: ISE nodes unable to see SFTP repository

Brian McPhillips — Wed, 30 Mar 2022 07:28:31 GMT

Hi Arne,

I randomly noticed the "ssh delete host X.X.X.X" command on ISE whilst troubleshooting and tried that and it fixed the issue. My steps were.

1. ssh delete host X.X.X.X

2. crypto host_key delete host X.X.X.X

3. Deleted Repo from GUI

4. Re Add repo from GUI

5. crypto host_key add host X.X.X.X

When i was running crypto host_key add host, it said the entry was added but didn't display the String output. the SSH command must remove the old entry properly.

Re: ISE nodes unable to see SFTP repository

Arne Bier — Wed, 30 Mar 2022 22:19:53 GMT

I am glad it all worked out. Hopefully it will help someone else in future too 🙂

Re: ISE nodes unable to see SFTP repository

rishisemwal — Sun, 19 Jun 2022 09:45:36 GMT

Hello,

1. I reconfigured SFTP after changing the IP address of ISE server. It's not generating the key

BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode

2. Because of that it is not validating. Please see the error

Repository validation failed due to error - CARS_RM_NOT_FOUND : -200 : Repository not found.

3. I can not delete it to reconfigure.. Please see the error

Unabled to delete repository(s): sftp (used in scheduled/on-demand backups)

Re: ISE nodes unable to see SFTP repository

Arne Bier — Sun, 19 Jun 2022 10:42:13 GMT

Don’t manage your repository configs via CLI. Make changes in the Admin UI because that will propagate the changes to all nodes in the deployment. The only thing you need to do on the CLI is to ensure that the crypto key is configured on every node on which you need to access that repo.

Re: ISE nodes unable to see SFTP repository

rishisemwal — Sun, 19 Jun 2022 11:27:19 GMT

Thanks for your response, I'm using CLI to add crypto key host only but response is showing as below, not showing[cid:024309ce-f287-4f05-a546-3f823932f22e] encrypted message.

# BEN-ISE/admin# crypto host_key add host 10.155.23.123
host key fingerprint added
Operating in CiscoSSL FIPS mode

#. I used following command to remove old key and reconfigure.

1. ssh delete host X.X.X.X

2. crypto host_key delete host X.X.X.X

Thanks

Re: ISE nodes unable to see SFTP repository

Arne Bier — Sun, 19 Jun 2022 20:41:28 GMT

I can't see the screenshot you tried to embed in the message (I think?).

I'm getting confused - you're able to delete the old crypto keys, yes?

But when you try adding a new crypto key then you get an error?

In my experience the error when adding crypto keys tends to be a network communication error - during this process, ISE tries to reach the remote host to exchange public keys - and if TCP/22 is being blocked (or the remote end fails to establish a response to ISE) then the crypto command will fail. You can try enabling the debug below, before you issue the the crypto add command

debug transfer 7

Re: ISE nodes unable to see SFTP repository

rishisemwal — Mon, 20 Jun 2022 13:17:19 GMT

Hello,

Please see result below,

[cid:dd3cf8fb-9d99-4ee3-8b8f-c217e5d6071c]
UNMISS-BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode

UNMISS-BEN-ISE/admin# sh repository sftp
3 [9587]:[error] transfer: cars_xfer.c[204] [admin]: couldn't get repository sftp
% Error: Repository sftp could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Repository not found
UNMISS-BEN-ISE/admin# debug transfer 7
UNMISS-BEN-ISE/admin#

Re: ISE nodes unable to see SFTP repository

Arne Bier — Mon, 20 Jun 2022 20:29:07 GMT

Hello @rishisemwal

You must enable the debug BEFORE you issue the "show repo" command - this debug command attempts to show you what ISE is doing during the network communications to the repo.

And have you also tried re-setting the password for the repo's user account? I sometimes validate the user credentials using a tool like WinSCP to ensure that I have the correct password. And then I re-enter the same password for this repo config in the ISE GUI.

Re: ISE nodes unable to see SFTP repository

rishisemwal — Tue, 21 Jun 2022 07:24:19 GMT

Good morning Arne,

1# I have changed the password and tried to configure host key but same issue.

UNMISS-BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode

UNMISS-BEN-ISE/admin# debug transfer 7
UNMISS-BEN-ISE/admin# sh repository sftp
% Error: Repository sftp could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
3 [18261]:[error] transfer: cars_xfer.c[204] [admin]: couldn't get repository sftp
% Repository not found
UNMISS-BEN-ISE/admin#

2# Getting following error when I tried to validate SFTP.
[cid:71f31336-ede9-401b-a798-03ebcab5ca02]

Thanks
Rishi

Re: ISE nodes unable to see SFTP repository

Arne Bier — Wed, 22 Jun 2022 21:37:48 GMT

Are you able to SSH from that ISE node to the SFTP IP address?

Re: ISE nodes unable to see SFTP repository

fitzie — Tue, 23 Aug 2022 14:45:59 GMT

Is it possible to have two SFTP repositories with the same IP? I want to have 1 repo for backups and one for patches. Whenever I try to add the second repo things go south. This is an air-gapped network, so it's not as easy to work with as my other nets.

Re: ISE nodes unable to see SFTP repository

Arne Bier — Tue, 23 Aug 2022 22:54:06 GMT

Hi @fitzie

I don't think ISE will let you add two repos with the same IP. I would do it differently. Have one repo, but create two different accounts - one for backups, and one for patches. The account/user login directory should then point to two different paths.

If you need two separate repos because of disk space constraints, then of course create another repo on the additional host.

Re: ISE nodes unable to see SFTP repository

ajc — Tue, 29 Aug 2023 13:53:32 GMT

Hi Arne,

I tried the proposed solution and it did not work. I can ping and SSH from my ISE 2.7 patch 7 into the SFTP server (see next), BUT I cannot get the KEY FINGERPRINT as per your example (I also used IP for the SFTP no DNS resolution). I mean, not getting the 2nd part of the CRYPTO HOST KEY ADD HOST "IP" (see also below). I have tried everything and still not working. Any thoughts? I am exploring if the option of adding in the KNOWN Hosts of my ISE the key/fingerprint

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4

myISE/admin# show repository myrepo13
6 [17055]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository myrepo13 requested
6 [17055]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.1.1.100
7 [17055]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.1.1.100 testing *** /home/isebck/ ls -l /home/isebck/
6 [17055]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.1.1.100 remote user: testing command: ls -l /home/isebck/
7 [17055]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [17057]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes nsmgmt@10.1.1.100
3 [17055]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
% Error: Repository myrepo13 could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
7 [17055]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
myISE/admin#

NOTHING from the following command output

myISE/admin#
myISE/admin show crypto host_keys
myISE/admin#

As I mentioned, when I followed the procedure of removing everything from CLI / GUI and recreated it, the next command did not provide me with the 2nd part of the output as your example

myISE/admin#
myISE/admin# crypto host_key add host 10.1.1.100
host key fingerprint added
Operating in CiscoSSL FIPS mode

**************NO RSA SHA 256 fingerprint output***************

# Host 10.48.148.10 found: line 1
10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4

myISE/admin#

********************SSH from ISE to SFTP Server*******************

my ISE/admin# ssh 10.1.1.100 testing
Operating in CiscoSSL FIPS mode
FIPS mode initialized
The authenticity of host '10.1.1.100 (10.1.1.100)' can't be established.
RSA key fingerprint is SHA256:0uSiedgerDEDGGG2341DDFTRVdfs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.100' (RSA) to the list of known hosts.
testing@10.1.1.100's password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-78-generic x86_64)

System information as of Mon Aug 28 02:01:21 PM EDT 2023

System load: 1.11767578125 Processes: 273
Usage of /: 7.8% of 588.04GB Users logged in: 1
Memory usage: 8% IPv4 address for ens160: 10.1.1.100
Swap usage: 0%

*** System restart required ***
Last login: Mon Aug 28 13:48:56 2023 from 172.16.1.1
testing@mytoolbox:~$ ^C
testing@mytoolbox:~$

Re: ISE nodes unable to see SFTP repository

Arne Bier — Tue, 29 Aug 2023 05:41:28 GMT

@ajc

The issue with the IP address you used in the crypto host_key add host command.

You must use the IP address of the SFTP server - your example shows:

crypto host_key add host 172.22.4.100

but according to your logs, the IP address of the SFTP server is 10.1.100 - this should work:

crypto host_key add host 10.1.100