12-16-2021 04:46 PM
Our ISE deployment consists of (4) nodes - (2) PANs and (2) PSNs - and only 1/4 can access the repository where the Log4J patch file is currently located.
I’ve recreated the repository via the ISE Admin console and the config deployed to all (4) of the nodes, but the only one that's able to connect and see the contents of the repository is our primary PAN. The secondary PAN can't connect, nor can the (2) PSNs in our deployment.
I validated the repository in the GUI after it was created.
As mentioned, I was able to see the contents of the repo using the show repository command on our primary PAN, but received the following error when running the same command on all other nodes:
show repository ISE_Repo
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
I've tried manually removing and then re-adding the repository and running the crypto host_key add host <host IP> command, but get the same result.
Any help would be greatly appreciated.
Solved! Go to Solution.
03-30-2022 12:28 AM
Hi Arne,
I randomly noticed the "ssh delete host X.X.X.X" command on ISE whilst troubleshooting and tried that and it fixed the issue. My steps were.
1. ssh delete host X.X.X.X
2. crypto host_key delete host X.X.X.X
3. Deleted Repo from GUI
4. Re Add repo from GUI
5. crypto host_key add host X.X.X.X
When i was running crypto host_key add host, it said the entry was added but didn't display the String output. the SSH command must remove the old entry properly.
12-16-2021 07:03 PM
Hello @paynewj
The crypto command MUST be issued on EVERY ISE node. The sftp config is replicated, but the crypto keys are individual per node. It's a manual once-off chore, but without it, you won't get access to the SFTP server.
regards
Arne
12-17-2021 05:24 AM
I appreciate the response, @Arne Bier .
I've issued the command on each of the nodes. I received confirmation that the host key fingerprint was added and that it was opperating in CiscoSSL FIPS mode.
Here's a debug from the one that's working:
ISESERVER/admin# show repo ISE_Repo
6 [16939]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [16939]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [16939]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [16939]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [16942]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
7 [16939]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read
7 [16939]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0
7 [16939]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded
7 [16939]:[debug] transfer: cars_xfer.c[268] [admin]: freed file list
ise-apply-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz
ise-rollback-CSCwa47133_Ver_24_30_allpatches-SPA.tar.gz
Debug from one that isn't:
ISESERVER/admin# show repo ISE_Repo
6 [30438]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ISE_Repo requested
6 [30438]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful <SFTPServer>
7 [30438]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: <SFTPServer> ise_backup *** /ise/ISE_Repo/ ls -l /ise/ISE_Repo/
6 [30438]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: <SFTPServer> remote user: ise_backup command: ls -l /ise/ISE_Repo/
7 [30438]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5
7 [30440]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes ise_backup@<SFTPServer>
3 [30438]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error
7 [30438]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306
% Error: Repository ISE_Repo could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour).
% Error : Operation failed due to one of the following reasons
1. host key is not configured
2. host key is removed because of re-image
3. host key is removed from some other repository having same ip/hostname
% Please add the host key using the crypto host_key exec command
% SSH connect error
Not sure why it's still saying "host key is not configured, or parsing error" after the crypto command has been entered.
I appreciate you time and assistance with this issue, Arne, and anyone else that's able to help.
12-17-2021 12:40 PM
I can reproduce your error. It looks like a fault with the crypto host key still.
nac2/admin# debug transfer 7 nac2/admin# show repository ubuntu 6 [11523]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested 6 [11523]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10 7 [11523]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/ 6 [11523]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/ 7 [11523]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5 7 [11525]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10 3 [11523]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error 7 [11523]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306 % Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour). % Error : Operation failed due to one of the following reasons 1. host key is not configured 2. host key is removed because of re-image 3. host key is removed from some other repository having same ip/hostname % Please add the host key using the crypto host_key exec command % SSH connect error nac2/admin# nac2/admin# show crypto host_keys nac2/admin#
Add the key - BUT - make sure it matches how it was defined in the repo - if you're using IP address in the repo, then add the key as an IP address. If using a hostname, then add the key as a hostname. I used an IP address in my repo URL.
nac2/admin# nac2/admin# crypto host_key add host 10.48.148.10 host key fingerprint added Operating in CiscoSSL FIPS mode # Host 10.48.148.10 found: line 1 10.48.148.10 RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4 nac2/admin# show repository ubuntu 6 [12976]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested 6 [12976]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10 7 [12976]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/ 6 [12976]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/ 7 [12976]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5 7 [12978]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10 7 [12976]:[debug] transfer: sftp_handler.c[478] [admin]: Found sftp prompt; No more data to read % Repository is empty 7 [12976]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status 0 7 [12976]:[debug] transfer: cars_xfer_util.c[2278] [admin]: ssh_list xfer succeeded nac2/admin#
My advice would be to list all the crypto keys, delete them all, and re-add
show crypto host_keys
And to demonstrate the disconnect between IP address and hostname in the crypto processing, here is an example where I add the crypto using the DNS hostname - and then the whole thing doesn't work again.
nac2/admin# nslookup ubuntu Trying "ubuntu.networks.local" ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54919 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;ubuntu.networks.local. IN ANY ;; ANSWER SECTION: ubuntu.networks.local 3600 IN A 10.48.148.10 Received 67 bytes from 10.48.148.11#53 in 15 ms nac2/admin# crypto host_key add host ubuntu host key fingerprint added Operating in CiscoSSL FIPS mode # Host ubuntu found: line 1 ubuntu RSA SHA256:e98rk9cO5vKtehEye+CgpvxcEkmoPpGhd1qi+MFMok4 nac2/admin# debug transfer 7 nac2/admin# show repository ubuntu 6 [32151]:[info] transfer: cars_xfer.c[224] [admin]: sftp dir of repository ubuntu requested 6 [32151]:[info] transfer: cars_xfer_util.c[2296] [admin]: Server validation successful 10.48.148.10 7 [32151]:[debug] transfer: sftp_handler.c[1093] [admin]: Running sftp command: 10.48.148.10 sftpuser *** /sftpuser/ ls -l /sftpuser/ 6 [32151]:[info] transfer: sftp_handler.c[583] [admin]: DEBUG: local user: admin UID: 0 sftp_run_parent FD: 5 remote host: 10.48.148.10 remote user: sftpuser command: ls -l /sftpuser/ 7 [32151]:[debug] transfer: sftp_handler.c[592] [admin]: fd is:5 7 [32153]:[debug] transfer: sftp_handler.c[290] [admin]: Executing SFTP command: 0 admin /usr/bin/sftp -oIdentityFile=/home/admin/.ssh/id_rsa -oUserKnownHostsFile=/home/admin/.ssh/known_hosts -oPasswordAuthentication=yes sftpuser@10.48.148.10 3 [32151]:[error] transfer: sftp_handler.c[652] [admin]: sftp_run_parent Error: host key is not configured, or parsing error % Error: Repository ubuntu could not be accessed. In case Backup was Restored on different setup, Please reconfigure the repository passwords (expected behaviour). 7 [32151]:[debug] transfer: sftp_handler.c[962] [admin]: sftp parent status -306 % Error : Operation failed due to one of the following reasons 1. host key is not configured 2. host key is removed because of re-image 3. host key is removed from some other repository having same ip/hostname % Please add the host key using the crypto host_key exec command % SSH connect error nac2/admin#
08-23-2022 07:45 AM
Is it possible to have two SFTP repositories with the same IP? I want to have 1 repo for backups and one for patches. Whenever I try to add the second repo things go south. This is an air-gapped network, so it's not as easy to work with as my other nets.
08-23-2022 03:54 PM
Hi @fitzie
I don't think ISE will let you add two repos with the same IP. I would do it differently. Have one repo, but create two different accounts - one for backups, and one for patches. The account/user login directory should then point to two different paths.
If you need two separate repos because of disk space constraints, then of course create another repo on the additional host.
03-28-2022 06:12 AM
Hi, Did you ever get this resolved? Experiencing a similiar issue on 3.0 patch 5. Backups were working and now have stopped. The error is saying the repository cannot be accessed.
I deleted the host keys to readd and now they are not showing in show crypto host_keys when i readd them. the command line displays they have been added correctly
03-28-2022 01:09 PM
Let's see what @paynewj has to say.
In my experience the crypto host_keys always add, even in ISE 3.0 patch 5.
If they don't get added then please validate 100% that the ISE node can reach the SFTP's IP/hostname of the host you're adding (e.g. ping/ssh to the SFTP host from ISE).
What does the "debug transfer 7" tell you when you try to add the host_key?
Do you have any other ISE hosts that are able to add the crypto host_keys to the same SFTP host?
After a config restore, the repository password has to be re-entered in the ISE GUI. This is an expected thing in ISE and is a separate step to the host_keys issue.
03-30-2022 12:28 AM
Hi Arne,
I randomly noticed the "ssh delete host X.X.X.X" command on ISE whilst troubleshooting and tried that and it fixed the issue. My steps were.
1. ssh delete host X.X.X.X
2. crypto host_key delete host X.X.X.X
3. Deleted Repo from GUI
4. Re Add repo from GUI
5. crypto host_key add host X.X.X.X
When i was running crypto host_key add host, it said the entry was added but didn't display the String output. the SSH command must remove the old entry properly.
03-30-2022 03:19 PM
I am glad it all worked out. Hopefully it will help someone else in future too
06-19-2022 02:45 AM
Hello,
1. I reconfigured SFTP after changing the IP address of ISE server. It's not generating the key
BEN-ISE/admin# crypto host_key add host 10.155.22.9
host key fingerprint added
Operating in CiscoSSL FIPS mode
2. Because of that it is not validating. Please see the error
Repository validation failed due to error - CARS_RM_NOT_FOUND : -200 : Repository not found.
3. I can not delete it to reconfigure.. Please see the error
Unabled to delete repository(s): sftp (used in scheduled/on-demand backups)
06-19-2022 03:42 AM
Don’t manage your repository configs via CLI. Make changes in the Admin UI because that will propagate the changes to all nodes in the deployment. The only thing you need to do on the CLI is to ensure that the crypto key is configured on every node on which you need to access that repo.
06-19-2022 04:27 AM
06-19-2022 01:41 PM
I can't see the screenshot you tried to embed in the message (I think?).
I'm getting confused - you're able to delete the old crypto keys, yes?
But when you try adding a new crypto key then you get an error?
In my experience the error when adding crypto keys tends to be a network communication error - during this process, ISE tries to reach the remote host to exchange public keys - and if TCP/22 is being blocked (or the remote end fails to establish a response to ISE) then the crypto command will fail. You can try enabling the debug below, before you issue the the crypto add command
debug transfer 7
06-20-2022 06:17 AM
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: