ISE 3.2 backup failure

[densto]

Hello,

I am trying to configure backup repository for ISE3.2 in Azure.  I can create it but backup would always fail.  If I use the same repository for ISE on prem it works just fine. From CLI if i run show repository i get an error message "% Failure occurred during request" sh backup history :  backup ISE-ADMIN-ConfigBackup-CFG10-221117-0000.tar.gpg to repository Cerberus-SFTP: error - transfer failed

I confirmed that permissions are good on the remote server.  We tried SFT, FTP.

Log from backup server session ends with ISE closing a session.

"Closing connection: An existing connection was forcibly closed by the remote host."

10/18/2022 1:43:41 PM 10.224.1.132 3193 Connection terminated
10/18/2022 1:43:41 PM 10.224.1.132 3193 Closing connection: An existing connection was forcibly closed by the remote host.
10/18/2022 1:42:55 PM Scheduled task check complete
10/18/2022 1:42:55 PM Checking scheduled tasks
10/18/2022 1:42:40 PM 10.224.1.132 3193 Using new keys for sending data to the client
10/18/2022 1:42:40 PM 10.224.1.132 3193 All keys derived
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length failed (16 -> 64) : invalid key length
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length (16 -> 64)
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length failed (16 -> 64) : invalid key length
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length (16 -> 64)
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1
10/18/2022 1:42:40 PM 10.224.1.132 3193 ECDH Key size: 256
10/18/2022 1:42:40 PM 10.224.1.132 3193 SSH key bits: 2048
10/18/2022 1:42:40 PM 10.224.1.132 3193 Agreed upon KEX: 'ecdh-sha2-nistp256' Host Key: 'rsa-sha2-512' C2S : 'aes128-ctr, hmac-sha2-512, none' S2C : 'aes128-ctr, hmac-sha2-512, none'
10/18/2022 1:42:40 PM 10.224.1.132 3193 Client Identification: SSH-2.0-OpenSSH_8.8 PKIX[13.2.3]
10/18/2022 1:42:40 PM 10.224.1.132 3193 Incoming connection request on SSH SFTP listener 2 at 10.226.1.16:22 accepted from 10.224.1.132:51378
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1

[marce1000]

         - Check the logs of the SFTP/SSH server (too).

 M.

[densto]

This is the log from the sftp server.  Connection is established successfully and then teminated by ISE

LOG is in reverse order.

10/18/2022 1:43:41 PM 10.224.1.132 3193 Connection
10/18/2022 1:43:41 PM 10.224.1.132 3193 Closing connection: An existing connection was forcibly closed by the remote host.
10/18/2022 1:42:55 PM Scheduled task check complete
10/18/2022 1:42:55 PM Checking scheduled tasks
10/18/2022 1:42:40 PM 10.224.1.132 3193 Using new keys for sending data to the client
10/18/2022 1:42:40 PM 10.224.1.132 3193 All keys derived
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length failed (16 -> 64) : invalid key length
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length (16 -> 64)
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length failed (16 -> 64) : invalid key length
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init: Set key length (16 -> 64)
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1
10/18/2022 1:42:40 PM 10.224.1.132 3193 ECDH Key size: 256
10/18/2022 1:42:40 PM 10.224.1.132 3193 SSH key bits: 2048
10/18/2022 1:42:40 PM 10.224.1.132 3193 Agreed upon KEX: 'ecdh-sha2-nistp256' Host Key: 'rsa-sha2-512' C2S : 'aes128-ctr, hmac-sha2-512, none' S2C : 'aes128-ctr, hmac-sha2-512, none'
10/18/2022 1:42:40 PM 10.224.1.132 3193 Client Identification: SSH-2.0-OpenSSH_8.8 PKIX[13.2.3]
10/18/2022 1:42:40 PM 10.224.1.132 3193 Incoming connection request on SSH SFTP listener 2 at 10.226.1.16:22 accepted from 10.224.1.132:51378
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1
10/18/2022 1:42:40 PM 10.224.1.132 3193 CCipher::init : 1ted by ISE

[ammahend]

Check firewall logs as well, once setup is done on port 21, server initiates traffic on port 20 from out to in, usually DPI takes care of this by default unless it’s disabled for certain protocols you are using.

[adamscottmaster2013]

Someone need to check your networking knowledge, seriously !!!!

The OP uses sFTP, NOT FTP.  sFTP uses tcp port 22.  There is NO command & control port and data port in sFTP.

[densto]

We tested with FTP too. but as i mentioned there is not firewalls between the nodes.  I am able to use the same repository on Prem ISE deployment 3.0 successfully. Thank you

[ammahend]

I get your point on sftp, I Wasn’t talking about sftp, you need to chill. 

[densto]

There is no firewall or NSG setup between the nodes.  

[adamscottmaster2013]

@densto:  Can you get on the command line of the ISE and perform the following:

ssh IP_address_of_the_sftp_server xxxx version 2

where xxxx is the username on the sftp server

You will like see a fail but at least it will give you some clues.  It is likely that the sftp server has the host key type that the ISE does not have.  In other words, the sFTP server does not support ssh-rsa host key type.

I have an issue with my 3.1 ise can not sFTP to the sftp server because Ubuntu 22.0.4 does not support ssh-rsa:

ciscoISE/admin#ssh 192.168.1.1 adamscott version 2
Operating in CiscoSSL FIPS mode
FIPS mode initialized
Unable to negotiate with 192.168.1.1 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
ciscoISE/admin#

[densto]

@adamscottmaster2013 

We use 3rd party backup server Cerberus that runs on windows.  It does not support ssh sessions.  I do have an Ubuntu server that I tested ssh from ISE 3.2 connection to ubuntu 20.04.4 successfully using your command.   I am wondering if new ISE 3.2 requires linux os to run SFTP like DNA server.   I am going to configure SFTP on ubuntu and will follow up with update.

Thank you,

Denis

[adamscottmaster2013]

@densto --> NO, NO, NO.... Ubutun server version 20.04.04 does NOT have this problem because it still support some of Cisco legacy host key type.  Try it on Ubuntu 22.04.x will produce the same issue as your 3rd party backup server, like mine did with Ubuntu 22.04.1.

My guess is that Cisco customizes it openssh/ssl not to support rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519.  I was able to reproduce this issue using CentOS-7 as the client (same as Cisco ISE), but that's just my guess.

[densto]

@adamscottmaster2013 

I tried creating a backup SFTP repository using Ubuntu 20.04.04 still did not work.  We opened a tac case.  I am going to test SFTP config with PKI  enabled for authentication.  

Thank you

[thomas]

I have created an Ubuntu instance in AWS using VSFTPD and I messed myself up originally by not allowing access to the Passive ports in my Security Group for the FTP Server. Make sure your and Azure NACLs and any other ACLs allow these additional ports!

These rules are for my Security Group Inbound to the SFTP server but your passive ports may differ based on your SFTP server configuration:

My VSFTPD Passive ports config:

pasv_enable=YES

pasv_min_port=1024

pasv_max_port=1048

[ajc]

Look at this post for additional information: https://community.cisco.com/t5/network-access-control/ise-nodes-unable-to-see-sftp-repository/td-p/4520304

TAC said that ISE 3.2 would have the fix for unsupported algorithm. I have not checked.

[hslai · Accepted solution]

@densto I believe you are using SSH key to connect to the file server in Azure. If that is the case, then CSCwd63717 is a new defect I opened just yesterday. Typically, it takes one or two business days before you are able to view it. The bug is impacting all repositories with PKI enabled (i.e. using SSH keys) configured in ISE 3.2 admin web UI or via OpenAPI. The workaround is to configure the repository in the ISE admin CLI but that could be removed or overwritten after a restart of ISE services.