ISE 3.1 patch 3 backup failure

[adamscottmaster2013]

I have a schedule job to back up the ISE configuration everyday to an external sFTP server, running on Ubuntu server 20.04.5 LTS and it has been working for the past two years.

Yesterday, I upgraded my Ubuntu server to 22.04.1 LTS and after that backup stopped working because the Ubuntu no longer accept the ssh-rsa host key from the ISE server.  This is what I see on the Ubuntu server log:

Nov 17 15:40:14 Ubuntu_22_04_1 sshd[145827]: Unable to negotiate with 192.168.1.1 port 17310: no matching host key type found. Their offer: ssh-rsa [preauth]

This is what I see on the ISE:  

ssh 192.168.1.2 adamscott version 2
Operating in CiscoSSL FIPS mode
FIPS mode initialized
Unable to negotiate with 192.168.1.2 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519

In other words, my ubuntu version 22.0.4.1 does not allow ssh-rsa from the ISE.  This option is no longer available in Ubuntu due to security risk.  At the same time, there is no option on the Cisco ISE to use the host key type that is acceptable to Ubuntu.  

How do you work around this problem?  Cisco ISE is running on CentOS 7.x so I assume that stronger host key type is definitely support.

Thoughts?

[hslai · Accepted solution]

@adamscottmaster2013 What you described appears addressed by the fix for CSCwa95889. The fix is to add rsa-sha2-512 and rsa-sha2-256 as HostKeyAlgorithms for SSH outbound from ISE.

ISE 3.1 Patch 4 includes this fix. Please try it out.

[adamscottmaster2013]

@hslai:  Any fix ISE version 3.0?

[adamscottmaster2013]

@hslai:  After upgrading to 3.1 patch-4, backup via sFTP is working again.  Thanks.

[adamscottmaster2013]

@hslai:  Do you mind sharing the workaround?  Is it as easy as editing the /etc/ssh/ssh_config file on the ISE?

[hslai]

@adamscottmaster2013 For ISE 3.0, the fix is coming in Patch 7 but that is months away. If you need it sooner, either open a TAC case to apply the workaround via root access or to request for a hot patch.

[adamscottmaster2013]

@hslai:  Do you mind sharing the workaround?  Is it as easy as editing the /etc/ssh/ssh_config file on the ISE?

[hslai]

@adamscottmaster2013 Yes, that is the main part. In case that the known_hosts file(s) not properly updated by "crypto host_key add host <>", manually add the missing entries.

[adamscottmaster2013]

@hslai:  Can you be specific about which line(s) in the /etc/ssh/ssh_config files?  Are you referring to these lines below:

# IdentityFile ~/.ssh/identity
# IdentityFile ~/.ssh/id_rsa
# IdentityFile ~/.ssh/id_dsa

I checked the /etc/ssh/ssh_config on in the CentOS 7.x and that's what I saw and the CentOS 7.x could ssh into the Ubuntu 22.0.4 LTS without any issues and yet the ISE 3.1 could not.  Therefore, I assume these lines are not the main culprit.

[hslai]

@adamscottmaster2013 Sorry for not being clear. The workaround is for Cisco TAC to apply to the affected ISE instances. That is why you would need a TAC case.

[adamscottmaster2013]

@hslai:  Can you be specific on the workaround?  I am trying to understand what is being changed.  Be specific.

[hslai]

@adamscottmaster2013  All the changes for the workaround are on ISE side and need root access.

If you have no TAC case on this, please open one. If you have one, please ask TAC to contact me if you need additional details. I wrote the internal note for TAC but that was done before our engineering fixed it so that note need some updates.