CUCM DRS backup device error

PACC · ‎09-19-2024

Hi all,

CUCM 11.5.1.12900-21
Rocky Linux 9.4, OpenSSH_8.7p1, OpenSSL 3.0.7 1

I am attempting to set the Rocky server as a backup device for CUCM. I initially got the errors regarding key exchange algorithms and host key types not matching, so I enabled these in Rocky's ssh config as below.

KexAlgorithms +diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
HostkeyAlgorithms +ssh-rsa,ssh-dss
Ciphers +aes128-cbc,3des-cbc

I then got a different error which isn't giving me a clue as to what the issue might be: com.jcraft.jsch.JSchException: verify: false [preauth]

Here is the ssh debug from the Rocky server:

debug1: Remote protocol version 2.0, remote software version JSCH-0.1.50
debug1: compat_banner: no match: JSCH-0.1.50
debug1: SELinux support enabled [preauth]
debug1: ssh_selinux_change_context: setting context from 'system_u:system_r:sshd_t:s0-s0:c0.c1023' to 'system_u:system_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: diffie-hellman-group14-sha1 [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: diffie-hellman-group14-sha1 need=32 dh_need=32 [preauth]
debug1: kex: diffie-hellman-group14-sha1 need=32 dh_need=32 [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth]
debug1: rekey out after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
error: Received disconnect from 172.16.101.10 port 52112:3: com.jcraft.jsch.JSchException: verify: false [preauth]

I can't tell if it is still an issue with key types or ciphers or something else entirely. Does anyone else have a clue what it might be?

thanks

j

Elliot Dierksen · ‎09-20-2024

Here is what I have used to modify the config of OpenSSH to support Cisco DRS.

Ciphers aes128-cbc,3des-cbc,blowfish-cbc

These ones can be added to the default with multiple addition lines:

Ciphers +aes128-cbc
Ciphers +3des-cbc
Ciphers +blowfish-cbc

Not sure if this is needed

KexAlgorithms +diffie-hellman-group1-sha1


in sshd_config:

KexAlgorithms +diffie-hellman-group1-sha1
KexAlgorithms +diffie-hellman-group-exchange-sha1

Ciphers +aes128-cbc
# 3DES isn't supported on newer version of DRS or in
# newer versions of OpenSSH
Ciphers +3des-cbc

Newer Ubuntu needs this:
PubkeyAcceptedAlgorithms +ssh-rsa
HostkeyAlgorithms +ssh-rsa

PACC · ‎09-22-2024

Thanks @Elliot Dierksen I have attempted to replicate your config but my version of OpenSSH does not support blowfish-cbc. The sshd daemon will not start with that in the config.

I am copying and pasting the credentials so I don't get them wrong.

Here are the drf logs from cucm. Not sure what other logs may show more info.

2024-09-23 09:23:06,551 DEBUG [NetMessageDispatch] - drfMessageHandler:HandleUpdateDestination: The total number of configured Devices (pre update):1
2024-09-23 09:23:06,551 INFO [NetMessageDispatch] - Entering decryptPassword
2024-09-23 09:23:06,552 INFO [NetMessageDispatch] - Use Dkey to decrypt data
2024-09-23 09:23:06,552 INFO [NetMessageDispatch] - CCMENC::ERROR : Dkey decryption failed. Use recovery mechanism to decrypt data.
2024-09-23 09:23:06,552 INFO [NetMessageDispatch] - Using static key to decrypt data
2024-09-23 09:23:06,552 INFO [NetMessageDispatch] - decryptPassword was successful
2024-09-23 09:23:06,552 DEBUG [NetMessageDispatch] - drfMessageHandler:HandleUpdateDestination: Check whether SFTP is accessible.
2024-09-23 09:23:06,553 INFO [NetMessageDispatch] - Entering decryptPassword
2024-09-23 09:23:06,553 INFO [NetMessageDispatch] - Use Dkey to decrypt data
2024-09-23 09:23:06,554 INFO [NetMessageDispatch] - CCMENC::ERROR : Dkey decryption failed. Use recovery mechanism to decrypt data.
2024-09-23 09:23:06,554 INFO [NetMessageDispatch] - Using static key to decrypt data
2024-09-23 09:23:06,554 INFO [NetMessageDispatch] - decryptPassword was successful
2024-09-23 09:23:06,554 DEBUG [NetMessageDispatch] - drfUtils:establishSftpConnection: Trying to connect to the SFTP server.
2024-09-23 09:23:06,554 INFO [NetMessageDispatch] - IN -- CryptoUtil.java - isCallManagerInFipsMode(..) - 
2024-09-23 09:23:06,560 INFO [NetMessageDispatch] - OUT -- CryptoUtil.java - isCallManagerInFipsMode - false
2024-09-23 09:23:06,560 DEBUG [NetMessageDispatch] - drfUtils:establishSftpConnection: Server is in NONFIPS mode
2024-09-23 09:23:06,560 DEBUG [NetMessageDispatch] - drfUtils:establishSftpConnection: Connecting SFTP server...
2024-09-23 09:23:06,715 INFO [NetMessageDispatch] - drfUtils:establishSftpConnection: JSCH exception has occurred.. retrying to connect. Authentication counter value = 1
2024-09-23 09:23:06,715 ERROR [NetMessageDispatch] - drfUtils:establishSftpConnection: Caught JSchException : verify: false
2024-09-23 09:23:06,856 INFO [NetMessageDispatch] - drfUtils:establishSftpConnection: JSCH exception has occurred.. retrying to connect. Authentication counter value = 2
2024-09-23 09:23:06,856 ERROR [NetMessageDispatch] - drfUtils:establishSftpConnection: Caught JSchException : verify: false
2024-09-23 09:23:06,999 INFO [NetMessageDispatch] - drfUtils:establishSftpConnection: JSCH exception has occurred.. retrying to connect. Authentication counter value = 3
2024-09-23 09:23:07,000 ERROR [NetMessageDispatch] - drfUtils:establishSftpConnection: Caught JSchException : verify: false
2024-09-23 09:23:07,000 ERROR [NetMessageDispatch] - drfUtils:establishSftpConnection: Unable to connect to SFTP server.. 
2024-09-23 09:23:07,000 ERROR [NetMessageDispatch] - drfUtils:isSftpLocationAccessible: Error . Failed to create SFTP connection to device, stdErr: Unable to access SFTP server. Please ensure the username and password are correct.

CUCM is currently backing up to a CentOS 7.4 box (we want to replace this). Here are the ssh debug logs from the CentOS box during a successful backup from CUCM; all these algorithms and ciphers show as being available on the Rocky box.

sshd[15489]: debug1: kex: algorithm: diffie-hellman-group1-sha1 [preauth]
sshd[15489]: debug1: kex: host key algorithm: ssh-rsa [preauth]
sshd[15489]: debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha1 compression: none [preauth]
sshd[15489]: debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha1 compression: none [preauth]
sshd[15489]: debug1: kex: diffie-hellman-group1-sha1 need=32 dh_need=32 [preauth]
sshd[15489]: debug1: kex: diffie-hellman-group1-sha1 need=32 dh_need=32 [preauth]

Elliot Dierksen · ‎09-23-2024

I am not familiar with the Rocky distribution, so it may require something I don't know about. I did spot an article that might help.

https://www.devdungeon.com/content/fix-broken-pipe-error-ssh-connection-fedoravmware

Also, the ssh debugs on the Rocky box look abbreviated. Is there more there? It might be advisable to try turning up the debug level on SSHD if you can. That code block is a note I have had for years so it makes sense to me, even if it seems a bit jumbled. The key part was towards the end.

KexAlgorithms +diffie-hellman-group1-sha1
KexAlgorithms +diffie-hellman-group-exchange-sha1

Ciphers +aes128-cbc

#Newer Ubuntu needs this:
PubkeyAcceptedAlgorithms +ssh-rsa
HostkeyAlgorithms +ssh-rsa

The OS on CUCM 11 is quite a bit older, and any newer distributions tend to have different and more secure default settings. The SFTP client in CUCM 11 is quite persnickety.

PACC · ‎09-23-2024

Thanks for your replies @Elliot Dierksen

I increased the debug level and have attached the result. I have a VMWare vCentre server backing up using the same credentials that is working successfully.

I tried that IPQoS parameter but it hasn't made a difference.

I thought about trying to ssh or sftp directly from the cucm shell just to see what happens, but am unsure how to do that (or if it is even possible).

Elliot Dierksen · ‎09-24-2024

@PACC Would you please post the complete contents of the sshd_config file from the Rocky box? It will probably be in /etc/ssh. I am sure there is something that the DRS clients wants but the SSH server isn't offering it. I have an older Ubuntu box (5.4.0-196-generic) that I have used for DRS in my lab, and these are the only modifications I have made to the stick sshd_config.

KexAlgorithms +diffie-hellman-group1-sha1
KexAlgorithms +diffie-hellman-group-exchange-sha1
Ciphers +aes128-cbc
Ciphers +3des-cbc

Here are the sshd_config modifications on a newer (6.5.0-41-generic) version of Ubuntu.

# Local additions
KexAlgorithms +diffie-hellman-group1-sha1
KexAlgorithms +diffie-hellman-group-exchange-sha1
Ciphers +aes128-cbc
PubkeyAcceptedAlgorithms +ssh-rsa
HostkeyAlgorithms +ssh-rsa

dpwaruna · ‎09-23-2024

Hi @PACC

Not sure if this helps. But I thought maybe I shared my experience. We have two CUCM clusters which are cucm11 and cucm14. I had a similar issue, and I raise a TAC case to resolve the issue. CISCO came up with a list of SFTP software that we can use with call manager.

==============================================================

Currently, there are four supported SFTP server solutions recommended for use with CUCM and CUC backup operations. Of these, only one is certified with the Cisco Technology Developer (CTD) program. The details of these SFTP solutions are listed below:

1. Open SSH
Runs On UNIX - currently the following versions are supported: AIX, HP-UX, Irix, Linux, NeXT, SCO, SNI/Reliant Unix, Solaris, Digital Unix/Tru64/OSF, Mac OS X, and Cygwin.
Cost Free
Recommended YES
Cisco Certified NO
URL http://sshwindows.sourceforge.net/

2. Cygwin - This is an application that provides a Unix-like environment and associated tools (SFTP is one), that runs on a modern x86 32-bit and 64-bit version of Windows OS (XP with SP3/Server 20xx/Vista/7/8)
Runs On Windows
Cost Free
Recommended YES
Cisco Certified NO
URL http://www.cygwin.com/

3. TitanFTP
Runs On Windows
Cost US $599.95 up to $1949.95 (depending on version purchased)
Recommended YES
Cisco Certified NO
URL http://www.titanftp.com/

4. GlobalscapeEFT
Runs On Windows
Cost From US $1495.00
Recommended YES
Cisco Certified YES
URL http://www.globalscape.com/gsftps/cisco.aspx

===================================================================

Then I choose cygwin as the SFTP server and install it on a windows server. CUCM14 backup was successful and CUCM11 had similar error as you mentioned above.

I agree with @Elliot Dierksen and CISCO also provided the same feed back

++ As per the error as OLD version of CUCM is not supporting the Algorithm of current SFTP sever so it is prompting experiencing authentication error on CUCM 11.

So, I had to try to use a different SFTP server which is using simple authentication method. I found freeFTPD server that I managed to setup with CUCM11.

Only concern is needed to install freeFTPD as a service. Otherwise, application will close when you logout from the server.

You can check the connectivity to sftp server by using bellow command.

utils network connectivity "SFTP server IP" port

PACC · ‎09-25-2024

admin:utils network connectivity 192.168.200.53 22
Connection to 192.168.200.53 22 port [tcp/ssh] succeeded!

Service accessible

I did get some jsch logs from cucm:

2024-09-26 09:16:08,274 INFO [NetMessageDispatch] - Connecting to 192.168.200.53 port 22
2024-09-26 09:16:08,275 INFO [NetMessageDispatch] - Connection established
2024-09-26 09:16:08,291 INFO [NetMessageDispatch] - Remote version string: SSH-2.0-OpenSSH_8.7
2024-09-26 09:16:08,291 INFO [NetMessageDispatch] - Local version string: SSH-2.0-JSCH-0.1.50
2024-09-26 09:16:08,291 INFO [NetMessageDispatch] - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2024-09-26 09:16:08,292 INFO [NetMessageDispatch] - CheckKexes: diffie-hellman-group14-sha1
2024-09-26 09:16:08,351 INFO [NetMessageDispatch] - SSH_MSG_KEXINIT sent
2024-09-26 09:16:08,351 INFO [NetMessageDispatch] - SSH_MSG_KEXINIT received
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group1-sha1,kex-strict-s-v00@openssh.com
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: none,zlib@openssh.com
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server: none,zlib@openssh.com
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server:
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: server:
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: client: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: client: ssh-rsa,ssh-dss
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: client: aes256-ctr,aes256-cbc,aes128-ctr,aes128-cbc,3des-cbc,blowfish-cbc
2024-09-26 09:16:08,352 INFO [NetMessageDispatch] - kex: client: aes256-ctr,aes256-cbc,aes128-ctr,aes128-cbc,3des-cbc,blowfish-cbc
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client: none
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client: none
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client:
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client:
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: server->client aes256-ctr hmac-sha1 none
2024-09-26 09:16:08,353 INFO [NetMessageDispatch] - kex: client->server aes256-ctr hmac-sha1 none
2024-09-26 09:16:08,363 INFO [NetMessageDispatch] - SSH_MSG_KEXDH_INIT sent
2024-09-26 09:16:08,363 INFO [NetMessageDispatch] - expecting SSH_MSG_KEXDH_REPLY
2024-09-26 09:16:08,401 INFO [NetMessageDispatch] - ssh_rsa_verify: signature false
2024-09-26 09:16:08,401 INFO [NetMessageDispatch] - Disconnecting from 192.168.200.53 port 22

Elliot Dierksen · ‎09-26-2024

I think the server sshd_config is still missing this.

PubkeyAcceptedAlgorithms +ssh-rsa
HostkeyAlgorithms +ssh-rsa

I say that because of these messages in the log.

2024-09-26 09:16:08,363 INFO [NetMessageDispatch] - expecting SSH_MSG_KEXDH_REPLY
2024-09-26 09:16:08,401 INFO [NetMessageDispatch] - ssh_rsa_verify: signature false

Do you have access to the SSH log on the Rocky server? That might be a little easier to decipher.