NSO 5.2 and UCSM 4.0.4c - ssh sync-from issue

[Sean Chandler]

Using UCSM package cisco-ucs-cli-3.3 I am trying to sync the configuration from a newly upgraded fabric.  Below is the command line failure and after that is the log in ncs-java-vm.log.  What I believe is happening is that the SSH client in NSO does not have sufficiently new ciphers.  Has anyone encountered this and or tested NSO against UCSM 4.0.4?

nsouser@ncs> request devices device pirl sync-from
result false
info Failed to connect to device pirl: connection refused: Key exchange was not finished, connection is closed. in new state
[ok][2019-08-26 20:15:16]

<ERROR> 26-Aug-2019::19:59:35.675 NedComCliBase Ned-Worker-Thread-1: - pirl
com.tailf.packages.ned.nedcom.connector.CliException: Key exchange was not finished, connection is closed. in new state
        at com.tailf.packages.ned.nedcom.connector.CliConnectInteractor.connect(CliConnectInteractor.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.tailf.packages.ned.nedcom.connector.CliInteractor.action(CliInteractor.java:553)
        at com.tailf.packages.ned.nedcom.connector.CliInteractor.access$600(CliInteractor.java:32)
        at com.tailf.packages.ned.nedcom.connector.CliInteractor$State.runState(CliInteractor.java:463)
        at com.tailf.packages.ned.nedcom.connector.CliInteractor$State.access$100(CliInteractor.java:379)
        at com.tailf.packages.ned.nedcom.connector.CliInteractor.run(CliInteractor.java:368)
        at com.tailf.packages.ned.nedcom.connector.CliConnectorNedUtils.doConnectorConnectDevice(CliConnectorNedUtils.java:95)
        at com.tailf.packages.ned.nedcom.connector.CliConnectorNedUtils.connectorConnectDevice(CliConnectorNedUtils.java:197)
        at com.tailf.packages.ned.nedcom.NedComCliBase.connectorConnectDevice(NedComCliBase.java:734)
        at com.tailf.packages.ned.nedcom.NedComCliBase.newConnection(NedComCliBase.java:1750)
        at com.tailf.ned.NedWorker.dorun(NedWorker.java:1492)
        at com.tailf.ned.NedWorker.run(NedWorker.java:312)
Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
        at ch.ethz.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:76)
        at ch.ethz.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:169)
        at ch.ethz.ssh2.Connection.connect(Connection.java:801)
        at com.tailf.ned.SSHConnection.connect(SSHConnection.java:113)
        at com.tailf.packages.ned.nedcom.connector.CliConnectInteractor.setupSSH(CliConnectInteractor.java:138)
        at com.tailf.packages.ned.nedcom.connector.CliConnectInteractor.connect(CliConnectInteractor.java:102)
        ... 15 more
Caused by: java.io.IOException: Cannot negotiate, proposals do not match.
        at ch.ethz.ssh2.transport.ClientKexManager.handleMessage(ClientKexManager.java:124)
        at ch.ethz.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:572)
        at ch.ethz.ssh2.transport.TransportManager$1.run(TransportManager.java:261)
        at java.lang.Thread.run(Thread.java:748)

[Sean Chandler]

Issue appears to be present in earlier versions as well (below version 4.6.1.3)

nsouser@ncs> request devices fetch-ssh-host-keys
fetch-result {
    device pirl
    result failed
    info internal error
}

[Sean Chandler]

Found a workaround for the time being by simply proxying to the host NSO is running on.  Not a solution but works for now.

set devices authgroups group jumphost umap nsouser remote-name localadmin

set devices authgroups group jumphost umap nsouser remote-password *******

set devices authgroups group jumphost umap nsouser remote-secondary-password *******

set devices device ucsm address 192.168.242.7

set devices device ucsm authgroup jumphost

set devices device ucsm device-type cli ned-id cisco-ucs

set devices device ucsm state admin-state unlocked

set devices device ucsm ned-settings cisco-ucs-meta:cisco-ucs proxy remote-connection ssh

set devices device pirl address 192.168.242.7

set devices device ucsm ned-settings cisco-ucs-meta:cisco-ucs proxy remote-address 10.10.1.3

set devices device ucsm ned-settings cisco-ucs-meta:cisco-ucs proxy remote-port 22

set devices device ucsm ned-settings cisco-ucs-meta:cisco-ucs proxy remote-password ********

set devices device ucsm ned-settings cisco-ucs-meta:cisco-ucs proxy remote-name admin

commit

[rogaglia]

Hi Sean,

I have a similar situation with IOS-XR 6.5.2 that may explain what is going on.

What happened to me was a bug in IOS-XR (CSCvo17475) that  rejected SSH interactive sessions (like NSO opens) while allowing non-interactive ssh sessions (like what you open manually).

As a user, SSH was working from the shell but not from NSO nor plain python scripts. The work-around of using a proxy (as you did) solved the problem temporarely. The final fix was a patch for IOS-XR.

Just a reference of a similar situation that had a logical explanation. You may want to check the support database of UCS if there is a similar problem already reported or with TAC.

Roque