SSH disconnect problems on IOS

[guerinp]

Hi,

I have a hub and spoke network topology, and in order to access the spoke routers I use SSH version 2 from the hub router.

Access is as follows:

1) I SSH onto a hub router from an UNIX server

2) Once logged into the hub router, I then SSH onto the spoke routers

this has been working fine for months, and now suddenly I cannot SSH from the hub router to any spoke. Here is an example of what happens when I try:

Hub router#ssh spoke

Connection to 10.136.19.8 closed by remote host.

Connection to 10.136.19.8 closed.

UNIXServer[/home/user]$

I have reset the SSH keys on both ends and rebooted both, but still this occurs. I suspect the problem is with the hub router as all spoke access causes the connection to be closed down by the hub.

Any ideas appreciated!

Thanks

Phil

[spremkumar]

Hi

Any config changes been done in recent past ??

Also is it possibl to post out the configs related to SSH taken out from both hub and one of the spoke site which you are trying out here ..

regds

[guerinp]

Hi,

Relevant config:

HUB router

hostname hub

ip domain name ibm.com

ip ssh time-out 60

ip ssh source-interface FastEthernet0/0

ip ssh version 2

Spoke Router

hostname spoke

ip domain name ibm.com

ip ssh time-out 60

ip ssh version 2

spoke#sh ssh

%No SSHv2 server connections running.

%No SSHv1 server connections running.

spoke#sh ip ssh

SSH Enabled - version 2.0

Authentication timeout: 60 secs; Authentication retries: 3

Hub#sh ssh

Connection Version Mode Encryption Hmac State Username

0 2.0 IN aes128-cbc hmac-md5 Session started campus

0 2.0 OUT aes128-cbc hmac-md5 Session started campus

%No SSHv1 server connections running.

Hub#sh ip ssh

SSH Enabled - version 2.0

Authentication timeout: 60 secs; Authentication retries: 3

[Richard Burts]

Is the 10.136.19.8 address in the messages the address of the spoke?

If SSH to the spoke does not work, does telnet to the spoke work?

Is it possible that an access-class has been configured on the spoke vty ports - and if so does it permit the address that is the source of your SSH packets? Is it possible that transport input has been configured on the spoke vty ports? Can you post the output of the vty ports on the spoke?

HTH

Rick

[guerinp]

Hi,

The 10.136.19.8 is the address of the hub router - it seems that the hub is disconnecting the session upon attempt to SSH to a spoke. I really dont understand why.

In terms of the VTY config of the spokes here it is:

line vty 0 4

password xxx

login local

transport input telnet ssh

Telnet works fine from hub to spoke - its just SSH that fails. I've tried all sorts of different flags (version, encryption etc.) and all results in a disconnect.

[Richard Burts]

Phil

It looks like the SSH to HUB works, but gets terminated if you initiate a SSH to SPOKE. That seems very strange to me. Can you verify that SSH to HUB is ok, that you can do various functions with no problem on HUB?

If you telnet to HUB and then initiate telnet to SPOKE what is the result?

Can you SSH from your Unix station directly to SPOKE or is that a problem also?

HTH

Rick

[guerinp]

Hi,

SSH to the hub is OK - works fine and even managed to use SCP to transfer latest IOS onto it from the UNIX server so pretty confident all OK there.

Currently the only method of access is by SSH to the hub, and then telnet onto the spokes thereafter. This works fine.

I cant SSH to spokes directly due to local site routing only allowing access back to the hub. As such all traffic from the spokes has to go to the hub first.

I've upgraded IOS as a last resort but still no joy -Cisco IOS Software, 3700 Software (C3725-ADVIPSERVICESK9-M), Version 12.3(14)T7, RELEASE SOFTWARE (fc2)

I'm stumped, as most others are who I've asked!!

[Richard Burts]

I interpret what you have told us to be that you SSH to the hub, and when you attempt to SSH to the spoke that the hub router is terminating your SSH session. Is that correct?

In the original post you stated that sSH to the spoke used to work and then started not working. Someone asked if there were changes that you were aware of. I do not think the question was answered. Knowing that answer would be helpful.

I wonder if running deb ip ssh (or debug ip ssh client) would have helpful information. You could turn it on for the hub, telnet to the spoke to turn it on, make sure that logging buffered in enabled so the messages can be examined after the event, and attempt SSH from hub to spoke.

HTH

Rick

[guerinp]

Hi,

There were no changes that I am aware of.

Debugged SSH to see if shows anything useful - this is debug on the spoke when attempting to connect from hub:

Apr 27 13:08:15.307 GMT: SSH1: starting SSH control process

Apr 27 13:08:15.307 GMT: SSH1: sent protocol version id SSH-2.0-Cisco-1.25

Apr 27 13:08:15.531 GMT: SSH1: protocol version id is - SSH-2.0-Cisco-1.25

Apr 27 13:08:15.535 GMT: SSH2 1: send: len 280 (includes padlen 4)

Apr 27 13:08:15.535 GMT: SSH2 1: SSH2_MSG_KEXINIT sent

Apr 27 13:08:15.535 GMT: SSH2 1: ssh_receive: 280 bytes received

Apr 27 13:08:15.535 GMT: SSH2 1: input: packet len 280

Apr 27 13:08:15.535 GMT: SSH2 1: partial packet 8, need 272, maclen 0

Apr 27 13:08:15.535 GMT: SSH2 1: input: padlen 4

Apr 27 13:08:15.539 GMT: SSH2 1: received packet type 20

Apr 27 13:08:15.539 GMT: SSH2 1: SSH2_MSG_KEXINIT received

Apr 27 13:08:15.543 GMT: SSH2: kex: client->server aes128-cbc hmac-sha1 none

Apr 27 13:08:15.543 GMT: SSH2: kex: server->client aes128-cbc hmac-sha1 none

Apr 27 13:08:15.727 GMT: SSH2 1: expecting SSH2_MSG_KEXDH_INIT

Apr 27 13:08:15.803 GMT: SSH2 1: ssh_receive: 144 bytes received

Apr 27 13:08:15.803 GMT: SSH2 1: input: packet len 144

Apr 27 13:08:15.803 GMT: SSH2 1: partial packet 8, need 136, maclen 0

Apr 27 13:08:15.803 GMT: SSH2 1: input: padlen 5

Apr 27 13:08:15.803 GMT: SSH2 1: received packet type 30

Apr 27 13:08:15.803 GMT: SSH2 1: SSH2_MSG_KEXDH_INIT received

Apr 27 13:08:16.051 GMT: SSH2 1: RSA_sign: private key not found

Apr 27 13:08:16.051 GMT: SSH2 1: signature creation failed, status -1

Apr 27 13:08:16.155 GMT: SSH1: Session disconnected - error 0x00

I think the below message is the key point, but not sure how to address this:

Apr 27 13:08:16.051 GMT: SSH2 1: RSA_sign: private key not found

Note that the hub still gets same disconnect message as it did on first post.

thanks

Phil

[Richard Burts]

Phil

Thanks for posting the debug output. I agree that the message:

RSA_sign: private key not found

is a significant indicator.

I wonder what you would get in debug on the hub router.

HTH

Rick

[guerinp]

Problem solved.

I manually created SSH keys on the routers and all is now working as it was before:

Hub#ip ssh rsa keypair-name hub.com

Then regenerated SSH keys to override.

Spoke#ip ssh rsa keypair-name spoke.com

Then regenerated SSH keys to override.

This had the effect of recreating new keys based on current hostnames. I suspect that hostname change many weeks OK has caused this problem. Not sure why it only manifested itself after several weeks away however,

Rgds

Phil

[Richard Burts]

Phil

Thanks for posting that the problem is solved and indicating the solution. It helps make the forum more useful when we can read about a problem and find what the solution turned out to be.

I am glad that you solved the problem and hope that the activity on the forum helped point you in the right direction.

HTH

Rick