SG300 ssh strange error: "A client is already connected"

[Tilman Schmidt]

Hi,

I've  got a few SG300-52 switches running software version  1.3.0.62 which I configured for ssh management access with public key  authentication via:

ip ssh server

ip ssh pubkey-auth auto-login

username mgmt password ... privilege 15

crypto key pubkey-chain ssh

user-key mgmt rsa

key-string ...

This is working fine if I connect interactively from my management system with:

ssh -i mgmt_id_rsa mgmt@switch

where mgmt_id_rsa is the name of a file containing the private key.

I get a privileged command prompt as intended, without being asked for a password.

However if I try to pass a command on the ssh command line like this:

ssh -i mgmt_id_rsa mgmt@switch show version

the command just hangs until I hit the Enter key a second time, and then emits the strange message:

Received disconnect from 10.11.12.13: 2:

A client is already connected

(Exactly like that, including the line break after the "2:" and the blank before "A client".)

The same happens if I pipe the command I want to send into ssh like this:

echo show version | ssh -i mgmt_id_rsa mgmt@switch

except the error message appears immediately and I don't have to hit Enter a second time.

This is unfortunate as the objective of the whole exercise is to send commands to the switch from a script.

Can anyone shed some light on why this is so? What is that strange message "a client is already connected" trying to tell me? Is that another bug in Cisco's ssh implementation? Ideas for a workaround, anyone?

Thanks,

Tilman

PS: I already asked that question over in the "big business" support community before noticing there's a separate small business section, but got no answer there.

PPS: The real objective of the exercise is to make scripted backups and updates of the switches' configurations, ie. what would be naturally expressed as

scp -i mgmt_id_rsa mgmt@switch:running-config /var/backup/switch.config

and

scp -i mgmt_id_rsa /var/conf/switch.configchange mgmt@switch:running-config

except it doesn't work that way because the SG300's ssh server lacks scp support. Trying to replace that by

ssh -i mgmt_id_rsa mgmt@switch copy running-config scp://server/var/backup/switch.config

and

ssh -i mgmt_id_rsa mgmt@switch copy scp://server/var/conf/switch.configchange running-config

led me straight to the problem above. Just in case someone feels inclined to ask the standard forum question: "Why do you want that anyway?" :-)

[Tom Watts]

Hi Tillman, please look in to the SCP options of this switch.

Chapter 19 of the admin guide

http://www.cisco.com/en/US/docs/switches/lan/csbms/sf30x_sg30x/administration_guide/78-19308-01.pdf

-Tom
Please mark answered for helpful posts

[Tilman Schmidt]

Hi Tom,

thanks for answering.

I read chapters 19 (SSH client) and 20 (SSH server) of the admin guide but didn't find anything that looked as if it might help solving the problem I described.

Could you be more specific?

Thanks,

Tilman

[Tom Watts]

Hi Tilman, I think a ssh session is establishing or has been established and it did not clear or was cancelled in the process of. I think it may be identifying a matching IP but potentially not a matching key or the matching key is in use or has already stored in to the table and is not available (at the time to be used).

Since you're using the specific keys, I feel like it hasn't "released" vs a "normal SSH" where the switch generates the keys each time.

-Tom
Please mark answered for helpful posts

[Tilman Schmidt]

Hi Tom,

I analyzed this a bit further on the client side with the ssh debug option (-vv), and this is what happens from the client point of view:

The key exchange is no different between an interactive session and one passing a command on the command line, so it doesn't look like a key exchange or session establishment issue. I'm omitting that (lengthy) part below.

After that, in the case of a command passed on the command line, ssh -vv reports:

debug1: Entering interactive session.

debug2: callback start

debug2: client_session2_setup: id 0

debug1: Sending environment.

debug1: Sending command: show version

debug2: channel 0: request exec confirm 0

debug2: callback done

debug2: channel 0: open confirm rwindow 256 rmax 128

At this point, the output stops. Anything I type is just echoed until I hit Enter, then the ssh client emits:

Received disconnect from 10.11.11.41: 2:

A client is already connected

and terminates without further debugging output.

In the case of an interactive session, the corresponding output is:

debug1: Entering interactive session.

debug2: callback start

debug2: client_session2_setup: id 0

debug2: channel 0: request pty-req confirm 0

debug1: Sending environment.

debug2: channel 0: request shell confirm 0

debug2: fd 3 setting TCP_NODELAY

debug2: callback done

debug2: channel 0: open confirm rwindow 256 rmax 128

followed by four newlines and the command prompt from the switch. Everything I type is then immediately interpreted by the switch, for example typing a question mark brings up the completion list. Nor does ctrl/C kill the ssh client; it's probably sent to the switch which ignores it.

The two differences I see are the additional "request pty-req" in the interactive case, and the command line case using "request exec" instead of "request shell". To elliminate the first one I added the -t (force pty allocation) option to ssh, which did indeed change the behaviour slightly to:

debug1: Entering interactive session.

debug2: callback start

debug2: client_session2_setup: id 0

debug2: channel 0: request pty-req confirm 0

debug1: Sending environment.

debug1: Sending command: show version

debug2: channel 0: request exec confirm 0

debug2: fd 3 setting TCP_NODELAY

debug2: callback done

debug2: channel 0: open confirm rwindow 256 rmax 128

show version

The "request pty-req" is now there, and the command passed on the ssh command line is echoed, but not executed. What's more, keyboard input has no effect anymore, ie. it's neither echoed, nor does hitting Enter or ctrl/C have any effect. I can terminate the session by typing the ssh client escape sequence "~." though.

In comparison, if I do the same with another Linux machine as SSH server instead of the switch (either with or without the -t option), the result is:

debug1: Entering interactive session.

debug2: callback start

debug2: client_session2_setup: id 0

debug2: channel 0: request pty-req confirm 0

debug1: Sending environment.

debug1: Sending command: show version

debug2: channel 0: request exec confirm 0

debug2: fd 3 setting TCP_NODELAY

debug2: callback done

debug2: channel 0: open confirm rwindow 0 rmax 32768

debug2: channel 0: rcvd adjust 131072

bash: show: command not found

debug1: client_input_channel_req: channel 0 rtype exit-status reply 0

debug2: channel 0: rcvd eof

debug2: channel 0: output open -> drain

debug2: channel 0: obuf empty

debug2: channel 0: close_write

debug2: channel 0: output drain -> closed

debug2: channel 0: rcvd close

debug2: channel 0: close_read

debug2: channel 0: input open -> closed

debug2: channel 0: almost dead

debug2: channel 0: gc: notify user

debug2: channel 0: gc: user detached

debug2: channel 0: send close

debug2: channel 0: is dead

debug2: channel 0: garbage collecting

debug1: channel 0: free: client-session, nchannels 1

Connection to arche closed.

("show: command not found" being of course the expected output from sending "show version" to a Linux host.)

The alternative, piping the command into ssh, does use "request shell" like the interactive case but fails anyway. Here's the debug log of that case:

debug1: Entering interactive session.

debug2: callback start

debug2: client_session2_setup: id 0

debug1: Sending environment.

debug2: channel 0: request shell confirm 0

debug2: callback done

debug2: channel 0: open confirm rwindow 256 rmax 128

debug2: channel 0: read<=0 rfd 4 len 0

debug2: channel 0: read failed

debug2: channel 0: close_read

debug2: channel 0: input open -> drain

debug2: channel 0: ibuf empty

debug2: channel 0: send eof

debug2: channel 0: input drain -> closed

Received disconnect from 10.11.11.41: 2:

A client is already connected

Again, no "rcvd adjust" after the "open confirm".

Nothing in all this changes if I have a second interactive  SSH session open in parallel on the same switch from the same client  machine, so it doesn't look like a session limit issue. The log messages from the switch (%AAA-I-CONNECT and %AAA-I-DISCONNECT) look identical for all the test cases above.

Any further ideas?

Thanks,

Tilman

[Tilman Schmidt]

Anyone?

[Tom Watts]

Hi Tilman, I think it is an interesting observation you've made. I think this particular scenario would need a further looking in to.

Please contact the small business support center to create a service request.

-Tom
Please mark answered for helpful posts

[Tilman Schmidt]

Hi Tom,

alas, the support center found my observation much less interesting than you did. This may be related to the fact that I don't have active support contracts for the switches in question. So this avenue seems barred.

Meanwhile, I have googled some more on the topic and it looks like nobody but me even expects SSH Exec to work. The common solution for sending commands to Cisco gear from a script via SSH is the same one as in good old Telnet times: using the expect tool to fake an interactive dialog so that the device never notices it isn't talking to a human operator. So I followed that path too, and came up with an expect script I called ciscosb-exec which allows me to run a command on a switch like this

ciscosb-exec myswitch "copy running-config scp://server/var/backup/myswitch.config"

If the posting function of this forum works again I'll try to post the script too so it might help others.

Thanks for your suggestions along the way.

Tilman

[Tilman Schmidt · Accepted solution]

Hi all,

I am now using the Unix/Linux tool "expect" as a workaround, with the following script I cobbled together from various suggestions on the 'net:

#!/usr/bin/expect

# Script to run an IOS command on a Cisco Small Business Switch via ssh

# Prerequisites:

# - Cisco Sx300 series switch with software version 1.3 or later

# - public key authentication with auto-logon configured

# Usage:

#   ciscosb-exec

# Args:

#         name or IP address of switch

#      command string to execute

# Result:

#   Switch response will appear on stdout

log_user 0

# configurable values

set userid "confmgmt"

set keyfile "$env(HOME)/.ssh/confmgmt_id_rsa"

set sshcmd "/usr/bin/ssh -c aes192-cbc"

set timeout 20

# end of configurable values

# below matches prompts such as "switch#", "switch>", "switch$"

set prompt "\[>#$\]\ *$"

set device  [lindex $argv 0]

set command [lindex $argv 1]

eval "spawn $sshcmd -l $userid -i $keyfile $device"

match_max [expr 32 * 1024]

while { 1 } {

  expect {

    # command prompt

    -nocase -re "$prompt"     {break}

    # confirmations (unknown fingerprint etc.)

    -nocase -re "\\(yes/no\\)"  {send "yes\r"}

    # username prompt

    -nocase -re "name:|^login:" {send "$userid\r"}

    # password prompt

    -nocase -re "word:"       {puts "Public key authentication failed";exit}

    # errors

    eof                       {puts "Connect failed: $expect_out(buffer)";exit}

    timeout                   {puts "Timeout waiting for command prompt"; exit}

  }

}

send "terminal datadump\r"

expect {

    -nocase -re "$prompt"     {send "$command\r"}

    eof                       {puts "Connection lost: $expect_out(buffer)";exit}

    timeout                   {puts "Timeout waiting for command prompt"; exit}

}

# actual command may take a long time

set timeout 120

expect {

    # skip command echo

    -re "$command\[\r\n\]*"   {exp_continue}

    # answer confirmation request

    -nocase -re " \\(Y/N\\).*\? *$" {

        # send confirmation, skip echo

        send "Y"

        expect -re "Y\[\r\n\]*"

        exp_continue

    }

    # collect response, excluding next prompt

    -nocase -re "(.*)\r\n.*$prompt" {set output $expect_out(1,string)}

    eof                       {puts "Connection lost: $expect_out(buffer)";exit}

    timeout                   {puts "Timeout waiting for command prompt"; exit}

}

puts "$output"

send "exit\r"

expect eof

Saved as an executable file named ciscosb-exec somewhere in $PATH, so far it seems to be working reasonably well for usages such as

ciscosb-exec myswitch "copy scp://myserver/workdir/myswitch.configchange running-config"

ciscosb-exec myswitch "copy running-config startup-config"

ciscosb-exec myswitch "copy startup-config scp://myserver/configdir/myswitch.config"

ciscosb-exec myswitch "show interfaces status detailed"

Hope it helps someone.

Tilman

[Tom Watts]

Tilman, thanks for the post. That is good info.

-Tom
Please mark answered for helpful posts

[Tilman Schmidt]

Hi all,

I've improved my expect script a bit to:

• allow specifying the SSH user and keyfile on the command line
• allow sending configuration mode commands
• correctly handle very long commands (line wrap) and commands producing no output

Extended usage:

ciscosb-exec confuser@myswitch -i ~/.ssh/confuser_id_rsa -c "ip ssh-client username memyself"

ciscosb-exec confuser@myswitch -i ~/.ssh/confuser_id_rsa "copy scp://myserver/workdir/myswitch.configchange running-config"

The "new and improved" script:

#!/usr/bin/expect

# Script to run an IOS command on a Cisco Small Business Switch via ssh

# Prerequisites:

# - Cisco Sx300 series switch with software version 1.3 or later

# - public key authentication with auto-logon configured

# Usage:

#   ciscosb-exec [] [@]

# Args:

#         username on switch

#         name or IP address of switch

#      command string to execute

# Options:

#   -c          execute in configuration mode

#   -i use SSH private key from

#   -d          activate debugging output

# Result:

#   Switch response will appear on stdout

# debug switches

log_user 0

exp_internal 0

# configurable values

set sshcmd "/usr/bin/ssh -c aes192-cbc"

# end of configurable values

# below matches prompts such as "switch#", "switch>", "switch$"

set prompt "\[>#$\]\ *$"

# getopt implementation snarfed from http://www2.tcl.tk/17342

proc getopt {_argv name {_var ""} {default ""}} {

    upvar 1 $_argv argv $_var var

    set pos [lsearch -regexp $argv ^$name]

    if {$pos>=0} {

        set to $pos

        if {$_var ne ""} {

            set var [lindex $argv [incr to]]

        }

        set argv [lreplace $argv $pos $to]

        return 1

    } else {

        if {[llength [info level 0]] == 5} {set var $default}

        return 0

    }

}

# parse command line

set configmode [getopt argv -c]

getopt argv -i idfile

if {[getopt argv -d]} {

  log_user 1

  exp_internal 1

}

if {[llength $argv] != 2} {

  send_user "Usage: ciscosb-exec \[\] \[@\] \"\"\n"

  send_user "Arguments:\n"

  send_user "        target username (default: current user)\n"

  send_user "          target host name or IP address\n"

  send_user "         command string to execute\n"

  send_user "Options:\n"

  send_user "    -c            execute in configuration mode\n"

  send_user "    -i    use SSH private key from \n"

  send_user "    -d            activate debugging output\n"

  exit 1

}

set target [split [lindex $argv 0] @]

if {[llength $target] == 1} {

  set device [lindex $target 0]

  set userid "$env(USER)"

} elseif {[llength $target] == 2} {

  set userid [lindex $target 0]

  set device [lindex $target 1]

} else {

  send_user "bad target: [lindex $argv 0]\n"

  exit 1

}

set command [lindex $argv 1]

if {[info exists idfile]} {

  set sshcmd "$sshcmd -i $idfile"

}

eval "spawn $sshcmd -l $userid $device"

match_max [expr 32 * 1024]

# handle initial noise

set timeout 20

while { 1 } {

  expect {

    # command prompt

    -nocase -re "$prompt"     {break}

    # confirmations (unknown fingerprint etc.)

    -nocase -re "\\(yes/no\\)"  {send "yes\r"}

    # username prompt

    -nocase -re "name:|^login:" {send "$userid\r"}

    # password prompt

    -nocase -re "word:" {send_user "Public key authentication failed\n"; exit}

    # errors

    timeout     {send_user "Timeout waiting for command prompt\n"; exit}

    eof         {send_user "Connect failed: $expect_out(buffer)\n"; exit}

  }

}

# disable terminal formatting junk

send "terminal datadump\r"

expect {

    -nocase -re "$prompt"     {}

    timeout     {send_user "Timeout waiting for command prompt\n"; exit}

    eof         {send_user "Connection lost: $expect_out(buffer)\n"; exit}

}

send "terminal width 0\r"

expect {

    -nocase -re "$prompt"     {}

    timeout     {send_user "Timeout waiting for command prompt\n"; exit}

    eof         {send_user "Connection lost: $expect_out(buffer)\n"; exit}

}

# switch to desired mode

if {$configmode} {

  send "configure terminal\r"

  expect {

    -nocase -re "$prompt"     {}

    timeout     {send_user "Timeout waiting for command prompt\n"; exit}

    eof         {send_user "Connection lost: $expect_out(buffer)\n"; exit}

  }

}

# actual command may take a long time

set timeout 180

send "$command\r"

expect {

    # skip command echo

    -re "$command\[\r\n\]*"   {exp_continue}

    # answer confirmation request

    -nocase -re " \\(Y/N\\).*\? *$" {

        # send confirmation, skip echo

        send "Y"

        expect -re "Y\[\r\n\]*"

        exp_continue

    }

    # collect response, excluding next prompt

    -re "\r\n"                {send_user "$expect_out(buffer)"; exp_continue}

    -nocase -re "$prompt"     {send "exit\r"}

    timeout     {send_user "Timeout waiting for command prompt\n"; exit}

    eof         {send_user "Connection lost: $expect_out(buffer)\n"; exit}

}

set timeout 20

expect {

    # second exit needed for logging out from configuration mode

    -nocase -re "$prompt"     {send "exit\r"}

    timeout     {send_user "Timeout waiting for hangup\n"; exit}

    eof         {exit}

}

expect {

    -nocase -re "$prompt"     {puts "Failed to log out, disconnecting"; exit}

    timeout                   {puts "Timeout waiting for hangup"; exit}

    eof                       {exit}

}

HTH

Tilman

[portmannth]

Hello all,

I was searching for the exact same things as Tilman (and for the same reasons, config backup from a management station). It is a pitty that neither scp nor ssh w/ specified command to the switch does not work. I will try your script, though.

Thanks and best regards

Tom

[advuni-md1]

Hi Tilman,

I don't know if you maintain this script anymore, but just FYI this is still not fixed in FW version 1.4.1.3

Small improvement:
In the function that sends the actual command, you have the line

 -re "\r\n" {send_user "$expect_out(buffer)"; exp_continue}

which should be changed to

 -re "\r\n" {send_user -- "$expect_out(buffer)"; exp_continue}

otherwise any commands that output strings starting with a dash ("-") will break the script:

[root@centos7 config_backups]# ./ciscosb-exec readonly@10.x.x.x -i cisco_authkey "show version"
Unit SW version Boot version HW version
bad flag "------------------- ------------------- ------------------- -------------------
": must be -i, -h, -s, -null, -0, -raw, -break, or --
 while executing
"send_user "$expect_out(buffer)""
 invoked from within
"expect {
 # skip command echo
 -re "$command\[\r\n\]*" {exp_continue}
# answer confirmation request
 -nocase -re " \\(Y/N\\).*\? *$" {..."
 (file "./ciscosb-exec" line 152)

after the fix, it works as EXPECTed (pun intended ;-)

[root@centos7 config_backups]# ./ciscosb-exec readonly@10.x.x.x -i cisco_authkey "show version"
    Unit                SW version          Boot version          HW version
------------------- ------------------- ------------------- -------------------
      1                  1.4.1.3            1.4.0.02                 V01
      2                  1.4.1.3            1.4.0.02                 V03