4510+R ssh error after upgrade xe 3.11.07 to xe 3.11.09

[Hubert Kupper]

Hi,

after a upgrade from xe 3.11.07 to 3.11.09 we cannot open a ssh session to our 4510+R Switch. Before the upgrade everything works fine. The error message in the log  is:

%SSH-3-BAD_PACK_LEN: Bad packet lenghth

We zeroised the  rsa key and generate it new but the error still occours.

Any idea?

Regards, Hubert

[marce1000]

  >...I also already tried with latest PuTTY 0.79. Also was running latest Debian 12, and MacOS 14.1 (which has newest OpenSSL libraries).
                   Use the following debugging commands :
                       debug ip ssh client
                          debug ip ssh detail
                         debug ip ssh packet

  And provide the resulting packet capture , 

 M.

[John.panzer]

Here is the result of debugging.

*Oct 27 14:51:04.073: SSH0: starting SSH control process
*Oct 27 14:51:04.073: SSH0: sent protocol version id SSH-2.0-Cisco-1.25
*Oct 27 14:51:04.078: SSH0: protocol version id is - SSH-2.0-OpenSSH_9.4
*Oct 27 14:51:04.078: SSH2 0: kexinit sent: kex algo = ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha1,di
*Oct 27 14:51:04.078: SSH2 0: Server certificate trustpoint not found. Skipping hostkey algo = x509v3-ssh-rsa
*Oct 27 14:51:04.078: SSH2 0: kexinit sent: hostkey algo = ssh-rsa
*Oct 27 14:51:04.078: SSH2 0: kexinit sent: encryption algo = aes128-ctr,aes192-ctr,aes256-ctr
*Oct 27 14:51:04.078: SSH2 0: kexinit sent: mac algo = hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
*Oct 27 14:51:04.079: SSH2 0: send:packet of length 376 (length also includes padlen of 11)
*Oct 27 14:51:04.079: SSH2 0: SSH2_MSG_KEXINIT sent
*Oct 27 14:51:04.079: SSH2 0: ssh_receive: 536 bytes received
*Oct 27 14:51:04.079: SSH2 0: input: total packet length of 1504 bytes
*Oct 27 14:51:04.079: SSH2 0: partial packet length(block size)8 bytes,needed 1496 bytes,
maclen 0
*Oct 27 14:51:04.079: SSH2 0: ssh_receive: 536 bytes received
*Oct 27 14:51:04.079: SSH2 0: partial packet length(block size)8 bytes,needed 1496 bytes,
maclen 0
*Oct 27 14:51:04.079: SSH2 0: ssh_receive: 432 bytes received
*Oct 27 14:51:04.079: SSH2 0: partial packet length(block size)8 bytes,needed 1496 bytes,
maclen 0
*Oct 27 14:51:04.079: SSH2 0: input: padlength 10 bytes
*Oct 27 14:51:04.079: SSH2 0: SSH2_MSG_KEXINIT received
*Oct 27 14:51:04.079: SSH2 0: kex: client->server enc:aes128-ctr mac:hmac-sha2-256
*Oct 27 14:51:04.079: SSH2 0: kex: server->client enc:aes128-ctr mac:hmac-sha2-256
*Oct 27 14:51:04.079: SSH2 0: Using hostkey algo = ssh-rsa
*Oct 27 14:51:04.079: SSH2 0: Using kex_algo = ecdh-sha2-nistp256
*Oct 27 14:51:04.084: SSH2 0: kexecdh_server: expecting SSH2_MSG_KEX_ECDH_INIT
*Oct 27 14:51:04.084: SSH2 0: ssh_receive: 80 bytes received
*Oct 27 14:51:04.084: SSH2 0: input: total packet length of 80 bytes
*Oct 27 14:51:04.084: SSH2 0: partial packet length(block size)8 bytes,needed 72 bytes,
maclen 0
*Oct 27 14:51:04.084: SSH2 0: input: padlength 5 bytes
*Oct 27 14:51:04.084: ssh2_calculate_modulus_length: modulus len 32
*Oct 27 14:51:04.084: SSH2 0: kexecdh_server: Calculated shared secret len 32
*Oct 27 14:51:04.650: SSH2 0: send:packet of length 1152 (length also includes padlen of 7)
*Oct 27 14:51:04.650: SSH0: TCP send failed enqueueing
*Oct 27 14:51:04.663: SSH2: kex_derive_keys complete
*Oct 27 14:51:04.663: SSH2 0: send:packet of length 16 (length also includes padlen of 10)
*Oct 27 14:51:04.663: SSH2 0: newkeys: mode 1
*Oct 27 14:51:04.663: SSH2 0: SSH2_MSG_NEWKEYS sent
*Oct 27 14:51:04.663: SSH2 0: waiting for SSH2_MSG_NEWKEYS
*Oct 27 14:51:04.670: SSH2 0: ssh_receive: 16 bytes received
*Oct 27 14:51:04.671: SSH2 0: input: total packet length of 16 bytes
*Oct 27 14:51:04.671: SSH2 0: partial packet length(block size)8 bytes,needed 8 bytes,
maclen 0
*Oct 27 14:51:04.671: SSH2 0: input: padlength 10 bytes
*Oct 27 14:51:04.671: SSH2 0: newkeys: mode 0
*Oct 27 14:51:04.671: SSH2 0: SSH2_MSG_NEWKEYS received
*Oct 27 14:51:04.870: SSH2 0: ssh_receive: 64 bytes received
*Oct 27 14:51:04.870: SSH2 0: input: total packet length of 129511862 bytes
*Oct 27 14:51:04.870: %SSH-3-BAD_PACK_LEN: Bad packet length 129511858
*Oct 27 14:51:04.870: SSH2 0: send:packet of length 64 (length also includes padlen of 18)
*Oct 27 14:51:04.870: SSH2 0: computed MAC for sequence no.#3 type 1
*Oct 27 14:51:04.870: SSH2 0: send:packet of length 80 (length also includes padlen of 16)
*Oct 27 14:51:04.870: SSH2 0: computed MAC for sequence no.#4 type 1
*Oct 27 14:51:04.967: SSH0: Session disconnected - error 0x00

[marce1000]

                       - Ref : debugging output provided
                               >...
                               >...: TCP send failed enqueueing
  Look like a bug to me indeed , provide the output of show  tcp statistics 

 M.

[John.panzer]

Rcvd: 64 Total, 0 no port
0 checksum error, 0 bad offset, 0 too short
28 packets (6740 bytes) in sequence
8 Control packets
0 packets (0 bytes) in CEF
0 dup packets (0 bytes)
0 partially dup packets (0 bytes)
0 out-of-order packets (0 bytes)
0 Retransmitted out-of-order packets
0 packets (0 bytes) with data after window
0 packets after close
0 window probe packets, 0 window update packets
0 dup ack packets, 0 ack packets with unsend data
28 ack packets (6260 bytes)
0 ack packets (0 bytes) in CEF
Sent: 64 Total, 0 urgent packets
8 control packets (including 0 retransmitted)
24 data packets (6252 bytes)
0 data packets (0 bytes) in CEF
0 data packets (0 bytes) retransmitted
0 data packets (0 bytes) fastretransmitted
28 ack only packets (8 delayed)
0 ack only packets in CEF
0 window probe packets, 4 window update packets
0 Connections initiated, 4 connections accepted, 4 connections established
4 Connections closed (including 0 dropped, 0 embryonic dropped)
0 Total rxmt timeout, 0 connections dropped in rxmt timeout
0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive, 0 Connections closed before socket close.

[marce1000]

- I see nothing relevant in there , so it becomes a case for TAC indeed,

M.

[Hubert Kupper]

I will try it next week, because we cannot switch the XE software many times.

[skuppand]

Hey, can you share the output of the following?

sho run all | i ip ssh

[RVTim]

I should have provided the version number in my post but yes, I fail with Putty 0.79, I fail with RoyalTS SSH, I fail with a fully updated Fedora 38 PC with current OpenSSH.   They all give:

Oct 27 09:14:30 my-sw 275: Oct 27 09:14:29: %SSH-3-BAD_PACK_LEN: Bad packet length -1984819621

In fact I just upgraded it again to 03.11.09 from 03.11.08 with no changes and it broke right away.   The whole Putty client thing may be a correct fix for other issues (and in fact it HAS fixed issues for me in the past with cisco)....but it's a total red herring on this issue.   This issue is a bug.

[tbarton]

I have opened a case with TAC and referenced this community post with them to show that I am not the only one with this issue. For reference in the TAC case, I am running on a 4500X-16 switch, but this appears to be across multiple platforms also.

[mistertom]

Any luck with the TAC case? 03.11.09 is unusable with this SSH bug.

[RVTim]

I was thinking about how painful it would be for someone who didn't know about the bug, to upgrade a switch remotely and then not be able to get in via SSH, and how even though we know about it due to this thread, someone may inadvertently download the update and install it.  So I left a review on the code on the download page.   I'd encourage at least a couple more of you to do the same.  Hopefully we can keep other people from having a big issue.  As it is, my switch that I first upgraded was in a remote datacenter, 5+ hours away, and if it weren't for the fact that I have a console server attached, I'd be stuck.

[skuppand]

Change the kex configuration to the following

ip ssh server algorithm kex diffie-hellman-group-exchange-sha1 diffie-hellman-group14-sha1

[John.panzer]

I was able to confirm this command does allow SSH to work again.

It does seem as though that is a large change that is made between a minor version change. If that is expected behavior, that needs to be documented plainly somewhere so that people don’t get stuck.

[CoreyH]

THANK YOU! That worked for me!

[RVTim]

I agree.  I can also confirm that the command does fix it.  But I do still think there's a bug and that implementing what they did without putting it in the release notes was definitely not a good idea.   Here's what I tested.

I did "show ip ssh" before I upgraded and this was my original config:

KEX Algorithms:diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

If you notice, this is exactly what "skuppand" asked us to enter to fix it.  I actually again ran that command and did a write mem before rebooting.  Sadly, that doesn't mean it survives the reboot.

Once the switch reboots, and you're locked out of SSH, if you log in via console and do "show ip ssh", you get this:

KEX Algorithms:ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

And when you go to enter the command you see the choices:

MySwitch(config)#ip ssh server algor kex ?
diffie-hellman-group-exchange-sha1 DH_GRPX_SHA1 diffie-hellman key exchange algorithm
diffie-hellman-group14-sha1 DH_GRP14_SHA1 diffie-hellman key exchange algorithm
ecdh-sha2-nistp256 ECDH_SHA2_P256 ecdh key exchange algorithm
ecdh-sha2-nistp384 ECDH_SHA2_P384 ecdh key exchange algorithm
ecdh-sha2-nistp521 ECDH_SHA2_P521 ecdh key exchange algorithm

I tried just entering those 2 and sure enough, SSH started working.

So then I tried rearranging them, and entered them with group 14 first, then group-exchange, and then the others.   Nope, that doesn't work.  Apparently when it has any of those ecdh ones enabled you're gonna fail. 

So then, thinking that maybe I could fix this on the Putty side, I looked at my settings in Putty.   Under SSH Kex settings, ecdh was actually 2nd in the priority list.  That SHOULD have been good.  I moved it to #1 position and tried it again.  Still no go.

So, we now know how to fix it, but we have zero information on what the heck is actually causing it to break.

I'm making an assumption that if those other algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521     were actually working, that Putty should have worked with them.  So are they just broken?

I went back to a different 4500 to see what options we get:

My-Other-4500X(config)#ip ssh server algo kex ?
diffie-hellman-group-exchange-sha1 DH_GRPX_SHA1 diffie-hellman key exchange algorithm
diffie-hellman-group14-sha1 DH_GRP14_SHA1 diffie-hellman key exchange algorithm

So apparently we don't have the option, in 03.11.08E to use those new ones.   And from the fix, we can infer that whatever they're doing with them, those new ones are breaking it.  If they planned for this, some release notes would have been pretty helpful.