SSH Stopped Working: Cisco 3650 Switch

[BashedRoot]

Been using this switch for years without issues, then suddenly for no reason SSH stopped working yesterday. I don't have strict ACL as I need to connect remotely from anywhere anytime (VPN or not).

Would appreciate help. I use SecureCRT and Termius. Tried via from multiple networks and devices. I'm the only one with access and no changes were made recently, so weird it just started.

cat3k-2a-io6-company.com#show ip ssh
SSH Enabled - version 1.99
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
MAC Algorithms:hmac-sha1,hmac-sha1-96
KEX Algorithms:diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Authentication timeout: 90 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 2048 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): cat3k-2a-io6-company.com
ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                                                                 

cat3k-2a-io6-nj.company.com#show ssh
%No SSHv2 server connections running.

[Leo Laohoo]

Re-generate the crypto key.

[BashedRoot]

I tried doing that, 4096 bits and at 3 "..." dots it just froze. I detached out of "screen" tool since I had to connect via serial console. I tried to reconnect again and all the characters are gibberish, can't type anything, just messed up. I have no idea what to do now. Switch itself is still up though. Just can't see normal prompt in my screen session.

[Leo Laohoo]

Re-generate the key at 2048.  4096 is going to take a very long time to get in.

[BashedRoot]

Problem is I can't get back in screen in normal mode. It's messed up. How can I fix it? Please see this: https://community.cisco.com/t5/server-networking/ssh-stopped-working-cisco-3650-switch/m-p/4768471/highlight/true#M13536

[Leo Laohoo]

Reboot the switch.

[BashedRoot]

Did that, didn't fix it. Still gibberish stuff when I enter screen mode.

[MHM Cisco World]

do change domain ?

[BashedRoot]

This is what I see when I re-enter screen mode via serial management console.

[root@kvm ~]# dmesg | grep -i tty
[    0.000000] console [tty0] enabled
[    1.994239] 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.014941] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[1823572.053980] cdc_acm 2-1.2:1.0: ttyACM0: USB ACM device

[root@kvm ~]# screen /dev/ttyACM0
sAresa

[balaji.bandi]

reboot the device again, connect the console (keep changing the baud rate until you see proper output)

rekey the RSA and change IP ssh v2 (I see your output show 1.99)

[BashedRoot]

How am I supposed to change the baud rate? Please explain. How do you "rekey the RSA" and change IP SSH to v2?

[balaji.bandi]

if you using Linux to connect  to console device follow below guide to change baud rate :

https://docs.kernel.org/admin-guide/serial-console.html#:~:text=The%20maximum%20baudrate%20is%20115200.&text=defines%20that%20opening%20%2Fdev%2Fconsole,type%20(serial%2C%20video).

config t 

IP ssh v 2 (change to version 2)

######### Generate SSH keys :
crypto key generate rsa

[MHM Cisco World]

https://www.youtube.com/watch?v=jIRRsIgfHU8

first adjust the baud and access SW via console, 
re-generate SSH key 

[BashedRoot]

Thanks for the help folks. Funny thing is, for no reason at all on my part (I was still stuck at the gibberish screen mode after reboots)...the switch gave me access again in normal SSH mode again. So, I started doing the erase RSA / regenerate method from Cisco's doc: https://www.cisco.com/c/en/us/support/docs/security-vpn/secure-shell-ssh/4145-ssh.html#anc38

I started at the "%No SSHv2 server connections running." portion.

Did steps 1-3 including re-generating rsa key in 2048 bits. [ok]

I also ran this after generating rsa key:

# IP ssh v 2

Now it says for steps 4 and 5

4. Configure the SSH server.

5. To enable and configure a Cisco router/switch for the SSH server, you must configure SSH parameters. If you do not configure SSH parameters, the default values are used.

ip ssh {[timeout seconds] | [authentication-retries integer]}

Can someone direct me from here so I don't mess this up and get locked out?

Thanks again.

[MHM Cisco World]

these only step to make SW/R accept SSH connection 
1. Configure the hostname command.

2. Configure the DNS domain.

3. Generate the public RSA key to be used.

4. Enable SSH transport support for the vtys (optional step). <<- please see Edit below 

not more steps, the steps you mention above is using specific RSA for each user, this not need in your case. 
""Setup an IOS Router as an SSH Server that Performs RSA-based User Authentication""

Edit:- as @balaji.bandi  mention always keep console and telnet in case that SSH is not work.