Cisco 9200L SSH Issues

[bristi]

We are experiencing an unusual issue concerning SSH connections to Cisco 9200L switches. We recently migrated from Cisco 2960 switches to Cisco 9200L models. When connecting from the same network using a Linux server, SSH to the switches works successfully. However, SSH connections from other networks are not successful.

The default gateway is correctly set, and SSH is enabled on the switches. Notably, we still have one Cisco 2960 switch that was not replaced, and we can SSH into it from other networks without any problems. The IP addresses of the switches are on the same subnet, and we can ping them from other networks, but SSH connections fail.

I have consulted with TAC support, and they have reviewed the configuration, indicating that it appears correct. I also provided them with packet captures but have not yet received further feedback. Has anyone else encountered a similar issue and could provide insights or suggestions?

Additionally, the SSH connection issue is intermittent from other networks, but consistently works when connecting from a server on the same subnet. Any recommendations would be appreciated.

Cisco 9200L version is - C9200L-24P-4X 17.14.01 CAT9K_LITE_IOSXE

[bristi]

Hello Paul

Yes, I attempted zeroizing SSH as well, but it did not make any difference

[Richard Pidcock]

Are you using the dedicated mgmt interface on the 9200L and do you have any access-lists applied to your vty lines?  If so have you specified use of the management vrf within your vty section.

i.e. 

line vty 0 4
 access-class "ACL-NAME" in vrfname Mgmt-vrf

[arogers97]

Was this ever fixed? 

[mcnuttj]

I am seeing this same issue on a Catalyst 9200L.

Additional information:

We could SSH to this switch when it was running IOS-XE 17.9.  When we upgraded to IOS-XE 17.12, this exact problem started happening.

I ran a packet capture on the SSH client and the strangest thing happens:  The client sends the normal TCP SYN to TCP port 22 on the switch... and the switch sends back a bare ACK.  Not SYN+ACK.  Just ACK.

Again, this is a device that was working before.  IP address, mask, and gateway are correct.  The "transport input ssh" lines are present in the "line vty" stanzas.  We have access lists applied using "access-class" statements, but these match the other 9200s that are working (that are still running 17.9.x).

We can log into the switch using HTTPS from the remote client.  It is NOT a routing problem.

I have read through the rest of this thread and tried similar things to get this one to work.  I have not zeroized the SSH keys, since SSH works just fine so long as the client is on the same subnet as the switch.

The switch is managed through an SVI on VLAN 23, if that makes any difference.  (It physically lives in a place where using the Gi0/0 interface isn't practical so we manage it in-band.)

I tried PuTTY as well as OpenSSH (command line) to make sure it wasn't just this client.  Same results.  SYN, then ACK.  No SYN+ACK from the switch.

[arogers97]

Yup same exact thing is happening to me. 

[Richard Pidcock]

Not sure if this was ever resolved.  Are you able to share config from your 9200L for review?

[mcnuttj]

It was not resolved.  We opened a case on it: 698402573

If the case comes to any resolution other than "don't upgrade to 17.12" I will post it here.

[Richard Pidcock]

Here's what I've found and hope this will help.  I took a 9200L off of my shelf that was running 17.6.5.  I connected the device into my network on my management subnet and successfully SSH'd to the device from a different subnet.  I then upgraded the 9200 to 17.12.4.  After completing the upgrade, upon my first attempt to SSH to the device I received this error (See image, I'm using Secure CRT).  

I answered yes to the prompt, and then I was able to successfully SSH into the 9200L.  Also see the snip of settings from my Secure CRT session.  

Lastly, I tried to SSH to the 9200 from another device using Putty rather than Secure CRT, this connection was also successful and I didn't have to make any modifications regarding key exchange within Putty.

[arogers97]

From what I've read 17.12.4 is the maintenance deployment version. So, I'm glad to see it worked. My 9200L came preloaded with 17.12.3 and at the time was the recommended version so I kept it there. The switch is in production, and I needed to troubleshoot with Cisco TAC before I could get the green light to upgrade. Thank you!

[mcnuttj]

What we are seeing is not related to the key exchange, as shown in the packet captures we took.  In our case, the 9200 is not completing the TCP handshake, which happens before the key exchange.

What is odd is that it doesn't exhibit this behavior if the client is on the same VLAN (and subnet) as the 9200.  In that case, SSH works fine.  But from any other network, it fails.

[mcnuttj]

We upgraded one of our Cat 9200L switches from 17.9 to 17.12 and it has this SSH issue.

Downgraded the switch from 17.12 to 17.9 and the problem went away.

Upgraded the switch from 17.9 to 17.12 again and the problem returned.

I'd say it's definitely an issue with 17.12.

[mcnuttj]

It is now a known bug:  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwk36412

The workaround is:

ip tcp window-size 4128
no ip ssh bulk-mode

OR upgrade to 17.15.2.

We have upgraded our test switch to 17.15 and observed that it is fixed.

We have NOT tested the 17.12.x workaround configuration snippet as shown above.

[arogers97]

This worked thank you!

[bristi]

Sorry for the delay, 

The work around is as follows, execute the following commands.

ip tcp window-size 4128
no ip ssh bulk-mode

[mcnuttj]

Huh.  That looks remarkably similar to what I posted almost ten days ago...