topic Re: SSH connections through asa hanging in Network Security

SSH connections through asa hanging

waqas gondal — Tue, 12 Mar 2019 03:24:39 GMT

Hi,

I have been troubleshooting this intermittent issue with Cisco TAC for about 3 weeks now.

The issue is that when users connect to external servers via ssh whether it is over a VPN or not, the connections hang. The timing is random and is not dependant on anything. After a user connects, the connection will hang and the show local host indicates that the connection is idle.

We are using an ASA 5505 with version 8.4(5) and a 5510 with 8.4(6) at another location with a site-to-site VPN in between them. The users who initiate the ssh connection are behind the 5505 and sometimes connect to servers behind the 5510, those connections also hang randomly.

When we create ssh traffic through the VPN to servers behind the 5510:

The packet captures don't show a reason for the connection to hang, they just show that packets have stopped going through.

Syslog messages show nothing on the 5505 when the connection hangs, syslogs on the 5510 sometimes show the "deny tcp (no connection)"

Any ideas as to what might cause this issue?

Re: SSH connections through asa hanging

Jon Are Endrerud — Fri, 03 Jan 2014 18:08:42 GMT

Is it possible this could have something to do with your ISP or other providers?

Sent from Cisco Technical Support iPhone App

SSH connections through asa hanging

waqas gondal — Fri, 03 Jan 2014 19:46:21 GMT

It is possible but I have not checked with them yet.

Re: SSH connections through asa hanging

Julio Carvajal — Fri, 03 Jan 2014 19:47:26 GMT

Hello,

So basically a SSH session that already was closed is still present on the local-host table of the ASA and the connection table??

Can you check the Timeout configuration on your firewall and also the MPF setup.

What's the Idle time you have configured for a TCP session?

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

waqas gondal — Fri, 03 Jan 2014 21:10:12 GMT

The timeout on both ASAs is 1 hour. However if an ssh connection is established from behind the 5505 to a destination behind the 5510 the hanging connection is not present in the table of the 5510, and idle on the 5505.

The problem isn't getting the connection out of the connection table. The problem is trying to figure out why the connections are hanging intermittently.

There is no MPF setup

SSH connections through asa hanging

Julio Carvajal — Fri, 03 Jan 2014 21:14:56 GMT

Hi,

When you say the hanging connection what do you mean?

Do you mean the connection is closed but still present on one of the FWs?

That let us know we can focus on the ASA 5505

Can you share the configuration used there

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

waqas gondal — Fri, 03 Jan 2014 21:33:22 GMT

By hanging connections I mean there is no packet in any of the wireshark captures that indicates the connection has closed. A connection remains on the 5505 but the amount of bytes passed through do not increase. The ssh connection window to the device behind the 5510 is still open but inactive with no messages indicating close.

Unfortuantly I cannot share configuration for policy reasons. Here I just wanted some ideas for things to look for when troubleshooting.

Thank you for your time.

Re: SSH connections through asa hanging

Julio Carvajal — Fri, 03 Jan 2014 21:57:03 GMT

Well,

I would create a capture on both of the interfaces (as you said you did).

I would check the MPF configuration for any specific set connection timeout

I would also check the Global timeout connection.

And of course enable logging on the FW to capture as much information as possible (between this sessions)

Is this the only traffic affected?

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

waqas gondal — Fri, 03 Jan 2014 23:53:34 GMT

Hi,

SSH is the only affected connection type.

And I have tried all of the things which you have mentioned. It is difficult to look through the firewall logs because they are extensive and don't show what caused the connection to hang.

Thanks,

Waqas

Re: SSH connections through asa hanging

Julio Carvajal — Fri, 03 Jan 2014 23:57:05 GMT

Well,

I am basically troubleshooting on blind mode so we cannot move forward bud.

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

waqas gondal — Sat, 04 Jan 2014 00:24:58 GMT

Thank you for your input though, I really appreciate it.

Re: SSH connections through asa hanging

Julio Carvajal — Sat, 04 Jan 2014 01:01:56 GMT

Sure,

If you are willing to work providing updates and config related to the problem let us know.

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

cciesec2011 — Sat, 04 Jan 2014 01:50:16 GMT

The problem can be resolved very easily without touching the Cisco device. By enabling ssh keep-alive on either the ssh client or the ssh server.

/etc/ssh/sshd_config

Look for TCPKeepAlive and make sure it is set to yes and add the following lines after it:

ClientAliveInterval 30

ClientAliveCountMax 10000

service sshd restart

This will help the ssh connection from disconnecting. If you still experience it, it is the cisco ASA

Re: SSH connections through asa hanging

Julio Carvajal — Sat, 04 Jan 2014 02:25:56 GMT

Hello,

I seriously do not think your configuration on the client/server side will make any difference.

We have already found where the problem is (ASA 5505) as the connection is hanging there while not traffic is being seeing.

On the other ASA (5510) the connection is succesfully removed from all of the respective tables.

So your work-around will not make any difference here as for the client/server the connection has been already closed (this after the customer description of the problem)

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

cciesec2011 — Sat, 04 Jan 2014 13:51:43 GMT

jcarvaja wrote:

Hello,

I seriously do not think your configuration on the client/server side will make any difference.

We have already found where the problem is (ASA 5505) as the connection is hanging there while not traffic is being seeing.

On the other ASA (5510) the connection is succesfully removed from all of the respective tables.

So your work-around will not make any difference here as for the client/server the connection has been already closed (this after the customer description of the problem)

You can not base that on what the user described. In order to understand the issue, you need packet capture.

What I am suggesting is commonly used for connectivity traversing the firewall to prove whether the issue is on network or application itself. By enabling keepalive on the application, you can see how it behaves.

Re: SSH connections through asa hanging

Oscar Castillo — Sat, 04 Jan 2014 15:08:17 GMT

I had this kind of issue long time ago.

I enabled an inside host to accept ssh connection and the ASA failed to enable it.

I remember that one of the steps that I've done, was.. reconfigure (ssh) the ASA from scratch, and test it many times to make sure, it was working properly.

Crypto, aaa, username, ip, inside/outside... All that.

Then I went to create object, nat (inside, outside) host or subnet, acl then access-group.

That way worked for me, but it tool me 3 days to figure that out.

You could try that, hope it works.

Regards,
Oscar

Sent from Cisco Technical Support iPhone App

Re: SSH connections through asa hanging

Julio Carvajal — Sat, 04 Jan 2014 15:37:23 GMT

You can not base that on what the user described. In order to understand the issue, you need packet capture.

If customer does not provide us access to the box, inputs that we request we got to trust what he says. This is the case!

Now, I do not think you understand what I am saying..

If this were an app issue then the orphaned sessions would exist on both firewalls! Not just on one. As simple as that

Looking for some Networking Assistance?
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Re: SSH connections through asa hanging

waqas gondal — Sat, 04 Jan 2014 17:24:47 GMT

Thanks,

This is something I will check.

Re: SSH connections through asa hanging

waqas gondal — Sat, 04 Jan 2014 17:29:49 GMT

This issue is a little different I think.

My ASA allows the connection through but after a random amount of time the connection hangs.

I do understand your logic though, I have had issues where I simply erased the config, applied it again and everything was functonal.

Thanks for your input,

Waqas

Re: SSH connections through asa hanging

Jon Are Endrerud — Sat, 04 Jan 2014 17:48:25 GMT

When does it hang? Is it possible to use it at all?

Is it after big blocks of text passing through the terminal?

What is the MTU between the ssh server and client?

Sent from Cisco Technical Support iPhone App