Firewall dropping connection every ~50 days

[RahilShaikh4066]

Hello Everyone,

I have been trying to figure it but unable to know what's the problem. 

Our Oracle database server is behind the Cisco ASA firewall and after every 50 days, it randomly drops/closes few connections from its side, but the connection object is still inside the connection pool of the application. 

As soon as I try to that connection which has been broken, my application gets stuck due to TCP Retransmissions and Timeout algorithm is kicked in. 

Application---------->ASA(Firewall)----------->OracleDatabase. 

This actually puts my application is in blocking state which is only recovered once TCP Timeout is reached. 

Can someone please help me...!!

thanks

--

Raj

[balaji.bandi]

what is the model of the ASA and what version of code running the ASA.

what you see interface stats both ASA and connected Switch ?

the problem resolves once reboot the ASA Fw ?

[RahilShaikh4066]

Hardware: ASA5585-SSP-20, 12288 MB RAM, CPU Xeon 5500 series 2133 MHz, 1 CPU (8 cores)
ASA Version 9.4(4)29

This Firewall is something which we don't control, this is customers firewall and we hardly get information from them on this. 

It is very hard to ask them to restart the Firewall for us without a solid answer. 

[Philip D'Ath]

Are you saying your application is keeping a TCP connection open for 50 days?

You may need to extend the timeouts using the "timeout" command if this is the case.  It is posisble you could be hitting idle times as well if the connection is held open for a long time without any data transmission.  The "timout" command also configures this.

[RahilShaikh4066]

Hi Philip,

Thank you for replying.
I checked the timeout settings on the firewall and those seem to be default settings when you say increase it what should be the baseline to increase and how it should be increased?
Thanks,
Rahil

[Maykol Rojas]

My 2 cents, you need to first identify why the connection is being closed prior extending the timeout of it. 

Nothing good comes out of changing the default timeout values. If the connection needs to stay up for periods longer than 1 hour, I recommend using DCD to verify if the peer has the socket open still and leave the conn open.

This should be done for the peers you are having troubles with only, and not a global setting. 

Here is the configuration example for it: 

https://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/firewall/asa_91_firewall_config/conns_connlimits.html#pgfId-1231374

[RahilShaikh4066]

Hi Maykol, 

The problem is not changing the default settings, the problem is with the default settings on CISCO ASA we see connection drops every 50 days, which is quite abnormal with default CISCO ASA settings. 

We basically want to understand is there a plausible reason why this might be happening. 

Thanks,

Rahil

[Maykol Rojas]

Hi; 

Yup, the way to do it would be using syslogs. That is the best way to troubleshoot it. 

If you have a syslog server, you can go back and correlate the last time if failed and check the reason for the teardown. 

[RahilShaikh4066]

Hi Maykol Rojas,



Biggest struggle is we can't do that because its customers Firewall and according to them that is not a problem.

We are in a state where we need to prove that the connection drop was due Firewall, once we do that then may be they would agree to give us more details on firewall.

The situation is real mess with our customers right now and we don't to push them hard on this.



Is there any other way we can figure this out ?

[Sheraz.Salim]

This does not seem to be issue with ASA.

Worth to check the application site. As firewall is acting as a middle man. unless you have set a connection limit on the ASA. however, you mentioned you using the default setting I am sure the problem is not the ASA box.

few precaution you can implement,

syslog,

netflow,

if there is a switch in between the application and firewall than do a remote-spam or monitor session the switch to see where is problem could be. I had a similar issue in past. digging into this for a long time a week we find out the application was sending the rest connection request.

similar to other issue like above you have we find out a fiber issue in the switch as the fiber was damage and drop the connection.

you have to be a more broad minded. do not focus only on the ASA (to be problem here).

[RahilShaikh4066]

Thanks Sheraz. 

But the reason we think it is firewall box because there are multiple TCP connections (part of database connection pool) which are made from the application but there are only 1 or 2 which gets affected on the 50th Day, the pattern is solid and doesn't deviate this kind of behavior can happen only in one case if something is configured that way, I would have agreed there could have been something broken on network which requires fixing until all connections were affected but its just 1 or 2 connection from application which gets impacted after 50th Day, all these connection are created on day 1 and will remain in pool till the time application is restarted. 

Thanks,

Rahil 

[Sheraz.Salim]

I am still not convince this is a firewall issue. however, you in this situation so it could be an issue with firewall. Oracle server is Linux platform (apologies for the dumb question). I can only think of one thing. configure a tcp by-pass for this server application. mean time do collect the syslog, wireshark, more data more possibility to pin point the issue.

ACL:

access-list TESTING_ACL line 1 extended permit tcp host X.X.X.X any4 eq X.X.X.X

CLASS-MAP:

class-map TESTING_MAP
match access-list TESTING_ACL

POLICY-MAP:

policy-map global_policy
class inspection_default

class TEST_MAPING
set connection random-sequence-number disable
set connection advanced-options tcp-state-bypass
class class-default
set connection timeout dcd

SERVICE-POLICY

service-policy global_policy interface

[Marvin Rhoads]

I had seen this once before with long lived database connections (tcp sessions). Coincidentally (or not!) the ASA maximum timeout for tcp is 1193 hours (49.7 days).

• timeout conn hh:mm:ss—The idle time after which a connection closes, between 0:5:0 and 1193:0:0. The default is 1 hour (1:0:0).

The problem was (in my environment) that some database tcp connections were opened and not active. However their being terminated at the ASA caused application layer issues. We were able to increase to 24 hours and avoid the problem. If the 1193 hours maximum does not suffice you would have to do a tcp bypass like another poster already responded.

In any case a filtered packet capture should be able to demonstrate whether or not the endpoint is terminating the connection.

[Sheraz.Salim]

as Marvin said he encounter this issue in past. and also i suggested earlier you have to create a tcp-by-pass rule in order to get this issue fixed.

give it a try.

[cciesec2011]

---

@Sheraz.Salim wrote:

as Marvin said he encounter this issue in past. and also i suggested earlier you have to create a tcp-by-pass rule in order to get this issue fixed.

give it a try.

Advise from people who don't know anything about database, LOL

You do not need to involve the firewall.  The simplest solution is to enable the database keep-alive on the Oracle database.  You need to set this parameter in the sqlnet.ora file on the database server.  For example, set it to 10 minutes:

SQLNET.EXPIRE_TIME=10

This way, the server will automatically refresh the connection with the client every 10 minutes even if there is no traffic between the client and server during that time.  That will prevent the connection from timing out.

---