11-08-2016 07:46 AM - edited 03-12-2019 01:30 AM
All,
I am having an intermittent issue with an SQL based application running on a point to point style network. At each end an ASA firewall protects the 2 application servers. When the application fails, a generic SQL error appears in relation to TCP timeouts. At this exact time, the firewall produces this error:
6 | Nov 08 2016 | 14:50:54 | 302014 | 1.1.1.1 | 1433 | 2.2.2.2 | 53062 | Teardown TCP connection 645242116 for outside:1.1.1.1/1433 to inside:2.2.2.2/53062 duration 0:13:27 bytes 1187358713 Flow closed by inspection |
Any ideas please how I can progress?
D
Solved! Go to Solution.
11-09-2016 05:13 AM
Ok, good to read.
Interesting is this part here:
Class-map: ips_class_map
IPS: card status Up, mode promiscuous fail-open
packet input 32, packet output 75530394851, drop 32, reset-drop 0
That looks like you also have the IPS card installed? In this case it might be a policy there, if you send the affected traffic to the IPS. I don't have any IPS experience though, can't help if the traffic goes through it.
Also some recommendation, you have some very old inspects active, which I recommend to disable for better compatibility. These are: dns (512 is not enough for ibm.com for example), esmtp (this breaks encrypted mail transfer in some cases), netbios (unless you need netbios traversing the firewall).
11-08-2016 08:26 AM
This sounds like you have an inspect configured for this type of traffic. Check your inspection rules, in ASDM that's under configuration -> Firewall -> Service Policy Rules.
11-08-2016 10:36 AM
Thanks. Is there a way I can determine which inspection rule is doing this?
D
11-08-2016 11:42 PM
As far as I can tell, not in an easy way if there is a lot of traffic on this firewall.
On a positive note, it must be (I hope) one of the inspections and not a default one. It must be a customized one as there is no default one for Port 1433.
I'm not sure if that will be dropped and logged in the ASP function, if so, this should help.
On the CLI you can see all asp-drop statistics with
show asp drop
See here for more info: http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s2.html
With 'clear asp drop counters' you can clear the counters.
11-09-2016 02:26 AM
Thanks for your response this is helpful.
I can't see anything obvious from an inspection point of view, however we are running 8.4(6) code on this ASA pair and I have seen a bug on 8.4 with 'flow closed by inspection'. CSCUJ54806. Perhaps now is the time to upgrade our code?
D
11-09-2016 02:43 AM
I believe that's the wrong bug id, for me it shows a security vulnerability with ICMP. But yeah, if there is such a bug in the 8.4.6 code, then I recommend you to switch to a fixed release.
In case you go to a 9.x code, read carefully the release notes of all previous versions (special attention to 9.0), there were some configuration changes involved. The biggest change are 'unified access-rules'.
11-09-2016 03:09 AM
This issue has futher developed and when the SQL application fails, I see the following log on the ASA:
6 Nov 09 2016 10:20:08 106015 1.1.1.1 1433 2.2.2.2 61520 Deny TCP (no connection) from 1.1.1.1/1433 to 2.2.2.2/61520 flags ACK on interface outside
Any ideas?
D
11-09-2016 03:34 AM
That might be normal, as the firewall has closed that connection with the previous message in your first thread.
Could you please post your complete inspection policy?
11-09-2016 04:24 AM
class-map tcp-bypass
match access-list tcp-bypass
class-map netflow-export-class
match access-list netflow-export
class-map tcp_conn_map
match access-list tcp_conn_acl
class-map inspection_default
match default-inspection-traffic
class-map NetFlow-traffic
match access-list netflow-export
class-map ips_class_map
match access-list traffic_to_ips
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map netflow-policy
class netflow-export-class
flow-export event-type all destination 4.4.4.4
class NetFlow-traffic
flow-export event-type all destination 4.4.4.4
policy-map tcp_con_policy
class tcp_conn_map
set connection timeout idle 8:00:00 reset
set connection advanced-options tcp-state-bypass
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect sqlnet
inspect tftp
inspect sip
inspect xdmcp
inspect esmtp
inspect waas
inspect icmp
inspect icmp error
class ips_class_map
ips promiscuous fail-open
class netflow-export-class
flow-export event-type all destination 4.4.4.4
class NetFlow-traffic
flow-export event-type all destination 4.4.4.4
class class-default
policy-map tcp-bypass-policy
class tcp-bypass
set connection advanced-options tcp-state-bypass
!
service-policy global_policy global
service-policy tcp_con_policy interface outside
service-policy tcp_con_policy interface inside
service-policy tcp-bypass-policy interface dmz
11-09-2016 04:28 AM
FYI have setup 'tcp-state-bypass' in relation to the ASA denying TCP connections with no SYN present. So far the SQL application is working all OK. *fingers crossed*
D
11-09-2016 05:13 AM
Ok, good to read.
Interesting is this part here:
Class-map: ips_class_map
IPS: card status Up, mode promiscuous fail-open
packet input 32, packet output 75530394851, drop 32, reset-drop 0
That looks like you also have the IPS card installed? In this case it might be a policy there, if you send the affected traffic to the IPS. I don't have any IPS experience though, can't help if the traffic goes through it.
Also some recommendation, you have some very old inspects active, which I recommend to disable for better compatibility. These are: dns (512 is not enough for ibm.com for example), esmtp (this breaks encrypted mail transfer in some cases), netbios (unless you need netbios traversing the firewall).
11-11-2016 02:18 AM
You also might want to check the "service resetinbound" and "service resetoutbound".
I believe in your case the firewall might terminate the open connection after the default tcp connection timeout, but it's not sending a tcp rst packet to the host(s). Because of this, the host doesn't know the connection was terminated and the host still tries to use the connection.
Source for the command and further explanation: http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s1.html
Please note, the defaults of this command changed in later software revisions. I'd turn it on for internal interfaces, that way your server knows that the connection was closed by the firewall for whatever reason. Thanks to this your host knows it needs to open a new connection.
Turning on tcp-state-bypass is dangerous, as it turns of many basic tcp security features.
11-09-2016 04:26 AM
Also if this helps:
Result of the command: "show service-policy"
Global policy:
Service-policy: global_policy
Class-map: inspection_default
Inspect: dns preset_dns_map, packet 400215268, drop 69814571, reset-drop 0
Inspect: ftp, packet 12842, drop 0, reset-drop 0
Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0
tcp-proxy: bytes in buffer 0, bytes dropped 0
Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0
Inspect: netbios, packet 138560785, drop 0, reset-drop 0
Inspect: rsh, packet 0, drop 0, reset-drop 0
Inspect: rtsp, packet 0, drop 0, reset-drop 0
tcp-proxy: bytes in buffer 0, bytes dropped 0
Inspect: skinny , packet 0, drop 0, reset-drop 0
tcp-proxy: bytes in buffer 0, bytes dropped 0
Inspect: sqlnet, packet 0, drop 0, reset-drop 0
Inspect: tftp, packet 2159516, drop 0, reset-drop 0
Inspect: sip , packet 0, drop 0, reset-drop 0
tcp-proxy: bytes in buffer 0, bytes dropped 0
Inspect: xdmcp, packet 0, drop 0, reset-drop 0
Inspect: esmtp _default_esmtp_map, packet 3017200, drop 0, reset-drop 0
Inspect: waas, packet 56690034, drop 0, reset-drop 0
Inspect: icmp, packet 92166119, drop 35324, reset-drop 0
Inspect: icmp error, packet 32304756, drop 7135, reset-drop 0
Class-map: ips_class_map
IPS: card status Up, mode promiscuous fail-open
packet input 32, packet output 75530394851, drop 32, reset-drop 0
Class-map: netflow-export-class
Class-map: NetFlow-traffic
Interface outside:
Service-policy: tcp_con_policy
Class-map: tcp_conn_map
Set connection policy: drop 0
Set connection timeout policy:
idle 8:00:00 reset
DCD: disabled, retry-interval 0:00:15, max-retries 5
DCD: client-probe 0, server-probe 0, conn-expiration 0
Set connection advanced-options: tcp-state-bypass
Interface inside:
Service-policy: tcp_con_policy
Class-map: tcp_conn_map
Set connection policy: drop 0
Set connection timeout policy:
idle 8:00:00 reset
DCD: disabled, retry-interval 0:00:15, max-retries 5
DCD: client-probe 0, server-probe 0, conn-expiration 0
Set connection advanced-options: tcp-state-bypass
Interface dmz:
Service-policy: tcp-bypass-policy
Class-map: tcp-bypass
Set connection policy: drop 0
Set connection advanced-options: tcp-state-bypass
11-09-2016 07:38 AM
Thank you for all your help.
D
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide