topic Thank you for all your help. in Network Security

Flow closed by inspection

drbabbers — Tue, 12 Mar 2019 08:30:23 GMT

All,

I am having an intermittent issue with an SQL based application running on a point to point style network. At each end an ASA firewall protects the 2 application servers. When the application fails, a generic SQL error appears in relation to TCP timeouts. At this exact time, the firewall produces this error:

Nov 08 2016

14:50:54

302014

1.1.1.1

1433

2.2.2.2

53062

Teardown TCP connection 645242116 for outside:1.1.1.1/1433 to inside:2.2.2.2/53062 duration 0:13:27 bytes 1187358713 Flow closed by inspection

Any ideas please how I can progress?

This sounds like you have an

patoberli — Tue, 08 Nov 2016 16:26:49 GMT

This sounds like you have an inspect configured for this type of traffic. Check your inspection rules, in ASDM that's under configuration -> Firewall -> Service Policy Rules.

Thanks. Is there a way I can

drbabbers — Tue, 08 Nov 2016 18:36:48 GMT

Thanks. Is there a way I can determine which inspection rule is doing this?

As far as I can tell, not in

patoberli — Wed, 09 Nov 2016 07:42:50 GMT

As far as I can tell, not in an easy way if there is a lot of traffic on this firewall.

On a positive note, it must be (I hope) one of the inspections and not a default one. It must be a customized one as there is no default one for Port 1433.

I'm not sure if that will be dropped and logged in the ASP function, if so, this should help.

On the CLI you can see all asp-drop statistics with

show asp drop

See here for more info: http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s2.html

With 'clear asp drop counters' you can clear the counters.

Thanks for your response this

drbabbers — Wed, 09 Nov 2016 10:26:26 GMT

Thanks for your response this is helpful.

I can't see anything obvious from an inspection point of view, however we are running 8.4(6) code on this ASA pair and I have seen a bug on 8.4 with 'flow closed by inspection'. CSCUJ54806. Perhaps now is the time to upgrade our code?

I believe that's the wrong

patoberli — Wed, 09 Nov 2016 10:43:32 GMT

I believe that's the wrong bug id, for me it shows a security vulnerability with ICMP. But yeah, if there is such a bug in the 8.4.6 code, then I recommend you to switch to a fixed release.

In case you go to a 9.x code, read carefully the release notes of all previous versions (special attention to 9.0), there were some configuration changes involved. The biggest change are 'unified access-rules'.

This issue has futher

drbabbers — Wed, 09 Nov 2016 11:09:23 GMT

This issue has futher developed and when the SQL application fails, I see the following log on the ASA:

6 Nov 09 2016 10:20:08 106015 1.1.1.1 1433 2.2.2.2 61520 Deny TCP (no connection) from 1.1.1.1/1433 to 2.2.2.2/61520 flags ACK on interface outside

Any ideas?

That might be normal, as the

patoberli — Wed, 09 Nov 2016 11:34:05 GMT

That might be normal, as the firewall has closed that connection with the previous message in your first thread.

Could you please post your complete inspection policy?

class-map tcp-bypass match

drbabbers — Wed, 09 Nov 2016 12:24:28 GMT

class-map tcp-bypass
match access-list tcp-bypass
class-map netflow-export-class
match access-list netflow-export
class-map tcp_conn_map
match access-list tcp_conn_acl
class-map inspection_default
match default-inspection-traffic
class-map NetFlow-traffic
match access-list netflow-export
class-map ips_class_map
match access-list traffic_to_ips
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map netflow-policy
class netflow-export-class
flow-export event-type all destination 4.4.4.4
class NetFlow-traffic
flow-export event-type all destination 4.4.4.4
policy-map tcp_con_policy
class tcp_conn_map
set connection timeout idle 8:00:00 reset
set connection advanced-options tcp-state-bypass
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect sqlnet
inspect tftp
inspect sip
inspect xdmcp
inspect esmtp
inspect waas
inspect icmp
inspect icmp error
class ips_class_map
ips promiscuous fail-open
class netflow-export-class
flow-export event-type all destination 4.4.4.4
class NetFlow-traffic
flow-export event-type all destination 4.4.4.4
class class-default
policy-map tcp-bypass-policy
class tcp-bypass
set connection advanced-options tcp-state-bypass
!
service-policy global_policy global
service-policy tcp_con_policy interface outside
service-policy tcp_con_policy interface inside
service-policy tcp-bypass-policy interface dmz

Also if this helps:

drbabbers — Wed, 09 Nov 2016 12:26:46 GMT

Also if this helps:

Result of the command: "show service-policy"

Global policy:
Service-policy: global_policy
    Class-map: inspection_default
      Inspect: dns preset_dns_map, packet 400215268, drop 69814571, reset-drop 0
      Inspect: ftp, packet 12842, drop 0, reset-drop 0
      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0
      Inspect: netbios, packet 138560785, drop 0, reset-drop 0
      Inspect: rsh, packet 0, drop 0, reset-drop 0
      Inspect: rtsp, packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: skinny , packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: sqlnet, packet 0, drop 0, reset-drop 0
      Inspect: tftp, packet 2159516, drop 0, reset-drop 0
      Inspect: sip , packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: xdmcp, packet 0, drop 0, reset-drop 0
      Inspect: esmtp _default_esmtp_map, packet 3017200, drop 0, reset-drop 0
      Inspect: waas, packet 56690034, drop 0, reset-drop 0
      Inspect: icmp, packet 92166119, drop 35324, reset-drop 0
      Inspect: icmp error, packet 32304756, drop 7135, reset-drop 0
    Class-map: ips_class_map
      IPS: card status Up, mode promiscuous fail-open
        packet input 32, packet output 75530394851, drop 32, reset-drop 0
    Class-map: netflow-export-class
    Class-map: NetFlow-traffic

Interface outside:
Service-policy: tcp_con_policy
    Class-map: tcp_conn_map
      Set connection policy:         drop 0
      Set connection timeout policy:
        idle 8:00:00 reset
        DCD: disabled, retry-interval 0:00:15, max-retries 5
        DCD: client-probe 0, server-probe 0, conn-expiration 0
      Set connection advanced-options: tcp-state-bypass

Interface inside:
Service-policy: tcp_con_policy
    Class-map: tcp_conn_map
      Set connection policy:         drop 0
      Set connection timeout policy:
        idle 8:00:00 reset
        DCD: disabled, retry-interval 0:00:15, max-retries 5
        DCD: client-probe 0, server-probe 0, conn-expiration 0
      Set connection advanced-options: tcp-state-bypass

Interface dmz:
Service-policy: tcp-bypass-policy
    Class-map: tcp-bypass
      Set connection policy:         drop 0
      Set connection advanced-options: tcp-state-bypass

FYI have setup 'tcp-state

drbabbers — Wed, 09 Nov 2016 12:28:13 GMT

FYI have setup 'tcp-state-bypass' in relation to the ASA denying TCP connections with no SYN present. So far the SQL application is working all OK. *fingers crossed*

Ok, good to read.

patoberli — Wed, 09 Nov 2016 13:13:40 GMT

Ok, good to read.

Interesting is this part here:

Class-map: ips_class_map
IPS: card status Up, mode promiscuous fail-open
packet input 32, packet output 75530394851, drop 32, reset-drop 0

That looks like you also have the IPS card installed? In this case it might be a policy there, if you send the affected traffic to the IPS. I don't have any IPS experience though, can't help if the traffic goes through it.

Also some recommendation, you have some very old inspects active, which I recommend to disable for better compatibility. These are: dns (512 is not enough for ibm.com for example), esmtp (this breaks encrypted mail transfer in some cases), netbios (unless you need netbios traversing the firewall).

Thank you for all your help.

drbabbers — Wed, 09 Nov 2016 15:38:32 GMT

Thank you for all your help.

You also might want to check

patoberli — Fri, 11 Nov 2016 10:18:51 GMT

You also might want to check the "service resetinbound" and "service resetoutbound".

I believe in your case the firewall might terminate the open connection after the default tcp connection timeout, but it's not sending a tcp rst packet to the host(s). Because of this, the host doesn't know the connection was terminated and the host still tries to use the connection.

Source for the command and further explanation: http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s1.html

Please note, the defaults of this command changed in later software revisions. I'd turn it on for internal interfaces, that way your server knows that the connection was closed by the firewall for whatever reason. Thanks to this your host knows it needs to open a new connection.

Turning on tcp-state-bypass is dangerous, as it turns of many basic tcp security features.