Flow closed by inspection

[drbabbers]

All,

I am having an intermittent issue with an SQL based application running on a point to point style network. At each end an ASA firewall protects the 2 application servers. When the application fails, a generic SQL error appears in relation to TCP timeouts. At this exact time, the firewall produces this error:

6

Nov 08 2016

14:50:54

302014

1.1.1.1

1433

2.2.2.2

53062

Teardown TCP connection 645242116 for outside:1.1.1.1/1433 to inside:2.2.2.2/53062 duration 0:13:27 bytes 1187358713 Flow closed by inspection

Any ideas please how I can progress?

D

[patoberli]

This sounds like you have an inspect configured for this type of traffic. Check your inspection rules, in ASDM that's under configuration -> Firewall -> Service Policy Rules.

[drbabbers]

Thanks. Is there a way I can determine which inspection rule is doing this?

D

[patoberli]

As far as I can tell, not in an easy way if there is a lot of traffic on this firewall.

On a positive note, it must be (I hope) one of the inspections and not a default one. It must be a customized one as there is no default one for Port 1433.

I'm not sure if that will be dropped and logged in the ASP function, if so, this should help.

On the CLI you can see all asp-drop statistics with

show asp drop

See here for more info: http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s2.html

With 'clear asp drop counters' you can clear the counters.

[drbabbers]

Thanks for your response this is helpful.

I can't see anything obvious from an inspection point of view, however we are running 8.4(6) code on this ASA pair and I have seen a bug on 8.4 with 'flow closed by inspection'. CSCUJ54806. Perhaps now is the time to upgrade our code?

D

[patoberli]

I believe that's the wrong bug id, for me it shows a security vulnerability with ICMP. But yeah, if there is such a bug in the 8.4.6 code, then I recommend you to switch to a fixed release.

In case you go to a 9.x code, read carefully the release notes of all previous versions (special attention to 9.0), there were some configuration changes involved. The biggest change are 'unified access-rules'.

[drbabbers]

This issue has futher developed and when the SQL application fails, I see the following log on the ASA:

6 Nov 09 2016 10:20:08 106015 1.1.1.1 1433 2.2.2.2 61520 Deny TCP (no connection) from 1.1.1.1/1433 to 2.2.2.2/61520 flags ACK  on interface outside

Any ideas?

D

[patoberli]

That might be normal, as the firewall has closed that connection with the previous message in your first thread.

Could you please post your complete inspection policy?

[drbabbers]

class-map tcp-bypass
 match access-list tcp-bypass
class-map netflow-export-class
 match access-list netflow-export
class-map tcp_conn_map
 match access-list tcp_conn_acl
class-map inspection_default
 match default-inspection-traffic
class-map NetFlow-traffic
 match access-list netflow-export
class-map ips_class_map
 match access-list traffic_to_ips
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 512
policy-map netflow-policy
 class netflow-export-class
  flow-export event-type all destination 4.4.4.4
 class NetFlow-traffic
  flow-export event-type all destination 4.4.4.4
policy-map tcp_con_policy
 class tcp_conn_map
  set connection timeout idle 8:00:00 reset
  set connection advanced-options tcp-state-bypass
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny 
  inspect sqlnet
  inspect tftp
  inspect sip 
  inspect xdmcp
  inspect esmtp
  inspect waas
  inspect icmp
  inspect icmp error
 class ips_class_map
  ips promiscuous fail-open
 class netflow-export-class
  flow-export event-type all destination 4.4.4.4
 class NetFlow-traffic
  flow-export event-type all destination 4.4.4.4
 class class-default
policy-map tcp-bypass-policy
 class tcp-bypass
  set connection advanced-options tcp-state-bypass
!
service-policy global_policy global
service-policy tcp_con_policy interface outside
service-policy tcp_con_policy interface inside
service-policy tcp-bypass-policy interface dmz

[drbabbers]

FYI have setup 'tcp-state-bypass' in relation to the ASA denying TCP connections with no SYN present. So far the SQL application is working all OK. *fingers crossed*

D

[patoberli · Accepted solution]

Ok, good to read.

Interesting is this part here:    

Class-map: ips_class_map
      IPS: card status Up, mode promiscuous fail-open
        packet input 32, packet output 75530394851, drop 32, reset-drop 0

That looks like you also have the IPS card installed? In this case it might be a policy there, if you send the affected traffic to the IPS. I don't have any IPS experience though, can't help if the traffic goes through it.

Also some recommendation, you have some very old inspects active, which I recommend to disable for better compatibility. These are: dns (512 is not enough for ibm.com for example), esmtp (this breaks encrypted mail transfer in some cases), netbios (unless you need netbios traversing the firewall).

[patoberli]

You also might want to check the "service resetinbound" and "service resetoutbound".

I believe in your case the firewall might terminate the open connection after the default tcp connection timeout, but it's not sending a tcp rst packet to the host(s). Because of this, the host doesn't know the connection was terminated and the host still tries to use the connection.

Source for the command and further explanation: http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s1.html

Please note, the defaults of this command changed in later software revisions. I'd turn it on for internal interfaces, that way your server knows that the connection was closed by the firewall for whatever reason. Thanks to this your host knows it needs to open a new connection.

Turning on tcp-state-bypass is dangerous, as it turns of many basic tcp security features.

[drbabbers]

Also if this helps:

Result of the command: "show service-policy"

Global policy:
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: dns preset_dns_map, packet 400215268, drop 69814571, reset-drop 0
      Inspect: ftp, packet 12842, drop 0, reset-drop 0
      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0
      Inspect: netbios, packet 138560785, drop 0, reset-drop 0
      Inspect: rsh, packet 0, drop 0, reset-drop 0
      Inspect: rtsp, packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: skinny , packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: sqlnet, packet 0, drop 0, reset-drop 0
      Inspect: tftp, packet 2159516, drop 0, reset-drop 0
      Inspect: sip , packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: xdmcp, packet 0, drop 0, reset-drop 0
      Inspect: esmtp _default_esmtp_map, packet 3017200, drop 0, reset-drop 0
      Inspect: waas, packet 56690034, drop 0, reset-drop 0
      Inspect: icmp, packet 92166119, drop 35324, reset-drop 0
      Inspect: icmp error, packet 32304756, drop 7135, reset-drop 0
    Class-map: ips_class_map
      IPS: card status Up, mode promiscuous fail-open
        packet input 32, packet output 75530394851, drop 32, reset-drop 0
    Class-map: netflow-export-class
    Class-map: NetFlow-traffic

Interface outside:
  Service-policy: tcp_con_policy
    Class-map: tcp_conn_map
      Set connection policy:         drop 0
      Set connection timeout policy:
        idle 8:00:00 reset
        DCD: disabled, retry-interval 0:00:15, max-retries 5
        DCD: client-probe 0, server-probe 0, conn-expiration 0
      Set connection advanced-options: tcp-state-bypass

Interface inside:
  Service-policy: tcp_con_policy
    Class-map: tcp_conn_map
      Set connection policy:         drop 0
      Set connection timeout policy:
        idle 8:00:00 reset
        DCD: disabled, retry-interval 0:00:15, max-retries 5
        DCD: client-probe 0, server-probe 0, conn-expiration 0
      Set connection advanced-options: tcp-state-bypass

Interface dmz:
  Service-policy: tcp-bypass-policy
    Class-map: tcp-bypass
      Set connection policy:         drop 0
      Set connection advanced-options: tcp-state-bypass

[drbabbers]

Thank you for all your help.

D