Re: ASA5585-SSP-20 9.1(2) Flow closed by inspection

ALEXANDR LYKOV · ‎06-28-2013

Hello,

We have site-2-site VPN between two the same ASAs and I'm connecting to the some switch beside of one ASA through internet.

We had default configuration for inspection:

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect skinny

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

inspect icmp

inspect ipsec-pass-thru

and after some times we have lost connection in the middle of ssh session or telnet or Radmin and others (like oracle sql)...

Debug show this messages:

Jun 29 2013 07:52:56: %ASA-6-302304: Teardown TCP state-bypass connection 356159 from outside:192.168.A.A/60669 to DB-PROD:10.B.B.B/1521 duration 0:00:23 bytes 2633284 Flow closed by inspection

Jun 29 2013 07:54:33: %ASA-6-302014: Teardown TCP connection 355841 for outside:10.X.X.X/59883 to management:10.Y.Y.Y/22 duration 0:02:48 bytes 176730 Flow closed by inspection

Jun 29 2013 07:54:38: %ASA-6-302014: Teardown TCP connection 356316 for outside:10.X.X.X/59902 to management:10.Y.Y.Y/23 duration 0:01:50 bytes 11250 Flow closed by inspection

This happened when i start ping A.A.A.A repeat 100000

and after many !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! we have a stop ssh or telnet sesssion.

If we set

set connection advanced-options tcp-state-bypass

everything working fine.

What can we do with this inspection?

thx,

Alex.

Julio Carvajal · ‎06-28-2013

Hello Alexandr,

We need more details on this ( a clear explanation as I am not sure I follow your explanation)

Now this log:

Teardown TCP state-bypass connection means you have configured a TCP state-by-pass policy and on the config you provide us it's missing so I want to ask you:

Are you 100 % sure you are providing the logs from the right device where you took the configuration?

Is that all of the MPF configuration u have on the device where you get the logs?

Note: I remember from a case that when the customer's VPN went down the connects were torn down showing that reason so it would make sense ( Next time this happens can you doble check that) If that;s the case then this is expected.

Add the following commands and let me know if it happens again:

Sysopt connection preserve-vpn-flows

Sysopt connection reclassify-vpn

Let me kwow

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

ALEXANDR LYKOV · ‎06-28-2013

Hello Jcarvaja,

I've already try this sysopt's commands - they are not working.

In fact I've reated some class maps now and use bypass for all traffic except from my host.

[....]

access-list alykov; 2 elements; name hash: 0x770e55

access-list alykov line 1 extended permit ip host 10.X.X.X any (hitcnt=224) 0xda70e89b

access-list alykov line 2 extended permit ip any host 10.X.X.X (hitcnt=40) 0x0f106e1e

!

class-map alykov

match access-list alykov

class-map inspection_default

match default-inspection-traffic

class-map tcp_bypass

match any

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect skinny

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

inspect icmp

inspect ipsec-pass-thru

class alykov

class tcp_bypass

set connection advanced-options tcp-state-bypass

[...]

There are logs from 10.X.X.X when I'm connecting by SSH:

Fw01# sh conn address 10.X.X.X

6529 in use, 7574 most used

TCP outside 10.X.X.X:49561 management 10.Y.Y.Y:22, idle 0:00:10, bytes 3266, flags UIOB

It's ok - I can work and enter commands in shell... but when I try to run ping A.B.C.D repeat 10000 I have sucked ssh session and disconnect:

Sw01# ping A.B.C.D repeat 10000

Type escape sequence to abort.

Sending 10000, 100-byte ICMP Echos to A.B.C.D, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[SSH disconnected]

and logs on Fw:

Jun 29 2013 11:17:38: %ASA-6-302013: Built inbound TCP connection 435354 for outside:10.X.X.X/65487 (10.X.X.X/65487) to management:10.Y.Y.Y/22 (10.Y.Y.Y/22)

Jun 29 2013 11:17:59: %ASA-6-302014: Teardown TCP connection 435354 for outside 10.X.X.X/65487 to management:10.Y.Y.Y/22 duration 0:00:21 bytes 143174 Flow closed by inspection

Jun 29 2013 11:17:59: %ASA-6-106015: Deny TCP (no connection) from 10.Y.Y.Y/22 to 10.X.X.X/65487 flags PSH ACK on interface management

Jun 29 2013 11:17:59: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/65487 to 10.Y.Y.Y/22 flags ACK on interface outside

Jun 29 2013 11:18:08: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/65487 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

capture asp type asp-drop all

Fw01# sh cap asp | i 10.X.X.X

838: 11:17:37.549730 10.Y.Y.Y.22 > 10.X.X.X.65404: R 1355770447:1355770447(0) win 3868

839: 11:17:37.554597 10.Y.Y.Y.22 > 10.X.X.X.65404: R 1355770447:1355770447(0) win 0

840: 11:17:37.554658 10.Y.Y.Y.22 > 10.X.X.X.65404: R 1355770447:1355770447(0) win 0

1035: 11:17:59.290573 10.Y.Y.Y.22 > 10.X.X.X.65487: P 2439716514:2439716566(52) ack 2130525119 win 3296

1037: 11:17:59.481298 10.X.X.X.65487 > 10.Y.Y.Y.22: . ack 2819499508 win 64808

1116: 11:18:08.173147 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039254(52) ack 2819499508 win 64808

1117: 11:18:08.406366 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039254:1715039306(52) ack 2819499508 win 64808

1118: 11:18:08.468908 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1125: 11:18:09.050259 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1138: 11:18:10.212086 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1144: 11:18:11.375651 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1155: 11:18:12.546480 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1180: 11:18:14.867630 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1224: 11:18:19.518543 10.X.X.X.65487 > 10.Y.Y.Y.22: P 1715039202:1715039306(104) ack 2819499508 win 64808

1304: 11:18:28.826861 10.X.X.X.65487 > 10.Y.Y.Y.22: R 1715039306:1715039306(0) ack 2819499508 win 0

Julio Carvajal · ‎06-28-2013

Hello Alexander,

Well you are showing different config and logs now,

In fact I've reated some class maps now and use bypass for all traffic except from my host.

Based on the config you post, this is not true as you are matching the traffic but nothing is being done.

class-map tcp_bypass

match any

Now this is not good at all:

First of all you are removing a layer of security from your network

Second match any will match any Layer 4/layer 3 protocol ( TCP state bypass only for TCP traffic so change that IF this is really required)

Can you leave the MPF config on it's default and then clear the local-host table,

Then attempt the same tests and share the logs again,

We will move from there

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

ALEXANDR LYKOV · ‎06-28-2013

Ok. My new config:

Fw01(config-pmap)# sh run policy

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect skinny

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

inspect icmp

inspect ipsec-pass-thru

inspect sqlnet

!

Fw01# clear local-host

Fw01# clear local-host all

------------------

next, connected by SSH:

-------------------

Fw01# sh conn | i 10.X.X.X

TCP outside 10.X.X.X:51573 management 10.Y.Y.Y:22, idle 0:00:32, bytes 3266, flags UIOB

-----------------

It's ok. ran ping or sh run or just pres enter and wait -> connection stopped

------------

Jun 29 2013 12:46:08: %ASA-6-302014: Teardown TCP connection 470272 for outside:10.X.X.X/51573 to management:10.Y.Y.Y/22 duration 0:00:46 bytes 22878 Flow closed by inspection

Jun 29 2013 12:46:08: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/51573 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

Jun 29 2013 12:46:09: %ASA-6-106015: Deny TCP (no connection) from 10.Y.Y.Y/22 to 10.X.X.X/51573 flags PSH ACK on interface management

Jun 29 2013 12:46:09: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/51573 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

Jun 29 2013 12:46:10: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/51573 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

Jun 29 2013 12:46:13: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/51573 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

Jun 29 2013 12:46:17: %ASA-6-106015: Deny TCP (no connection) from 10.X.X.X/51573 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

sh cap asp | i 10.X.X.X

23: 12:46:27.293685 10.X.X.X.51573 > 10.Y.Y.Y.22: R 3447707071:3447707071(0) ack 242859695 win 0 Drop-reason: (tcp-not-syn) First TCP packet not SYN

Julio Carvajal · ‎06-29-2013

Hello Alexandr,

Great job, we are getting closer,

Deny TCP (no connection) from 10.X.X.X/51573 to 10.Y.Y.Y/22 flags PSH ACK on interface outside

So 10.Y.Y.Y it's the Outside interface IP address, right?

This SSH connection does not come via the L2L or does it?

Can you share the entire config, this is odd enough with just this lines of config (I do not see anything wrong yet)

Can you do the following and provide the output you get:

sh service-policy flow tcp host your_client_ip_address host Interface_IP_address_you_connect eq 22

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

ALEXANDR LYKOV · ‎06-30-2013

Hello!

It's not quite right - 10.X.X.x it's an admin's station on outside network. 10.y.y.y - It;s a switch inside (management) network on management port.

SSH connection is going fine but only if you slowly enter commands and switch doesn't send lots of text back.

Often, It's stuck in the middle of "sh run' command and always it happen when I run 'ping repeat 10000' and after many !!!!!!!!!!!!!!!!!!!!!! we have a stuck and lost connection to ssh terminal. In additional to it, we can't work by RDP when try to connect to servers on this switch - always loosing connection.

Fw01# sh service-policy flow tcp host 10.X.X.X host 10.Y.Y.Y eq 22

Global policy:
Service-policy: global_policy
    Class-map: class-default
      Match: any
      Action:
Fw01# ut flow:

^^^^^^^^^^^^^^^^^^ - bug?

Julio Carvajal · ‎06-30-2013

Could be a bug,

Without the show run I can go any further

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

ALEXANDR LYKOV · ‎06-30-2013

I'vel sent the tech-support file to you by email.

Thx.

Julio Carvajal · ‎06-30-2013

Hello,

Okey, so you send it to my personal email

Just review it,

There was nothing that caught my attention what point us to a really weird behavior here,

Does this happens with traffic to any other interface? I mean if you go to any other interface than managment do you still see the same behavior?

What's the CPU usage, Memory of the firewall when this happens?

Is there a way that you could enable debugging logs and take as much as you can while connecting?

I am trying to get a pattern in order to look for a vulnerability here or a bug condition

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

ALEXANDR LYKOV · ‎07-01-2013

Hello,

Yes, it happens on other interfaces too. I've tried through 201 vlan - situation was the same.

CPU and Mem are ok (below 10% of usage). Could you explain what kind of debug shall I enabel?

We are going to try downgrade to 9.1(1) soon. before - I can enable debug for this situation.

Wbr,

Alex

ALEXANDR LYKOV · ‎07-01-2013

Hello Julio,

We've downgrade to 9.1(1) and everything is ok now.

In fact the same ASA with the almost same config in other city working fine with 9.1(2)

Thank you

Alex

Julio Carvajal · ‎07-01-2013

Hello Alexander,

Yeah, pretty weird definetly looks like a bug,

one of my concerns would be if we had reboot it the ASA, what would happen (without changing the code).

This communication comes from the VPN tunnel right?

I mean both endpoints are behind each of the L2L sites ?

Here is one bug I found:CSCtg17779

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

ALEXANDR LYKOV · ‎07-01-2013

Hello Julio,

Yes, Everythin comes through l2l tunnel.

This ASA had been installed about one month ago and everything was ok, and then happened this situation last week.

I tried to reboot ASA but unsuccessfully.

Thank for you help,

Alexadner.

Julio Carvajal · ‎07-01-2013

Hello Alexander,

What do you think about the bug I mentioned?

Regards

Remember to rate all of the helpful posts.

For this community that's as important as a thanks.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC