Re: %FWSM-6-106028: Deny TCP (Connection marked for deletion) fr

JMCNEL · ‎12-02-2009

I received the following error on the FWSM. the IP in question is for a particular server which is part of a group servers and all rules in FWSM apply to the group servers. The user however only have problems accessing this 1 server and not all the servers.

%FWSM-6-106028: Deny TCP (Connection marked for deletion) from src
ip-address/src-port to dst ip-address/dst-port flags flag on interface intf-name

A user which have the correct rights to access all servers is getting a generic "explorer cannot view this page" for this particular server.

I did do a search on the error on the cisco site but need a little bit help troubleshooting this. Any ideas.

Thanks

JNel

Panos Kampanakis · ‎12-02-2009

What is the flag of the packet deleted?

Usually this log shows up if we see FINs or RSTs and we are ready to close the conn and we see some more packets for that conn.

PK

JMCNEL · ‎12-03-2009

sh local x.x.x.x

ipv4 local hosts:

local host: (x.x.x.x), tcp conn(s)/limit - 0/0, embryonic(s)/limit = 0/0 udp conn(s)/limit = 24/0

xlate(s):

global x.x.x.x local x.x.x.x

Kureli Sankar · ‎12-02-2009

Pls. book mark this link below for fwsm documentation:

http://www.cisco.com/en/US/products/hw/modules/ps2706/ps4452/tsd_products_support_model_home.html

If you search for "system logs messges" in the above link it will take you to the syslog messages link where you can search for 106028 message.

http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/system/message/logmsgs.html#wp1279924

106028

Error Message    %FWSM-6-106028: Deny TCP (Connection marked for deletion) from src 
ip-address/src-port to dst ip-address/dst-port flags flag on interface intf-name

Explanation This syslog message is generated because a TCP port (source, destination, or both) was reused to start a new connection within the TIME_WAIT period prescribed in the TCP RFC for an existing connection. This message appears only for a short period of time (until the connection is actually cleaned up), and only for new connections for SYN-only TCP packets whose flow has been marked for deletion. Although reuse is not mandatory, it is recommended. If a VLAN is shared in more than one context, then the VLAN number appears instead of the intf-name variable.

Recommended Action Investigate why the client is reusing the port. After the connection is deleted in the FWSM, if the same port is reused for a new connection, then the connection is established.

-KS

Giorgio Romano · ‎05-05-2011

HI,

I'm having the same problem.

My proxies has the time_wait value set to 120 seconds.

How can I get from my fwsm 4.1 the time_wait value?

Is it possible to change it?

Thank you very much for any answer

giorgio

Giorgio Romano · ‎05-08-2011

Hi,

an update.

I found in the genuine cisco doc something like this:

(http://www.cisco.com/en/US/docs/security/pix/pix61/command/reference/s.html#wp1026942)

"sysopt connection timewait

The sysopt connection timewait command is necessary for end host applications whose default TCP terminating sequence is a simultaneous close instead of the normal shutdown sequence (see RFC 793). In a simultaneous close, both ends of the transaction initiate the closing sequence, as opposed to the normal sequence where one end closes and the other end acknowledges prior to initiating its own closing sequence.

The default behavior of the PIX Firewall is to track the normal shutdown sequence and release the connection after two FINs and the ACKnowledgment of the last FIN segment. This quick release heuristic enables the PIX Firewall to sustain a high connection rate.

However with a simultaneous close, the quick release forces one side of the connection to linger in the CLOSING state (see RFC 793). Many sockets in the CLOSING state can degrade the performance of an end host. For instance, some WinSock mainframe clients are known to exhibit this behavior and degrade the performance of the mainframe server. Old versions of HP/UX are also susceptible to this behavior. Enabling the sysopt connection timewait command enables a quiet time window for the abnormal close down sequence to complete.

The no sysopt connection timewait command disables the option, which is off by default.

Note Use of the sysopt connection timewait command may impact PIX Firewall performance especially with low memory configuration and highly dynamic traffic pattern such as HTTP. "

but this doc is referred to cisco pix 6.1 FWOS version and I don't know if it is applicable to the fwsm version 4.1.(3).

Instead into the cisco fwsm 4.1 command reference I found this:

(http://www.cisco.com/en/US/docs/security/fwsm/fwsm41/command/reference/s6.html#wp2763407)

The "sysopt connection timewait" command is used but when I try to enter it in the FWSM CLI, the output I get is:

"FWSM(config)# sysopt connection timewait

^

ERROR: % Invalid input detected at '^' marker."

It seems not exist!

Where is the mistake?

Thank you very much!

giorgio

varrao · ‎05-08-2011

This command does not exist on the FWSM, here is the command reference for 4.1(x)

http://www.cisco.com/en/US/partner/docs/security/fwsm/fwsm41/command/reference/s8.html#wp2775118

Hope that helps,

Thanks,

Varun

Thanks,
Varun Rao

Kureli Sankar · ‎05-08-2011

I beleive the mistake may have come from ASA command reference.

Varun is right that the command doesn't exist on the FWSM. But, the link that you posted is for the command "show run sysopt"

That section of the document may have come from the ASA guide. Thanks for pointing it out. We will file a documentation defect and have this corrected.

See if the half-closed timeout will help in this case.

http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/command/reference/s1.html#wp2725339

Thanks,

KS

Giorgio Romano · ‎05-08-2011

Hi,

thank you for your answer!

Yes, I posted link for the command "show run sysopt" but in the same argument there is (as related command) "sysopt connection timeout" command.

Anyway I think I didn't understand well.

The "half-closed timeout" connection you seggested to me is the same thing as "time_wait" described in RFC 973?

I have got a BlueCoat Proxies with the "time_wait" timeout set to 120 seconds.

On my FWSM 4.1.(3) the timeout conn half-closed is set to 10 minutes.

May be this difference to generate the error "%FWSM-6-106028"?

Thank you very much again!

giorgio

Kureli Sankar · ‎05-08-2011

Yes, pls. follow this sample below and see if this helps. It should.

hostname(config)# access-list CONNS permit ip any host 10.1.1.1
hostname(config)# class-map conns
hostname(config-cmap)# match access-list CONNS
hostname(config-cmap)# policy-map conns
hostname(config-pmap)# class conns
hostname(config-pmap-c)# set connection timeout half-closed 0:20:0

They sysopt connection time-wait is supported on the ASA platform but, not int he FWSM.

Pls. follow the above MPF (Modular Policy Framework) method above to configure half-closed timeout selectively 
for certain traffic.

-KS

Giorgio Romano · ‎05-08-2011

Hi, thank you for your advise. Tomorrow I 'll try it and then I'm gonna report you the result. Thank you giorgiol

Giorgio Romano · ‎05-09-2011

Hi,

I applied the following commands. the "interface-name" in the service-policy command is the interface where the error occurs. The access-list name "half-closed" doesn't match. Is it normal?

The error still occurs.

access-list half-closed extended permit ip any any

!

class-map half-closed

match access-list half-closed

!

policy-map half-closed

class half-closed

set connection timeout half-closed 0:02:00

!

service-policy half-closed interface "interface-name"

Following the error messsages.

May 09 2011 08:38:02: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 192.168.1.22/65266 to 122.167.64.13/443 flags SYN on interface dmz_proxy

May 09 2011 08:38:03: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 192.168.1.22/56257 to 122.170.3.155/443 flags SYN on interface dmz_proxy

May 09 2011 08:38:03: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 192.168.1.21/55900 to 83.211.26.2/443 flags SYN on interface dmz_proxy

May 09 2011 08:38:04: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 192.168.1.22/51982 to 122.167.152.52/443 flags SYN on interface dmz_proxy

May 09 2011 08:38:04: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 192.168.1.22/62148 to 122.162.237.149/443 flags SYN on interface dmz_proxy

May 09 2011 08:38:04: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 192.168.1.22/60103 to 122.168.48.16/443 flags SYN on interface dmz_proxy

Thank you

giorgio

Kureli Sankar · ‎05-09-2011

These logs show SYN flag set in the packets. So I am assuming this is because TCP port (source,

destination or both) was reused to start a new connection  within the
TIME_WAIT period prescribed in the TCP RFC (793) for an  existing connection. 
Pls. check logs prior to these to see if the same hosts sent same source port and dest port to establish the 
connection. If so, these logs are to be expected.


-KS

Giorgio Romano · ‎05-09-2011

HI,

it is for this reason that I'm trying to find out the TIME_WAIT value set on the FWSM.

The logs prior are present.

The destination ports are always TCP 80 e TCP 443 (web server). The destination hosts are many servers on internet.

The source hosts are some bluecoat proxy. Thay have the TIME_WAIT value set to 120 seconds. May be this thing to create the error? Is there a command on FWSM to find out the TIME_WAIT value on the FWSM?

thank you, thank you very much

giorgio

Kureli Sankar · ‎05-09-2011

Time_wait is set on the end hosts.

Here is an example:
Once a host sends a FIN, it moves to CLOSING state in the TCP
state-machine. Later, once it receives the ACK for this FIN, it moves to
CLOSED state. From this time to again open the same port is called the
TIME_WAIT period. This is defined by the RFC; however, this is also a
configurable item in some of the hosts. However, for fail-safe
mechanisms, there will be a timeout for the CLOSING state, after which
even if there is no ACK, the state-machine moves to the CLOSED state.

However, in case of the FWSM, once it receives a FIN in one direction,
the conn moves to the half-closed state. Once the ACK for this is
received, the conn is "marked-for-deletion". Later a separate thread
running in the FWSM, (which runs periodically), scans for conns marked
for deletion and permanently deletes it from the system. Similar to
end-hosts, if the ACK for the FIN is not received, we have a timeout
(for half-closed, which can be configured), after which even if there is
no ACK, we "mark" the connection for deletion and the aging thread takes
over as described.

What you configured via MPF is half-closed timeout.

-KS