After upgrade from 6.2(2) to 6.3(3) we see more denys

[cdoyle]

One of the functions of our PIX 525 is to permit internet access for our users. This PIX does not have direct access to the Internet, it relays upstream to another device that does.

Immediately after upgrading to 6.3(3) our syslog server began reporting an almost constant stream of deny messages generated by the PIX for what appears to be normal response traffic to the http sessions. The traffic is tcp/80 inbound to a random source port between 1024-65535.

We did not modify any of the connection parms or timeouts during the upgrade. The ASA should be expecting this return traffic and allow it through. The really strange part is that Internet access appears to be functioning fine.

Is 6.3(3) simply providing optics on denys that were occuring all along or is this something new ?

Thanks,

Craig.

[ehirsel]

Can you post a few of the messages that you see on your syslog server? I have upgraded from pix 6.2.2 to 6.3.2 (have not yet gone to 6.3.3) and I have not seen any more deny messages than before.

[hugodrax]

you should go to 6.3.3, 6.3.2 is buggy and was pulled quickly.

[cdoyle]

Hi Edward,

Sorry for the delay, attached is a sample of the log. It only shows several minutes of activity but it's fairly constant as long as there are users browsing out. What you'll see in the log is traffic inbound from the Internet back into the inside and it's being blocked.

Appreciate your intrest, let me know what you think.

Craig.

[ehirsel]

I noted that you are using a proxy-server on your network. I think that the proxy-server and the firewall think that the connection is closed, but for some reason the remote server still thinks that it is open and it may be trying to do a normal close where as the proxy does a quick disconnect.

I know that there is some mention on the cisco web site about configuring the firewall for quick disconnect. Once I find out where it is, I'll post it here.

In the meantime, to see if this is really the case, can you run a capture command on the pix firewall between the proxy-server and the upstream (closest to isp) gateway? Pick a source, such as www.cisco.com, and filter for only that address to make the capture trace easier to read. I'd be interested in looking at the capture trace to see if the remote end does not receive, or ignores, the connection close sequence.

[cdoyle]

You are right on the mark with your understanding of the topology.

I do have, and have attached, a sample of a capture I ran earlier. It was originally done as part of a completely separate troubleshooting exercise for the remote server as you've identified it in your above response. That remote server is actually another firewall (non-Cisco) that connects us to the Internet, the PIX does not have a direct connection to the Internet and must pass Internet bound traffic through this upstream firewall.

The capture (filtered for brevity) shows an outbound SYN request that ultimately gets ignored by the upstream firewall because - apparently - the SYN request parameters match what the device considers is an already open conversation (a matching five-tuple (spelling?) is what the vendor referred to this condition as) so it ignores the SYN. The two conversation are interleaved and visible.

It appears now that the support call we've had with this other vendor and this PIX call are most likely related as the PIX blocks this "late" traffic the upstream firewall is trying to send back through it - and - the users are complaining of a condition where there Web pages are not loading completely because - as far as we can tell - the upstream firewall is selectively ignoring SYN request form the inside Web Proxy Server.

A Pandora's box all around.

[cdoyle]

The cause of these Deny messages appearing right after the upgrade from 6.2(2) to 6.3(3) has been found.

In short, the message level of this particular deny message was promoted from 4 to 6 with 6.3(3). Our config was set to monitor level 6 so all of a sudden we were seeing these messages. Cisco TAC is in the process of documenting this in the support site, however, a more detailed explanation provided by TAC follows:

Before 6.3.3, when a connection was torn down, and any continuation packets after the connection is torn down, (for this same connection session) you would receive these messages

Teardown TCP connection 77654 for outside:63.211.153.89/80 to inside:64.0.182.228/1322 duration 0:00:01 bytes 711

106015: Deny TCP (no connection) from 63.211.153.89/80 to 64.0.182.228/1322 flags ACK on interface outside

With 6.3.3 when a connection is torn down and any continuation packets after the connection is torn down you will see these messages

Teardown TCP connection 77654 for outside:63.211.153.111/80 to inside:64.0.182.226/10351 duration 0:00:01 bytes 711

106023: Deny tcp src outside:63.211.153.111/80 dst inside:64.0.182.226/10351 by access-group "outside"

Therefore, if you have level 6 logging turned on, (with 6.3.3) before you see this message 106023: Deny tcp src outside:63.211.153.111/80 dst inside:64.0.182.226/10351

by access-group "outside"

You should also have a message that states that this connection is torn down. The reason the customer probably did not notice these messages before 6.3.3 is because 106015: Deny TCP (no connection) message is a level 6 message. While 106023: Deny tcp src outside:63.211.153.111/80 is a level 4 message.

Thanks for the responses to my posting !

Regards,

Craig.