7.11 Not maintaning state on TCP conns?

[slug420]

We recently upgraded to 7.11. I say upgraded but in fact we installed a new pair of firewalls with 7.11 whose config we moved over from the existing 6.3(4) firewall.

I am seeing a lot of messages in the logs like the following:

%PIX-4-106023: Deny tcp src dmz:172.17.122.10/443 dst

outside:10.122.0.71/3908 by access-group "dmz" [0x0, 0x0]

Note that this is obviously a reply from an existing TCP connection to the web server (on 443). Also note that it is being denied because of ACCESS-LISTS and *not* because of "no connection" as would at least make sense. Based on the log message the apparent solution would be to add ACLs on the DMZ interface allowing the traffic out, but that woudl be admitting that my stateful PIX firewall is not stateful at all.

Anyone else run into this? I am seeing this across all interfaces on all different ports, in some instances it appears it may slow down the connections or possibly cause them to not work altogether (i say possibly because I am having difficulty seeing the latency users are seeing so frequently and thus having trouble tying ot not tying it directly to these log messages). The latency and lack of functionality COULD be due to something else but as long as the pix is denying traffic because it cant keep state for some reason, I cannot rule the firewall out or begin looking elsewhere for problems.

Anyone else see this?

[b.nystrand]

Yes indeed, our syslog file gets flooded with these messages after upgrading to an ASA 5510 with 7.1.1 binary.

Our http and https traffic seems to get through the firewall but we get the same messages you do every 3 seconds or so during daytime (approx. 150 users inside).

Our sales partner in sweden, Ementor, says they're talking to Cisco about this, let us hope for an answer soon.

[slug420]

I also have and have had for over a week, a ticket open with Cisco about this. One engineer told me it was a bug but when he did not get back to me in a timely manner I had it assigned to a new engineer who says he cannot find any bug that this could be related to.

In your instance are you seeing latency or dropped connections from this or do you suspect it is not affecting the traffic? Some users claim to be experiencing latency to the web server in question but I cant prove whether or not this syslog message is related to that.

[b.nystrand]

We have'nt really experienced any problems, but the cpu on the new firewall is a lot faster than the old one so maybe that's why the latency is not noticed? I hope Cisco will provide us with an answer soon...

[b.nystrand]

Read Ciscos answer below. I can't find the bug-id anywhere at Ciscos website though...

"I have done some reasearch on this issue, and have come across an instance that matches your problem...

When the connection to an outside web server terminates, the inside host sends a FIN packet back to the web server. On some occasions, the we server returns a FIN ACK back to the host. But since the stateful firewall has already seen the first FIN go outbound it closed the connection and removed it from the state table. THis will cause the acknowledgement of the FIN to be denied on the outside interface because the state no longer exists. This is generating the 106023 syslog message. This is related to an old bug that affected PIX 6.3 and has reappeared in PIX 7.1, this bug ID is CSCef38784."

[slug420]

shouldnt that be:

%PIX-6-106015: Deny TCP (no connection)

instead of "denied by access-group blahblah"?

I see tons of denies from connections that have been terminated and always have but being denied by an access-group is different....or at least used to be pre 7.11

Are they saying the bug is the behavior itself or the fact that it is denied due to the access-group instead of denied due to "no-connection"?

[bjr]

I have a couple of pix with 7.1.2 in them and still see the same result. Anyone heard anything from Cisco?

[jwalker]

I experienced the same latency issues with an ASA and a PIX after upgrading to 7.x. After troubleshooting the issue for a while, I noticed the problem was because the firewall was dropping out-of-order packets due to full buffers. If you do a show asp drop, you will see what I am referring to. To solve the issue you need to add more buffers to the queue. The only problem is these commmands are only available on the ASAs. Cisco is supposedly adding them to the PIX OS. Be careful about how many buffers you add because they take up memory on your ASA. I changed mine to 100 and it is working great now. No latency any more!! Here are some steps to do this:

1) Define an access-list for interesting traffic:

access-list tcp-queue-limit extended permit ip any any

2) Define your tcp-map:

tcp-map queue-limit-map

queue-limit 100

3) Define a class map:

class-map tcp-queue-limit

match access-list tcp-queue-limit

4) now match everything up in your policy map that is/will be applied globally:

policy-map global_policy

class tcp-queue-limit

set connection advanced-options queue-limit-map

** Remember this is a fix for ASAs only...

[heronmb]

I experienced this issue on version 7.0.5.

Any one have the Cisco bug id of this case ?