Disappearing Access Lists

[mike_jones]

Early this morning, one of our applications which communicates across the firewall stopped working. As it turned out, the ACL that was in place specifically allowing that app. to communicate was no longer present on the firewall.

After peeling through our syslog server, it's apparent that there were no system failures, failover events, or managment sessions that took place anywhere near the time that this happened. It's like the ACL just disappeared.

Any ideas as to what could've happened? We're running a PIX 535 v6.3.3

[jaime.pedraza]

2 months ago, I had the same problem. Owed to an outage, the PIX 535 (6.3.3) and its failover shut down. After the boot we found some missing connections. We searched in the syslog: nothing, the firewall local log: nothing. Doing the comparison with the backed up configuration, several differences were found. We opened a case in the TAC but we haven't received any response.

If you find the cause please tell me.

[stalljh]

Well, sometimes a sh access-list outside could be erroneously done as no access-list outside, etc., and when you hit return, poof, the acl is history, or better yet....when trying to clear the counters, you try to put in clear access-list outside counters, but you forget the work counters and hit return and poof...your acl is gone. But if this happened, you could immediately just command a reload on the PIX, and your original config would come back up. It only takes about 30 seconds for the PIX to reload. This assumes of course that you are using the command line interface to do this.

[layer9]

This is a interesting problem. I would like to know more. There are some rare instances when access-lists are dynamically written and torn down without user interaction but this sounds different. I would like to know the following before making any assumptions.

1. Was there only the ONE line in the Access-List or were there OTHER lines?

2. If there were OTHER lines in the ACL did THOSE lines disappear as well?

3. What service is the ACL permitting to the server (i.e. SMTP, HTTP, TELNET, etc)?

I will look for your responses.

Chris Weber CCDP

[mike_jones]

Our access lists are huge, but this is the only line I can find that has disappeared. I would say there is a decent possibility that something else has gone missing and we just haven't noticed it yet considering the size of the lists.

Here's the actual line. It's for our blackberry server.

access-list inside permit tcp host 10.X.X.X any eq 3101

I can pinpoint the exact time the application stopped working. There were no managment sessions or configuration changes listed in the syslog anywhere near the time it stopped.

[mike_jones]

Ok. I've found another disappearing act. One of our static NAT translations has also disappeared. It looks like it left around the same time as the ACL.

[layer9]

There are only two things that can be causing the anomolies you are experiencing. One is that your PIX is giving up the ghost. Something is wrong with it, like a bad flash chip or something.

The only other explanation is someone is having fun with you. Someone is accessing the PIX and removing those lines manually. There is no "configuration" issue that will stop those lines from disappearing.

Only a really weird hardware anomaly or someone is having fun on your PIX.

Period.

Chris Weber CCDP

cw@layer9corp.com

[dro]

I've run into this problem atleast twice before where I was the ONLY person with access to the device. However, it was back around the 6.2 releases and on a pair of 515's.

-Joshua

[layer9]

I feel your pain Joshua. But what you are describing I have never seen on any PIX's, other than those with a hardware problem, user error or unauthorized users tampering with the PIX. I have been working with the PIX for almost a decade and I know of no anomaly which would account for your code changing by itself.

Of course if you ACL's are really big maybe your PIX is running out of resources. I have heard where configurations started dropping lines when the overall config was too big for the PIX it was on. You might look at your resources on the PIX and I would keep focused on a hardware issue.

Good Luck

Chris Weber CCDP

cw@layer9corp.com