Access Control Policy Blocks all traffic below block rules

[Herald Sison]

Hi Everyone,

I have this very weird incident that happened just a few hours ago. I have access control policy running in my FTD 7.0.4 with FMC 7.0.4 and Snort 3.

This was running smooth and did not even encountered any problem with url blocking for a year already, up until today that the blocking was messed up even if we did not change anything form the ACP but we do deployed an SRU update18 hours ago and the problem just came up an hour ago.

The problem is that only ACP RULE number 1 and 2 are accessible and all the rest are blocked. So in short only facebook and youtube are accessible which are allowed in ACP policy number 1 and 2 and all the rest are blocked (even teams, outlook etc), even the ACP policy number 6 (all users can access all sites basically all all configuration) is not working.

is this problem related to the latest SRU update that was deployed 18 hours ago? if yes, what it took a long time to make some changes and how can we fix this one? is this some kind of a bug?

my number 1 ACP policy is to allow certain domain users to access youtube and facebook

my number 2 ACP policy is to allow certain mobile devices to access youtube and facebook

my number 3, 4 and 5 ACP policies are blocke youtube, facebook and adult sites to all

my number 6 ACP policy is to allow all users to all sites

My ACP arrangement looks like this in the image below.

i tried moving the all the block rule to the bottom end and all the allow rule at the top and the result is that all websites are accessible even those sites that are supposed to be blocked including adult sites. 

So right now i just disabled the blocked ACP so that all users can access office.com and other work related sites.

This is very unusual since this config has been running every since day 1 and i did not change anything in ACP rules, only SRU updates has been deployed to the FTD.

Hope someone can help me on this since all employees can access restricted sites right now.

[Herald Sison]

Is 7.0.5 upgrade for FTD considered reliable and stable? because i am thinking about upgrading both my FMC and FTD hoping that this may fix the issue that i am facing right now. it is really weird why the Access Control Policy only reads the ALLOW RULE 1 and 2 and all the rest are considered blocked below it, even if i have an ALLOW (any) RULE in my RULE 6.

[Eric R. Jones]

s this the SRU,   2022-12-08-001-vrt?

[Herald Sison]

Hi Sir, you are right i am running that version of SRU and just deployed last week and after maybe around 16-18 hours this mess happened. Did you experience the same problem?

 

[Herald Sison]

any idea anyone? TAC cant even pinpoint the problem. its taking a long time already. i need the url blocking to be back ASAP.

[Eric R. Jones]

Not to be disrespectful of skills, have you opened the failing rules and checked the "inspection" column on the far right to see what the what those settings are? The defaults I have are Intrusion policy = None, variable set = default set and file policy = none. Check under Policies >Intrusion Policies and see if you have any particular configurations in there that are getting triggered or not with the new SRU. That's about all I can think of. We are currently on 7.1.X and I know that's been stable for us for several months. 

[Herald Sison]

Hi Sir upon checking we have the same settings in the inspection tab of the ACP.

 

[Eric R. Jones]

We haven't experienced this issue with ours. We don't have specific SNORT configurations setup for V3, we don't use the earlier versions. From my understanding you have been on version 7.0.X for some time and are using snort 3, which is the default for all FMC's on 7.0 and above. Are you using the basic snort rules, or have you modified them? Also is there any way that reversion to snort 2 could have happened? Devices > Device Managment > Device > Inspection Engine. I too am not sure what it could be but if everything worked fine up to now, no changes to the FMC or FTD's that are managed I would look at what snort is doing. Unfortunately, there doesn't seem to be a way to back out an installed SRU and revert to an older one.

[Herald Sison]

Hi sir the history for this device is i was running 7.0.1 with snort 2 before and during that time i always get (snort - high memory utilization) and so i decided to upgrade to snort 3 but i got errors and everyone in the office got blocked (except for this users and devices that are included in the rule 1,2 in ACP and all the rest of the users are blocked even my FMC cannot get to the internet) so i decided to revert back to snort 2 and upgrade my FTD and FMC to 7.0.4 then i upgrade my snort again to version 3 then for about 2 weeks of running 7.0.4 and snort 3 eveyrthing went well and my memory utilization reaches is at 30-50% only up until last week that i experienced this problem i have right now. And it is very unusual since i did not change any ACP since then.

so now if i revert back to snort 2 again i know this ACP problem that i am facing right now will be fixed but the (snort - high memory utilization) error come up again.

 

[Herald Sison]

Hi sir i was also thinking if maybe the OGS (object group search) affects the ACP's performance? Not really sure but just asking. Because i enabled OGS in my FTD hoping to normalize my memory usage when it was still running 7.0.1 with snort2 but that setting did not help in anyway but i still kept it as is (enabled). So after upgrading to 7.0.4 with snort 3 the OGS was still enabled so maybe or maybe not that setting affects the ACP's performance.

correct my please if im wrong about this since i have not really touched deeply about OGS.

[Herald Sison]

A summery update from the TAC engineer below:

I have gone through the collected outputs. After checking the outputs and referring internal cases, the device appears to be hitting following defect:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa85492
The defect has been resolved in 7.0.5 release:
https://www.cisco.com/c/en/us/td/docs/security/firepower/70/relnotes/firepower-release-notes-700/bugs.html

In my opinion its is better to stay with my snort version rather than going back to snort 2 since i have a very disappointing problem encountered with high memory usage, So is there anyone tried upgrading to 7.0.5 FMC and FTD? do you find it pretty stable?

[Eric R. Jones]

I was curious about the OGS, I've not messed with that on our system, and I found this in the community, https://community.cisco.com/t5/network-security/object-group-search-feature-in-ftd-2110-version-6-6-1/td-p/4388733. I read the section on device groups, and it states that it's on by default. I checked our setting in the FMC and ours is disabled. I don't remember if I disabled it on purpose or if this was the default when we started on version 6.X.  The document referenced in the link explains what OGS is, I didn't remember this from FW class. At first I thought you may want to raise the resource on your VM that holds the FMC but I don't think that's it. The issue is with the FTD's CPU and Memory.

I understand better where you're coming from. you had been on a previous version and did some upgrades and that's why you're on SNORT3 now. We did the same but only after getting an advisory that directed us to not use SNORT2. I'll check my notes and see if we had to deal with this OGS issue.

[Herald Sison]

Thanks for the information sir. OGS in my case was disabled by default and i am the one who enabled it manually. On the other hand i had a call and remote session with TAC engineer few hours ago and after further checking he restarted the snort service, SFDataCorrelator and Cloud Agent in FMC in also in FTD, after the service restart everything went well but as per the TAC engineer this fix is just a temporary one and maybe after 3-5 days this problem will occur again since this is considered bug on the Snort on the side of URL CATEGORIES that are not categorized properly. He will keep me posted on any development of this problem

but for now his recommendation is to either go back to Snort 2 or leave the block ACP's disabled since he is not confident on reverting it back to normal since the problem may occur again anytime during peak office hours.

on my end i am not confident on going back to Snort 2 since i had a memory utilization problem with that version before so i rather open all access to all users than suffer 92%-100% memory utilization. Anyway we have another url blocking tool that atleast block unwanted sites but still not reliable (better than nothing). 

Has anyone experienced this weird problem?

[Eric R. Jones]

We haven't experienced this issue, but we also went from 7.0.1 to 7.1 skipping over the interim versions between the two. Is there a reason you can't leap from up to 7.1 or 7.2? I'm familiar with constraints on upgrades so I'll understand if policy forbids an upgrade to higher version, but I can say that I've had zero issues with our 7.0.1.0.1 version running SNORT 3.

[Herald Sison]

I had no choice at that time when my FTD was running 7.0.1 because it hit a bug also. It is very unlucky for me to always hit a bug in my FTD/FMC

the bug is it got stuck at 5% during deployment and i need to clear the DB inside the FTD to clear it.

here is the community thread i started before when i hit a bug running 7.0.1

https://community.cisco.com/t5/network-security/fmc-deployment-got-stuck-at-5-for-almost-3-hours/m-p/4718057#M1094867