topic Re: Security Intelligence URL: memcap exceeded in Network Security

Security Intelligence URL: memcap exceeded

kicmar — Wed, 28 Dec 2022 10:00:18 GMT

Does anybody else noticed this warning started this weekend on their devices ?

XXX : Security Intelligence URL: memcap exceeded (loaded 2167178 of 2939377)

This started showing up since this saturday, with no change to any policy/configuration, and only for low memory/older devices (aka ASA 5516 running FTD/Firepower 1010). This is not afecting Firepower 1120 or above models.

One interesting observation is that it seems like feed is constantly growing by each day:

Time: Sat Dec 24 04:54:44 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2317133)

Time: Sat Dec 24 20:39:59 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2354548)

Time: Sun Dec 25 04:33:19 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2365381)

Time: Sun Dec 25 20:19:23 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2413343)

Time: Mon Dec 26 04:14:15 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2444498)

Time: Mon Dec 26 19:59:33 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2612033)

Time: Tue Dec 27 03:49:22 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2667956)

Time: Tue Dec 27 19:37:55 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2891657)

Time: Wed Dec 28 03:32:46 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2939377)

Re: Security Intelligence URL: memcap exceeded

MHM Cisco World — Wed, 28 Dec 2022 11:14:15 GMT

https://www.lammle.com/post/cisco-firepower-security-intelligence-error-url-memcap-exceeded/

Re: Security Intelligence URL: memcap exceeded

Mark Elsen — Wed, 28 Dec 2022 11:25:03 GMT

- Review these bug reports : https://bst.cloudapps.cisco.com/bugsearch?pf=prdNm&kw=memcap%20exceeded&bt=custV&sb=anfr

Re: Security Intelligence URL: memcap exceeded

kicmar — Wed, 28 Dec 2022 11:29:45 GMT

I am asking mainly to check if anybody is seeing this error since last few days or it's just us ?

I am aware of this article and older bugs, and have some thought that maybe this time it is also issue with SI feeds that hotfix from Cisco will resolve. Cause number of entries seems to be growing by each day by hundreds of thousands.

Re: Security Intelligence URL: memcap exceeded

Marvin Rhoads — Wed, 28 Dec 2022 12:49:36 GMT

For what it's worth, I have a pair of ASA5516-X running FTD 7.0.4 with the latest SI feeds etc and they are not showing this error.

Re: Security Intelligence URL: memcap exceeded

mcw1217 — Wed, 28 Dec 2022 18:43:29 GMT

We encountered this, it happened this past few days and there's no changes from our end. Do you have any idea about this?

x.x.x.x : Security Intelligence URL: memcap exceeded (loaded 2176989 of 3113153)

Re: Security Intelligence URL: memcap exceeded

Stephen Hess — Thu, 29 Dec 2022 03:38:44 GMT

We are seeing the same issue, and it seems to be a likely cause of a severe outage for us actually. Our 5525s (with virtual sensors, not running FTD directly) are configured for whitelist-only web access, and it seems that the Talos feed gets pushed to the device before custom feeds - so our whitelist is not getting applied. Wonderful to have business systems failing because they cannot get to Azure...

Re: Security Intelligence URL: memcap exceeded

Marvin Rhoads — Thu, 29 Dec 2022 12:59:58 GMT

I did see the same error on a customer yesterday. They were running an HA pair of ASA5545-X with Firepower service module version 6.6.5.

Re: Security Intelligence URL: memcap exceeded

drobson — Thu, 29 Dec 2022 13:41:16 GMT

Seeing this same issue on our ASA5516-X running version 6.6.7. Also have the issue with our whitelists not getting applied for users that are set for whitelist-only access.

Re: Security Intelligence URL: memcap exceeded

kicmar — Thu, 29 Dec 2022 13:53:33 GMT

So based on our Cisco TAC update, this is the reason:

"TALOS team performed a large update to the Security Intelligence feeds and made them significantly larger."

Solutions proposed so far are:

1. Lower number of feeds in SI

2. Disable URL filtering if it is being used in same policy (of course it is - c'mon Cisco, otherwise what's the point of using NGFW's)

3. Install bigger/better/newer platform (which is a problem cause this is affecting FPR1010 as well).

Re: Security Intelligence URL: memcap exceeded

mcw1217 — Thu, 29 Dec 2022 16:53:08 GMT

Hello Kicmar, Where did you get this information? Would you mind posting the link here. Thanks!

Re: Security Intelligence URL: memcap exceeded

Stephen Hess — Fri, 30 Dec 2022 00:31:29 GMT

On my TAC call last night, the engineer shared that most of the relevant engineering team is off until the 3rd due to the holidays. That leads me to suspect that *maybe* this issue is due to an automated feed addition that has gone rouge over the past week. For now, disabling some of the URL feeds is the best option available for those using custom whitelist feeds.

Re: Security Intelligence URL: memcap exceeded

kicmar — Mon, 02 Jan 2023 10:20:01 GMT

I get this info from TAC engineer, while working on case. I have asked for official documentation on this change, so far we didn't get it.

At the same time, I am leaning toward interpretation from Stephen Hess more - automated feeds have gone rough over last week, rather than it is a planned activity.

Let's say that before, feeds consisted of ~2 million entries. We haven't gotten any warnings, everyhing has been working fine for years.

Issue started when devices were not able to load more than ~2.2 million entires:

Time: Sat Dec 24 04:54:44 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2317133)

Let's assume that this was indeed planned activity, and Talos made feeds bigger. The open question then becomes - why this "making feeds bigger" happens constantly over and over.

On 24th feeds have 2317133 entires, on 28th feeds have 2939377 entires, and just today our warning shows that feeds are having 3521854 entries. That would mean that feeds grown by 50% just over last 7 days, sicne they were expected to be bigger, and with this speed, this would be above the limit for newer platforms as well.

I am expecting more that relevant engineering team went off till 3rd Jan and hopefully this will fix itself when they return.

Re: Security Intelligence URL: memcap exceeded

dpeldo22 — Tue, 03 Jan 2023 14:24:15 GMT

I too came in today to find this message. I can confirm 100% that this error was not showing up on Sunday and is sitting at "loaded 1914588 of 33533414 on my 5516-x running 7.0.4 fmc & ftd. I would rather not shut off SI feeds to fix this.

Re: Security Intelligence URL: memcap exceeded

TCSPB — Tue, 03 Jan 2023 16:48:03 GMT

I'm showing the same issues on all of our 1010's today.

Security Intelligence URL: memcap exceeded (loaded 1913477 of 3535927)

Is there a workaround for this? We are running 7.0.4 on FMC and FTD's.

Re: Security Intelligence URL: memcap exceeded

Stephen Hess — Tue, 03 Jan 2023 16:54:45 GMT

No real workaround, only option is to reduce what feeds you are using in your access policy. And open a TAC case to get them to prioritize this ASAP. Users with 1010 + models are going to be able to get past TAC blaming it on old hardware not being able to handle things.

Re: Security Intelligence URL: memcap exceeded

casnetadmin — Tue, 03 Jan 2023 20:03:02 GMT

I am also seeing these errors which brought me to this page. Started this weekend.

Re: Security Intelligence URL: memcap exceeded

kicmar — Wed, 04 Jan 2023 11:06:09 GMT

I am really leaning toward saying that something went wrong with feeds and is causing this problem.

You can check feeds on your own:

find /var/sf/siurl_download/. -name '*' | xargs wc -l - this commands shows which feeds consumes most entries.

That allows to check specific feeds content. In this checking, it can be seen that some of the domains are duplicated, while for others we have main domain, as well as subdomain entries.

Ideally - if more people raise a TAC case for this, maybe this will get necessary attention on proper team from Cisco side.

Examples are below:

cat /var/sf/siurl_download/./23f2a124-8278-4c03-8c9d-d28fe08b8fc9.lf | grep stilogo.it
stilogo.it/mobile/authentication/
stilogo.it/mobile/authentication

cat /var/sf/siurl_download/./23f2a124-8278-4c03-8c9d-d28fe08b8fc9.lf | grep csgo-collect.com
csgo-collect.com
csgo-collect.com/twltch
csgo-collect.com/twitch
csgo-collect.com/get
csgo-collect.com/g
csgo-collect.com/cdn-cgi/phish-bypass?atok=_imlix8xsgqllgsckwlnfys6mwbw0ugftlguymful8s-1671136044-0-/g
csgo-collect.com/go

Re: Security Intelligence URL: memcap exceeded

razelle — Wed, 04 Jan 2023 15:32:57 GMT

I've been seeing this issue with my FMCv managed FP1010 running 7.0.5. for almost 2 weeks. I'm waiting to see if there is a resolution posted before I contact TAC

Re: Security Intelligence URL: memcap exceeded

jeckert — Thu, 05 Jan 2023 15:21:12 GMT

I am seeing this as well across all of my customers on versions from 6.5 .5 to 7.1.2