Security Intelligence URL: memcap exceeded

[kicmar]

Hi 

Does anybody else noticed this warning started this weekend on their devices ? 

XXX : Security Intelligence URL: memcap exceeded (loaded    2167178 of    2939377)

This started showing up since this saturday, with no change to any policy/configuration, and only for low memory/older devices (aka ASA 5516 running FTD/Firepower 1010). This is not afecting Firepower 1120 or above models. 

One interesting observation is that it seems like feed is constantly growing by each day:

Time: Sat Dec 24 04:54:44 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2317133)

Time: Sat Dec 24 20:39:59 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2354548)

Time: Sun Dec 25 04:33:19 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2365381)

Time: Sun Dec 25 20:19:23 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2413343)

Time: Mon Dec 26 04:14:15 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2444498)

Time: Mon Dec 26 19:59:33 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2612033)

Time: Tue Dec 27 03:49:22 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2667956)

Time: Tue Dec 27 19:37:55 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2891657)

Time: Wed Dec 28 03:32:46 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2939377)

[MHM Cisco World]

https://www.lammle.com/post/cisco-firepower-security-intelligence-error-url-memcap-exceeded/

[marce1000]

 - Review these bug reports : https://bst.cloudapps.cisco.com/bugsearch?pf=prdNm&kw=memcap%20exceeded&bt=custV&sb=anfr

 M.

[kicmar]

I am asking mainly to check if anybody is seeing this error since last few days or it's just us ? 

I am aware of this article and older bugs, and have some thought that maybe this time it is also issue with SI feeds that hotfix from Cisco will resolve. Cause number of entries seems to be growing by each day by hundreds of thousands. 

[Marvin Rhoads]

For what it's worth, I have a pair of ASA5516-X running FTD 7.0.4 with the latest SI feeds etc and they are not showing this error.

[Marvin Rhoads]

I did see the same error on a customer yesterday. They were running an HA pair of ASA5545-X with Firepower service module version 6.6.5.

[casnetadmin]

I am also seeing these errors which brought me to this page. Started this weekend. 

[mcw1217]

We encountered this, it happened this past few days and there's no changes from our end. Do you have any idea about this?

x.x.x.x : Security Intelligence URL: memcap exceeded (loaded 2176989 of 3113153)

[mcw1217]

Does anyone still have issue? The error is gone on our FMC. Is there any announcement from Cisco that they already resolved the issue?

[Stephen Hess]

We are seeing the same issue, and it seems to be a likely cause of a severe outage for us actually. Our 5525s (with virtual sensors, not running FTD directly) are configured for whitelist-only web access, and it seems that the Talos feed gets pushed to the device before custom feeds - so our whitelist is not getting applied. Wonderful to have business systems failing because they cannot get to Azure...

[drobson]

Seeing this same issue on our ASA5516-X running version 6.6.7. Also have the issue with our whitelists not getting applied for users that are set for whitelist-only access.

[kicmar]

So based on our Cisco TAC update, this is the reason:

"TALOS team performed a large update to the Security Intelligence feeds and made them significantly larger." 

Solutions proposed so far are:

1. Lower number of feeds in SI 

2. Disable URL filtering if it is being used in same policy (of course it is - c'mon Cisco, otherwise what's the point of using NGFW's) 

3. Install bigger/better/newer platform (which is a problem cause this is affecting FPR1010 as well). 

[mcw1217]

Hello Kicmar, Where did you get this information? Would you mind posting the link here. Thanks!

[kicmar]

I get this info from TAC engineer, while working on case. I have asked for official documentation on this change, so far we didn't get it. 

At the same time, I am leaning toward interpretation from Stephen Hess more - automated feeds have gone rough over last week, rather than it is a planned activity. 

Let's say that before, feeds consisted of ~2 million entries. We haven't gotten any warnings, everyhing has been working fine for years. 

Issue started when devices were not able to load more than ~2.2 million entires: 

Time: Sat Dec 24 04:54:44 2022 UTC - Security Intelligence URL: memcap exceeded (loaded XXX of 2317133)

Let's assume that this was indeed planned activity, and Talos made feeds bigger. The open question then becomes - why this "making feeds bigger" happens constantly over and over. 

On 24th feeds have 2317133 entires, on 28th feeds have 2939377 entires, and just today our warning shows that feeds are having 3521854 entries. That would mean that feeds grown by 50% just over last 7 days, sicne they were expected to be bigger, and with this speed, this would be above the limit for newer platforms as well. 

I am expecting more that relevant engineering team went off till 3rd Jan and hopefully this will fix itself when they return. 

[Stephen Hess]

On my TAC call last night, the engineer shared that most of the relevant engineering team is off until the 3rd due to the holidays. That leads me to suspect that *maybe* this issue is due to an automated feed addition that has gone rouge over the past week. For now, disabling some of the URL feeds is the best option available for those using custom whitelist feeds.