ETF The storage limit exceeded

[Steflstefan]

Hello

sometimes I recieve a warning from ESA

The Warning message is:

THREAT_FEEDS: The storage limit of 1250000 observables exceeded for the observable type: URL

I have two ETF souces configured; Haila_phishtank and OTX Alienvault.

Is there a option to increase the storage for ETF observable URL?
Maybe, the same URLs stored two times, one from Haila and the other from Alienvault? Could that be?

Can I have a look into ETF URL store or is there an option to download the stored data for further investigation?

All in all, we are satisfied with the possibility to discover URLs using the ETF sources.

Regards
Stefan 

[SriramV]

ETF Feature was brought to mitigate with very recent Threat or to use Internal Threat source. Mostly within a few hours/ days, the threat is updated in AV, antispam or other engines in ESA.

Even though we has provided 356 days of feed data to poll, it's not recommended configure it unless its needed. 

Answering your questions.

For a same source, ESA does not save duplicate URL's from feeds.

ESA stores an URL in 3 formats, <http and https>://<url> and only URL without protocol. 3 entries will be added in DB table for a URL. its done to increase search performance.

Currently we don't have any option increase DB limits. Can request for enhancement.

[Steflstefan]

Hello Siram,

Thank you for your feedback.

The following is currently configured:
Age of Threat Feeds: 10
Time Span of Poll Segment: 10
Should this be reduced?

About the duplicates:
If I understand it correctly, the URL http://bad.hacker.com, for example, is stored twice if it is in Haila's feed and also in Alienvault's feed.
The URLs are not correlated across all ETF sources?

Regards Stefan