03-04-2020 09:18 PM
Dear all. i configured external threat feeds on ESA. I use HalwaiTaxii and AilenVault open source TAXII servers. Hailwai is synchronizing successfully every 1 hour but AlienVault is not? When i manually poll it it successfully fetches feeds. I let it to be updated every 1 hour automatically but it fails. When i look at threatfeed logs observed as below
Thu Mar 5 04:23:46 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Thu Mar 5 04:23:46 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: xxxxxxx
Thu Mar 5 04:23:46 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-05 04:23:46.092405
Thu Mar 5 04:30:39 2020 Warning: THREAT_FEEDS: Unable to fetch the observables from the source: AlienVault after 3 failed attempts. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
03-05-2020 01:36 AM
We have tested with following config with AilenVault.
collection: user_AlienVault
username : <created in account>
password : <empty> <<< for my account
use https : yes and Port 443
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: Adding the source: alienvault to the threat feeds database
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: alienvault
Thu Mar 5 09:29:28 2020 Debug: THREAT_FEEDS: The length of the full poll job queue is: 1
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: A full poll has started for the source: alienvault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: alienvault between 2020-02-04 09:29:27.928443 and 2020-03-05 09:29:27.928458
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: The external threat feeds engine has started
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: STIX Packages of 5011696 bytes size were fetched from the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: 70 STIX Packages were fetched from the source: alienvault
Thu Mar 5 09:30:04 2020 Info: THREAT_FEEDS: 3443 observables were fetched from the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 09:29:27.928458 for the last full poll for the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: Updating the partial download timestamp: None for the last full poll for the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 09:29:27.928458 for the last attempted poll for the source: alienvault
hopefully this helps
03-05-2020 02:06 AM
Hello Sriram
i have done the same. when we configure as you said it is successfully connects and fetches feeds. but when 1 hour interval reaches it is not able to connect to server. You can wait 1 hour to see whether it is successfully polled or not. please let me know.
03-05-2020 07:38 AM
Hi,
Can you share ESA ETF alienvault config and also can you share debug log of ETF when is issue happens.
i have just shared the output of my setup after 1hr, still don't see any issue. even though there is no threat info available at this movement, there is no polling error at the movement.
Thu Mar 5 10:29:29 2020 Debug: THREAT_FEEDS: The last full poll was done at: 2020-03-05 09:29:27.928458 for the source: alienvault
Thu Mar 5 10:29:29 2020 Debug: THREAT_FEEDS: The last attempted poll was done at: 2020-03-05 09:29:27.928458 for the source: alienvault
Thu Mar 5 10:29:29 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: alienvault
Thu Mar 5 10:29:29 2020 Debug: THREAT_FEEDS: The length of the delta poll job queue is: 1
Thu Mar 5 10:29:29 2020 Info: THREAT_FEEDS: A delta poll has started for the source: alienvault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 10:29:29 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: alienvault between 2020-03-05 08:29:27.928458 and 2020-03-05 10:29:28.415184
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: STIX Packages of 0 bytes size were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: 0 STIX Packages were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Info: THREAT_FEEDS: No new observables were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Info: THREAT_FEEDS: 0 observables were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 10:29:28.415184 for the last delta poll for the source: alienvault
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 10:29:28.415184 for the last attempted poll for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The last delta poll was done at: 2020-03-05 10:29:28.415184 for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The last full poll was done at: 2020-03-05 09:29:27.928458 for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The last attempted poll was done at: 2020-03-05 10:29:28.415184 for the source: alienvault
Thu Mar 5 11:29:29 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The length of the delta poll job queue is: 1
Thu Mar 5 11:29:29 2020 Info: THREAT_FEEDS: A delta poll has started for the source: alienvault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 11:29:29 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: alienvault between 2020-03-05 09:29:28.415184 and 2020-03-05 11:29:28.579657
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: STIX Packages of 0 bytes size were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: 0 STIX Packages were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Info: THREAT_FEEDS: No new observables were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Info: THREAT_FEEDS: 0 observables were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 11:29:28.579657 for the last delta poll for the source: alienvault
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 11:29:28.579657 for the last attempted poll for the source: alienvault
03-05-2020 07:54 AM
03-05-2020 09:31 PM
dear all
i just looked at status of service on ESA observed that 2 minutes ago it had been updated. i am bewildered know. sometimes it is not automatically updated but today it shows otherwise.
here is my configuration
Fri Mar 6 02:20:23 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Fri Mar 6 02:20:23 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 02:20:23 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-06 02:20:23.588518
Fri Mar 6 02:24:25 2020 Warning: THREAT_FEEDS: Unable to fetch the observables from the source: AlienVault after 3 failed attempts. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 02:24:25 2020 Info: THREAT_FEEDS: Job failed with exception : Source: AlienVault. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 03:20:23 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Fri Mar 6 03:20:23 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 03:20:23 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-06 03:20:23.594613
Fri Mar 6 03:24:26 2020 Warning: THREAT_FEEDS: Unable to fetch the observables from the source: AlienVault after 3 failed attempts. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 03:24:26 2020 Info: THREAT_FEEDS: Job failed with exception : Source: AlienVault. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 04:20:24 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Fri Mar 6 04:20:24 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 04:20:24 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-06 04:20:23.985923
Fri Mar 6 04:21:15 2020 Info: THREAT_FEEDS: 11233 observables were fetched from the source: AlienVault
Fri Mar 6 04:21:44 2020 Info: THREAT_FEEDS: 19068 observables were fetched from the source: AlienVault
Fri Mar 6 04:22:17 2020 Info: THREAT_FEEDS: 26863 observables were fetched from the source: AlienVault
Fri Mar 6 04:23:02 2020 Info: THREAT_FEEDS: 42256 observables were fetched from the source: AlienVault
Fri Mar 6 04:23:35 2020 Info: THREAT_FEEDS: 48591 observables were fetched from the source: AlienVault
Fri Mar 6 04:23:57 2020 Info: THREAT_FEEDS: 52794 observables were fetched from the source: AlienVault
Fri Mar 6 04:24:28 2020 Info: THREAT_FEEDS: 59476 observables were fetched from the source: AlienVault
Fri Mar 6 04:24:51 2020 Info: THREAT_FEEDS: 62693 observables were fetched from the source: AlienVault
Fri Mar 6 04:25:18 2020 Info: THREAT_FEEDS: 66223 observables were fetched from the source: AlienVault
Fri Mar 6 04:25:40 2020 Info: THREAT_FEEDS: 69919 observables were fetched from the source: AlienVault
Fri Mar 6 04:25:59 2020 Info: THREAT_FEEDS: 71595 observables were fetched from the source: AlienVault
Fri Mar 6 05:20:24 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: AlienVault
Fri Mar 6 05:20:24 2020 Info: THREAT_FEEDS: A delta poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 05:20:24 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2020-03-06 03:20:23.985923 and 2020-03-06 05:20:24.424532
*********the rest of logs not added here
03-05-2020 10:12 PM
Hi,
I've seen this kind of errors when integrating with third-party-providers for some kind of updates. To avoid issues on your side, try to upgrade to a stable ESA version.
Regards,
Cristian Matei.
03-05-2020 10:52 PM
Which version do you recommend. currently i use 13 version
03-06-2020 09:26 AM
ETF code is not changed from 12.0 release till now.
Also as per the logs, ETF is fetched lot of observables. everything looks good now.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: