Buy or Renew
Buy or Renew
Duo Security forums now LIVE! Get answers to all your Duo Security questions. Learn more
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
4980
Views
15
Helpful
8
Replies

Cisco ESA External Threat Feeds

ccna_security
Level 3
Level 3

‎03-04-2020 09:18 PM

Dear all. i configured external threat feeds on ESA. I use HalwaiTaxii and AilenVault open source TAXII servers. Hailwai is synchronizing successfully every 1 hour but AlienVault is not? When i manually poll it it successfully fetches feeds. I let it to be updated every 1 hour automatically but it fails. When i look at threatfeed logs observed as below

 

Thu Mar 5 04:23:46 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Thu Mar 5 04:23:46 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: xxxxxxx
Thu Mar 5 04:23:46 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-05 04:23:46.092405
Thu Mar 5 04:30:39 2020 Warning: THREAT_FEEDS: Unable to fetch the observables from the source: AlienVault after 3 failed attempts. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler

Labels:
0 Helpful
8 Replies 8

SriramV
Cisco Employee
Cisco Employee

‎03-05-2020 01:36 AM

We have tested with following config with AilenVault.

 

Hostname: otx.alienvault.com

Polling Path : /taxii/poll

collection: user_AlienVault

username : <created in account>

password : <empty>     <<< for my account 

use https : yes and Port 443

 

Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: Adding the source: alienvault to the threat feeds database
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: alienvault
Thu Mar 5 09:29:28 2020 Debug: THREAT_FEEDS: The length of the full poll job queue is: 1
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: A full poll has started for the source: alienvault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: alienvault between 2020-02-04 09:29:27.928443 and 2020-03-05 09:29:27.928458
Thu Mar 5 09:29:28 2020 Info: THREAT_FEEDS: The external threat feeds engine has started
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: STIX Packages of 5011696 bytes size were fetched from the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: 70 STIX Packages were fetched from the source: alienvault
Thu Mar 5 09:30:04 2020 Info: THREAT_FEEDS: 3443 observables were fetched from the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 09:29:27.928458 for the last full poll for the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: Updating the partial download timestamp: None for the last full poll for the source: alienvault
Thu Mar 5 09:30:04 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 09:29:27.928458 for the last attempted poll for the source: alienvault

 

hopefully this helps

0 Helpful

ccna_security
Level 3
Level 3
In response to SriramV

‎03-05-2020 02:06 AM

Hello Sriram

 

i have done the same. when we configure as you said it is successfully connects and fetches feeds. but when 1 hour interval reaches it is not able to connect to server. You can wait 1 hour to see whether it is successfully polled or not. please let me know.

0 Helpful

SriramV
Cisco Employee
Cisco Employee
In response to ccna_security

‎03-05-2020 07:38 AM

Hi,

 

Can you share ESA ETF alienvault config and also can you share debug log of ETF when is issue happens.

 

i have just shared the output of my setup after 1hr, still don't see any issue. even though there is no threat info available at this movement, there is no polling error at the movement. 

 

Thu Mar 5 10:29:29 2020 Debug: THREAT_FEEDS: The last full poll was done at: 2020-03-05 09:29:27.928458 for the source: alienvault
Thu Mar 5 10:29:29 2020 Debug: THREAT_FEEDS: The last attempted poll was done at: 2020-03-05 09:29:27.928458 for the source: alienvault
Thu Mar 5 10:29:29 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: alienvault
Thu Mar 5 10:29:29 2020 Debug: THREAT_FEEDS: The length of the delta poll job queue is: 1
Thu Mar 5 10:29:29 2020 Info: THREAT_FEEDS: A delta poll has started for the source: alienvault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 10:29:29 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: alienvault between 2020-03-05 08:29:27.928458 and 2020-03-05 10:29:28.415184
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: STIX Packages of 0 bytes size were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: 0 STIX Packages were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Info: THREAT_FEEDS: No new observables were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Info: THREAT_FEEDS: 0 observables were fetched from the source: alienvault
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 10:29:28.415184 for the last delta poll for the source: alienvault
Thu Mar 5 10:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 10:29:28.415184 for the last attempted poll for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The last delta poll was done at: 2020-03-05 10:29:28.415184 for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The last full poll was done at: 2020-03-05 09:29:27.928458 for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The last attempted poll was done at: 2020-03-05 10:29:28.415184 for the source: alienvault
Thu Mar 5 11:29:29 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: alienvault
Thu Mar 5 11:29:29 2020 Debug: THREAT_FEEDS: The length of the delta poll job queue is: 1
Thu Mar 5 11:29:29 2020 Info: THREAT_FEEDS: A delta poll has started for the source: alienvault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 11:29:29 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: alienvault between 2020-03-05 09:29:28.415184 and 2020-03-05 11:29:28.579657
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: STIX Packages of 0 bytes size were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: 0 STIX Packages were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Info: THREAT_FEEDS: No new observables were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Info: THREAT_FEEDS: 0 observables were fetched from the source: alienvault
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 11:29:28.579657 for the last delta poll for the source: alienvault
Thu Mar 5 11:29:30 2020 Debug: THREAT_FEEDS: Updating the timestamp: 2020-03-05 11:29:28.579657 for the last attempted poll for the source: alienvault

0 Helpful

charella
Cisco Employee
Cisco Employee

‎03-05-2020 07:54 AM

Hello ccna_security,

I just went through the chore of creating an account for for otx.alienvault.com.
Creating it on ESA in 2 different environments
It worked nicely with no failure

My settings
Source Name: user_AlienVault
Hostname: otx.alienvault.com
Polling Path: /taxi/poll
Collection Name: user_AlienVault
Polling Interval: 1 Hour
Age of Threat Feeds: 30 days
Time San of Poll Segment: 30 Days
Use HTTPS: YES
Polling port 443
Configure User Credentials: Yes
Username = my API KEY
Password = blank
My proxy settings are : NO


Thu Mar 5 09:24:32 2020 Info: THREAT_FEEDS: Adding the source: user_AlienVault to the threat feeds database
Thu Mar 5 09:24:32 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: user_AlienVault
Thu Mar 5 09:24:32 2020 Info: THREAT_FEEDS: A full poll has started for the source: user_AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 09:24:32 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: user_AlienVault between 2020-02-04 09:24:32.838958 and 2020-03-05 09:24:32.838971
Thu Mar 5 09:25:03 2020 Info: THREAT_FEEDS: 3443 observables were fetched from the source: user_AlienVault
Thu Mar 5 10:24:33 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: user_AlienVault
Thu Mar 5 10:24:33 2020 Info: THREAT_FEEDS: A delta poll has started for the source: user_AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Thu Mar 5 10:24:33 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: user_AlienVault between 2020-03-05 08:24:32.838971 and 2020-03-05 10:24:33.247384
Thu Mar 5 10:24:34 2020 Info: THREAT_FEEDS: No new observables were fetched from the source: user_AlienVault
Thu Mar 5 10:24:34 2020 Info: THREAT_FEEDS: 0 observables were fetched from the source: user_AlienVault


Options:

1. Delete the External Threat Feed profile and recreate.
* Just in case something isn’t right with it in the system. (doubtful)
2. Check your network environment for path restrictions or inspections/decryptions on the traffic.

Not all feeds are identical.
If I can perform the action on multiple hosts with different versions, it may not be the ESA as a source of the problem. (If you are using the same settings).




For the Forum, I did this to get the user account and API key.

>>> this requires registration using your email account. Once registered the account receives an API key to use as the username.
Navigate to the API Integration tab (top of page) > There are subtle transparent tabs with blue font above the icons, select TAXII.

https://otx.alienvault.com/api
Poll Address: https://otx.alienvault.com/taxii/poll

Collection Name: user_AlienVault

Collection Type: DATA_FEED

Available: True

Collection Description: Data feed for user: AlienVault

Supported Content: All


Collection Name: group_999

Collection Type: DATA_FEED

Available: True

Collection Description: Data feed for group: id=999 name=Secret Group #1

Supported Content: All


ccna_security
Level 3
Level 3
In response to charella

‎03-05-2020 09:31 PM

dear all

i just looked at status of service on ESA observed that 2 minutes ago it had been updated. i am bewildered know. sometimes it is not automatically updated but today it shows otherwise.

here is my configuration 

 

1.JPG

Fri Mar 6 02:20:23 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Fri Mar 6 02:20:23 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 02:20:23 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-06 02:20:23.588518
Fri Mar 6 02:24:25 2020 Warning: THREAT_FEEDS: Unable to fetch the observables from the source: AlienVault after 3 failed attempts. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 02:24:25 2020 Info: THREAT_FEEDS: Job failed with exception : Source: AlienVault. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 03:20:23 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Fri Mar 6 03:20:23 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 03:20:23 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-06 03:20:23.594613
Fri Mar 6 03:24:26 2020 Warning: THREAT_FEEDS: Unable to fetch the observables from the source: AlienVault after 3 failed attempts. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 03:24:26 2020 Info: THREAT_FEEDS: Job failed with exception : Source: AlienVault. Reason for failure: Taxii Error: FAILURE: There was a failure while executing the message handler
Fri Mar 6 04:20:24 2020 Info: THREAT_FEEDS: A full poll is scheduled for the source: AlienVault
Fri Mar 6 04:20:24 2020 Info: THREAT_FEEDS: A full poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 04:20:24 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2019-03-16 12:51:38 and 2020-03-06 04:20:23.985923
Fri Mar 6 04:21:15 2020 Info: THREAT_FEEDS: 11233 observables were fetched from the source: AlienVault
Fri Mar 6 04:21:44 2020 Info: THREAT_FEEDS: 19068 observables were fetched from the source: AlienVault
Fri Mar 6 04:22:17 2020 Info: THREAT_FEEDS: 26863 observables were fetched from the source: AlienVault
Fri Mar 6 04:23:02 2020 Info: THREAT_FEEDS: 42256 observables were fetched from the source: AlienVault
Fri Mar 6 04:23:35 2020 Info: THREAT_FEEDS: 48591 observables were fetched from the source: AlienVault
Fri Mar 6 04:23:57 2020 Info: THREAT_FEEDS: 52794 observables were fetched from the source: AlienVault
Fri Mar 6 04:24:28 2020 Info: THREAT_FEEDS: 59476 observables were fetched from the source: AlienVault
Fri Mar 6 04:24:51 2020 Info: THREAT_FEEDS: 62693 observables were fetched from the source: AlienVault
Fri Mar 6 04:25:18 2020 Info: THREAT_FEEDS: 66223 observables were fetched from the source: AlienVault
Fri Mar 6 04:25:40 2020 Info: THREAT_FEEDS: 69919 observables were fetched from the source: AlienVault
Fri Mar 6 04:25:59 2020 Info: THREAT_FEEDS: 71595 observables were fetched from the source: AlienVault
Fri Mar 6 05:20:24 2020 Info: THREAT_FEEDS: A delta poll is scheduled for the source: AlienVault
Fri Mar 6 05:20:24 2020 Info: THREAT_FEEDS: A delta poll has started for the source: AlienVault, domain: otx.alienvault.com, collection: user_AlienVault
Fri Mar 6 05:20:24 2020 Info: THREAT_FEEDS: Observables are being fetched from the source: AlienVault between 2020-03-06 03:20:23.985923 and 2020-03-06 05:20:24.424532

*********the rest of logs not added here

0 Helpful

Cristian Matei
VIP Alumni
VIP Alumni
In response to ccna_security

‎03-05-2020 10:12 PM

Hi,

 

    I've seen this kind of errors when integrating with third-party-providers for some kind of updates. To avoid issues on your side, try to upgrade to a stable ESA version.

 

Regards,

Cristian Matei.

ccna_security
Level 3
Level 3
In response to Cristian Matei

‎03-05-2020 10:52 PM

Which version do you recommend. currently i use 13 version

0 Helpful

SriramV
Cisco Employee
Cisco Employee
In response to ccna_security

‎03-06-2020 09:26 AM

ETF code is not changed from 12.0 release till now. 

Also as per the logs, ETF is fetched lot of observables. everything looks good now.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents