Tripwire IP360 Host Input API Connector

[dohurd]

This community supported utility is designed to run on a Sourcefire Defense Center and to connect to a Tripwire IP360 VnE server to pull the host, service, and vulnerability
data(*) into the Sourcefire Defense Center. It uses the Tripwire API v2 and
thus should support any Tripwire version 6.9 or later.  This utility was written and tested against Tripwire IP360 7.3.2.

[dev-zero_]

Any idea on following error: hit query rows limit 100000? Due to too many IPs? I've limited it to query=networks and limited the networks to little subnets, and even set the max_age to 1 day.

Reading the config (ip360_hi.conf)
Validating the configuration and setting up the environment
Creating User Agent
Creating the RPC client
 - Logging into IP360
 - Logged in
Validating requested IP360 networks
 - Querying IP360 for data from the Network class
Starting processing of IP360 ASPL
 - Found previous ASPL
 - Getting current ASPL details
 - ASPL file located at: https://<snip>
 - No need to update ASPL
Getting hosts, and the vulnerability and app results from IP360
 - Querying IP360 for data from the Host class
    * Query filter: timeStamp >= 1459296000 AND network IN ('Network.127','Network.125')
 - Querying IP360 for data from the Audit class
    * Query filter: endTime >= 1459296000 AND network IN ('Network.127','Network.125')
 - Querying IP360 for data from the AppResult class
Error querying IP360 API: ELIMIT: hit query rows limit 100000, the search may need to specify explicit limit and/or page results

Printing stack trace:
        called from ./ip360_hi.pl (571)
        called from ./ip360_hi.pl (684)
        called from ./ip360_hi.pl (1024

[josmith2013]

The easiest workaround is going to be to reduce the amount of data that you are holding in IP360. This is done by going to IP360 UI->Administer->System->Database->Archive->Delete Settings. The delete process runs 1x a day (default time is midnight), so you'll have to wait up to 24 hrs before it runs.

However, if you're in charge of a large environment, that's not always going to work since you might still have more than 100k rows in the appresult table even after reducing the amount of data in IP360. Unfortunately, the fix in the ip360_hi.pl script isn't just a couple of lines of code. As the error indicates, the API limits you to only retrieving 100k rows at a time.

The issue is that the script is trying to retrieve all appresult rows and then only filter out those that are associated with the hosts that are in scope. That's why limiting the max_age isn't helping you - it doesn't affect the appresult call at all.

In order for the script to work, it has to be modified in 1 of 2 ways:

1) Put the appresult fetch call into a loop and use the limit/offset arguments to retrieve batches of 100k rows. Note that this would increase script run time and some customers with very, very large environments may still run into issues if this uses up all available memory on the script's server.

2) Only retrieve appresults associated with the host records that are in scope using the argument query and filtering by host records. Depending on data, this may still require a limit/offset loop.

HTH - wish it were better news.

[dev-zero_]

updated line 568, and included 'limit' => '200000':

$result = $rpc_client->simple_request('call', $rpc_client->{_ip360_cookie}, 'class.'.$class, 'fetch', {'limit' => '200000' , 'format' => 'table'});

now successful. Thanks for the guidance provided.

[phil.hydev]

Any idea on this one? Conf file is configured to look at all objects:

Reading the config (ip360_hi.conf)
Validating the configuration and setting up the environment
Creating User Agent
Creating the RPC client
 - Logging into IP360
 - Logged in
Populating bits from the Host Input API
 - Source Type: 2, Source ID: 1000000
Starting processing of IP360 ASPL
 - Getting current ASPL details
<username> Use of uninitialized value $aspl_url in concatenation (.) or string at ip360_hi.pl line 474.
 - ASPL file located at: https://x.x.x.x
 - Downloading IP360 ASPL
Getting hosts, and the vulnerability and app results from IP360
 - Querying IP360 for data from the Host class
    * Query filter: timeStamp >= 1472428800
Can't use an undefined value as an ARRAY reference at ip360_hi.pl line 796.

Printing stack trace:
        called from ip360_hi.pl (796)
        called from ip360_hi.pl (574)
        called from ip360_hi.pl (682)
        called from ip360_hi.pl (1024)

[dohurd]

I'll send this to a few Cisco techs.  Might be able to make a suggestion to fix.

[phil.hydev]

Thank you.

[phil.hydev]

Do you know what Tripwire permissions are required for the API user?

Currently it has read-only access to a set of reports.