Stealthwatch API flows query max. results 10,000?

[MJonkers]

Hi,

I'm using Python with Pandas to alter the results I get from the following snippet of code. I do get max 10,000 records back but there are more records. Is there another solution so I get all the flows back from the last 5 minutes?

URL = 'https://' + SMC_HOST + '/sw-reporting/v2/tenants/' + SMC_TENANT_ID + '/flows/queries'

    # Set the pe timestamps for the filters, in the correct format, for last 5 minutes
    end_datetime = datetime.datetime.utcnow()
    start_datetime = end_datetime - datetime.timedelta(minutes=5)
    end_timestamp = end_datetime.strftime('%Y-%m-%dT%H:%M:%SZ')
    start_timestamp = start_datetime.strftime('%Y-%m-%dT%H:%M:%SZ')

    # Set the filter with the request data
    request_data = {
        "startDateTime": start_timestamp,
        "endDateTime": end_timestamp,
        "recordLimit": 100000,
        "subject": {
            "ipAddresses": {
                "includes": [],
                "excludes": []
            },
        },
        "flow": {
          "includeInterfaceData": "false"
        },
    }

    # Perform the query to initiate the search
    request_headers = {'Content-type': 'application/json', 'Accept': 'application/json'}
    response = api_session.request("POST", URL, verify=False, data=json.dumps(request_data), headers=request_headers)  

[Lionel VERSCHEURE]

Hello,

I have got the same limitation with security API /tenants/{tenantId}/security-events/queries with a limit of 2 000.

Extract from API documentation (https://developer.cisco.com/docs/stealthwatch/enterprise/#!reporting-api-version-1) : Results are limited to 2,000 records. If you reach this limit, narrow your request so it returns a more inclusive data set.

Cisco support team tell me this limit is by design. After I open a case and several talks with different Cisco people, they finally accept to open a new feature request : to have the whole result and not only the 2 000 first one.

I am waiting for this new feature and for now, to avoid this limit, I use 2 methods :

• Request a shorter time range the API
• Limit the request by security event type by ID instead of the whole security event type.

Your trouble is not exactly the same but the 10 000 limit seams by design and not expandable too (https://developer.cisco.com/docs/stealthwatch/enterprise/#!reporting-api-version-2) : Limit your number of search results to the minimum needed to accomplish your task. The maximum number of results returned is 10,000.

Regards

---

[MJonkers]

Hi Lionel,

Thank you for the reply. I tried shortening the time range, but because we have a large network I get +60K flows when I ask for a 5 minute window, which seems the minimum (5 min windows) and I only get 10K results. These sort of tools need to accomodate us and not introduce limits. We have nfsen running, where I can easily get the large number of flows with nfdump. The networking department wants to phase out nfsen and use stealthwatch for that. Cisco needs to think when they introduce a new product and make the data within easily accessable. We analysts need the info stored in the product. When I ask for 1 milion records it needs to deliver. I asked the network engineers to open a TAC case. Let's see where it takes us.

Kind regards Marc

[benugent]

Marc,

Did you ever hear back on this?  Did a TAC case get opened and was there any resolution?  Thanks!

Beth

[MJonkers]

Morning (I'm in Europe ;-)) Beth,

TAC provided me with a workaround, scheduling a csv export from the desktop software and download the csv from the appliance. I can work with that as a temp solution, but have asked for a rfc so I can use the Python script with the API call. 

Kind regards 

Marc