amp logs 'upload denied [...] queue full'

[daro]

Hello,

a file  was denied for the amp file analysis.
This is the output from the amp logs:

Thu Mar 2 10:52:39 2017 Info: File analysis upload skipped. SHA256: e24d3f1a17f44be8b750968fa3e8386d637bf14833a4dc82501e42119ad26bb8, file name: 573244.xlsTimestamp[1488448022] details[File SHA256[e24d3f1a17f44be8b750968fa3e8386d637bf14833a4dc82501e42119ad26bb8] upload denied, file mime[application/vnd.ms-excel], upload priority[High] queue full

I could not find this error message in the ESA user guide, so do you guys know what that means?


thanks
cheers

Daniel

[Libin Varghese]

Hi Daniel.

This error is normally ran into when you have exceeded the daily maximum files uploaded for ThreatGrid. When the limit is reached as you saw the file is not uploaded for analysis from ThreatGrid. We just rely on the File Reputation piece for the file.

I would recommend reaching out to your account team to discuss options on increasing the upload limit for your ESA appliances, if you hit the limit quite often.

Please review the below link to determine the upload limits based on the hardware model.

http://www.cisco.com/c/dam/en/us/td/docs/security/content_security/content_security_general/Content-security-file-reputation-and-analysis-criteria.pdf

Thank You!

Libin Varghese

[daro]

Hello,
are you sure? I can see a second line with the same sha256 and the following description:

Reason: Local analysis queue full

Wed Mar 29 17:57:04 2017 Info:  File analysis upload skipped. SHA256: c0f47865cb109c64e05eaf27c7538e59d1f158c1e9b2d792e45adad5a2ca4fa8, file name: F.11.3_Fassaden-LP_1826_SM Labin_rmX-29.03.2017.pdfTimestamp[1490802487] details[File SHA256[c0f47865cb109c64e05eaf27c7538e59d1f158c1e9b2d792e45adad5a2ca4fa8] upload denied, file mime[application/pdf], upload priority[Low] queue full

Wed Mar 29 17:57:04 2017 Info:  File not uploaded for analysis.  MID = 74266298 File SHA256[c0f47865cb109c64e05eaf27c7538e59d1f158c1e9b2d792e45adad5a2ca4fa8] file mime[application/pdf] Reason: Local analysis queue full

and I still get the follwing logs 3 hours LATER that same day:
Info:  File Analysis complete. [...] Details: Analysis is completed for the File

Wed Mar 29 21:30:33 2017 Info:  File Analysis complete. SHA256: 7b6f1ed49ac1942893f066bb6934a09ebf4a4693aad94f1d3c60a16db1af32e8, File name: Mileta Bewerbung2.doc, Submit Timestamp: 1490813311, Update Timestamp: 1490815567, Disposition: 1 Score: 0, run_id: 262946492 Details: Analysis is completed for the File SHA256[7b6f1ed49ac1942893f066bb6934a09ebf4a4693aad94f1d3c60a16db1af32e8]  Spyname:[None]

that would mean that files are still processed by the file analysis, right?
thanks

cheers
Daniel

[Libin Varghese]

There are two rate limiting reasons the local upload queue may become full- either you have reached your maximum uploads for the day, or a high volume of uploads are being made over a brief, sustained period.

When a file has a slot in the upload queue it will retry three times to upload while waiting for a daily limit to rollover or backoff to be resolved. The "upload denied... queue full" errors indicate that there was no slot open for this attachment so it would not have been uploaded.

So I would suspect you hit the per hour limit in this scenario.

- Libin V

[daro]

Hello,
so to be clear, "upload denied [...] queue full" does NOT indicate that the ThreatGrid upload limit has been reached?
the log message to tell that the upload limit has been reached would be something like this:
Sat Feb 6 13:22:56 2016 Info: File analysis upload skipped. SHA256:
b5c7e26491983baa713c9a2910ee868efd891661c6a0553b28f17b8fdc8cc3ef,Timestamp[1454782976]
details[File SHA256[b5c7e26491983baa713c9a2910ee868efd891661c6a0553b28f17b8fdc8cc3ef] file
mime[application/pdf], upload priority[Low] not uploaded, re-tries[3], backoff[986]
discarding ...]

is that correct?

if the "queue full" logs only appear if a local analysis/upload queue is full, what would be the solution to avoid that?

thank you
Daniel

[dmccabej]

Hello,

The 'queue full' messages are indicative of your appliances reaching the daily upload limit to ThreatGrid, which you can think of as a rolling 24/hr period. Local queue full alerts are most likely a secondary symptom to the same root cause.

As Libin suggested, you should reach out to your Account/Sales teams as they can provide licensing options to increase this limit if need be.

Thanks!

-Dennis M.

[daro]

Hello,
there is something not quite as expected.
As mentioned I received the 'queue full' message on WED 29 March 2017 quite a few times, 269 times to be exact. Looking into the SMA and the File Analysis reporting I can see that 190 files have been uploaded and sandboxed.

as we are using 2x C100V appliances in a cluster with AMP grouping, we should have 200 files/day across the appliances. The reporting features tells me that this limit has NOT been reached, but the specific files were definitely not uploaded due to 'queue full [...] Reason: Local analysis queue full'. That means 269 files should have been uploaded, but have been skipped (despite the fact that the file upload limit has not yet been reached).

so with the two facts that I can still upload files AFTER i get the 'queue full' message and with the SMA reporting stating that the upload limit has not been reached, I do not think that those 'queue full' messages are directly connected to the ThreatGrid upload limit.

so what exactly is the root cause for those messages?

thank you
cheers
Daniel

[dmccabej · Accepted solution]

Hello,

The 24/hr limit is more of a rolling window and not a daily cap. There was a recent email thread which provides a good explanation. (below - credit to another Cisco employee)

If you're looking for further confirmation I would recommend opening a TAC case.

Thanks!

-Dennis M.

+++

It is a rolling 24 hour window.   So if you want to put a line in the ground…

During any 24 hour period – time to time + 24 hours – they can hit no more than their total submission count.

If they have 1000 samples, and are brand new, never used, they submit 500 files quickly at 12:30p, then 400 more at 1:30p, then 100 at 2p,  the system would not accept more until 12:30p the next day as those 500 would  ‘drop out’ and from 12:30p to 1:30p, they would have 500 left.

+++

[daro]

Hello,

thank you very much,
the 'rolling window' was the missing and therefore confusing part.

cheers
Daniel

[dmccabej]

You're very welcome! I'm glad I could help clarify. :)

Thanks!

-Dennis M.