File dependency issue

[vamsivaliveti.edu1]

I have tidal job with file dependency.The size(approximately:10MB) of file dependency is high.
currently my job is triggering when the file is present but the problem is job is triggering even before entire data is present in file.The file was palced by third part and i want to trigger the tidal job when entire file data is present.

I think one of the propery a)should Has Changed In DD:HH:MM or b)Size Stable For DD:HH:MM should be useful but not sure which property should be used

[Derrick Au]

Hi vamsivaliveti.edu1

A file event has the following options: [1] wait until file is at least this size (bytes), [2] milliseconds delay before processing, [3] file size stable for (milliseconds). In the case when the approximate size is 10 MB option [1] may not be a good choice because you will end up with partial data the action gets triggered as soon as minimum byte size is met; option [2] relies solely on time and may not be a good choice as there may intermittent LAN/WAN connectivity and you may also end up with partial data; option [3] waits for a file size to become stable for however many milliseconds specified. This would be a good choice but what happens if there is a connectivity issue and data stops transferring? To bulletproof this option, your process would have to be able to ingest the file and perform some file checks to ensure that all data is present.

BR,

Derrick Au

[vamsivaliveti.edu1]

1)Why should i use File event ?Is it possible to achieve same thing using File Dependency in a job?Just i want to trigger the job when entire data is present in a file.Please correct me if i am wrong since i am new to tidal

[Derrick Au]

Hi vamsivaliveti.edu1

Thanks for correcting me, yes you can use a File Dependency instead of a File Event. Both Has Changed in DDHHMM checks to see when the file was last changed (if the file is up to date and not stale by DDHHMM), whereas Stable for DDHHMM checks to see if all data has been received and file is stable for DDHHMM. Stable for DDHHMM would be the best option in your case. But as mentioned before, in order to bulletproof this option, your process should be able to ingest the file and perform some file checks to ensure that all data is present before any further processing :-)

BR,

Derrick Au