Tidal check for files that are in a certain format

[aknight03]

Hello,

I am new to Tidal and I have been able to get a simple job to work that simply logs into our client's SFTP server, gets all files and brings them back to our system. Since SFTP is smart enough to know if we have a duplicate file, using wildcards has proved as an easy way to get this job started.

However, we certainly would rather have this job running a lot tighter and error proof for the future. So I figured I would come here to ask a few questions and see if anyone could provide me with some assistance on getting this script running a lot better.

Basically, we need to check this client's site every four hours for a file that they will be uploading daily. They don't put it on the site at the same time everyday, so that makes things a tad bit more difficult, but certainly not a big deal. The format of the file is always going to be the same and this is where I wanted some assistance.

The file will always be in the format of yyyymmdd_this_is_same_every_time.gz

So I basically just need a way to define a varaible in Tidal for this file that will take into consideration the date stamp of the file. Grab that certain file and then move it to an archive directory. We then plan on deleting files that are over a week old in the archive, but I feel like that is the easier task here.

Again, any assistance would be appreciated.

Thanks guys.

[Joe Fletcher]

You can define date format variables under the variables tab in you job/group definition. So, click variables tab, then variables button at the bottom, select "System variables" then "System Date" and click " Date Format". Fill in the box then once you've created and named it you can substitute the value into the job at runtime. Thus in your case your sftp get will look like  <what_I_called_my_date_var>_this_is_the_same.gz.

Cheers

[aknight03]

Thank you for the quick reply, this seems like it will do exactly what I need it to do.

[aknight03]

A follow up question on this. Our client is putting out files on their site from the previous day, so the issue I am running into is that Tidal is using the current system date as the search variable, but the file will actually have the previous days date. Is there anyway to tell Tidal to use the previous date? I didn't see any option for this in the variables menu.

Thanks

[Joe Fletcher]

If you go to the variable definition item, you can define a variable of date and check the "calculated" box. This allows you to apply various offsets such as +1, -1, previous working day and so on. Judicious combination of these offsets can give you all sorts of capabilities like "previous working day unless its the 29th February and unless its the year 2025 but it is my birthday".

"Working day -1 except if its Monday so last working day was Friday" is a common one for people to use.

Just remember you are now looking in the variables definitions menu ie outside the job definition as discussed above.

Cheers