Tidal 5.3.1 - rerun same occurrence quetion

[frederickmspinney]

I have a job group with two jobs in the group. It has a time window of 04:00 AM - 07:00 PM and it will rerun the same occurrence every minute if the file dependency requirement in the first job is met. The second job has a dependency of the first job. The first job loads the file and the second job moves the file. If both these jobs run successfully this works fine. My problem is if the second job that moves the files completes abnormally because the file to be moved is in use my job just resets itself to waiting on dependencies and then the first job triggers off again after a minute because the file is still out there. The cause the same file to keep getting loaded over and over again. I have the rerun same occurrence defined in the group level and not in the jobs. So I trying to set it up so the first job won't continue to run over and over. Does anyone out here have any suggestions.

[Derrick Au]

Frederick,

How are you specifying the frequency rate of every minute? Are you sending files every minute? Whatever the case is, Job 1 to load file can be evaluated as the job that takes the longest time. Simply switch the ordering of Job 1 with Job 2, along with the file dependency added to Job 2, and you will be able to observe as soon as file gets moved, then job will load data.  And if you want to modify the behavior of how jobs are running, you can also select from the Options tab, if job is running, then select Defer Until Normal -- this way, only a single instance of the job can run at any given time or Run Anway can be selected if you wish to handle moving and load data for multiple files.You will also be able to observe that the Move Job will now have a status of completed normally (and will just re-run each time when file dependency is met); and the Load Job can now trigger correctly.

BR,

Derrick Au

[frederickmspinney]

Derreck,

I can not reverse the order that these jobs are running. The file must be loaded before it can be moved. If done in reverse order there would be no file for the job to load. The move job is just moving the file to another location so the load job doesn't continue to run because the same file is there.

I am specifying o n the schedule tag to rerun same occurrence, repeat every 1 minutes with a dependency of the file.

The problem is sometime the file can't be moved and the job completes abnormally but reset itslef. After a minute the move job run again because the same file is out there.

I don't see how setting the options tab to defer until normal will help.

Hopefully I am misunderstanding something you were trying to tell me.

Appreciate the assistance.

[Derrick Au]

Hi Frederick,

My initial thought is to change the load process and have it point to where the files get moved to, if reversing the sequence of these jobs is an option. So, I have a few more questions,

How is your file dependency setup at the group level?

Is there a single file or are there multiple files?

Is the file name static or does it vary?

What is the frequency of these files coming in?

Also, since file dependency is defined at group-level, I believe if you remove repeat 1 minute, re-run same occurrence from the schedule tab, BUT in your dependencies tab keep Rerun each time dependencies are met. When file arrives this should automatically kick off the group and load job, then upon successful completion of load job the move job kicks off. Let me know if this helps.

BR,

Derrick Au

[frederickmspinney]

Derrick,

How is your file dependency setup at the group level? file dependency is actually set up in the first job.

Single file

File name static

They want the job to run every time the files gets dropped. Could be 0 times or it could be 50 times.

The problem isn't getting it to resubmit, problem is if the second job that moves this file completes abnormally (without moving the file) it resets itself and then the first job resubmits because the file is still out on that server. So if the second job (move file) completes abnormally won't the first job run if I have rerun each time dependencies are met.

For now I have had success by adding a sleeper job but figure there has to be a better way.

Appreciate all your feedback.

[azhar15c1]

I hope you guys did find your answer.

I have similar issue - 

I have a check status job which is a web service job and if the web service returns the 'Process Still Running', then this job should repeat itself after 'x' mins.

Is it possible?

thanks in advance!!

[jlayton99]

I suggest that you create a resource and make both jobs use the same resource. This way, the jobs will only run when they have control of the resource.