I've got a scenario that I'm desperately trying to find a solution for.
We use an integration tool called SnapLogic that exposes APIs that allow you to kick off an integration "pipeline". SnapLogic also provides APIs for monitoring the status of pipeline status at a summary level or for each individual pipeline. We are able to successfully create a webservice job that calls the API. The problem comes in when trying to determine the status of the job that was kicked off and subsequently kick off dependent jobs.
As we're dealing with a REST API, the response to the call to execute the pipeline is an immediate "200 - OK". That just tells us that the API accepted the request and the pipeline has started executing. I need to be able to set up dependencies on that first job that will only execute once the first pipeline has actually completed.
My initial thought was to have a secondary job (b) that would run after the first (a) with the sole purpose of calling the status API every minute or so. The response for that status API would be either 0 - Completed, 1 - Still Running, or 2 - Failed. If 1 was the response, the job would continue to execute until either 0 or 2 was returned. In the case of 0, the status job would be marked as a success and the next job would fire as it's dependency would be met. If a 2 was returned, the job would fail.
I'm rather new to tidal and I've been going down rabbit trails regarding job events, variables to hold the output values, parsing the outputs t get the 0, 1, 2 two out of the json response. I've not been able to figure out how to implement this idea, but I can't be the only one with this problem.
Can anyone provide some insight as to how to accomplish this or perhaps suggest an alternative method?
Thanks in advance!