Tidal Scheduler - job event - Tidal job appears to have ended (viewing in client), but process is still running...

[Dave.Carnahan]

As the title suggests -

-

I've got a Tidal job that runs (basically) 7 by 24, perpetually going off every 2 minutes - when the last Tidal job iteration finishes.

I can see the Tidal job iteration count start to grow throughout the day, then coming in the next day I see the job (for that day) cap off with xxx iterations and the next days job is on its way for the current day.

-

But at times, the project developer will come to me and point out the job (process) really didn't end and (sometimes) this process has gone off on it's own (attempting to call a web service that gets toasted waiting for a SQL connection) and never ends - but the Tidal job does end... 

-

When the second situation happens there will be many multiple processes running on the server, all of the same name.

In this last case, the job is attempting to execute older SSIS packages written in SQL2005 in a SQL2008R2 environment.

The process on the server remaining is an app called 'vsta.exe' which I believe is a SQL converter app taking the original 2005 app and converting the SQL2005 API calls into 2008R2 API calls at run time.

With both of these scenarios, I've added two job events.

One Tidal event to kill the Tidal job - after job duration exceeds the set max time allowed.

Then a second Tidal event to change the status to completed normal - so the perpetual job can continue.

But apparently, killing the Tidal job isn't the only thing needed, its actually stopping the specific processes on the server running the Tidal agent.

==

Can anyone explain the disassociation between the Tidal job ending, but the process still continuing (it's probably spawning a process...)

-

And

How to trap (in Tidal) for this occurrence, so the actual job (process) doesn't continue to run.

(Killing the job after xxxx minutes is acceptable per the developer).

Thanks

[Derrick Au]

Hi Dave,

How many CPUs does your Master have? Is this a VM?

I suspect more than likely this is a systems performance issue (and not Tidal :-]). So, this is a high frequency job that runs every 2 minutes perpetually. The problem is due to not having enough CPUs on your server. Even though it may be okay to kill the Tidal job after X max minutes, Tidal is still spawning more processes faster than it can kill them, which I believe is the case here. Try logging on to the machine and then pop taskmgr open, you will undoubtedly find under the Processes and Performance tabs a whole slew of Tidal processes hanging and CPU utilization is pegged to the max.

I would recommend increasing the CPUs on the machine. There is a guide as to how many CPUs a machine should have at the very minimum, but from experience, it is always better to add more at the beginning. Please work with a Cisco Support specialist who can guide you further with fine tuning your machines based on your environment size, and also verify with them just how much more crunch power to add to your machines. 

BR,

Derrick Au