Error in creating IO file handles for job (number 3152513)

[wlockwich]

Hi All -

I am using Tidal 5.3.1.307. And the Windows agent that is running these jobs is at 3.0.2.05.

Basically the error in the subject was received when starting a particular job once it was cancelled and a couple of other different jobs a few days before. These jobs have run successfully in the past.

This particular job was running for 500+ minutes when it should run at an estimated 40 minutes. At that time it would not allow for a re-start of the job, it just stayed in a launched status.

Trying to figure out what causes this error.

Error in creating IO file handles for job 3152513

Note - from that being said we were to see 2 instances of this process running at the same time, we noticed some blocking on the DB side of things.

Trying to figure out if this is a known tidal issue or a coding issue or both.

Another side note, after cancelling the 2nd rerun attempt the following error was encountered: Error activating job, Duplicate.

When we did receive the Error creating IO file, the job did actually restart, but Tidal actually lost hooks into it and the query was still running as an orphan on the db server.

Thanks All!

[Tracy Donmoyer]

I've seen this happen when a job does not terminate cleanly, which matches what you describe.  The problem is the workfiles used by the agent are still locked by the initial process that created them, so when you try to rerun the job the new process cannot lock the workfiles because they are already locked.

The simple solution is to reboot the server which will release the locks.

A cleaner solution is to release the lock on the workfile.  If you search you can find tools to do this.  Try "Unlocker" or "Process Explorer from sysinternals"

The work files can be found in the Tidal Agent install directory:

[Tidal Agent Install]\TIDAL_AGENT_1\ES_{master_server_name}_1\services.  Search the sub-directories for the job ID.  In your case it would be "3152513".

Thanks.

Tracy D.

[wlockwich]

Hi Tracy -

First and foremost - thanks for the reply!

When you are saying reboot the server, are you talking master or agent? And are you talking about re-booting the whole machine or just re-starting the tidal service for the master or agent?

Isnt there also some directory in the tidal install that has numerical folders? Like 2, 4, 16, etc....?

Also, what do those #'s mean? Also, what's the ext on those files?

Thanks,

Bill

[wlockwich]

Also, when you say the job does does not terminate cleanly, do you find that to generally be a Tidal, code, or some other issue causing that?

[Tracy Donmoyer]

The server to reboot is the agent server.  You can try stopping the agent and then manually deleting the file.  That may work.  When the agent is running the agent process may keep the file locked, so rebooting may not be sufficient.

The numerical folders are found as sub-directories off of the services directory I mentioned.  I think the numbers correspond to the job type, so one number corresponds to standard jobs, another to FTP jobs.  I'd just look in the numbered directories until you find a filename matching the job number.

The extensions don't really matter since you will want to delete all files that match your job number.  There should only be one or two files that you need to delete and they should all be in the same numbered sub-directory.

As to the root cause of the problem, I can't really say since it doesn't happen very often.  My recollection is that it is either caused by a job blowing up spectacularly (e.g. a memory leak in the program being launched by Tidal) or someone doing something atypical with the client.

[rami_rgare]

I would say stop the tidal agent on your server.. wait few seconds until the connection is fully dropped to the master. now explore to the tidal folder and look for the folder tidal_agent_1 (this is the default name when you first install the agent so it might be somehtng else.. but typically that folder will be found under the Program files\Tidal\Agent\

once you locate that tidal_agent_1 folder you simply delete it . now start the tidal agent again. that folder should be recreated now ... and you should have a green connection on the tidal client.

i would also review the events on the box itself to check if there is a memory or java heap issues..