Agent dropping every 11 minutes and reconnects

[paul.standre1]

Tidal master 5.3.1.396 windows

Agent added windows 3.1.0.05

Just added new Agent and agent is dropping every 11 minutes

Re-installed agent and rebooted no luck

Checked agent log and I am seeing this:

_20141217 13:36:53 <err.TIDAL_AGENT_1.ES_INVJOBSCHED01_1> Resource Config - Settings : Active = 0 Master = ES_INVJOBSCHED01_1 CpuPoll = -1 VmemPoll = -1

_20141217 13:47:56 <inf.TIDAL_AGENT_1.ES_INVJOBSCHED01_1> Master Agent ID changed from 265 to 94

_20141217 13:47:56 <inf.TIDAL_AGENT_1.ES_INVJOBSCHED01_1> Connecting to master port 5591@INVJOBSCHED01

_20141217 13:47:56 <inf.TIDAL_AGENT_1.ES_INVJOBSCHED01_1> Suspend request received from master ES_INVJOBSCHED01_1, Unregistering master ES_INVJOBSCHED01_1

_20141217 13:48:01 <inf.TIDAL_AGENT_1.ES_INVJOBSCHED01_1> Master Agent ID changed from 94 to 265

In props file on agent ID=265

Checked nodmst table I see ID 265 and is assigned to right node, this node, there is no node 94.

Any ideas??

[Dinesh Sankaranarayanan · Accepted solution]

I know there is a newer version of the windows agent available. Not sure if there is any compatibility issue with 5.3.x. May be try using that to see if the issue still occurs ?

[hrechek4470]

We have 6.2.1 SP2 masters with fault tolerance. Almost all of our agents are 3.1.0.14 and issue still occurs for us

[David Rickard]

hrechek4470    

Thank you for the response.  I had been hoping that upgrading would resolve our issue.  Seems like that is not the case. 

Were you ever able to get a work around for this?  I am concerned about jobs going to orphaned status and hanging up production.

[Joe Fletcher]

I've had this problem start for a single windows agent since installing a new license.

It was an extension key to add some more agents but since then this one host has been bleating about dropping its connection. Its got the latest agent and we're patched up on 6.2.1 just shy of SP3.

I'm going to wait until we load SP3 before I start playing about with the license files since its always such fun getting new ones.

:)

[fgannon01]

Paul,

We were on version 6.1.0.483 (Windows Masters), with 3.1.0.05 agents which kept restarting every 11 mins....  turns out for us it was a bad license on the master.  Once we received and installed the new license, the problem was resolved.

Hope this is the answer to your problem.

Best regards,

Fred Gannon

fgannon@west.com

[Bill Slobodnik]

Fred, 

I have this issue on a 6.1 install as well, only my disconnects are every 55 minutes.   Do you have a Fault monitor setup in your environment?   Reason I ask is that when the Backup master takes over job execution in my environment I do not see the disconnects, and the license file is only installed on the primary master.

thanks, Bill S

[David Rickard]

We have the same issue in our Production and Sandbox environment.  We have loaded multiple licenses in each environment and still are experiencing this issue.  Currently we are running on a Demo license as a work around.  Please see CSCuv74153.

Has anyone seen this before?  If so, has anyone found a solution?

Thanks,

Bud Rickard

Senior Tidal Administrator

Voya

Bud.Rickard@voya.com

[Mark Jaeger]

John Laird suggested this fix:

• Remove all .lic files from Program Files\TIDAL\Scheduler\Master\config\
• Write a new copy of your master license to the folder.
• Shut down the Scheduler Master.
• Start the Scheduler Master WITHOUT starting the Client Manager.
• Shut down the Scheduler Master again.
• Start the Scheduler Master. Look in the Master.log file for a line like:

02/26 11:25:41:130[1:main]: (mem=3551847384/4116709376) Scheduler: No license [Demo or Permanent] to process

Now you can start the Client Manager.

Since we went through this process (over a month ago), we have not experienced the dropped connections.

[rocketscientist1]

how much memory do you have allocated to your agent?  

how many other windows agents are in your environment...and do they exhibit the same behavior?

have you begun scheduling any jobs yet on this newest agent?

[hrechek4470]

We have floating licenses for all of our agents.

9 of our 12 Windows agents have this behavior every 55 minutes.

11 of 12 are at 3.1.0.14,1 of the offending 9 is 3.1.0.18, We have jobs scheduled on all of them.

When you say "how much memory do you have allocated to your agent? " are you referring to on the  VM's or within Tidal?