cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2040
Views
0
Helpful
6
Replies

Creating an SFTP job using a Unix Agent

stevejj11
Level 1
Level 1

Hi All,

I was wondering if anyone in this forum has created an SFTP job using a Unix Agent via the T.E.S. GUI?  If so, how do you go about doing this?  I am new to Unix, since I operate in a pure windows environment and am trying to learn the basics of Unix and shell scripting. 

I know how to set up an FTP and SFTP job using a Windows agent, but don't know how to do it with the Unix agent? I tried, but am having issues and getting errors in the job output that keep telling me that the "Connection is refused."   Do you have to use shell scripts to do this?  Any help and/or feedback would be much appreciated.  Thanks!

-Steve

6 Replies 6

Joe Fletcher
Level 1
Level 1

Hi,

There can be a number of complexities here including whether or not your target unix machine has a suitable sftp service available.

You may find you get some mileage from adding the following to the agent.ini file at the unix end:

SSLVLDCRT=N

Cheers

I'm using sftp (i have no issues connecting) but my problem is how to pass sftp command in the command parameters.

Usage: sftp [-vC1] [-b batchfile] [-o option] [-s subsystem|path]
            [-F config] [-P direct server path] [-S program] [-B buffer_size]
            [-R num_requests] [user@]host[:file [file]]

 

here's how I do this

 

1.  create the TES job on linux agent:

command:  /bin/bash

parameter:  /path/to/shell/script.sh

 

2.  on your linux system, install sshpass.   You can look this up and see how to install it.  If you need assistance in installing it, contact me and I can provide more detailed instructions.

 

3.  create shell script.   This is the shell script that you call in your job created in step #1.  Only use the portions after ---begin--- and before ---end---.  Populate the variables with your own.  This script will download a file (or files if you use the wildcard) and delete them from the remote server

 

---begin---

#!/bin/bash

loc_initial="/remote/system/directory"

loc_staging="/path/to/local/directory"

 

ext_addy="10.11.12.13"                  

ext_user="username"

ext_pass="password"

var_filename="filename.txt"                           #NOTE:  you can also specify filetypes like *.txt here

 

export SSHPASS=$ext_pass

/usr/local/bin/sshpass -e sftp -v -oBatchMode=no -b - $ext_user@$ext_addy << EOT

    cd $loc_initial

    mget $var_filename $loc_staging

    rm $var_filetype

    bye

EOT

 

echo -e "SCRIPT COMPLETE"

exit 0

---end---

 

4.  make this script executable    (chmod 755 script.sh)

5.  test this script by running it first from the command line.    

./script.sh

6.  troubleshoot and correct any errors, rerunning until the script executes as you desire it.

7.  run the job in tidal

 

As Joe has mentioned earlier, there are LOTS of variables involved with FTP and SFTP.  If this is not what you need, let me know and I can help you out.  I do this a lot and actually prefer using scripts to the adapter.

 

 

I am using GNU bash, version 3.2.25(1)  running on RHEL 5.9

I generally find the adapter is fine for SFTP. By and large its just a case of filling in the boxes although you do need to be careful about configuring the agent correctly (SSLVLDCRT=N).

Check the agent configuration. Also that "connection refused" error is possibly firewall related.

You'll need to verify that source and target addresses can communicate on the desired ports.

 

Joe,

I encountered a few situations when transmitting large files (~3GB) in size that the transmission time via the FTP Adapter took much longer 10x than with a scripted solution.  Was never able to figure out why, even after checking all the typical suspects.  This is one of the reasons I use the scripted solution.

 

 

too bad, it's no possible to install sshpass. it won't passed the audit. i created a script but its working as a workaround. What I want to accomplish is to when i run the sftp in the command line is to accept the DATE parameters in the command parameter field,

ie:

sftp

/tmp/load_<JobDate.yyyymmdd>.dat
/tmp/load_<JobDate.yyyymmdd>.MRK

so that when it passed the 12:00am it will still honor the date issued before 12:00 which what tidal (sysadmiral) batch job does.

 

regards

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: