The National Institute for Computational Sciences

File Transfer Utilities

Standard file transfer utilities, scp, sftp and rsync are available on all NICS systems. These are good for transferring a few small files, however, we strongly recommend using other transfer tools like gridftp or bbcp that make a better usage of the network bandwidth for larger files.


rsync:

rsync allows for synchronization between two sets of files across a network. It will only copy the data that is different between the two machines. It does not allow you to copy between two remote machines.

EXAMPLES:

rsync [options] [source] [destination]

From remote host to local machine (there to here):

rsync -avz -e ssh remoteuser@remotehost:/remote/dir /this/dir/

From local machine to remote machine (here to there):

rsync -avz -e ssh /this/dir/* remoteuser@remotehost:/remote/dir/

For more information, type "man rsync" at the command line.

Some rsync options can be very resource intensive. Please do not run more than a single instance of rsync simultaneously on the same node, and avoid using rsync on directories with very many files. Processes may be killed if they cause a node to become unresponsive.


scp:   

scp allows you to copy between two machines over a network. It uses ssh to transfer the data, requiring authentication in order to proceed with the transfer. It will even allow you to transfer files between two remote machines.

EXAMPLES:

scp [options] [source] [destination]

From remote machine to local machine (there to here):

scp your_username@remotehost.edu:thisfile.txt /some/local/directory

From the local machine to the remote machine (here to there):

scp thisfile.txt your_username@remotehost.edu:/some/remote/directory

From one remote machine to another remote machine (there to there)

scp your_username@rh1.edu:/some/remote/directory/thisfile.txt \ your_username@rh2.edu:/some/remote/directory/

For more information, type "man scp" at the command line.

In case you have shell commands in your shell initialization files, you will need to modify your files with code as follows, in order for scp to work.

The following code is Bourne-shell compatible:

TTY=`/usr/bin/tty`
if [ $? = 0 ]; then
   /usr/bin/echo "interactive stuff goes here"
fi

The following code is C-shell compatible:

( /usr/bin/tty ) > /dev/null
if ( $status == 0 ) then
   /usr/bin/echo "interactive stuff goes here"
endif

sftp:

sftp is an interactive file transfer program which, like "scp" and "rsync -e ssh ...", provides encrypted transfers between two machines over a network.

Examples:

Open a session:

sftp your_username@remotehost.edu

You will be asked for a password. Upon successful entry, you will see the sftp prompt:

sftp>

To get a file from the remote machine and save it to your local machine:

sftp> get /some/remote/directory/file /some/local/directory/

To put a file from the local machine into the remote machine:

sftp> get /some/remote/directory/file /some/local/directory/

To exit, type "exit". For more information, type "man sftp" at the command line.

Category: