BBCP

BBCP is an alternative to Gridftp when transferring large amounts of data, capable of breaking up your transfer into multiple simultaneous transferring streams, thereby transferring data much faster than single-streaming utilities such as SCP and SFTP.

Usage

The basic syntax looks like:

bbcp [-options] filename user_name@target_machine_name:filename

The most common options to be aware are:

-a

Continue a previously failed transfer (only if -k was used in the previous attempt).

-h

Prints the help information.

-k

Use in case your transfer fails and you want to recover.

-P secs

How often in secs do you want a progress report.

-r

Recursive copy.

-s snum

Number of parallel network streams (Default is 4).

-v

Use if you want some verbose information like transfer rates.

-w size

TCP window size (Default is 64K).

-z

Useful when connecting to a hostname served by different IP addresses by doing a reverse connection (target to source).

Examples:

This example shows a simple file transfer using the default parameters:

bbcp /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

This example shows a file transfer with optimal transfer parameters to Sooner at OSCER, that uses six streams and a 10MB TCP window size:

bbcp -s 6 -w 10M /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

This example shows a file transfer with optimal transfer parameters to Ranger at TACC, that uses seven streams, a 12MB TCP window size, the capability to recover and prints an update every 2 secs:

bbcp -k -s 7 -w 12M -P 2 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

The following example shows how to transfer a directory to a system like Triton at SDSC where 'bbcp' needs to be installed in a user home area. This transfer uses 10 parallel streams and 2MB TCP buffer sizes. It will do a recursive copy, and it will allow to recover from a failed previous transfer. Note that it is necessary to provide the full path to the bccp executable.

bbcp -a -k -r -P 2 -V -w 10m -s 16 -T 'ssh -x -a -oFallBackToRsh=no    \
   %I -l %U %H /home/myhomearea/bin/bbcp' SourceFile account@:TargetFile

Documentation

More information on BBCP can be found by typing “bbcp -h on the NICS systems as well as on the BBCP site.

How to install it if not already available

Before you can use the BBCP utility, it must be installed on both the local and remote systems. BBCP is already available in Kraken at /usr/local/bin/bbcp. Several precompiled binaries as well as the source can be downloaded from the Stanford Linear Accelerator Center (SLAC) BBCP download page.

Installation from Source Tips

• Download the BBCP source code from download page.
• Decompress the downloaded file into a directory on the local system.

> tar -Zxvf bbcp.tar.Z

• From within the decompressed BBCP directory, run make. This should build the BBCP executable into the created bin directory. The build has been tested on Linux-based systems and should build with few or no modifications. If you systems uname command does not return Linux, you may need to modify the Makefile.

> cd bbcp
> uname
Linux
> gmake

Common variable modifications

• In MakeSname, the test command is hard coded to /usr/bin/test. If this is not the location of test on your system, you can change the following line to the correct path (which test should return the path to test on your system):

  if /usr/bin/test -${1} $2; then

• If the uname command is not in /bin on your system, change the uname variable in the MakeSname file. You will also need to change the following line in the file Makefile:

 @cd src;$(MAKE) make`/bin/uname` OSVER=`../MakeSname`

• If the libz.a library is not located at /usr/local/lib/libz.a on your system, change the libzMakefile file.

• The file Makefile contains compiler and compiler flag options for the BBCP build. You can change the compilers and flags by modifying variables in this file. For example, to change the compilers used on a Linux system, modify the variables LNXCC and LNXcc.