The National Institute for Computational Sciences

Data Transfer - ACF SIP

  Data Transfer to/from the ACF SIP

Under Construction This ACF-SIP Data Transfer documentation is still under development

SIP Data Transfer Nodes (DTN)

The ACF SIP provides several ways for transferring files to/from the NFS home directories, project directories, and Lustre scratch directories. The ACF SIP provide a capability called a Data Transfer Node, also known as a DTN. Currently, each login node is a DTN. The table below shows the ACF SIP login node/DTNs and the relevant information.
Data Transfer NodeIP AddressAuthentication SupportedFile Transfer Protocol SupportedFile System Access
x.509 certificate
Globus File Transfer
x.509 certificate
Globus File Transfer

Data Transfer Protocols

The ACF SIP support team provides support on the DTNs for the following file transfer capabilities: SCP, SFTP, GSISCP, Globus File Transfer.

Performance note:SCP and SFTP utilities are available for transferring files but will usually perform slower than GSISCP and Globus File Transfer. GSISCP and Globus file transfers will usually be the fastest file transfer protocol methods due to their high-performance networking (HPN) support.


The DTNs support file transfer with OpenSSH file transfer utilities SCP and SFTP. SCP and SFTP are installed and available on most Linux/Unix machines. To perform a file transfer using SCP or SFTP to an ACF SIP DTN you can use scp or sftp on the command line. For the syntax see the man pages for scp or sftp on a SIP login node.

The DTNs also support file transfer with GSI-OpenSSH file transfer utility GSISCP. GSI-OpenSSH is a variant of OpenSSH developed by the Globus project that supports high performance file transfer and authentication with Grid Security Infrastructure (GSI). GSISCP is installed on the SIP and the software is available from GSISCP will work with GSI user certificate authentication (x.509 and MyProxy certificates) and with NetID+RSA authentication. GSISCP will attempt GSI certificate authentication first and then RSA authentication.

Globus Web-based File Transfer

ACF SIP users can use the web-based Globus file transfer interface to perform data transfers to/from ACF supported resources. The visual interface makes it quite easy to move, back up or restore relevant data. To get you started, visit the Globus website and consult the Getting Started guide. There are some fantastic documentation on this capability located in the Globus How-To documentation.

Please note: Using the Globus Web-based interface and Globus API only works with the University of Tennessee CILogon InCommon credential. The ACF NetID, password, and RSA two factor authentication credentials will not work with SIP DTNs when using Globus. You would not want to use this method anyway as you would have to authenticate for every set of data transfers. Using the X.509 CILogon InCommon credential issued by University of Tennessee will allow for unattended data transfers initiated on the Globus website, retry of data transfers, and use of the Globus file transfer API without having to use a username/password based authentication credential.

A sample view of Globus GUI for file transfer between two DTNs

Globus Endpoints

The Globus endpoints to access SIP are the following:

  • TBD
  • TBD

Setting up x.509 authentication

In order to use the GSISCP and Globus file transfer services each user needs to do three things:

  1. In the NICS portal associate their NetID with their NICS account (see the image below) and
  2. In the NICS portal setup their X.509 user certificate by associating their CILogon InCommon credential with their NICS account
  3. Authenticate to the Globus web-based interface for file transfers using the University of Tennessee X.509 based CILogon InCommon credential
Both of these are shown in the image below. To start off login to the NICS portal at and click on the "To associate your UTK or UTHSC NetID with your NICS account" follow the prompts, then click on the button to associate your InCommon credential with the NICS infrastructure. Click on the buttons shown in this example portal view as shown below:

To setup this credential you will select "University of Tennessee" as the identity provider and login using your University of Tennessee NetID username and password when prompted by the InCommon CILogon interface. You will set a password for your X.509 credential. Please note and remember this password as you will use it in setting up Globus or GSISCP with X.509 credentials. Once you go through the CILogon process the Distinguished Name (DN) of your X.509 credential will be associated with the NICS ACF infrastructure and will be available for use. Screeshots of the step by step process is shown below.

Step 0: Login to the Newton login node in order to save the credential you are about to create in Step 4

Step 1: select University of Tennessee as the Identity Provider

Step 2: Authenticate with your UT NetID and Password

Step 3: enter a password for your new InCommon credential (and remember this!)

Step 4: you will get a screen that shows you can click to download your certificate. Click to download and save locally. You could also use wget to this URL from Newton to save to your Newton home directory. There is a time limit for access to this certificate so be aware of that. You may have to move quickly to download the certificate.

This X.509 distinguished name (DN) information is put into the /etc/grid-security/grid-mapfile on the SIP DTNs. Once you have this setup and your credential is in the /etc/grid-security/grid-mapfile on the DTNs you are ready to start using Globus for data transfers. If you want to use GSISCP you will need to follow the instructions in the below paragraph to set that up. The SIP DTNs are configured to use CILogon OAuth credentials. For the example, the nics#datamover1 Globus endpoint is setup to use your CILogon credential so just login to Globus, select the nics#datamover1 endpoint and authenticate with your CILogon password. No other authentication method will work for the SIP DTNs with Globus and the GSISCP protocols (one cannot use NetID and password, for example).

To use your new X.509 credential with GSISCP you will need to obtain a credential pem file and put it in your home directory. The file specifically needs to go into the in ~/.globus/usercert.pem with permissions 600. If you didn't save the credential following the instructions above you can get a new credential pem file by going back to the page and go through the process again to generate a new certificate. This will then prompt you for a credential password so go ahead and type one in. Again, be sure to remember what this password is for future reference. The CILogon page will give you a link to download the certificated needed as shown below.

Once you have this credential in the ~/.globus/usercred.pem file then login to one of the DTNs and run grid-proxy-init. grid-proxy-init will prompt you for your CILogon credential password. This will create a proxy credential which can be used with GSISCP. Once you have done the grid-proxy-init you can then do a gsiscp without having to type a username or password. The default credential lifetime is 12 hours. See the following transcript for an example.

Using WinSCP to transfer files to/from the SIP DTNs

WinSCP does work for file transfers to/from the ACF SIP. Please only use the DTNs described in the Data Transfer Nodes section for data transfer.

To use WinSCP client with your NetID, password and RSA multi-factor authentication follow these steps: (Note: WinSCP will work with the DTNs that use RSA, but the example below is shown using Duo to the ACF Open DTNs.)

  • Open the WinSCP client and click on "New Site"

  • Fill in "Host name", "User name", and "Password". Port should be 22 (the default) and hit return.

  • If you have not logged into this server before you will get a Warning dialog to add the server host key. Click "Yes".

  • The authentication banner will be displayed. Click "continue"

  • The login will continue and it will prompt for the Duo push. Enter "1" and click ok or press return

  • Once you authenticate you will get the WinSCP application screen showing left side of the local machine and the right side being the system you logged into.