The National Institute for Computational Sciences

Darter Quick Start Guide

Overview

Darter is a Cray XC30 system with an Aries interconnect and a Lustre storage system, that provide both high scalability and sustained performance. The Darter supercomputer has a peak performance of 240.9 Tflops (1012 floating point operations per second).

System Overview

  • Cray XC30 (Cascade) supercomputer
  • Cray Linux Environment CLE 5.2 UP01 (based on SLES 11.3)
  • 11,968 physical compute cores (23,936 logical cores with Hyper-Threading enabled)
  • 22.6 TB of compute memory
  • Peak performance of 240.9 TF
  • 1.3PB parallel Lustre file system for scratch
  • Interconnect has a Dragonfly network topology with Cray Aries technology
  • 724 compute nodes
  • 3 login nodes with 10GigE uplinks
  • Long term archival/storage available through HPSS

Compute Node Details

  • Two 2.6 GHz 64bit Intel 8-core XEON E5-2600 (Sandy Bridge) Series processors
  • 16 physical cores (32 logical cores if using Hyper-Threading)
  • 32 GB of memory
  • Connection via Cray Aries router with a bandwidth of 8GB/sec
  • Diskless nodes
  • Shared libraries access available via DVS service

Access and Login

In order to use ssh to access Darter, you must use a one-time password (OTP) token. For more information on obtaining a token, see the Access and Login page.

To connect to Darter, type the following at a shell prompt:

ssh username@darter.nics.xsede.org
or access via GSI by doing the following:
gsissh gsissh.darter.nics.xsede.org

Currently, gridftp, gsissh, ftp, sftp, scp, hsi, and htar are available on Darter. See the Data Transfer page for more information.

Compiling

The easiest way to get your code compiled on Darter can be achieved by using the default Cray compilers. (Note: The default compilers on Kraken were from PGI. The PGI compilers are not available on Darter.) See the following examples to get an idea about how it works:

Serial code

cc serialcode.c -o serialprogram

MPI and C program

cc myprogram.c -o myprogram

MPI and C++ program

CC myprogram.cxx -o myprogram

MPI and Fortran program

ftn hello_mpi.f90 -o hello_mpi

To use a different compiler, you need to use the module command. For more information about other available compilers and options, please go to Compiling on Darter page.

Running Jobs

To submit a job to PBS, use the qsub command. The most common way to use qsub is to simply pass it the name of a 'batch script' that contains all of the information about running your job:

> qsub myscript.pbs

The following is an example batch script.

#PBS -S /bin/bash
#PBS -A UT-NTNL0121
#PBS -l size=96,walltime=01:35:00

cd $PBS_O_WORKDIR
aprun -n 96 ./a.out

Note: The #PBS -l size option is the number of physical cores to use. It must be a multiple of 16.

For more information about running jobs, please go to the Running Jobs page.

Filesystems

Darter has access to two file systems:

  • $HOME points to /nics/[a-d]/home/$USER, the Network File System (NFS home space, quota enforced)
  • Lustre Medusa File System is located at /lustre/medusa/$USER (Lustre scratch space, no quota and 30 day purge policy)

Network File System (NFS)

The Network File Service (NFS) server contains user's home directories, project directories, and software directories. Home and project directories can be used to store frequently used items such as source code, binaries, and scripts. Both are accessible from all interactive NICS resources. See the File systems page for more information.

Lustre Medusa File System

This space is intended for production work and not long term storage. This space is temporary and is subject to frequent purging to ensure the stability of the system for all users. Compute nodes do not have access to home and project directories on NFS. Compute nodes only have access to the Lustre filesystems.

Optimizing your IO performance will not only lessen the load on Lustre, it will save you compute time as well. Please consider reading the Lustre Striping Guide which we believe will help you make the best use of the parallel Lustre filesystem and improve your application's I/O performance.

Modules

module avail

Lists all available installed packages/software.

module list

Lists all currently loaded modules in your environment.

module avail {package}

Lists all currently installed versions of the specified package. Notice the "default" version of the package. If you want a different version, you will have to use the module load {package/version#} command.

module load {package/version#}

Loads package.

module unload {package}

Removes package from environment.

module swap {A} {B}

Swaps loaded {A} package with {B} package.

module show {package}

Shows the modified paths of the compiler/software, in case you would like to see libraries or the executables.