The National Institute for Computational Sciences

Running Jobs on Nautilus

Nautilus was decommissioned on May 1, 2015. For more information, see Nautilus Decommission FAQs.

Quick Start

Nautilus consists of one UV1000, which has 1024 cores, 4 terabytes of global shared memory in a single system image, 960 TB Lustre file system, access to the NICS HPSS storage system. Nautilus also has four UV10 Harpoon Nodes, containing 32 nodes, 128 GB memory, 4 NVIDIA Tesla (codename Fermi) GPUs with 3GB of GDDR5 memory per GPU, supporting CUDA C/C++/Fortran, OpenCL, and DirectCompute Toolkits.

Nautilus provides C, C++ and Fortran compilers from Intel, GNU and PGI, with Intel being the default. Special programming environment or PE modules are used to easily switch compilers. MPI is provided on Nautilus by SGI's Message Passing Toolkit (MPT) which supports version 2.2 of the MPI standard as well as providing the SHMEM programming model. MPT is loaded by default with any of the PE modules. When compiling your MPI code, use the -lmpi linker flag.

For both batch and interactive jobs that utilize GPUs one can request this resource by -l gpus=N. Currently, there is a limitation in that Moab interprets this value as GPUs-per-core.

Login Session

When you log into Nautilus at login.nautilus.nics.tennessee.edu or login.nautilus.nics.xsede.org, you are placed on one of two login nodes, arronax or conseil. An additional login node, nedland, is also available for gsissh connections. Login nodes should be used for basic tasks such as file editing, file transfers, code compilation, data backup, and job submission. The login nodes should not be used to run production jobs. Production work should be performed on the systems compute resources.

Submitting Jobs

To submit a job to PBS, use the qsub command. The most common way to use qsub is to simply pass it the name of a 'batch script' that contains all of the information about running your job:

> qsub myscript.pbs

For information on how to write a batch script and the various PBS options available, see the Batch Scripts page.

You can also use qsub to start an interactive session on Nautilus by including the -I option:

> qsub -I -A XXXYYY -l ncpus=16,walltime=1:00:00,mem=64GB

As you can see in this case, you must specify the PBS options along with the qsub command (again, see the Batch Scripts page for an explanation of PBS options.)

The Nautilus system consists of one UV1000 (Nautilus) with 1016 available cores and four UV10 (Harpoon) nodes with 32 cores each. By default, jobs with 32 cores or less will be placed on a Harpoon node if available. You may use the -l feature PBS option to explicitly request Nautilus or a Harpoon node. To guarantee job placement on Nautilus use:

#PBS -l feature=nautilus

To guarantee placement on a Harpoon node (with 32 cores or less) use:

#PBS -l feature=uv10

Upon submission, your job will be placed in a particular queue. The job scheduler, Moab, uses queues to aid in the organization and scheduling of jobs. See the Queues page for the available queues on Nautilus and their benefits.

Parallel Execution

Nautilus supports MPI and OpenMP for running parallel jobs. See the Parallel Execution page for details on their use along with various examples.

If you need to run many concurrent instances of serial code, we provide the Eden tool for managing such jobs. Please see the Eden documentation page for instructions on use.

File I/O and Ramdisk

The $SCRATCH_RAMDISK environment variable contains the pathname to a job-specific directory residing on a RAM-based file system. By using $SCRATCH_RAMDISK, applications can perform I/O to blade memory rather than to persistent storage. Memory I/O is, in general, several orders of magnitude faster than mechanical disks and certain applications may see improved I/O speeds by using $SCRATCH_RAMDISK to create files in memory. While the actual performance will vary between applications, some I/O patterns are more likely to benefit than others. Examples include:

  • Frequent small I/O requests
  • Shared read-only files
  • File-per-process with large numbers of small files

Please note that:

  • $SCRATCH_RAMDISK is not persistent and any data that needs to be saved must be copied to /lustre/medusa before the job terminates. Once a job terminates, all data on $SCRATCH_RAMDISK will be lost. The amount of time needed to copy data should be taken into account when weighing the benefits of using $SCRATCH_RAMDISK.
  • $SCRATCH_RAMDISK takes advantage of free memory (within a job allocation) and, as a result, is relatively limited in size compared to /lustre/medusa. Therefore, users should not save files whose total size are larger than the memory available.

Job Accounting

For information on how projects are charged for time on Nautilus, see the Job Accounting page.

Scheduling Policy - Nautilus

Moab on Nautilus is configured to do “first fit” backfill. Backfilling allows smaller, shorter jobs to use otherwise idle resources.

Computation Queue Policies

  • The scheduler places a limit of one running computation job per user.
  • Computation jobs can be preempted by analysis jobs at any time. Preempted jobs will be re-queued.
  • The maximum wall time for computation jobs is 48 hours.

Analysis Queue Policies

  • There is no limit on the number of running analysis jobs per user.
  • The maximum wall time for analysis jobs is 48 hours.

VNC Session on Nautilus

For instructions on how to start a VNC session on Nautilus, see the VNC Session page.

NX Session on Nautilus

For instructions on how to start a NX session on Nautilus, see the NX Session page.