The National Institute for Computational Sciences

Beacon System Overview

System Overview

Overview

Beacon is an energy efficient cluster that utilizes Intel® Xeon Phi™ coprocessors.  It is funded by NSF through the Beacon project to port and optimize scientific codes to the coprocessors based on Intel's Many Integrated Core (MIC) architecture.



MIC Programming Models

Native Mode:

  • All code runs directly on the coprocessors
  • Any libraries used will need to be recompiled for native mode
  • Use the compiler flag flag '-mmic' to compile for native mode
  • Offload Mode:

  • Code starts running on host
  • Parallel regions of code can be manually specified to run on the coprocessors using pragmas/directives
  • Data is either copied explicitly to the card or implicitly (used for complex data types involving pointers, only available in C++)
  • Automatic Offload (AO) is available for certain Intel Math Kernel Library (MKL) functions: ?GEMM, ?TRSM, ? PORTF, ?GEQRF, and ?GETRF


  • System Configuration

    The Beacon system offers access to the following:

  • 48 compute nodes and 6 I/O nodes
  • FDR InfiniBand interconnect providing 56 Gb/s of bi-directional bandwidth
  • Each compute node is equipped with:

  • 2 8-core Intel® Xeon® E5-2670 processors
  • 256 GB of memory
  • 4 Intel® Xeon Phi™ coprocessors 5110P with 8 GB of memory each
  • 960 GB of SSD storage
  • We have one compute node configured with 4x Xeon Phi 7120P, and one compute node configured with 4x NVIDIA Tesla K20Xm GPUs. Instructions on how to access these nodes can be found here.

    Each I/O node provides:

  • Access to an additional 4.8 TB of SSD storage
  • Overall, Beacon provides 768 conventional cores and 11,520 accelerator cores that provide over 210 TFLOP/s of combined computational performance, 12 TB of system memory, 1.5 TB of coprocessor memory, and over 73 TB of SSD storage, in aggregate.



    Beacon

    Computing Environment

    The default account setup each user has is their home directory, Lustre scratch space, and their unix group (typically beacon-users).

    Unix shell

    The command-line interpreter (a.k.a. shell) is the traditional Unix/Linux operating system. The default shell environment on Beacon is bash. There are other shells available, such as sh, csh, tcsh, and zsh. Users may change their default shell in the NICS User Portal. To log into the portal, you need to use your RSA SecurID OTP token.

    Modules

    The modules software package allows you to dynamically modify your user environment by using module files. Modules are very useful to compile and running on Beacon. 

    Modules can make your computing experience easy. Here is a short list and description of commonly used module commands. Note, if no version number is preceding the package name, it will use the default package.

    module list

    Shows all loaded modules.

    module swap <package A> <package B>

    This will swap package B for package A. Useful to switch to versions of other modules.

    module avail <package>

    If no package is given, it will list all available modules. This command is useful to see which versions of particular software are installed. Try: module avail zlib

    module show <package>

    This gives you the information concerning the installed software. One will see the setenv commands that will modify your environment if you decide to load that module. This is useful for two major reasons. First, you can make sure what executable you like to run. Here, you can perform a ls on the outputted bin directory. Secondly, some environmental variables could be introduced. For instance, the FFTW module will provide environmental variables that point to the library and include directories.  Simply include these variables into your makefile versus including the full path. Note that all modules are now complete for Xeon and Xeon Phi.

    Useful Links