Frequently Asked Questions

Python's multiprocessing module is similar to threading, so you should use the following in your Darter batch script to launch the python script on a single node:

module load python
aprun -d 16 python script.py

Listed below are the limits on the compute nodes of NICS operated resources. Here are the results from some basic tests that were run on our resources to check the real maximum values for allocatable memory and open files:

System   | MaxMem | MaxOpenFiles
------------------------------
Darter    | 31.1GB | 1015
Nautilus  | 32.1GB | 48
Keeneland | 32.1GB | >2048

They should be stored at $TMPDIR/mic# and they need to be copied to either the home or Lustre filesystem before the submitted job is complete

Each Darter node has 16 cores and 32 Gbytes of memory: about 2 GB per core if all cores are used. Sometimes it is necessary to leave some cores idle to make more memory available per core. For example, if you use 8 cores per node, each core has access to about 4 Gbytes of memory.

#PBS -l walltime=01:00:00,size=1500

aprun -n 1500 -S 4 $PBS_O_WORKDIR/a.out

The above aprun command won't work. The nodes on Darter have 2 sockets and each socket has 8 cores. That makes a total of 16 cores per node. Your size should be a multiple of 16. To make a long story short, use the following formula to get close to a multiple of 16 with what you want to do.

cores per socket on Darter * size / mpi processes per socket

8*(1500)/4 = 3000

The next number that is a multiple of 16 is 3008. Change your size = to 3008 in your pbs option and you should be fine.

The shell initiation line in PBS scripts is not guaranteed to be used to determine the interpreting shell. The default behavior is to use the user's default login shell or the value of the PBS option -S (i.e. #PBS -S /bin/bash or qsub -S /bin/bash). If you are using a shell for a PBS script which is different than your default shell, please use the PBS -S option.

The MPI_IN_PLACE option causes communication on an intra-communicator to happen in place, rather than being copied into buffers. This reduces the required number of operations (it is only possible within a node, not between nodes).

In order to use this option with MPI_Alltoall, you need to disable Cray's optimization for that call:

export MPICH_COLL_OPT_OFF=mpi_alltoall

Dynamic shared libraries are supported on Darter but it may have a performance impact. See Dynamic Linking on Darter.

When you connect to a system, your environment is set up with default limits for stack size, core file size, number of open files, etc. The system sets both soft and hard limits for these parameters. The soft limit is the actual limit imposed by the system. For example, the soft stack limit is the maximum stack size the system will allow a process to use. Users cannot increase their hard limits. Hard Limits can be decreased, but its not recommended.

While it is rarely necessary to change shell limits on Darter or Nautilus, there may be times when limits must be changed to get your program to run properly. However, users occasionally need to increase the default limits. This is where the hard limit becomes important. The system allows users to increase their soft limits, but it uses the hard limit as the upper bound. So, users cannot increase their soft limit to a value greater than their hard limit.

The command to modify limits varies by shell. The C shell (csh) and its derivatives (such as tcsh) use the limit command to modify limits. The Bourne shell (sh) and its derivatives (such as ksh and bash) use the ulimit command. The syntax for these commands varies slightly and is shown below. More detailed information can be found in the man page for the shell you are using.

Limit commands

With any shell, you can always reset both soft and hard limits to their default values by logging out and back in.

On the Cray XT, both RLIMIT_CORE and RLIMIT_CPU limits are always forwarded to the compute nodes. If you wish to set any other user resource limits, you must set APRUN_XFER_LIMITS environment variable to 1 along the new limits within the job script before the aprun call:

export APRUN_XFER_LIMITS=1
 #or
setenv APRUN_XFER_LIMITS 1

Default user resource limits

The default user resource limits in the compute nodes are:

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        unlimited
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 512
process              unlimited
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited

This message occurs always when running C-shell style job scripts. It is not really an error message, it is a friendly reminder that this is a remote batch job which can not be acted upon (such as ^C or ^Z for suspension).

Linux uses "virtual memory" for each process, which creates the illusion of a contiguous memory block when a process starts, even if physical memory is fragmented, or residing on a hard disk. When a process calls malloc, it is given a pointer to an address in this virtual memory. When the virtual memory is first used, it is then mapped to physical memory.

Optimistic Memory Allocation means that Linux is willing to allocate more virtual memory than there is physical memory, based on the assumption that a program may not need to use all the memory it asks for. When a node has used all its physical memory, and there is another call to malloc, instead of giving a null pointer, the program will receive a seemingly good pointer to virtual memory. When the memory is used, the kernel will try to map the virtual memory to physical memory, and enter an "Out of Memory" condition. To recover, the kernel will kill one (or more) process. On Darter, this will almost certainly be your executable. You should see "OOM killer terminated this process."

For more information, see O'Reilly's writeup or man malloc under "Bugs".

While using tools is a preferable method of debugging to simply using print statements, sometimes the latter option is the only method to find the bug. In this case, the most effective way to isolate the error in your code is through the method of bisection, which is an iterative process for tracing the program manually.

Step 1: In the main routine of your code, comment out the second half of the code (or approximately the second half).

Step 2: Compile and run the code. Did it crash as before?

Step 3A: If yes, return to step one and comment out the second half of the part of the main routine that ran successfully. Repeat until you have narrowed it down to the line/routine causing issues, which may include following this same tack within a subroutine.

Step 3B: If no, then swap out the half which was commented and try compiling and running again. Then, go to Step 3A.

Additionally, the use of print statements to see variable values can give insight into some earlier piece of code that might have been run through just fine but is creating an errant, unacceptable value that causes a later routine to crash.

Finally, if there is any way to duplicate the error in serial, this makes the print statements more consistent (as far as being ordered chronologically, since they are not all coming from different processors' buffers).

Now, while this might sound like a lot of work, and it is non-trivial, here is a tip to make your burden lighter: Have three sessions open on Darter simultaneously.

1. One session to edit the code.
2. Another session to compile the code.
3. Another session in which you submit for an Interactive Job so that you do not have to submit your job every time and wait in the queue.

In order to determine memory usage for a given process on a compute node, one would normally simply issue the command "top" and look at the memory usage of the process in question. However, this cannot be done on a Darter compute node, since they are not accessible to the user. Also, OOM (Out of Memory) errors often occur even when a problem has been discretized finely enough but memory leaks in the code occur in the worst case scenario, causing the program to crash.

This crashing behavior means that the user needs to instrument their code and fix the memory leaks, and the Scientific Computing staff at NICS have created a simple method to add to your current program in spots where memory usage is suspect due to possible leaks. This can assist with finding potential memory leaks as well as diagnosing situations where memory is growing in a manner not commensurate with what the user expected. While tools like valgrind and electric fence exist, they often slow the code execution to the point where the memory issue cannot be found within the prescribed wall time, making the run a waste of SUs and user time.

The following is a C function "GetMemoryUsage" which can be added into the source tree and compiled along with the rest of the user code. This function returns a program's memory usage on the compute node at the point in the program at which it is called. The idea is that one can insert "GetMemoryUsage" function calls at different places in the source, recompile, and run to observe memory leaks. To test if a function / subroutine has memory leak, one can call GetMemoryUsage at the beginning and end of the function and check if there is noticeable different in memory usage. If there is, that means there is some memory leak in that function, unless it is allocating memory of its own. If the latter is true, then the user should be able to note that the growth was by the exact amount allocated, otherwise a memory leak still exists. Regardless, the user should be able to see how much memory is allocated for a given function and determine if that is commensurate with what they were expecting. Through repeated insertion of the GetMemoryUsage function call, one can narrow down which part of large code is contributing to the memory leak.

The sample program "memusage_test.c" is to show how the function can be used, and running this should assist the user in becoming familiar with how the application works to prepare for use in a larger code base. In the sample program, a code with memory leak is created intentionally, and therefore GetMemoryUsage will keep returning higher and higher memory usage as the program continues. A sample makefile is also provided for convenience.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MEMORY_INFO_FILE "/proc/self/status"
#define BUFFER_SIZE 1024

void GetMemoryUsage ( HWM, RSS )
double *HWM, *RSS;
  {
  FILE *fp;
  size_t n = BUFFER_SIZE;
  char buffer [ BUFFER_SIZE ], scratch [ BUFFER_SIZE ];
  char *loc;

  fp = fopen ( MEMORY_INFO_FILE, "r" );
  while ( fgets ( buffer, BUFFER_SIZE, fp ) )
    {
    if ( strncmp ( buffer, "VmHWM:", 6 ) == 0 )
      {
      loc = strchr(&buffer [ 7 ], 'k');
      n = loc - &buffer [ 7 ];
      strncpy ( scratch, &buffer [ 7 ], n );
      *HWM = strtod ( scratch, NULL );
      }
    if ( strncmp ( buffer, "VmRSS:", 6 ) == 0 )
      {
      loc = strchr(&buffer [ 7 ], 'k');
      n = loc - &buffer [ 7 ];
      strncpy ( scratch, &buffer [ 7 ], n );
      *RSS = strtod ( scratch, NULL );
      }
    }
  }

#include <stdio.h>
#include <stdlib.h>

int main ( int argc, char **argv)
  {
  int i, j;
  double HWM, RSS;
  double *Array;
  GetMemoryUsage ( &HWM, &RSS );
  printf ( "Initial Usage: \nHWM : %f kB \nRSS : %f kB\n\n", HWM, RSS );
   // Create leaky code
  for ( j = 1; j < 100; j++ )
    {
    Array = malloc ( sizeof ( double ) * 100000 );
    for ( i = 0; i < 100000; i++ )
      Array [ i ] = 0.0;
    Array = NULL;

    GetMemoryUsage ( &HWM, &RSS );
    printf ( "Usage at j = %d \nHWM : %f kB \nRSS : %f kB\n\n", j, HWM, RSS );
    }
  return 0;
  }

all:
        cc -c GetMemoryUsage.c
        cc -o memusage_test.exe memusage_test.c GetMemoryUsage.o

clean:
        rm -f *.o *.exe

Unlike Darter's compute nodes, its login nodes have modest hardware specs: a single quad-core processor with 8 gigabytes of memory. However, each of the Darter login nodes may have up to 30 user login sessions active at any given time. As a result, a single user who runs a very processor- or memory-intensive task on a Darter login node can affect the work of several dozen other users. As a result, NICS recommends that concurrent makes ("make -j N") on Darter be done with an N of 2 or less.

For Darter, consult the “Cray online documentation” (http://docs.cray.com).

For C, search for the Cray “C and C++ Reference Manual” and for Fortran, consult the “Cray Fortran Compiler Commands and Directives Reference Manual”.

#include is the Standard C++ way to include header files. The 'iostream' is an identifier that maps to the file iostream.h. In older C++ versions you had to specify the file name of the header file, hence #include . Older compilers may not recognize the modern method but newer compilers will accept both methods even though the old method is obsolete.

NICS provides different methods for logging into their resources. To log in via the XSEDE portal, please click on the link below for step-by-step instructions.

XSEDE login instructions

Apple handles the local "DISPLAY" variable necessary for X11 connections differently than Linux/Unix, therefore older versions of gsissh had trouble parsing the variable, yielding this error:

$ xlogo
/tmp/launch-xZ1piK/: unknown host. (no address associated with name)
X connection to localhost:14.0 broken (explicit kill or server shutdown).

Note that if the remote DISPLAY variable had been broken, it would have given the error "Can't open display".

Updating to the most recent release of gsissh should resolve this issue.

For instructions on activating and using your RSA SecurID, see the connecting page.

In order to check one's usage on HPSS, enter the "hsi" command. Then, use the HPSS "du" command in the most top level directory, or "du -s" (summary for the entire directly only) option.

For example:

O:[/home/username]: du
2137614 4 /home/std00/
305920049 1 /home/std00/directory1/
86648223 1 /home/std00/directory2/
211942420 1 /home/std00/directory3/
156677661 47 /home/std00/direcotry4/
6455083743 1 /home/std00/directory5/
0 0 /home/std00/
-----------------------
7218409709 total 512-byte blocks, 55 Files (3,695,825,770,765 bytes)

The ~ is appended if the user has "autobackup=on" in their .hsirc file. Otherwise, the file is simply overwritten. Another option is to use "hsi cput" instead of "hsi put". Using cput will cause hsi to give a warning message if the file exists. The file that the user is attempting to store won't be written to HPSS, but the old one won't be overwritten. (The user also needs to pay careful attention to the output from hsi so that they'll notice the file wasn't stored.)

When storing files with similar names, it is best to append a date (and time if necessary) in 4-digit year, 2-digit month, 2-digit day, 2-digit hour, 2-digit minute form to the filename. This provides a unique name but also causes the files to be automatically sorted by ls based on the date for which they contain information (which might not always be the date/time they were written). An example might be file.tar.201212032250 for a date at 10:50 PM.

To find out what groups you are a member of on HPSS use the groups command.

K:[/home/username]: groups
K:HPSS Group List:
  1045: nsf008       1928: nsf008q4s 

This shows the user is a member of groups nsf008 and nsf008q4s.

If other members of your team are listed in the same group you can simply log into HPSS using HSI and change the group and permissions to share the files or directories.

For example, if both you and other members are all in nsf008q4s you will simply need to do a chgrp.

K:[/home/yourusername]: chgrp nsf008q4s filename

Then you will need to do a chmod to make the file group readable.

K:[/home/yourusername]: chmod 750 filename

The other members of the group should then be able to access your files on HPSS.

If you are unsure of the HPSS group that correlates to the NICS project, or the other members of your group are not members of the same group you will need to submit a ticket to help@xsede.org and request they be added to the group on HPSS. Please reference this FAQ in your request.

Use hsi -ls to show the tar file in HPSS

>hsi ls -l file.tar
...
-rw-------   1 username     username          12800 Oct  2  2008 file.tar

> htar -tvf file.tar
HTAR: drwxr-xr-x  username/nicsstaff          0 2008-10-02 10:47  dir2/
HTAR: -rw-r--r--  username/nicsstaff       1492 2008-10-02 10:47  dir2/data.pbs
HTAR: -rw-r--r--  username/nicsstaff       1924 2008-10-02 10:47  dir2/mpi.pbs

> htar -xvf file.tar dir2/data.pbs
HTAR: -rw-r--r--  username/nicsstaff       1492 2008-10-02 10:47  dir2/data.pbs

Administrators may disable users for archiving too many small files at a time. Archiving too many small files introduces a lot of overhead on the system, and this archiving system is not designed to handle a lot of small files. Please use htar to tar together your files. Documentation can be found here.

We should contact you if this happens, but if you are concerned that your access to HPSS has been disabled, contact us at help@xsede.org. We can re-enable your HPSS access provided that it is used correctly.

One easy way to increase file size on HPSS is to use 'htar'. For the most part, this works the same as the regular tar. We would prefer that you perform htar on ~10GB chunks. After you confirm that you will be using htar from now on, we will proceed to provide you access to HPSS. Our system staff would like you to remove all of your archived small files from HPSS and archive them again using htar.

Because HSI is a third-party package, clients may be available for your system; however, NICS currently supports access to HPSS only through HSI clients on Darter and Nautilus.

HPSS performance is greatly improved when the transfer size is between 8 GB and 256 GB. For that reason, users with large numbers of relatively small files should combine those files into one or a few 8 GB to 256 GB files and then transfer the larger files. The files can be combined with tar on the HPC system, or they can be created on the fly with a command similar to tar cvf some_dir -|hsi put - : somedir.tar. This command will tar all files in the some_dir subdirectory into a file named somedir.tar on HPSS. HPSS also supports the htar command.

Some sites recommend using the .modulerc file to set your default modules. Do not do so on NICS systems : the .modulerc file is read every time module is called. This causes issues with some of the Cray software on Darter, the global default module list, and can lead to unexpected results (if you unload a module in the .modulerc file, it will be re-loaded next time you use the module command). Instead, set your default environment in your .bashrc file (or analogue). It is best to send the output (stderr in particular) to a log or /dev/null to prevent .bashrc from printing anything, which may cause errors.

Different operating systems use different methods of indicating the end of a line in a text file. UNIX uses only a new line, whereas Windows uses a carriage return and a line feed. If you have transferred text files from your PC to a UNIX machine, you may need to remove the carriage-return characters. (The line-feed character in Windows is the same as the new-line character under UNIX, so it doesn’t need to be changed.) Some systems provide a command dos2unix that can perform the translation. However, it can also be done with a simple perl command. In the following example, win.txt is the file transferred from your PC, and unix.txt is the new file in UNIX text format:

perl -p -e 's/\r$//' win.txt unix.txt

Users may change their default shell in the NICS User Portal . To log into the portal, you need to use your RSA SecurID.

In the event of a lustre slowdown, there are many things to consider as lustre has many working parts and is shared by all users on the system. NICS continually monitors lustre's performance and seeks to improve researcher's data communications. If you notice your code's I/O performance or the lustre filesystem is slower than usual, please answer the following questions to the best of your knowledge and email XSEDE Help Desk your answers.

• When did you first notice the slowdown? How long did it last?
• Which login node were you on?
• Can you estimate the magnitude of the slowdown? (ex - "It took 2 min instead of 3 secs", "batch job exceeded walltime limit of 10 hours, but normally finishes in 8 hours")
• What were you doing? Interactive command (like "ls")? Batch job?
• For interactive commands:
  • Which host were you using?
  • Did you see the same behavior on other hosts?
  • Can you provide the exact command that was run and the directory in which it was run?
• For batch jobs:
  • Can you supply the job IDs for jobs that were affected?
  • Can you provide any details about the IO pattern for your job?

A user can change the striping settings for a file or directory in Lustre by using the lfs command. The usage for the lfs command is

lfs setstripe  -s  -i  -c 

where

size - the number of bytes on each OST (0 indicating default of 1 MB) specified with k, m, or g to indicate units of KB, MB, or GB, respectively.
index - the OST index of first stripe (-1 indicating default)
count - the number of OSTs to stripe over (0 indicating default of 4 and -1 indicating all OSTs [limit of 160]).

NOTE: If you change the settings for existing files, the file will get the new settings only if it is recreated.

To change the settings for an existing directory, you will need to rename the directory, create a new directory with the proper settings, and then copy (not move) the files to the new directory to inherit the new settings.

If your application is the type in which each separate process writes to its own file, then we believe that the best option is to not use striping. This can be set by using this command:

> lfs setstripe  -c 1

Then we see that

> lfs find -v testdirectory
OBDS:
0: ost1_UUID ACTIVE
--snip--
testdirectory/
default stripe_count: 1 stripe_size: 0 stripe_offset: -1

This shows we have a stripe count of 1 (no striping), the stripe size is set to 0 (which means use the default), and the stripe offset is set to -1 (which means to round-robin the files across the OSTs).

NOTE: You should always use -1 for stripe_offset.

The stripe count and stripe size are something you can tweak for performance. If your application writes very large files, then we believe that the best option is to stripe across all or a subset of the OSTs on the file system. Striping across all OSTs can be set by using this command:

> lfs setstripe  -c -1

Caution: Not striping large files may cause a write error if the file's size is larger than the space on a single OST. Each OST has a finite size which is smaller than the total Lustre area of all OSTs.

Use micmpiexec instead of mpiexec The necessary SSH keys are provided through the micmpiexec script

micmpiexec –n 1 ./program

micmpiexec –n 1 program

When the C shell (or one of its derivatives, such as tcsh) is starting up and encounters an error in one of its initialization files, it stops processing its initialization files. So, any aliases, environment settings, etc., that occur after the line that caused the error will not be processed. For help in troubleshooting the startup files, contact the User Assistance Center.

You can use the showusage utility to view all projects for which you are a member.

The SSH-based SCP and SFTP utilities can be used to transfer files to and from NICS systems.

For larger files, the multistreaming transfer utility BBCP may be used (not available on Darter or Beacon). The BBCP utility is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data faster than single-streaming utilities such as SCP and SFTP.

For more information on data transfers, see the remote data section of the data management page.