HPSS

Your file transfer has caused a Lustre storage server (OST) to become full, resulting in an error like:

ead_cond_timedwait() return error 22, errno=0 OUT OF SPACE condition detected while writing local file

This usually happens because the stripe count is too small (often 1). To solve this issue, remove the partially transferred file and change the stripe count of the directory before transferring the file. To change the stripe count of the directory, first cd to that directory. Second, type the following command:

lfs setstripe . -c 8

where 8 is the new stripe count, meaning that any new files in that directory will be striped across 8 OSTs. The larger the stripe count, the more OSTs the file will be striped across. Typically, a stripe count that results in a file using less than 100 GB per OST should usually work.

To find out what groups you are a member of on HPSS use the groups command.

K:[/home/username]: groups
K:HPSS Group List:
  1045: nsf008       1928: nsf008q4s 

This shows the user is a member of groups nsf008 and nsf008q4s.

If other members of your team are listed in the same group you can simply log into HPSS using HSI and change the group and permissions to share the files or directories.

For example, if both you and other members are all in nsf008q4s you will simply need to do a chgrp.

K:[/home/yourusername]: chgrp nsf008q4s filename

Then you will need to do a chmod to make the file group readable.

K:[/home/yourusername]: chmod 750 filename

The other members of the group should then be able to access your files on HPSS.

If you are unsure of the HPSS group that correlates to the NICS project, or the other members of your group are not members of the same group you will need to submit a ticket to help@xsede.org and request they be added to the group on HPSS. Please reference this FAQ in your request.

HTAR provides the “-Hverify=option[,option...]” command line option, which causes HTAR to first create the archive file normally, and then to go back and check its work by performing a series of checks on the archive file. You choose the types of checks to be performed by specifying one or more comma-separated options. The options can be either individual items, or the keyword “all”, or a numeric level between 0, 1 or 2. Each numeric level includes all of the checks for lower-valued levels and adds additional checks. The verification options are:

htar -cvf TEST_VERIFY.TAR /lustre/scratch/ -Hcrc -Hverify=2
htar -Hcrc -tvf TEST_VERIFY.TAR

Use hsi -ls to show the tar file in HPSS

>hsi ls -l file.tar
...
-rw-------   1 username     username          12800 Oct  2  2008 file.tar

> htar -tvf file.tar
HTAR: drwxr-xr-x  username/nicsstaff          0 2008-10-02 10:47  dir2/
HTAR: -rw-r--r--  username/nicsstaff       1492 2008-10-02 10:47  dir2/data.pbs
HTAR: -rw-r--r--  username/nicsstaff       1924 2008-10-02 10:47  dir2/mpi.pbs

> htar -xvf file.tar dir2/data.pbs
HTAR: -rw-r--r--  username/nicsstaff       1492 2008-10-02 10:47  dir2/data.pbs

To retrieve a single directory from HPSS use the -R option. For example,

>hsi
>get -R dir1

Administrators may disable users for archiving too many small files at a time. Archiving too many small files introduces a lot of overhead on the system, and this archiving system is not designed to handle a lot of small files. Please use htar to tar together your files. Documentation can be found here.

We should contact you if this happens, but if you are concerned that your access to HPSS has been disabled, contact us at help@xsede.org. We can re-enable your HPSS access provided that it is used correctly.

One easy way to increase file size on HPSS is to use 'htar'. For the most part, this works the same as the regular tar. We would prefer that you perform htar on ~10GB chunks. After you confirm that you will be using htar from now on, we will proceed to provide you access to HPSS. Our system staff would like you to remove all of your archived small files from HPSS and archive them again using htar.

There is nothing that should prevent you from running a script that creates multiple simultaneous connections to HPSS. The HPSS system administrator recommends that you should not create more than 1 or 2 connections at a time. Every time you introduce a new instance, the performance of the overall system is degraded.

Because HSI is a third-party package, clients may be available for your system; however, NICS currently supports access to HPSS only through HSI clients on the HPC systems.

If you log into kraken using your passcode from your OTP token, you can run HSI without entering your passcode each time. You can also run batch scripts that use HSI in the "hpss" queue. If you logged into kraken-gsi using GSI authentication you will be prompted for your passcode each time you use HSI.

HPSS performance is greatly improved when the transfer size is between 1 and 10 GB. For that reason, users with large numbers of relatively small files should combine those files into one or a few 1 GB to 10 GB files and then transfer the larger files. The files can be combined with tar on the HPC system, or they can be created on the fly with a command similar to tar cvf some_dir -|hsi put - : somedir.tar. This command will tar all files in the some_dir subdirectory into a file named somedir.tar on HPSS. HPSS also supports the htar command.

The HSI utility allows automatic authentication and provides a user-friendly command line and interactive interface to HPSS.

Users may access HPSS from any NICS high-performance computing (HPC) system with the Hierarchical Storage Interface (HSI) utility. Access it by typing the command hsi in your linux environment. To exit, simply type quit.