The National Institute for Computational Sciences

File Systems - SIP


At the time of this writing, three filesystems run on the ACF-SIP: NFS (Network File System), EncFS, and Lustre SIP. Home directories are stored on NFS. EncFS holds encrypted project directories. Lustre SIP provides high-performance storage for job-related data, data transfers, and project directories. Table 1.1 summarizes the filesystems available to the ACF-SIP.

Table 1.1 - SIP File System Summary
File SystemPathQuotaPurgedEncrypted
NFS Home Directory /nics/[a,b,c,d]/home/<username> 1GB Not Purged No
EncFS Project Space /projects/<project> By Request Not Purged Yes
Lustre SIP Project Space /lustre/sip/proj/<project>/<username> No Quota Not Purged No
Lustre SIP Scratch Space /lustre/sip/scratch No Quota Purged No

While all these filesystems are reliable, errors and corruptions can still occur. It is your responsibility to back up your data. To learn about this process on the ACF-SIP, please consult the Data Transfer document.

Home Directories

For home directories, the ACF-SIP uses NFS. 500GB of storage space is available to this filesystem. When you receive a new account on the ACF-SIP, a home directory is automatically created for your use. You will always start in your home directory on the ACF-SIP. No other users can access your home directory. Here you may store job scripts, virtual environments, and other types of data up to the quota limit. For convenience, refer to your home directory with the tilde (~) character or with the $HOME environment variable.

Users on the ACF-SIP possess 1GB of storage space in their home directories. Due to the limited size of this space, it is not suitable for large amounts of data. For job-related storage, use the Lustre SIP filesystem. To learn how to back up your data from the ACF-SIP to your own storage resources, please refer to the Data Transfer document.

Home directories are not purged and are regularly backed up. Please note that your home directory is not encrypted.

Project Directories

Encrypted project space is available on each login node for each project. The location of each project's project space is at the /project/<project-name> location. As part of the project creation process the PI will be contacted to create a password for this encrypted project space. The current process is there is one password for this project space known to all project users. However, up to 7 additional passwords can be created if there is a need to do so. Each user should create their own subdirectory under this project space the first time one makes use of the project space. There is a special process to mount this project space on the UTK or UTHSC login node. Each project will get a password that allows users to mount and use this space. Use the following command to mount your project space: sudo sipmount <project-name>Figure 3.1 shows an example of how to use this command.

[user-x@sip-login1 ~]$ sudo sipmount SIP-STA0001
Enter PASSCODE: (TFA for your SIP account)
Enter passphrase for /dev/VolGroup/lv_SIP-STA0001: (password for the encrypted project space)
Figure 3.1 - Mounting an Encrypted Project Directory

Once the above commands are completed then the project directory will be mounted.

Scratch Directories

The Lustre SIP file system provides about 15 terabytes (TB) of global high performance scratch space for data sets related to running jobs on the SIP resources and transferring data in and out of the data transfer nodes. Every user has their own scratch directory created at account creation time located in their lustre project space /lustre/sip/proj/{project}/{username}.The environment variable $SCRATCHDIR points to each users scratch directory location. Scratch space on SIP can be purged weekly, but has no storage space or quota limit associated with it. Scratch space on the SIP is for storage of data in use and is not required to be encrypted as described in the ACF SIP Security Plan Data that is at rest (not being used) is required to be encrypted in the project space.

Lustre SIP Scratch directories are NOT backed up.

Important Points for Users Using Lustre SIP Scratch

  • The Lustre SIP Scratch file system is scratch space, intended for work related to job setup, running jobs, and job cleanup and post-processing on SIP resources and not for long term data storage. Files in scratch directories are not backed up and data that has not been used for 30 days is subject to being purged. It is the user's responsibility to back up all important data to another storage resource.

    The Lustre find command can be used to determine files that are eligible to purge:

    > lfs find /lustre/sip/{project}/$USER -mtime +30 -type f
  • This will recursively list all regular files in your Lustre scratch area that are in eligible to be purged.

  • Striping is an important concept with Lustre—. Striping is the ability to break files into chunks and spread them across multiple storage targets (called OSTs). The striping defaults set up for NICS resources are usually sufficient but may need to be altered in certain use cases, like when dealing with very large files. Please see our Lustre Striping Guide for details.

  • Beware of using normal Linux commands for inspecting and managing your files and directories in Lustre scratch space. Using ls -l can cause undue load and may hang because it necessitates access to all OSTs holding your files. Make sure that your ls is not aliased to ls -l.

  • Use lfs quota to see your total usage on the Lustre system. You must specify your username and the Lustre path with this command, for example:

    > lfs quota -u <username> /lustre/haven

For more detailed information regarding Lustre usage, see the following pages:

NICS will be developing additional storage policies and will notify users about any storage policy changes.

Last Updated: 01 / 15 / 2020