The National Institute for Computational Sciences

File Systems - SIP


At the time of this writing, three filesystems run on the ACF-SIP: NFS (Network File System), EncFS, and Lustre SIP. Home directories are stored on NFS. EncFS holds encrypted project directories. Lustre SIP provides high-performance storage for job-related data, data transfers, and project directories. Table 1.1 summarizes the filesystems available to the ACF-SIP.

Table 1.1 - SIP File System Summary
File SystemPathQuotaPurgedEncrypted
NFS Home Directory /nics/[a,b,c,d]/home/<username> 1GB Not Purged No
EncFS Project Space /projects/<project> By Request Not Purged Yes
Lustre SIP Project Space /lustre/sip/proj/<project>/<username> No Quota Not Purged No
Lustre SIP Scratch Space /lustre/sip/scratch No Quota Purged No

While all these filesystems are reliable, errors and corruptions can still occur. It is your responsibility to back up your data. To learn about this process on the ACF-SIP, please consult the Data Transfer document.

Home Directories

For home directories, the ACF-SIP uses NFS. 500GB of storage space is available to this filesystem. When you receive a new account on the ACF-SIP, a home directory is automatically created for your use. You will always start in your home directory on the ACF-SIP. No other users can access your home directory. Here you may store job scripts, virtual environments, and other types of data up to the quota limit. For convenience, refer to your home directory with the tilde (~) character or with the $HOME environment variable.

Users on the ACF-SIP possess 1GB of storage space in their home directories. Due to the limited size of this space, it is not suitable for large amounts of data. For job-related storage, use the Lustre SIP filesystem. To learn how to back up your data from the ACF-SIP to your own storage resources, please refer to the Data Transfer document.

Home directories are not purged and are regularly backed up. Please note that your home directory is not encrypted.

Project Directories

Encrypted project directories on the cluster uses the EncFS filesystem for encryption. Unless they are explicitly mounted, they will be unavailable. Users can mount these directories on the cluster with the sipmount command. Figure 3.1 shows the syntax for this command.

sudo sipmount <project-directory>
Figure 3.1 - Usage of the sipmount Command

The sudo command must be used with sipmount because mounting is a privileged action on the cluster. When you use sudo, you will be prompted to supply your NetID password and authenticate with Duo TFA. After you successfully authenticate, you can use the mount command to verify that the project directory was successfully made available to you. Figure 3.2 shows the process for mounting a project directory. You can access the contents of the project directory from the /projects/SIP-<project-ID> path.

[user@sip-login1 ~]$ sudo sipmount SIP-UTK9999
[sudo] password for user: 
Duo two-factor login for user

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-1234
 2. SMS passcodes to XXX-XXX-1234

Passcode or option (1-2): 1

Pushed a login request to your device...
Success. Logging you in...
[user@sip-login1 ~]$ mount | grep SIP-UTK9999
encfs on /projects/SIP-UTK9999 type fuse.encfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
Figure 3.2 - Mounting an Encrypted Project Directory

Mounted project directories will automatically be unmounted after fifteen minutes of inactivity, but if you wish to ensure that the project directory is secured, you can unmount it with the sipunmount command. The syntax and usage of the command is equivalent to the sipmount command.

If you wish to transfer data from an encrypted project directory to another storage resource, please refer to the Data Transfer document.

Scratch Directories

The Lustre SIP file system provides about 15 terabytes (TB) of global high performance scratch space for data sets related to running jobs on the SIP resources and transferring data in and out of the data transfer nodes. Every user has their own scratch directory created at account creation time located in their lustre project space /lustre/sip/proj/{project}/{username}.The environment variable $SCRATCHDIR points to each users scratch directory location. Scratch space on SIP can be purged weekly, but has no storage space or quota limit associated with it. Scratch space on the SIP is for storage of data in use and is not required to be encrypted as described in the ACF SIP Security Plan Data that is at rest (not being used) is required to be encrypted in the project space.

Lustre SIP Scratch directories are NOT backed up.

Important Points for Users Using Lustre SIP Scratch

  • The Lustre SIP Scratch file system is scratch space, intended for work related to job setup, running jobs, and job cleanup and post-processing on SIP resources and not for long term data storage. Files in scratch directories are not backed up and data that has not been used for 30 days is subject to being purged. It is the user's responsibility to back up all important data to another storage resource.

    The Lustre find command can be used to determine files that are eligible to purge:

    > lfs find /lustre/sip/{project}/$USER -mtime +30 -type f
  • This will recursively list all regular files in your Lustre scratch area that are in eligible to be purged.

  • Striping is an important concept with Lustre—. Striping is the ability to break files into chunks and spread them across multiple storage targets (called OSTs). The striping defaults set up for NICS resources are usually sufficient but may need to be altered in certain use cases, like when dealing with very large files. Please see our Lustre Striping Guide for details.

  • Beware of using normal Linux commands for inspecting and managing your files and directories in Lustre scratch space. Using ls -l can cause undue load and may hang because it necessitates access to all OSTs holding your files. Make sure that your ls is not aliased to ls -l.

  • Use lfs quota to see your total usage on the Lustre system. You must specify your username and the Lustre path with this command, for example:

    > lfs quota -u <username> /lustre/haven

For more detailed information regarding Lustre usage, see the following pages:

NICS will be developing additional storage policies and will notify users about any storage policy changes.

Last Updated: 01 / 15 / 2020