The National Institute for Computational Sciences

File Systems


Introduction


Two filesystems are available to ACF users for storing user files: the Network File System (NFS) and Lustre Haven. NFS contains and is used for home directories and Lustre Haven contains and is used for project and scratch directories. Table 1.1 summarizes the available filesystems.

Table 1.1 - File System Summary
File SystemPathQuotaPurged
NFS Home Directory /nics/[a,b,c,d]/home/<username> 10GB Not Purged
Lustre Scratch Directory /lustre/haven/user/<username> No Quota Purged
Lustre Project Directory /lustre/haven/proj/<project> 1 TB default
more by request
Not Purged

Please note that while both NFS and Lustre are reliable filesystems, errors and corruptions can still occur. It is each user's responsibility to back up your data. To learn about data transfer to/from the ACF, please review the Data Transfer document.

For more information on the Lustre file system, please refer to the Lustre User Guide.

Home Directories


On the ACF, fifteen terabytes of storage space is available via NFS for home directories. Home directories on NFS are periodically backed up for disaster recovery. Each new account on the ACF receives a home directory on NFS. This is each account's personal storage location for a small amount of files. Here you can store job scripts, virtual environments, and other types of files and data. In a Linux environment, you can refer to your home directory with the environment variable $HOME or with the tilde (~) character.

By default, your home directory is limited to 10GB of storage space. It is not intended to store large amounts of project or job-related data. For job-related data, please use your scratch directory. For project data that you do not want to be purged request and use project space.

To determine how much storage space you have consumed in your home directory, execute the quota -s command. Figure 2.1 shows the possible output of this command. Of interest are the first “space,” “quota,” and “limit” fields. The “space” field shows how much storage space is currently in use. The “quota” field displays the soft quota placed on your home directory. The “limit” field defines the hard quota, which is the absolute maximum storage space you can consume. When you exceed the quota, you will start a grace period that gives you time to reduce your storage space usage. If you do not reduce your storage space usage during this period, the soft limit defined by the “quota” field will be enforced.

Disk quotas for user user-x (uid 00001): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
nfs.nics.utk.edu:/nfs/b
                  3144M  10240M  10752M           58432       0       0
Figure 2.1 - Output of quota -s

Project Directories


All users are initially put into one of the ACF institutional projects: ACF-UTK0011 for UTK users and ACF-UTHSC0001 for UTHSC users. These projects do not have a project space. All courses and research projects requested by a University faculty are created with a default 1 terabyte project space. For graduate student Master's Thesis or Dissertations it is suggested that a project should be requested with the graduate student's advisor as the project principal investigator to obtain a project directory. If a 1 terabyte project space is not sufficient please submit a project directory request on the portal to request more project storage space. Specify all the information requested on the form in order to process the request. Lustre project directories will be accessible under the /lustre/haven/courses and /lustre/haven/proj/ paths.

Lustre file systems are protected by RAID and are generally reliable. However, if requested Lustre project directories can be backed up for a fee. Submit a service request if you want this service.

Quotas on Lustre project directories will be set to the total storage space requested (default 1 terabyte) by the PI of the project.

Note: Lustre project directories are NOT purged.

Scratch Directories


Scratch directories are available for all users on the Lustre Haven filesystem. Approximately 2.7 petabytes of Lustre storage space is available on /lustre/haven which is shared with scratch directories and project directories.

Important: Lustre scratch directories are NOT backed up.

Important Purging Notice: Lustre Haven scratch space is purged monthly on approximately the 3rd Monday of each month. Files in Lustre Haven scratch directories are deleted by the purging process if they have not been accessed or modified within 180 days. We will be consulting with the UT Research Computing Faculty Advisory Board on whether to gradually reduced the purge to shorter than 180 days days. See the purging schedule below.

The ACF staff are charged with managing storage resources for the benefit and use of all ACF users. The ACF Open Research cluster has a a petabyte scale Lustre file system, mounted as /lustre/haven, that is used for scratch space and project space. In general, users may accumulate many temporary files in their scratch directory on Lustre Haven, located at /lustre/haven/user/{username}, that are no longer needed once a job completes and results are returned. Often these files, other orphaned files, and unneeded files are not deleted by end users, they accumulate in scratch directories, and can fill the file system which is detrimental to all users. To stay ahead of the file system filling up, Lustre Haven scratch directory purging will be restarted on the ACF Open Research cluster on July 20, 2020. For many university research HPC clusters, this is a normal business practice to maintain adequate storage resources for a user community.

ACF projects are available for request by UT Faculty for undergraduate, graduate (masters and doctoral) and research group purposes and they are set up by default with a 1 TB project directory that is not purged. If you or your research team needs an ACF project for your research to have unpurged storage space, please have your Faculty Sponsor request an ACF research project on the ACF portal at https://portal.acf.tennessee.edu. This applies to graduate students working on a Master’s Thesis or Dissertation, please work with your Faculty advisor to request an ACF project which comes with a corresponding project directory. When more than 1 terabyte of storage is needed for a project directory, a project directory increase can be requested as well on the ACF portal.

To request a temporary purge exemption submit a service request with "ACF Open Lustre Haven purge exemption request" in the subject. Project requests along with their corresponding project directory can be made at the portal project request page.

Table 2 - Lustre Haven Purge Schedule 2020
Purge Estimated DatePathFiles Purge if not accessed or modified in
Jul 20, 2020 /lustre/haven/user/<username> <180 days
Aug 17, 2020 /lustre/haven/user/<username> <180 days
Sep 21, 2020 /lustre/haven/user/<username> <180 days
Oct 19, 2020 /lustre/haven/user/<username> <180 days
Nov 16, 2020 /lustre/haven/user/<username> <180 days
Dec 21, 2020 /lustre/haven/user/<username> <180 days

To transfer data out of your scratch space see the ACF Data Transfer documenation.

Each user has access to a scratch directory in Lustre which is located at /lustre/haven/user/<username>. For convenience, use the $SCRATCHDIR environment variable to refer to your Lustre scratch directory.

If you wish to determine which files are eligible to be purged from Lustre space, execute the lfs find $SCRATCHDIR -mtime +180 -type f command. Files that will be purged from Lustre space are those that are not modified or accessed for 180 days. If you wish to view your total usage of Lustre space, execute the lfs quota -u <user> /lustre/haven command.

Any attempts to circumvent purging, such as using the touch command on all files in a users scratch directory, will be considered a violation of the ACF acceptable use policy. Instead of taking the time to circumvent purging, why not request a project with corresponding project space. As we are all Tennessee Volunteers, our research community will be improved with positive user actions and behaviors, such as cleaning up unneeded files or requesting a project, instead of circumventing ACF file purging policy. This will result in less wasted staff support time.


Return to Top


Last Updated: 01 / 29 / 2020