Temporary Disk Space

All users should use the $TMPDIR, /staging, or $SCRATCHDIR storage areas as their working space for jobs.

$TMPDIR

The $TMPDIR storage area is available locally on each node under /tmp. Please refer to the Node Allocation page for the /tmp disk space available on each node. Users should use the directory created exclusively for each job defined by the environment variable $TMPDIR. Users have access to $TMPDIR only when a job is running. All files created in $TMPDIR are deleted before the next job starts.

/staging

/staging is a 328-terabyte parallel file system that is hosted on dedicated storage machines. This is a shared resource for use by all HPC researchers. Data stored in /staging is retained and not deleted between jobs.

A directory is automatically created for each HPC project under /staging so that data can be stored there temporarily. Project-specific user subdirectories are also created. All subdirectories created under /staging will automatically be set to allow project group access for easier data sharing.

Managing Files in /staging

Although files on /staging are not deleted between jobs, they are never backed up and it is very important for users to periodically copy results to a permanent project file system. Files on /staging that have not been accessed for more than 10 days are subject to deletion by an automated cleaning system. In addition, the /staging file system will be completely cleared of data and rebuilt during HPC’s twice-yearly cluster downtimes.

There are several quotas on the /staging directory that HPC users should be aware of:

  • The default file number quota is 1,000,000 files per account.
  • Currently, there is a disk usage quota of 10 terabyte (TB) per account. Users are welcome to stage their large data sets in this directory before running jobs; however, please remember that /staging is a resource shared among all users, so it will be monitored for overuse.

NOTE: When the /staging file system reaches 85% data use capacity, HPC will send an email requesting that users with the largest amount of data delete some of it. If no one responds and the file system reaches 90% capacity, HPC staff will begin deleting the oldest data, targeting larger data sets, in order to bring the usage back under 85%.

$SCRATCHDIR

$SCRATCHDIR is a storage area that is created within /staging when a new job starts and is deleted at the end of the job. The $SCRATCHDIR storage area should be used to store temporary files that need to be accessed from all nodes in a job. If you need the files in the $SCRATCHDIR file system saved, copy them before the end of the job to a permanent storage disk. Note that the quota in $SCRATCHDIR is shared with your quota in /staging.

Managing Files in the Temporary Storage Areas

Files in all of the temporary storage areas are not backed up and it is the user’s responsibility to copy important data to a permanent project file system for safekeeping. While HPC will try to project user files on these systems, any software or hardware failures can result in data loss.

Managing Files in $TMPDIR and $SCRATCHDIR
Files in these storage areas are automatically cleared at the termination of a job, so it is important for jobs that use them to copy off any needed data before exiting.

Some compute nodes are shared among multiple users and if the performance of a computing resource is degraded by a temporary storage area that has too little available free space, HPC management may compress some user files, move some files, or delete some files.