Temporary Disk Space
All users must use the /tmp, /scratch, or /staging file systems as their working directory for all jobs.
The /tmp File System
The /tmp file system is available locally on each node. Please refer to the Node Allocation page for the /tmp disk space available on each node. Users should use the directory created exclusively for each job defined by the environment variable $TMPDIR. Users have access to the /tmp directory of a particular node only when a job is running on that node. All files created in /tmp are deleted before the next job starts.
The /scratch File System
/scratch is a shared temporary file system that is created when a new job starts and is deleted at the end of the job. The /scratch file system should be used to store temporary files that need to be accessed from all the nodes. If you need the files in the /scratch file system saved, copy them before the end of the job to a permanent storage disk.
The /staging File System
/staging is a 328-terabyte parallel file system that is hosted on dedicated storage machines. This is a shared resource for use by all HPC researchers. Data stored in /staging is retained and not deleted between jobs.
A directory is automatically created for each HPC project under /staging so that data can be stored there temporarily. Project-specific user subdirectories are also created. All subdirectories created under /staging will automatically be set to allow project group access for easier data sharing.
Managing Files in /staging
Although files on /staging are not deleted between jobs, they are never backed up and it is very important for users to periodically copy results to a permanent project file system. Files on /staging that have not been accessed for more than 10 days are subject to deletion by an automated cleaning system. In addition, the /staging file system will be completely cleared of data and rebuilt during HPC’s twice-yearly cluster downtimes.
We have implemented a total number of files quota on the /staging directory. The default quota is 1,000,000 files per HPC account. There are currently no disk usage quotas on the /staging directory, and users are welcome to stage their large data sets before running jobs; however, please remember that /staging is a resource shared among all users, so it will be monitored for overuse. When the /staging file system reaches 85% data use capacity, HPC will send an email requesting that users with the largest amount of data delete some of it. If no one responds and the file system reaches 90% capacity, HPC staff will begin deleting the oldest data, targeting larger data sets, in order to bring the usage back under 85%.
Managing Files in the Temporary File Systems
Files on all of the temporary file systems are not backed up and it is the user’s responsibility to copy important data to a permanent project file system for safekeeping. While HPC will try to project user files on these systems, any software or hardware failures can result in data loss.
Managing Files in /tmp and /scratch
Files on these systems are automatically cleared at the termination of a job, so it is important for jobs that use them to copy off any needed data before exiting.
Some compute nodes are shared among multiple users and if the performance of a computing resource is degraded by a temporary file system that has too little available free space, HPC management may compress some user files, move some files to another file system, or delete some files.