Working File System
All users should use the /staging storage area as their working space for jobs.
/staging is a parallel file system that is hosted on dedicated storage machines. This is a shared resource for use by all HPC researchers. Data stored in /staging is retained and not deleted between jobs.
A directory is automatically created for each HPC project under /staging so that data can be stored there temporarily. Project-specific user subdirectories are also created. All subdirectories created under /staging will automatically be set to allow project group access for easier data sharing.
Managing Files in /staging
Although files on /staging are not deleted between jobs, they are never backed up and it is very important for users to periodically copy results to a permanent project file system. Files on /staging that have not been accessed for more than 10 days are subject to deletion by an automated cleaning system. In addition, the /staging file system will be completely cleared of data and rebuilt during HPC’s twice-yearly cluster downtimes.
There are several quotas on the /staging directory that HPC users should be aware of:
- The default file number quota is 1,000,000 files per account.
- Currently, there is a disk usage quota of 10 terabyte (TB) per account. Users are welcome to stage their large data sets in this directory before running jobs; however, please remember that /staging is a resource shared among all users, so it will be monitored for overuse.
NOTE: When the /staging file system reaches 85% data use capacity, HPC will send an email requesting that users with the largest amount of data delete some of it. If no one responds and the file system reaches 90% capacity, HPC staff will begin deleting the oldest data, targeting larger data sets, in order to bring the usage back under 85%.
Managing Files in the /staging
Files in the /staging storage area are not backed up and it is the user’s responsibility to copy important data to a permanent project file system for safekeeping. While HPC will try to protect user files on these systems, any software or hardware failures can result in data loss.