HPCC

Center for High-Performance Computing and Communications

Frequently Asked Questions

Click the links below for answers to frequently asked questions regarding HPCC accounts, using the HPCC cluster, or software on HPCC.

Accounts

How do I get an HPCC account?

In order to apply for a new HPCC account, please complete the account application at www-rcf.usc.edu/rcfdocs/hpcc/allocations. You must access this application web page from a USC IP address. If you are off-campus or are otherwise using a non-USC network connection, you will need to use the USC Virtual Private Network (VPN) in order to access this web page. For information on accessing the USC VPN, see the VPN Overview page on the ITS website.

For additional information on applying for a HPCC account, see the Applying for a HPCC Account page.

I forgot my password. How can I reset it?

HPCC does not have access to passwords and cannot reset them. Please contact the ITS Customer Support Center at 213-740-5555 for assistance with resetting your account password.

What is a quota?

Every applicant for an HPCC allocation needs to justify his/her request for storage resources. Once your allocation has been approved, you will be granted a certain amount of disk space. Please be aware that the amount allocated to you may not match your request.

Your HPCC disk space will be located at

[/home/rcf-proj/[hpcc_allocation_id]

Inside this directory, every user of this allocation will have a subdirectory, identified by their RCF login. This disk space is called the project directory. All users associated with the allocation will share the disk quota for the project directory.

In addition to your space in the project directory, you will also be granted your own disk quota, which is located in your RCF home directory.  This directory has a fixed 1 GB quota and is strictly for personal use.  Always start your HPCC jobs from the project directory.  The most common cause of “disk quota exceeded” error messages is a direct result of users starting jobs from their home directory and not their project directory.

You can check the quota on your RCF home directory by running the following command:

myquota

You can also check your RCF home directory quota and your allocation’s project directory quota on the Checking Your Quotas page.

If your allocation’s project directory has become full, please send an email to hpcc@usc.edu for assistance.  Please note that HPCC is unable to increase your RCF home directory’s quota.

How do I use the temporary disk space on my account?

You must use either the /tmp or /scratch file system as your working directory for all jobs.

The /tmp File System

The /tmp file system is available locally on each node. Please refer to the Node Allocation page for the /tmp disk space available on each node. A directory is created exclusively for each job and is defined by the environment variable $TMPDIR. You have access to the /tmp directory of a particular node only when you are running a job on that node. All files created in /tmp are deleted before the next job starts.

The /scratch File System

The /scratch directory is a shared temporary file system. It is created when a new job starts and it is deleted at the end of the job. The /scratch file system should be used to store temporary files that need to be accessed from all the nodes. If you need to save the files in your /scratch file system, copy them to a permanent storage disk before your job ends.

Managing Files in the Temporary File Systems

Files located in the temporary file systems, /tmp and/scratch, are not backed up. It is your responsibility to copy important data to permanent file systems. While HPCC will try to protect your files for the stated retention periods, hardware failures could result in data loss.

If the performance of a computing resource is degraded by a temporary file system that has too little available free space, HPCC management may compress, move, or delete files that have not been accessed within the last five days.

Using the HPCC Cluster

How do I log in to the HPCC cluster?

To log in to the Linux cluster resource, you will need to use ssh to access either hpc-login1 for the 32-bit head node or hpc-login2 for the 64-bit head node.

These head nodes should only be used for editing and compiling programs; any computing should be done on the compute nodes.  Computing jobs submitted to the head nodes may be terminated before they complete. To submit jobs to the compute nodes, use the Portable Batch System (PBS) .

How do I run jobs on the cluster?

To a run job on the HPCC cluster, you will need to set up a Portable Batch System (PBS) file, in order to define the commands and cluster resources you wish to use for this job.  This PBS file is a simple text file that can be easily edited by using a UNIX editor, such as vi, pico, or emacs. Information on UNIX editors can be found in the UNIX Overview section of the ITS website.

Setting Up a PBS File

Here is a sample PBS file, named myjobs.pbs, followed by an explanation of each line of the file.

#!/bin/bash
#PBS -l nodes=1:ppn=2
#PBS -l walltime=00:00:59
cd /home/rcf-proj3/pv/test/
source /usr/usc/sas/default/setup.sh
sas my.sas

The 1st line in the file sets what shell will be used for the job.  Bash is used in this example, but csh or other valid shells will also work.

The 2nd line specifies the number of nodes and processors desired for this job. In this example, 1 node with 2 processors is requested.

The 3rd line in the PBS file states how much wall-clock time is requested. In this example 59 seconds of wall-clock time are requested.

The 4th line tells the HPCC cluster to access the directory where the data is located for this job. In this example, the cluster is instructed to cd into the /home/rcf-proj3/pv/test/ directory.

The 5th line tells the cluster which program you would like to use to analyze your data. In this case, the cluster sources the environment for SAS.

The 6th line tells the cluster to run the program. In this case, it runs SAS, specifying my.sas as the argument in the current directory, /home/rcf-proj3/pv/test, defined in the previous line.

Running Your Job

After creating your PBS file, type the following command to submit your job:

qsub myjob.pbs

Once you issue the command, you should receive a job id and 2 output files, one for standard output and one for standard error, in traditional UNIX fashion.

For more details and examples on how to run jobs, please see the Running a Job on the HPCC Cluster page.

How do I report a problem with a job submission?

If a job submission results in an error, please send an email to hpcc@usc.edu. Be sure to include the job ID, error message, and any additional information you can provide.

How do I know which allocation I should use to submit a job if I am in multiple HPCC allocations? How do I specify a certain allocation?

To see a listing of all of your available accounts and your current core hour allocations in these accounts, use the following command:

mybalance -h

If you have accounts in multiple allocations, the following command will show you your default allocation:

glsuser [-u <username>]

The default HPCC allocation is used to run a job when no allocation is specified in the qsub command line.

You can override this by using the following command:

qsub -A [hpcc_allocation_name] myjob.pbs

For further details on qsub, please read the official man pages available by typing the following on any HPCC login node:

man qsub
How do I configure the standard user environment for my HPCC account?

For information on the standard user environment for HPCC accounts, please see the Standard User Environment page.

How do I set up a non-standard compiler?

If you wish to use a non-standard compiler, i.e. a compiler other than one of the MPI compilers, HPCC has GNU, Portland Group, Intel, and Absoft compilers installed under the /usr/usc directory. In order to use on of these compilers, you must source the appropriate setup.csh or setup.sh file each time you login.

For example, here is how you would set up the Portland Group compiler for use on your account.

Setting Up the Portland Group Compiler

The following command sets up the Portland Group compiler:

source /usr/usc/pgi/default/setup.csh

If you are planning to use only the Portland Group compilers, add the following to your .login or .profile file to set up your environment.

if (-r /usr/usc/pgi/default/setup.csh) then
source /usr/usc/pgi/default/setup.csh
endif

Software on HPCC

How can I find out what software is available on the HPCC?

A comprehensive list of all software installed in the HPCC environment can be found in the /usr/usc directory. Most traditional UNIX utilities can be found in /usr/bin.

Why am I getting a “command not found” error when I try to run applications, such as MATLAB or SAS, on HPCC?

In order to use software installed on HPCC, its location must be defined in your PATH. Since /usr/usc is not in your PATH by default, the system is unable to find the requested software.

This can be resolved by sourcing the application’s environment. Below is an example of how to source MATLAB:

If you are using sh or bash, type:

source /usr/usc/matlab/default/setup.sh

If you are running csh or tcsh, type:

source /usr/usc/matlab/default/setup.csh

After running this command, your PATH will be set up properly, with MATLAB in it, and the program will be available to be run.

If you wish to avoid sourcing programs manually in the future, the source command can be placed in your .cshrc file if you are running the csh or tcsh shells or your .bashrc file if you are running the sh or bash shells.

Getting Help

If you have additional questions about HPCC, please send an email to hpcc@usc.edu.