The HPC GPU Cluster

The HPC graphics processing unit (GPU) cluster consists of 264 Hewlett-Packard SL250 servers, each with dual 8-core 2.4 gigahertz Intel E5-2665 processors, 64 gigabytes of memory, 1 terabyte of internal disk, and two NVIDIA K20 Kepler GPU accelerators. A 56.6 gigabit, low-latency, FDR InfiniBand network connects these servers. These GPU nodes are accessible from the main computing queue.

CUDA and OpenACC Program Development

In order to use the GPUs, you will need to program your jobs using the compute unified device architecture application platform interface (CUDA API) or the OpenACC API.

Adding CUDA Configuration Files

You will need to add configuration files to your HPC account in order to create CUDA-based applications.

To add CUDA program development capabilities to your account, add the following source commands to your account’s configuration file:

For accounts using the bash, sh, or ksh shells:

  • If you are using the gnu compiler (nvcc) with C and C++, add the following source command to your account’s .profile:
source /usr/usc/cuda/default/setup.sh
  • If you are using the PGI compiler (pgfortran) with Fortran, add the following source command to your account’s .profile:
source /usr/usc/pgi/13.3/setup.sh
  • If you are using the gnu compilers (mpicc or mpicxx) with C and C++ message passing interface (MPI) programs, add the following source commands to your account’s .profile:
source /usr/usc/cuda/default/setup.sh
source /usr/usc/openmpi/1.6.4/setup.sh
  • If you are using the Intel compiler (mpif90) with Fortran MPI programs, add the following source commands to your account’s .profile:
source /usr/usc/cuda/default/setup.sh
source /usr/usc/openmpi/1.6.4/share/setup-intel.sh
source /usr/usc/intel/default/setup.sh

For accounts using the tcsh or csh shells:

  • If you are using the gnu compiler (nvcc) with C and C++, add the following source command to your account’s .login:
source /usr/usc/cuda/default/setup.csh
  • If you are using the PGI compiler (pgfortran) with Fortran, add the following source command to your account’s .login:
source /usr/usc/pgi/13.3/setup.csh
  • If you are using the gnu compilers (mpicc or mpicxx) with C and C++ MPI programs, add the following source commands to your account’s .login:
source /usr/usc/cuda/default/setup.csh
source /usr/usc/openmpi/1.6.4/setup.csh
  • If you are using the Intel compiler (mpif90) with Fortran MPI programs, add the following source commands to your account’s .login:
source /usr/usc/cuda/default/setup.csh
source /usr/usc/openmpi/1.6.4/share/setup-intel.csh
source /usr/usc/intel/default/setup.csh

If you are uncertain which of these combination will best suit your needs, we recommend that you use one of the combinations listed below for program development.

Single Node Programming

For single node GPU program development, you may use the following API and programming language combinations.

For accounts using the bash, sh, or ksh shells:

  • To use the CUDA API with the C or C++ programming languages:
    • 1. Add this source command to your account’s .profile

      source /usr/usc/cuda/default/setup.sh

      2. Use nvcc as your C or C++ compiler.

  • To use the CUDA API with the Fortan programming language:
    • 1. Add these source commands to your account’s .profile:

      source /usr/usc/cuda/default/setup.sh
      source /usr/usc/pgi/13.3/setup.sh

      2. Use pgfortran as your Fortran compiler.

  • To use the OpenACC API with the C or C++ programming languages:
    • 1. Add this source command to your account’s .profile:

      source /usr/usc/pgi/13.3/setup.sh

      2. Use pgcc as your C or C++ compiler.

  • To use the OpenACC API with the Fortran programming language:
    • 1. Add this source command to your account’s .profile:

      source /usr/usc/pgi/13.3/setup.sh

      2. Use pgfortran as your Fortran compiler.

For accounts using the tcsh or csh shells:

  • To use the CUDA API with the C or C++ programming languages:
    • 1. Add this source command to your account’s .login:

      source /usr/usc/cuda/default/setup.csh

      2. Use nvcc as your C or C++ compiler.

  • To use the CUDA API with the Fortan programming language:
    • 1. Add these source commands to your account’s .login:

      source /usr/usc/cuda/default/setup.csh 
      source /usr/usc/pgi/13.3/setup.csh

      2. Use pgfortran as your Fortran compiler.

  •  To use the OpenACC API with the C or C++ programming languages:
    • 1. Add this source command to your account’s .login:

      source /usr/usc/pgi/13.3/setup.csh

      2. Use pgcc as your C or C++ compiler.

  • To use the OpenACC API with the Fortran programming language:
    • 1. Add this source command to your account’s .login:

      source /usr/usc/pgi/13.3/setup.csh

      2. Use pgfortran as your Fortran compiler.

Multiple Node Programming

OpenMPI, which takes advantage of the Infiniband network, should be used when creating a multiple-node CUDA application based on MPI. OpenMPI also allows users to use different compilers.

Here are some examples of configuration files for CUDA program development.

For accounts using the bash, sh, or ksh shells:

  • To use the CUDA API with the C or C++ programming languages:
    • 1. Add the following source commands to your account’s .profile:

      source /usr/usc/cuda/default/setup.sh 
      source /usr/usc/openmpi/1.6.4/setup.sh

      2. Use mpicc as your C compiler or use mpicxx as your C++ compiler.

  • To use the CUDA API with the Fortan programming language:
    • 1. Add the following source commands to your account’s .profile:

      source /usr/usc/cuda/default/setup.sh 
      source /usr/usc/openmpi/1.6.4/share/setup-intel.sh 
      source /usr/usc/intel/default/setup.sh

      2. Use mpif90 as your Fortran compiler.

For accounts using the tcsh or csh shells:

  • To use the CUDA API with the C or C++ programming languages:
    • 1. Add the following source commands to your account’s .login:

      source /usr/usc/cuda/default/setup.csh 
      source /usr/usc/openmpi/1.6.4/setup.csh

      2. Use mpicc as your C compiler or use mpicxx as your C++ compiler.

  • To use the CUDA API with the Fortan programming language:
    • 1. Add the following source commands to your account’s .login:

      source /usr/usc/cuda/default/setup.csh 
      source /usr/usc/openmpi/1.6.4/share/setup-intel.csh 
      source /usr/usc/intel/default/setup.csh

      2. Use mpif90 as your Fortran compiler.

Applications with GPU support

The following applications have GPU support and can be run on HPC’s GPU cluster. To run these programs on the GPUs, you will need to add the application-specific configuration files to your account’s configuration file.

For accounts using the bash, sh, or ksh shells:

  • To add GROMACS (4.6.1), add the following line to your account’s .profile:
 source /usr/usc/gromacs/4.6.1/setup.sh
  • To add NAMD 2.9, add the following line to your account’s .profile:
 source /usr/usc/NAMD/2.9-gpu/setup.sh
  • To add Matlab 2013a, add the following line to your account’s .profile:
source /usr/usc/matlab/default/setup.sh

For accounts using the tcsh or csh shells:

  • To add GROMACS (4.6.1), add the following line to your account’s .login:
source /usr/usc/gromacs/4.6.1/setup.csh
  • To add NAMD 2.9, add the following line to your account’s .login:
 source /usr/usc/NAMD/2.9-gpu/setup.csh
  • To add Matlab 2013a, add the following line to your account’s .login:
 source /usr/usc/matlab/default/setup.csh

Requesting GPU nodes

For details on how to request GPU-enabled nodes in your portable batch system (PBS) script, please see the section on requesting GPU nodes on the following page: Running a Job on HPC using PBS.

Getting Help

If you have any questions, please contact us at hpc@usc.edu.