Q-Chem on HPC

Q-Chem is a software package for analyzing “molecular structures, reactivities, and vibrational, electronic, and NMR spectra” that is available to HPC researchers by permission. To request access to this software on HPC, please email hpc@usc.edu

NOTE: Due to licensing restrictions with Gaussian, HPC users may not use Q-Chem and Guassian concurrently. Please email hpc@usc.edu to switch access.

This page will help you set up Q-Chem on your HPC account.

Setting Up Your Q-Chem Environment

Q-Chem requires a number of environment variables to run properly on HPC. Some environment settings are configured automatically when you run Q-Chem’s HPC setup script:

$ source /usr/usc/qchem/5.1/setup.sh
$ env | grep QC
QC=/auto/usc/qchem/5.1
QCPROG_S=/auto/usc/qchem/5.1/exe/qcprog.exe_s
QCRSH=ssh
QCPLATFORM=LINUX_Ix86_64
QCAUX=/auto/usc/qchem/5.1/qcaux
QCPROG=/auto/usc/qchem/5.1/exe/qcprog.exe
QCMPI=mpich

Other variables that you will need to configure include the scratch directory, the machine file, and a location for your temporary files. We also recommend that you configure your Q-Chem job for checkpointing.

Scratch Directory

Q-Chem requires that you set the environment variable QCSCRATCH. You should set it to the directory that Q-Chem will use for storing temporary files when running a job.

One suggestion is to set QCSCRATCH to $SCRATCHDIR. SCRATCHDIR is the temporary directory that Slurm creates for its own temporary job files.

export QCSCRATCH=$SCRATCHDIR

Machine File

Q-Chem also requires that you set the environment variable QCMACHINEFILE. This should be set to a text file that contains a list of all hostnames that your job has access to.

This information is only relevant for the duration of the job, so we suggest you place the file in $QCSCRATCH.

export QCMACHINEFILE=$QCSCRATCH/machinefile

You can create machinefile by redirecting the output of the Slurm command scontrol show hostname to $QCMACHINEFILE which will display the machines being used for the current job.

scontrol show hostname > machinefile

Each Q-Chem job may have different requirements for the machine file; it may be necessary to modify the output of the scontrol command depending on your job’s requirements.

Saving Temporary Files

If you wish to save KEY temporary files produced by Q-Chem during a run, you can run the program with a save directory argument. By default, Q-Chem will save the files to $QCSCRATCH/savedir.

NOTE: Slurm’s $SCRATCHDIR is deleted at the end of each job. If you have set QCSCRATCH to $SCRATCHDIR, you MUST copy the files to a persistent location before the end of your job.

export QCSCRATCH=$SCRATCHDIR
qchem data.in data.out data.savedir 
cp -r $QCSCRATCH/data.savedir .

Alternatively, you can set QCSCRATCH to a persistent location. Do this if you want to save ALL temporary files (normally not required) using the qchem –save option.

export QCSCRATCH=/staging/tt1/ttrojan
qchem -save data.in data.out data.savedir 

Checkpointing Option

HPC recommends checkpointing your Q-Chem job, especially if you plan to use the scavenge partition. A Q-Chem checkpoint file can be requested by setting GUI = 2 in the $rem section of the input file. The checkpoint file name is determined by the $GUIFILE environment variable.

export GUIFILE=data.fchk

Running a Q-Chem Job

The most basic way to run Q-Chem on a single processor (default) would be to run a command in the following form:

qchem data.in data.out data.savedir &

where data.in is the name of your Q-Chem input file, data.out is the name of the file where Q-Chem will place its output, and data.dir is the directory where Q-Chem will save key temporary files.

NOTE: If an output file already exists, it will be overwritten.

The & places the job into the background so that you may continue to work in the current shell.

See the Running Q-Chem manual page on the Q-Chem website for more details.

Command-Line Options

To run Q-Chem in parallel, the number of the number of threads, nt, (assumes 1 thread=1 CPU core) and the number of MPI processes (CPUs), np, may need be specified. If np is not specified, Q-Chem will default to running locally on a single node.

The default command, above, is equivalent to:

qchem -nt 1 data.in data.out data.savedir &

When -np is not given, Q-Chem will default to running locally on a single node.

Parallel Options

Q-Chem supports multiple methods to run in parallel, as described in the table below.

Type Description Slurm Requirements
OpenMP Multiple threads (CPUs) Single task
(Multiple CPUs per task)
MPI Multiple MPI processes
(One CPU each)
Multiple tasks
(One CPU per task)
MPI+OpenMP (Hybrid) Multiple MPI processes
(Multiple CPUs each)
Multiple tasks
(Multiple CPUs per task)

Example Q-Chem Script

There are a number of sample Q-Chem jobs available in /usr/usc/qchem/<version>/samples. You can use these sample input files as a starting place to building your own files.

To use a sample directory, copy it to your project directory and run it there. Compare your output with the reference files (.out) of the same name.

cp -r /auto/usc/qchem/5.1/samples/freq .

Below is a template script that can be used to run Q-Chem on either a single- or multi-node job using a sample input file.

#!/bin/bash
#SBATCH --export=none
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=12
#SBATCH --time=1:00:00
#SBATCH --mem-per-cpu=2GB
#SBATCH --job-name=qchem

source /usr/usc/qchem/5.1/setup.sh

# Define the scratch dir for savedir and machine file
export QCSCRATCH=$SCRATCHDIR
export QCMACHINEFILE=$SCRATCHDIR/machinefile

# Create machine file for this job
scontrol show hostname > $QCMACHINEFILE

# Sample files were copied to this directory
cd $SLURM_SUBMIT_DIR/freq

# $SLURM_NTASKS will take the value provided by --ntasks option
# $SLURM_CPUS_PER_TASK will take the value provided by --cpus-per-task option

qchem -np $SLURM_NTASKS -nt $SLURM_CPUS_PER_TASK FREQ_water.in FREQ_water.out FREQ_water

# Save key temporary files for curiosity
cp -r $QCSCRATCH/FREQ_water .

Getting Help

For assistance with running Q-Chem on HPC, see our Getting Help page or send an email to hpc@usc.edu.