Follow

GROMACS

Description

GROMACS (the GROningen MAchine for Chemical Simulations) is a versatile package to perform molecular dynamics, i.e., simulate the Newtonian equations of motion for systems with hundreds to millions of particles.

The GROMACS home page is at http://www.gromacs.org.

Available modules

Packages with modules

Module NeSI Cluster
GROMACS/5.1.2-intel-2015a-hybrid-PLUMED pan
GROMACS/4.6.5-intel-2015a-hybrid pan
GROMACS/5.1.2-foss-2015a-hybrid pan

Licensing requirements

GROMACS is available to anyone at no cost under the terms of the GNU Lesser General Public Licence.

Example scripts

Example scripts for the Pan cluster

Basic usage

#!/bin/bash -e

#SBATCH --job-name      MyGROMACSJob
#SBATCH --account       nesi99999
#SBATCH --time          01:00:00
#SBATCH --ntasks        8                   # One CPU per task is assumed
#SBATCH --mem-per-cpu   4G
#SBATCH --output        MyGROMACSJob.%j.out # Include the job ID in the names
#SBATCH --error         MyGROMACSJob.%j.err # of the output and error files

module load GROMACS/4.6.5-intel-2015a-hybrid

# This script assumes that grompp has already been run, as grompp is a
# single-cpu program.

# Note that not all options shown here are backwards compatible with old
# versions of Gromacs.

srun mdrun_mpi \
    -cpt 30 \
    -pin on \
    -s preprocessed_input.tpr \
    -o output_trajectory.trr \
    -c output_coordinates.gro \
    -e output_energies.edr \
    -g MyGromacsJob.log

Fast simulation using GPU support provided by CUDA

#!/bin/bash -e

#SBATCH --job-name        MyGromacsJob
#SBATCH --account         nesi99999
#SBATCH --time            01:00:00
#SBATCH --ntasks          16
#SBATCH --nodes           8
#SBATCH --ntasks-per-node 2
#SBATCH --cpus-per-task   1
#SBATCH --mem-per-cpu     32G
#SBATCH --constraint      avx
#SBATCH --gres            gpu:2
#SBATCH --output          MyGromacsJob.%j.out # Include the job ID in the names
#SBATCH --error           MyGromacsJob.%j.err # of the output and error files

module load CUDA/6.0.37
module load GROMACS/4.6.5-goolf-1.5.14-hybrid-CUDA-6.0.37

# Note that not all options shown here are backwards compatible with old
# versions of Gromacs.

# Use parallel grompp
srun grompp_mpi \
    -f md_gpu.mdp \
    -c input_coordinates.gro \
    -p topology.top \
    -o preprocssed_input.tpr

srun mdrun_mpi \
    -nb gpu \
    -cpt 30 \
    -s preprocessed_input.tpr \
    -o output_trajectory.trr \
    -c output_coordinates.gro \
    -e output_energies.edr \
    -g MyGromacsJob.log

Advanced usage with fast temporary disk

#!/bin/bash -e

#SBATCH --job-name      MyGromacsJob
#SBATCH --account       nesi99999
#SBATCH --time          01:00:00
#SBATCH --ntasks        8                   # One CPU per task is assumed
#SBATCH --mem-per-cpu   4G
#SBATCH --output        MyGromacsJob.%j.out # Include the job ID in the names
#SBATCH --error         MyGromacsJob.%j.err # of the output and error files

module load GROMACS/4.6.5-intel-2015a-hybrid

# Set the working directory where the job will
# actually be run, and create that directory
thisdir=$(pwd -P)
workdir="${CHK_DIR}/${SLURM_JOB_NAME}-${SLURM_JOB_ID}"
mkdir -p "${workdir}"
cd "${workdir}"

# This script assumes that grompp has already been run, as grompp is a
# single-cpu program.

# Note that not all options shown here are backwards compatible with old
# versions of Gromacs.

srun mdrun_mpi \
    -cpt 30 \
    -pin on \
    -s preprocessed_input.tpr \
    -o output_trajectory.trr \
    -c output_coordinates.gro \
    -e output_energies.edr \
    -g MyGromacsJob.log

# Copy output back to this directory, and
# remove it if the copy is successful
for file in ${workdir}/*
do
    cp -arv --no-preserve=mode "${file}" "${thisdir}" && rm -rfv "${file}"
done

Further notes

Checkpointing and restarting

In the examples given above, the -cpt 30 option instructs Gromacs to write a full checkpoint file every 30 minutes. You can restart from a checkpoint file using the -cpi flag, thus: -cpi state.cpt.

Using default filenames as input to mdrun

When running mdrun or mdrun_mpi, you may use the -deffnm flag to instruct the executable to use default file names for its input and output, instead of having to specify each argument one at a time. In the above job commands, for instance, -deffnm would replace the -s, -o, -c, -e and -g flags.

Warnings regarding CPU affinity

If you run GROMACS on a node that is simultaneously running other jobs (even other GROMACS jobs), you may see warnings like this in your output:

WARNING: In MPI process #0: Affinity setting failed. This can cause performance degradation! If you think your setting are correct, contact the GROMACS developers.

One way to prevent these warnings, which is also useful for reducing the risk of inefficient CPU usage, is to request entire nodes. On the Pan cluster, this can be done using the following lines in your input, altered as appropriate:

  • On Sandy Bridge nodes, using MPI parallelisation only:
#SBATCH --ntasks          128  # May vary as long as divisible by 16
#SBATCH --nodes           8    # Must be ntasks/16
#SBATCH --ntasks-per-node 16   # Must be 16
                               # (the number of cores per Sandy Bridge node)
#SBATCH --cpus-per-task   1    # Must be 1
#SBATCH --constraint      sb   # Specifies Sandy Bridge nodes
  • On Ivy Bridge nodes, using MPI parallelisation only:
#SBATCH --ntasks          192  # May vary as long as divisible by 24
#SBATCH --nodes           8    # Must be ntasks/24
#SBATCH --ntasks-per-node 24   # Must be 24
                               # (the number of cores per Ivy Bridge node)
#SBATCH --cpus-per-task   1    # Must be 1
#SBATCH --constraint      ib   # Specifies Ivy Bridge nodes
  • On Sandy Bridge nodes, using hybrid (OpenMP + MPI) parallelisation:
#SBATCH --ntasks          8    # Must be equal to the number of nodes
#SBATCH --nodes           8    # May vary
#SBATCH --ntasks-per-node 1    # Must be 1
#SBATCH --cpus-per-task   16   # Must be 16
                               # (the number of cores per Sandy Bridge node)
#SBATCH --constraint      sb   # Specifies Sandy Bridge nodes
  • On Ivy Bridge nodes, using hybrid (OpenMP + MPI) parallelisation:
#SBATCH --ntasks          8    # Must be equal to the number of nodes
#SBATCH --nodes           8    # May vary
#SBATCH --ntasks-per-node 1    # Must be 1
#SBATCH --cpus-per-task   24   # Must be 24
                               # (the number of cores per Ivy Bridge node)
#SBATCH --constraint      ib   # Specifies Ivy Bridge nodes

If using hybrid parallelisation, it is also important to run mdrun_mpi with the -ntomp <number> option, where <number> should be the number of CPUs per task (i.e., 16 for Sandy Bridge or 24 for Ivy Bridge).

Comments

Powered by Zendesk