SLURM: Reference Sheet

If you are unsure about using our job scheduler SLURM, more details can be found here.

SLURM Commands

A complete list of SLURM commands can be found here, or by entering man slurm into a terminal

sbatch sbatch submit.sl Submits the SLURM script submit.sl
squeue squeue Displays entire queue.
squeue -u usr9999 Displays queued jobs submitted by usr9999.
squeue -p long Displays queued jobs on the long partition.
sacct sacct Displays all the jobs run by you that day.
sacct -S 2019-01-01 Displays all the jobs run by you since the 1st Jan 2019
sacct -j 123456789 Displays job 123456789
scancel scancel 123456789 Cancels job 123456789
scancel -u usr9999 Cancels all your jobs (assuming you are usr9999). If you are not usr9999, you will get an error message.
sshare sshare -U usr9999 Shows the Fair Share scores for all projects of which usr9999 is a member.
sinfo sinfo Shows the current state of our SLURM partitions.
sview sview Launches a GUI for monitoring SLURM jobs.

 


 

SBATCH Variables

A complete list of SBATCH variables can be found here, or by running man sbatch

Options can be delimited using an '=' sign e.g. #SBATCH --account=nesi99999 or a space e.g. #SBATCH --account nesi99999But not both!

General variables

--job-name #SBATCH --job-name=MyJob The name that will appear when using squeue or sacct
--account #SBATCH --account=nesi99999 The account your core hours will be 'charged' too.
--time #SBATCH --time=DD-HH:MM:SS Job max walltime
--mem #SBATCH --mem=3000 Memory required.
--partition #SBATCH --partition=long Specified job partition. (Default large)
--output #SBATCH --output=%j_output.out Path and name of standard output file.
--mail-user #SBATCH --mail-user=bob123@gmail.com Address to send mail notifications.
--mail-type #SBATCH --mail-type=ALL Will send a mail notification at BEGIN END FAIL
#SBATCH --mail-type=TIME_LIMIT_80 Will send message at 80% walltime

Parallel variables

--ntasks #SBATCH --ntasks=2 Will start 2 MPI tasks.
--mem-per-cpu #SBATCH --mem-per-cpu=1500 Memory Per logical CPU
--array #SBATCH --array=1-10 Will submit job 10 times each with a different $SLURM_ARRAY_TASK_ID

 

Other

--qos #SBATCH --qos=debug Adding this line gives your job a very high priority. (Limited to one job at a time, max 15 minutes) 
--profile #SBATCH --profile=ALL Allows generation of a .h5 file containing job profile information. More here.
--dependency #SBATCH --dependency=afterok:123456789 Will only start after the job 123456789 has completed.
--hint #SBATCH --hint=nomultithread Disables hyperthreading, be aware that this will significantly change how your job is defined.

Tip

Variables have a short and long form e.g. #SBATCH --job-name=MyJob & #SBATCH -J=MyJob.

echo "Completed task ${SLURM_ARRAY_TASK_ID} / ${SLURM_ARRAY_TASK_COUNT} successfully"

Tokens

A token is a predefined variable that can be used in the SBATCH header

%x Job name
%u User name.
%j Job ID 
%a Job array Index

Environment variables

Common examples.

$SLURM_JOB_ID Useful for naming output files that wont clash.
$SLURM_JOB_NAME  
$SLURM_ARRAY_TASK_ID The current index of your array job. 
$SLURM_CPUS_PER_TASK Useful as an input for multi-threaded functions.
$SLURM_NTASKS Useful as an input for MPI functions.
$SLURM_SUBMIT_DIR Directory where sbatch was called.

Tip

In order to decrease the chance of a variable (environment or otherwise) being misinterpreted you should use the syntax ${NAME_OF_VARIABLE} and define in strings if possible. e.g.

echo "Completed task ${SLURM_ARRAY_TASK_ID} / ${SLURM_ARRAY_TASK_COUNT} successfully"
Was this article helpful?
3 out of 3 found this helpful