If you are unsure about using our job scheduler Slurm, more details can be found here.
Slurm Commands
A complete list of Slurm commands can be found here, or by entering man slurm into a terminal
| sbatch | sbatch submit.sl |
Submits the SLURM script submit.sl |
| squeue | squeue | Displays entire queue. |
| squeue -u usr9999 | Displays queued jobs submitted by usr9999. | |
| squeue -p long | Displays queued jobs on the long partition. | |
| sacct | sacct | Displays all the jobs run by you that day. |
| sacct -S 2019-01-01 | Displays all the jobs run by you since the 1st Jan 2019 | |
| sacct -j 123456789 | Displays job 123456789 | |
| scancel | scancel 123456789 | Cancels job 123456789 |
| scancel -u usr9999 | Cancels all your jobs (assuming you are usr9999). If you are not usr9999, you will get an error message. | |
| sshare | sshare -U usr9999 | Shows the Fair Share scores for all projects of which usr9999 is a member. |
| sinfo | sinfo | Shows the current state of our SLURM partitions. |
| sview | sview | Launches a GUI for monitoring SLURM jobs. |
SBATCH Variables
A complete list of SBATCH variables can be found here, or by running man sbatch
Options can be delimited using an '=' sign e.g. #SBATCH --account=nesi99999 or a space e.g. #SBATCH --account nesi99999. But not both!
General variables
| --job-name | #SBATCH --job-name=MyJob |
The name that will appear when using squeue or sacct |
| --account | #SBATCH --account=nesi99999 |
The account your core hours will be 'charged' too. |
| --time | #SBATCH --time=DD-HH:MM:SS |
Job max walltime |
| --mem | #SBATCH --mem=512MB |
Memory required. |
| --partition | #SBATCH --partition=long |
Specified job partition. (Default large) |
| --output | #SBATCH --output=%j_output.out |
Path and name of standard output file. |
| --mail-user | #SBATCH --mail-user=bob123@gmail.com |
Address to send mail notifications. |
| --mail-type | #SBATCH --mail-type=ALL |
Will send a mail notification at BEGIN END FAIL |
#SBATCH --mail-type=TIME_LIMIT_80 |
Will send message at 80% walltime |
Parallel variables
| --ntasks | #SBATCH --ntasks=2 |
Will start 2 MPI tasks. |
| --mem-per-cpu | #SBATCH --mem-per-cpu=512MB |
Memory Per logical CPU |
| --array | #SBATCH --array=1-10 |
Will submit job 10 times each with a different $SLURM_ARRAY_TASK_ID |
Other
| --qos | #SBATCH --qos=debug |
Adding this line gives your job a very high priority. (Limited to one job at a time, max 15 minutes) |
| --profile | #SBATCH --profile=ALL |
Allows generation of a .h5 file containing job profile information. More here. |
| --dependency | #SBATCH --dependency=afterok:123456789 |
Will only start after the job 123456789 has completed. |
| --hint | #SBATCH --hint=nomultithread |
Disables hyperthreading, be aware that this will significantly change how your job is defined. |
Tip
Variables have a short and long form e.g.
#SBATCH --job-name=MyJob&#SBATCH -J=MyJob.echo "Completed task ${SLURM_ARRAY_TASK_ID} / ${SLURM_ARRAY_TASK_COUNT} successfully"
Tokens
A token is a predefined variable that can be used in the SBATCH header
%x |
Job name |
%u |
User name. |
%j |
Job ID |
%a |
Job array Index |
Environment variables
Common examples.
$SLURM_JOB_ID |
Useful for naming output files that won't clash. |
$SLURM_JOB_NAME |
|
$SLURM_ARRAY_TASK_ID |
The current index of your array job. |
$SLURM_CPUS_PER_TASK |
Useful as an input for multi-threaded functions. |
$SLURM_NTASKS |
Useful as an input for MPI functions. |
$SLURM_SUBMIT_DIR |
Directory where sbatch was called. |
Tip
In order to decrease the chance of a variable (environment or otherwise) being misinterpreted you should use the syntax
${NAME_OF_VARIABLE}and define in strings if possible. e.g.echo "Completed task ${SLURM_ARRAY_TASK_ID} / ${SLURM_ARRAY_TASK_COUNT} successfully"