Job Prioritisation on Mahuika

Introduction to Job Priorities

In Slurm, every pending job receives a priority score. This priority score is used to determine the order in which jobs will start. A job is only eligible to start if its execution will not delay a higher-priority job.

Job Size

In addition to the job's priority, the actual order in which jobs start depends on the resources those jobs request. Jobs which request more resources (CPUs, memory, GPUs etc.) may have to wait longer for those resources to become available.

We have default maximum allowable job size of 512 cores. You can request permission to run jobs larger than that by emailing support@nesi.org.nz.

Job Duration

Jobs with shorter wall time limits have a natural advantage in the scheduler (see What is Backfill? for more information) and jobs of less than 3 days duration have the advantage of being allowed to run on any of the general-purpose compute nodes, while longer jobs are restricted to a subset of these via the long partition (see Mahuika Partitions). 

Multi-Factor Prioritisation

Slurm offers several prioritisation schemes. Our current scheme is the multi-factor prioritisation scheme. This means the priority is made up of weighted contributions from various aspects of the job. 

How are Priorities Calculated on Mahuika?

We calculate priorities as outlined in the following table.

Factor How Calculated
QoS Between 10 and 40, based on the project allocation class, or 1000 if "--qos=debug" is specified.
Age Time spent in the queue in hours. A job will reach its maximum possible age score after three weeks in the queue.
Size The number of cores requested (approximately). We prioritise large jobs because small jobs naturally have more opportunities to start, and bumping the expected start time of a large job forward in time makes the scheduler's job that much harder.
Fair Share Between 1 and 1000 based on the degree to which your share of recent (exponentially weighted average) cluster usage is over or under what is expected for your project, as determined by your allocation of CPU hours. For more information on fair share, please see the Slurm fair share documentation.
Nice When submitting a job you can choose to lower its priority by a "nice" value of up to 1,000 points, e.g., sbatch --nice=100 ....

 

How Can I See Priority Weighting Factors?

Run the following command at the command prompt on a login node:

sprio -w

How Can I See the Priorities of Currently Pending Jobs?

The following command shows how the priorities for your currently queued jobs were calculated:

sprio -u $USER

What is Backfill?

Backfill is a scheduling strategy that allows small, short jobs to jump the queue if by doing so they can use otherwise idle machinery. If not for backfill, jobs would start in strict order of priority, and the cluster as a whole would be less productive. You can take advantage of backfill by breaking your work up into small, short jobs.

While the kinds of jobs that can be backfilled will also get a low job size score, it is our general experience that an ability to be backfilled is on the whole more useful when it comes to getting work done on the cluster.

A job will not be backfilled if by doing so it would delay the start of another higher-priority job.

More Information

The Slurm documentation on multi-factor prioritisation is available here. More information about backfill can be found here.

Labels: mahuika
Was this article helpful?
0 out of 0 found this helpful