Mahuika Slurm Partitions

General Limits

  • No individual job can request more than 20,000 CPU hours.
  • No user can have more than 1000 jobs in the queue at a time. This limit can be relaxed for those who need to submit large numbers of jobs, provided that they undertake to do it with job arrays.

Partitions

Partition can be specified via sbatch option, eg:

#SBATCH --partition=long

If undefined the large partition will be used.

Definitions

CPU - Logical Core, also known as a hardware thread. Refereed to as a 'CPU' in slurm documentations.

Fairshare Weight - Core hours are multiplied by this factor to determine usage.

Node - A group of CPU's that share memory.

Walltime - Real world time, as opposed to CPU time (walltime x CPUs)

Name

Max Walltime

Nodes

CPUs/Node

Mem/CPU

Mem/Node

Fairshare
Weight

Description

large

3 days

226

72

1500 MB

108 GB

1

Standard partition, allows large core count jobs.  Due to the above job size limit, large jobs can not be extremely long.

long

3 weeks

69

72

1500 MB

108 GB

1

Standard partition to corral long-duration jobs.  Due to the above job size limit, long jobs can not be extremely large.

prepost

3 hours

5

72

7500 MB

540 GB

1

Short jobs only.  User for pre and post processing tasks in a workflow. More memory per CPU.

bigmem

7 days

4

72

7500 MB

540 GB

2

Standard partition for all other “large memory” jobs. 

hugemem

7 days

0.5

128

32 GB

4 096 GB

4

Can be used to run jobs that need up to 2000 GB of memory.  This is a limited resource and should only be used when "huge" memory is required.

gpu

3 days

4

8

1500 MB

12 GB

56 (per GPU)

Jobs must be set to the gpu partition in order to utilise a GPU. You must also specify;

#SBATCH --gres gpu:1

You should ask for 2 CPU's per GPU

ga_bigmem

7 days

1

72

7500 MB

108 GB

2

Only available to Genomics Aotearoa and debug QoS.

ga_hugemem

7 days

1

128

32 GB

4 096 GB

4

Only available to Genomics Aotearoa and debug QoS.

 

Quality of Service: Debug

Orthogonal to the partitions, each job has a "QoS", with the default QoS for a job being determined by the allocation class of its project. Specifying --qos=debug will override that and give the job very high priority, but is subject to strict limits: 15 minutes per job, and only 1 job at a time per user. Debug jobs are also limited to 72 CPUS.

 

Mahuika Infiniband Islands

The design of Mahuika is optimised for Capacity workloads. However, the Fat Tree (CLOS) Infiniband Network on Mahuika provides full non-blocking InfiniBand network connectivity up to “islands” of 26 nodes (or 936 physical cores). Accordingly, jobs that run entirely within an InfiniBand Island will achieve better application scaling performance than those that cross InfiniBand Island boundaries.

Users can request that the job run within InfiniBand Island by adding the SLURM flag #SBATCH --switches=1 to their batch script, which defines the maximum count of switches desired for the job allocation. We strongly advise that you manually set a maximum waiting time for the selected number of switches, e.g. #SBATCH --switches=1@01:00:00 will make the scheduler wait for maximum one hour before ignoring the switches request.

Caution: If Slurm finds an allocation containing more switches than the count specified, the job remains pending until it either finds an allocation with the desired switch count or the time limit expires. To determine the default wait time see scontrol show config | grep max_switch_wait.

 

 

 

 

Labels: mahuika slurm
Was this article helpful?
2 out of 3 found this helpful