Māui Slurm Partitions

NeSI Access to Māui (XC50) and Māui_Ancil (CS500)

Partitions on these systems that may be used for NeSI workloads carry the prefix "nesi_". 

Definitions

CPU - Logical Core, also known as a hardware thread. Referred to as a "CPU" in the Slurm documentation.  Since Hyperthreading is enabled, there are two CPUs per physical core and every job is allocated an even number of CPUs.

Charging on Māui (XC50)

By default nodes are not shared on Maui, and the minimum charging unit is node-hours, where 1 node-hour is 40 core-hours, or 80 Slurm CPU-hours.

Māui (XC50) Slurm Partitions

 Name 

Nodes

Max Walltime

Avail / Node

Max / Job

Max / User

Max in Queue

Description

nesi_research

264

24 hours

80 CPUs

80 or 160 GB RAM

66 nodes

168 node-hours

4 jobs

10 jobs

Standard partition for all  jobs.

Quality of Service: nesi_debug

Orthogonal to the Māui partition, each job has a "QoS", with the default QoS for a job being determined by the allocation class of its project. Specifying --qos=nesi_debug will override that and give the job very high priority, but is subject to strict limits: 20 minutes per job, and only 1 job at a time per user. Debug jobs are limited to 8 nodes.

Māui Electrical Groups

The XC nodes in Māui are connected using a Dragonfly network. On each blade are 4 nodes, sharing a network interconnect (NICs). A chassis is build of 16 blades, where all NICs are connected all-to-all. Six chassis build one electrical group (two cabinets), where every NIC is connected to NICs of all other chassis (copper cables). Furthermore, the electrical groups are connect in an all-to-all fashion using optical cables.

Māui consist of 3 cabinets, where the first two contain 336 compute nodes, and Cabinet 2 (an electrical group of only one cabinet) holds 128 compute nodes (only two of the three chassis in Cabinet 2 are populated). In certain circumstances, applications might experience a slowdown if the application is placed across both electrical groups.

Users might prevent this situation by adding the SLURM flag #SBATCH --switches=1 to their batch script, which defines the maximum count of switches desired for the job allocation. We strongly advise that you manually set a maximum waiting time for the selected number of switches, e.g. #SBATCH --switches=1@00:01:00 will make the scheduler wait for maximum one hour before ignoring the switches request.

Caution: If SLURM finds an allocation containing more switches than the count specified, the job remains pending until it either finds an allocation with the desired switch count or the time limit expires. To determine the default wait time see scontrol show config | grep max_switch_wait.

 

maui_ancil Slurm Partitions

Partition  

Nodes

Max Walltime

Avail / Node

Max / Job

Max / User

Description

nesi_prepost

4

24 hours

80 CPUs

720 GB RAM

20 CPUs

700 GB RAM

80 CPUs

700 GB RAM

Pre and post processing tasks.

nesi_gpu

4 to 5

72 hours

4 CPUs

12 GB RAM

1 GPU

4 CPUs

12 GB RAM

1 GPU

4 CPUs

12 GB RAM

1 GPU

GPU jobs and visualisation. 

nesi_igpu

0 to 1

2 hours

4 CPUs

12 GB RAM

1 GPU

4 CPUs

12 GB RAM

1 GPU

4 CPUs

12 GB RAM

1 GPU

Interactive GPU access 7am - 8pm.

 

 

Labels: maui slurm
Was this article helpful?
0 out of 0 found this helpful