Hyperthreading is enabled on the NeSI machines, so for each physical CPU core, there are two logical cores. This increases the efficiency of some multithreaded jobs, but the fact that Slurm is counting in logical cores makes aspects of running non-hyperthreaded jobs confusing, even when hyperthreading is turned off in the job with --hint=nomultithread.
- Non-hyperthreaded jobs which use --mem-per-cpu requests should halve their memory requests as those are based on memory per logical core, not per the number of threads or tasks. For non-MPI jobs it may be clearer to just specify --mem (ie: memory per node) instead.
- Non-MPI jobs which specify --cpus-per-task and use srun should also set --ntasks=1, otherwise the program will be run twice in parallel, halving the efficiency of the job.
The precise rules about when hyperthreading applies are:
- Jobs never share physical cores, so are always allocated an even number of logical cores.
- Tasks do share physical cores by default on Māui, but that can be overridden with --hint=nomultithread.
- Threads do share physical cores by default, but that can be overridden with --hint=nomultithread.
Since tasks (by default) and jobs (always) do not share physical cores, if you request an odd number of CPUs you can expect the number actually allocated to be rounded up to the nearest even number, so we recommend that multithreaded jobs request an even numbers of CPUs.