ABAQUS

Licences

A list of commands can be found with:

abaqus help

Hyperthreading can provide significant speedup to your computations, however hyperthreaded CPUs will use twice the number of licence tokens. It may be worth adding  #SBATCH --hint nomultithread to your slurm script if licence tokens are your main limiting factor.

Tips

Required ABAQUS licences can be determined by this simple and intuitive formula ⌊ 5 x N0.422 where N is number of CPUs.

Solver Compatibility

Not all solvers are compatible with all types of parallelisation.

  Element operations Iterative solver Direct solver Lanczos solver
mp_mode=threads
mp_mode=mpi

Note

If your input files were created using an older version of ABAQUS you will need to update them using the command,

abaqus -upgrade -job new_job_name -odb old.odb

or

abaqus -upgrade -job new_job_name -inp old.inp

Serial


For when only one CPU is required, generally as part of an job array.

 

#!/bin/bash -e

#SBATCH --job-name      ABAQUS-Shared
#SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 1 #SBATCH --mem 1500 # total mem module load ABAQUS/2019
abaqus job="propeller_s4rs_c3d8r" verbose=2 interactive

Shared Memory


mp_mode=threads

Uses a nodes shared memory for communication. 

May have a small speedup compared to MPI when using a low number of CPUs, scales poorly. Needs significantly less memory than MPI.

Hyperthreading may be enabled if using shared memory but it is not recommended.
#!/bin/bash -e

#SBATCH --job-name      ABAQUS-Shared
#SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 4 #SBATCH --mem 2G # total mem module load ABAQUS/2019
abaqus job="propeller_s4rs_c3d8r" verbose=2 interactive \
cpus=${SLURM_CPUS_PER_TASK} mp_mode=threads

UDF


Shared memory run with user defined function (fortran or C). 

user=<name_of_function> 

Function will be compiled at start of run. 

You may need to chance the function suffix if you usually compile on windows.

#!/bin/bash -e

#SBATCH --job-name      ABAQUS-SharedUDF
#SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 4 #SBATCH --mem 2G # total mem
module load gimkl module load ABAQUS/2019
abaqus job="propeller_s4rs_c3d8r" user=my_udf.f90 verbose=2 interactive \
cpus=${SLURM_CPUS_PER_TASK} mp_mode=threads

Distributed Memory


mp_mode=mpi

Multiple processes each with a single thread.

Not limited to one node.
Model will be segmented into -np pieces which should be equal to --ntasks.

Each task could be running on a different node leading to increased communication overhead
.Jobs can be limited to a single node by adding  --nodes=1 however this will increase your time in the queue as contiguous cpu's are harder to schedule.

This is the default method if mp_mode is left unspecified.

#!/bin/bash -e

#SBATCH --job-name      ABAQUS-Distributed 
#SBATCH --time 00:05:00 # Walltime #SBATCH --ntasks 8 #SBATCH --mem-per-cpu 1500 # Each CPU needs it's own.
#SBATCH --nodes 1 module load ABAQUS/2019
abaqus job="propeller_s4rs_c3d8r" verbose=2 interactive \
cpus=${SLURM_NTASKS} mp_mode=mpi

GPUs


The GPU nodes are limited to 16 CPUs

In order for the GPUs to be worthwhile, you should see a speedup equivalent to 56 CPU's per GPU used. GPU modes will generally have less memory/cpus

#!/bin/bash -e

#SBATCH --job-name      ABAQUS-gpu
#SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 4 #SBATCH --mem 4G # total mem
#SBATCH --gpus-per-node 1 module load ABAQUS/2019
module load CUDA
abaqus job="propeller_s4rs_c3d8r" verbose=2 interactive \
cpus=${SLURM_CPUS_PER_TASK} gpus=${SLURM_GPUS_PER_NODE} mp_mode=threads

User Defined Functions 

User defined functions (UDFs) can be included on the command line with the argument user=<filename> where <filename> is the C or fortran source code.

Make sure you have loaded the correct compiler with module load (for example module load gimkl/2020a). Extra compiler options can be set in your local abaqus_v6.env file. 

Environment file

The ABAQUS environment file contains a number of parameters that define how the your job will run, some of these you may with to change.

These parameters are read, 

../ABAQUS/SMA/site/abaqus_v6.env Set by NeSI and cannot be changed.

~/abaqus_v6.env (your home directory) If exists, will be used in all jobs submitted by you.

<working directory>/abaqus_v6.env If exists, will used in this job only.

 

Useful Links

 

ABAQUS_speedup_SharedVMPI.png

 

Note: Hyperthreading off, testing done on small mechanical FEA model. Results highly model dependant. Do your own tests.

Labels: mahuika application tier2 engineering gpu mpi omp
Was this article helpful?
1 out of 2 found this helpful
a.homepage:before