ANSYS

The ANSYS package is a commercial bundle of engineering software which includes the ANSYS finite element analysis software together with other components such as Fluent and CFX for computational fluid dynamics. Homepage here.

ANSYS is currently supported on both Māui and Mahuika. If your job can efficiently use an entire node or more, it is probably best run on Māui.

Licensing requirements

Currently the only ANSYS licencss available on our clusters belong to University of Auckland, Engineering faculty. If you are not a member of this group, please contact our support desk.

License Types

The three main ANSYS licenses are;

  • ANSYS Teaching License (ansys_t)
      This is the default license type, it can be used on up to 16 CPUs on models with less than 512k nodes.
  • ANSYS Research license (ansys_r)
      No node restrictions. Can be used on up to 16 CPUs, for every additional CPU over 16 you must request additional 'ansys_hpc' licenses.
  • ANSYS HPC License (ansys_hpc)
      One of these is required for each CPU over 16 when using a research license.

License Order

Whether to use a teaching or research license must be set manually. If your job is greater than the node limit, not switching to the research license before submitting a job will cause the job to fail.

The license order can be changed in workbench under tools > license preferences (provided you have X11 forwarding set up), or by running either of the following (ANSYS module must be loaded first using module load ANSYS).

prefer_research_license
prefer_teaching_license

Note

License preferences are individually tracked by each version of ANSYS. Make sure you set preferences using the same version as in your script.

License Count

License count can be set in the Slurm header using --licenses for example, a 64 cpu job will have to request.

#SBATCH --licenses=ansys_hpc:48,ansys

Journal files

Some ANSYS applications take a 'journal' text file as input. It is often useful to create this journal file in your SLURM script (tidyness, submitting jobs programmatically, etc). This can be done by using cat to make a file from a 'heredoc'.

Below is an example of this from a fluent script.

#!/bin/bash -e

#SBATCH --job-name      Fluent_Array
#SBATCH --time          01:00:00          # Wall time
#SBATCH --mem           3G                # Memory per node
#SBATCH --licenses      ansys_hpc:1       # One license token per CPU, less 16
#SBATCH --array 1:100
#SBATCH --hint nomultithread # No hyperthreading module load ANSYS/19.2
JOURNAL_FILE=fluent_${SLURM_JOB_ID}.in
cat <<EOF > ${JOURNAL_FILE}
rcd testCase${SLURM_ARRAY_TASK_ID}.cas
/solve/dual-time-iterate 10
/file/write-case-data testOut${SLURM_ARRAY_TASK_ID}.cas
exit
EOF
# Use one of the -v options 2d, 2ddp, 3d, or 3ddp fluent -v3ddp -g -i ${JOURNAL_FILE}
rm ${JOURNAL_FILE}

JOURNAL_FILE is a variable holding the name of a file, the next line cat creates the file then writes a block of text into it. The block of text written is everything between an arbitrary string (in this case EOF) and its next occurrence.

In this case (assuming it is the first run of the array and the jobid=1234567), the file  fluent_1234567.in will be created:

rcd testCase1.cas
/solve/dual-time-iterate 10
/file/write-case-data testOut1.cas
exit

then called as an input fluent -v3ddp -g -i fluent_1234567.in,
then deleted rm fluent_1234567.in

This can be used with variable substitution to great effect as it allows the use of variables in what might otherwise be a fixed input.

Fluent

fluent -help for a list of commands.

Must have one of these flags. 

-v2d 2D solver, single point precision.
-v3d 3D solver, single point precision.
-v2dpp 2D solver, double point precision.
-v3dpp 3D solver, double point precision.

 

Serial


Single process with a single thread (2 threads if hyperthreading enabled).

Usually submitted as part of an array, as in the case of parameter sweeps.

#!/bin/bash -e

#SBATCH --job-name      Fluent-Serial
#SBATCH --licenses ansys_r@uoa_foe:1 #One research license.
#SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 1 # Double if hyperthreading enabled #SBATCH --mem 3000 # total mem #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2
JOURNAL_FILE=/share/test/ansys/fluent/wing.in fluent -v3ddp -g -i ${JOURNAL_FILE}

Distributed Memory 


Multiple processes each with a single thread.

Not limited to one node.
Model will be segmented into -np pieces which should be equal to --ntasks.

Each task could be running on a different node leading to increased communication overhead
.Jobs can be limited to a single node by adding  --nodes=1 however this will increase your time in the queue as contiguous cpu's are harder to schedule.

#!/bin/bash -e

#SBATCH --job-name          Fluent-Dis
#SBATCH --time              00:05:00          # Walltime
#SBATCH --licenses ansys_r@uoa_foe:1,ansys_hpc@uoa_foe:20
##One research license, (ntasks-16) hpc licenses
#SBATCH --nodes 1 # Limit to n nodes (Optional) #SBATCH --ntasks 36 # Number processes
#SBATCH --cpus-per-task 1 # Double if hyperthreading enabled #SBATCH --mem-per-cpu 1500 # Standard for large partition #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2 JOURNAL_FILE=/share/test/ansys/fluent/wing.in fluent -v3ddp -g -t ${SLURM_NTASKS} -i ${JOURNAL_FILE}

Useful Links

CFX

cfx5solve -help for a list of commands.

Serial


Single process with a single thread (2 threads if hyperthreading enabled).

Usually submitted as part of an array, as in the case of parameter sweeps.

#!/bin/bash -e

#SBATCH --job-name      CFX-serial
#SBATCH --licenses ansys_r@uoa_foe:1 #One research license.
#SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 1 # Double if hyperthreading enabled #SBATCH --mem 3000 # total mem #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2 input=/share/test/ansys/cfx/pump.def cfx5solve -batch -def "$input"

Distributed Memory 


Multiple processes each with a single thread.

Not limited to one node.
Model will be segmented into -np pieces which should be equal to --ntasks.

Each task could be running on a different node leading to increased communication overhead
.Jobs can be limited to a single node by adding  --nodes=1 however this will increase your time in the queue as contiguous cpu's are harder to schedule.

#!/bin/bash -e

#SBATCH --job-name          ANSYS-Dis
#SBATCH --time              00:05:00          # Walltime
#SBATCH --licenses ansys_r@uoa_foe:1,ansys_hpc@uoa_foe:20
##One research license, (ntasks-16) hpc licenses
#SBATCH --nodes 1 # Limit to n nodes (Optional) #SBATCH --ntasks 36 # Number processes
#SBATCH --cpus-per-task 1 # Double if hyperthreading enabled #SBATCH --mem-per-cpu 1500 # Standard for large partition #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2 input=/share/test/ansys/mechanical/structural.dat cfx5solve -batch -def "$input" -part $SLURM_NTASKS

Mechanical APDL

 

Serial


Single process with a single thread (2 threads if hyperthreading enabled).

Usually submitted as part of an array, as in the case of parameter sweeps.

#!/bin/bash -e

#SBATCH --job-name      ANSYS-serial
#SBATCH --licenses ansys_r@uoa_foe:1 #SBATCH --time 00:05:00 # Walltime #SBATCH --cpus-per-task 1 # Double if hyperthreading enabled #SBATCH --mem 3000 # total mem #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2 input=/share/test/ansys/mechanical/structural.dat mapdl -b -i "$input"

Shared memory


Single process multiple threads.

All threads must be on the same node, limiting scalability.
Number of threads is set by -np and should be equal to --cpus-per-task.


Not recommended if using more than 8 cores (16 CPUs if hyperthreading enabled).

#!/bin/bash -e

#SBATCH --job-name      ANSYS-Shared
#SBATCH --licenses ansys_r@uoa_foe:1 #SBATCH --time 00:05:00 # Walltime #SBATCH --ntasks 1 # Any more than 1 will not be utilised. #SBATCH --cpus-per-task 8 # Double if hyperthreading enabled #SBATCH --mem 12G # 8 threads at 1.5 GB per thread #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2 input=/share/test/ansys/mechanical/structural.dat mapdl -b -np ${SLURM_CPUS_PER_TASK} -i "$input"

Distributed Memory 


Multiple processes each with a single thread.

Not limited to one node.
Model will be segmented into -np pieces which should be equal to --ntasks.

Each task could be running on a different node leading to increased communication overhead
.Jobs can be limited to a single node by adding  --nodes=1 however this will increase your time in the queue as contiguous cpu's are harder to schedule.

Distributed Memory Parallel is currently not supported on Māui.

#!/bin/bash -e

#SBATCH --job-name          ANSYS-Dis
#SBATCH --licenses ansys_r@uoa_foe:1,ansys_hpc@uoa_foe:4 #SBATCH --time 00:05:00 # Walltime
#SBATCH --nodes 1 # Limit to n nodes #SBATCH --ntasks 16 # Number processes
#SBATCH --cpus-per-task 1 # Double if hyperthreading enabled #SBATCH --mem-per-cpu 1500 # Standard for large partition #SBATCH --hint nomultithread # Hyperthreading disabled module load ANSYS/19.2 input=/share/test/ansys/mechanical/structural.dat mapdl -b -dis -np ${SLURM_NTASKS} -i "$input"

Not all MAPDL solvers work using distributed memory. 

Sparse
PCG
ICCG
JCG
QMR
Block Lanczos eigensolver
PCG Lanczos eigensolver
Supernode eigensolver
Subspace eigensolver
Unsymmetric eigensolver
Damped eigensolver
QRDAMP eigensolver
Element formulation
Results calculation
Pre/Postprocessing

Useful Links

LS-DYNA

Example

#!/bin/bash -e
#SBATCH --job-name      LS-DYNA
#SBATCH --account       nesi99999         # Project Account
#SBATCH --time          01:00:00          # Walltime
#SBATCH --ntasks        16                # Number of CPUs to use
#SBATCH --mem-per-cpu   1500              # Memory per cpu
#SBATCH --hint          nomultithread     # No hyperthreading

module load ANSYS/18.1
input=3cars_shell2_150ms.k
lsdyna -dis -np $SLURM_NTASKS i="$input" memory=$(($SLURM_MEM_PER_CPU/8))M

Best Practices

GPU acceleration support

GPUs can be slow for smaller jobs because it takes time to transfer data from the main memory to the GPU memory. We therefore suggest that you only use them for larger jobs, unless benchmarking reveals otherwise.

Interactive use

It is best to use journal files etc to automate ANSYS so that you can submit batch jobs, but when interactivity is really needed alongside more CPU power and/or memory than is reasonable to take from a login node (maybe postprocessing a large output file) then an alternative which may work is to run the GUI frontend on a login node while the MPI tasks it launches run on a compute node. This requires using salloc instead of sbatch, for example:

salloc -A nesi99999 -t 30 -n 16 -C avx --mem-per-cpu=2G bash -c 'module load ANSYS; fluent -v3ddp -t$SLURM_NTASKS' 

As with any job, you may have to wait a while before the resource is granted and you can begin, so you might want to use the --mail-type=BEGIN and --mail-user= options.

Hyperthreading

Utilising hyperthreading (ie: removing the "--hint=nomultithread" sbatch directive and doubling the number of tasks) will give a small speedup on most jobs with less than 8 cores, but also doubles the number of ansys_hpc license tokens required.

Labels: mahuika tier1 engineering app
Was this article helpful?
0 out of 0 found this helpful