Licences
OpenFOAM (Open Field Operation And Manipulation) is a open-source C++ toolbox maintained by the OpenFOAM foundation and ESI Group. Although primarily used for CFD (Computational Fluid Dynamics) OpenFOAM can be used in a wide range of fields from solid mechanics to chemistry.
The lack of licence limitations and native parallelisation makes OpenFOAM well suited for a HPC environment. OpenFOAM is an incredibly powerful tool, but does require a moderate degree of computer literacy to use effectively.
OpenFOAM can be loaded using;
module load OpenFOAM
source $FOAM_BASH
We maintain multiple different versions of OpenFOAM, an up to date list can be found here or by running module show OpenFOAM
.
Care should be taken when choosing what version to use as execution may vary, especially between the OFF and ESI branches. The two can be told apart as OFF uses incremental version notation (5.0,6.0, etc), while the ESI uses YY-MM (1706, 1812, etc).
We highly recommend specifying the version in your scripts. e.g. module load OpenFOAM/v1812-gimkl-2017a
as opposed to just module load OpenFOAM
.
Example Script
#!/bin/bash -e
#=================================================#
#title OpenFoamTest.sl
#description Run simpleFoam in paralell.
#author NeSI
#=================================================#
#SBATCH --time 04:00:00
#SBATCH --job-name OF_16CORES
#SBATCH --output %x.output #set output to job name
#SBATCH --ntasks 16
#SBATCH --mem-per-cpu 512MB
#=================================================#
#Working directory always needs to contain 'system', 'constant', and '0'
DIR_WORKING="/nesi/nobackup/nesi99999/OpenFOAM/testRun"
#Add this script to start of output
cat ${0}
cd ${DIR_WORKING}
module load OpenFOAM/v1712-gimkl-2017a
source ${FOAM_BASH}
decomposePar #Break domain into pieces for parallel execution.
srun simpleFoam -parallel
reconstructPar -latestTime #Collect
Filesystem Limitations
OpenFOAM generates a large number of files during run-time. In addition to the I/O load there is also the danger of using up available inodes.
Filesystems in excess of their allocation will cause any job trying to write there to crash.
There are a few ways to mitigate this
- Use
/nesi/nobackup
The nobackup directory has a significantly higher inode count and no disk space limits. - ControlDict Settings
WriteInterval
Using a high write interval reduce number of output files and I/O load.deltaT
Consider carefully an appropriate time-step, use adjustTimeStep if suitable.purgeWrite
Not applicable for many jobs, this keeps only the last n steps, e.g. purgeWrite 5 will keep the last 5 time-steps, with the directories being constantly overwritten.runTimeModifiable
When true, dictionaries will be re-read at the start of every time step. Setting this to false will decrease I/O load.writeFormat
Setting this to binary as opposed to ascii will decrease disk use and I/O load.
- Monitor Filesystem
The commandnn_storage_quota
should be used to track filesystem usage. There is a delay between making changes to a filesystem and seeing it onnn_storage_quota
.Filesystem Available Used Use% Inodes IUsed IUse%
home_cwal219 20G 1.957G 9.79% 92160 21052 22.84%
project_nesi99999 2T 798G 38.96% 100000 66951 66.95%
nobackup_nesi99999 6.833T 10000000 2691383 26.91% - Contact Support
If you are following the recommendations here yet are still concerned about indoes, open a support ticket and we can raise the limit for you.
Environment Variables
You may find it useful to use environment variables in your dictionaries e.g.
numberOfSubdomains ${SLURM_NTASKS};
Or create your variables to be set in your Slurm script.
startFrom ${START_TIME};
This is essential when running parameter sweeps.
You can also directly edit your dictionaries with sed
, e.g.
NSUBDOMAINS=10
sed -i "s/\(numberOfSubdomains \)[[:digit:]]*\(;\)/\1 $NSUBDOMAINS\2/g" system/controlDict
Recommended Resources
Generally, using 16 or less tasks will keep your job reasonably efficient. However this is highly dependant on the type of simulation and how the model was decomposed.
Adding custom solvers
Rather than installing custom solvers centrally, we encourage users to install under their user or project.
Downloading
Generally your custom solver will be stored in a git repo. Make sure you have the same version as the OpenFOAM you plan to use, this may require changing branch.
If releases are available
Open the 'releases' tab, right click on the '.tar' or '.zip' of the version you want and select 'copy link address', then paste that link into your terminal after wget
. For example:
wget https://github.com/vincentcasseau/hyStrath/archive/Concordia.tar.gz
wget can also be used to fetch files from other sources.
If repo only
Use the command git clone <path to repo>.git
For example:
git clone https://github.com/vincentcasseau/hyStrath.git
Decompress
If your source is a .zip file use the command unzip <filename>
if it is a .tar.gz use the command tar -xvzf <filename>
From here steps will vary, it is best to check the README of the package you are installing.
wmake
wmake
is an OpenFOAM tool used to compile code, based on make
.
A more comprehensive description of the use of wmake can be found here.
A library/application named 'newApp' would have the structure.
To build `newApp` one would run:
module load OpenFOAM
source $FOAM_BASH
wmake
However, most applications will involve building multiple libraries and solvers, and will generally come with a shell script (.sh
) that saves the user having to compile each item manually.
Some apps will try to place the new libraries in $FOAM_LIBBIN
and objects in $FOAM_APPBIN
, this will cause the build to fail as you will not have permission to write there. Make sure the Make/options
file specifies variables $FOAM_USER_LIBBIN
or $FOAM_USER_APPBIN
instead.
Location
User compiled libraries are kept in $FOAM_USER_LIBBIN
, by default this is set to ~/$USER/OpenFOAM/$USER-<version>/platforms/linux64GccDPInt32Opt/lib
User compiled objects are kept in $FOAM_USER_APPBIN
, by default this is set to ~/$USER/OpenFOAM/$USER-<version>/platforms/linux64GccDPInt32Opt/bin
You will need to change these locations if you want the rest of your project members to have access.
For example
module load OpenFOAM
source $FOAM_BASH
export FOAM_USER_LIBBIN=/nesi/project/nesi99999/custom_OF/lib
export FOAM_USER_APPBIN=/nesi/project/nesi99999/custom_OF/bin
These variables need to be set to the same chosen paths before compiling and before running the solvers.
Warning
Make sure to
export
your custom paths beforesource $FOAM_BASH
else they will be reset to default.