OpenFOAM (Open Field Operation And Manipulation) is a open-source C++ toolbox maintained by the OpenFOAM foundation and ESI Group. Although primarily used for CFD (Computational Fluid Dynamics) OpenFOAM can be used in a wide range of fields from solid mechanics to chemistry.
The lack of licence limitations and native parallelisation makes OpenFOAM well suited for a HPC environment. OpenFOAM is an incredibly powerful tool, but does require a moderate degree of computer literacy to use effectively.
OpenFOAM can be loaded using;
module load OpenFOAM
We maintain multiple different versions of OpenFOAM, an up to date list can be found here or by running
module show OpenFOAM.
Care should be taken when choosing what version to use as execution may vary, especially between the OFF and ESI branches. The two can be told apart as OFF uses incremental version notation (5.0,6.0, etc), while the ESI uses YY-MM (1706, 1812, etc).
We highly recommend specifying the version in your scripts. e.g.
module load OpenFOAM/v1812-gimkl-2017a as opposed to just
module load OpenFOAM.
#description Run simpleFoam in paralell.
#SBATCH --time 04:00:00
#SBATCH --job-name OF_16CORES
#SBATCH --output %x.output #set output to job name
#SBATCH --ntasks 16
#SBATCH --mem-per-cpu 512MB
#Working directory always needs to contain 'system', 'constant', and '0'
#Add this script to start of output
module load OpenFOAM/v1712-gimkl-2017a
decomposePar #Break domain into pieces for parallel execution.
srun simpleFoam -parallel
reconstructPar -latestTime #Collect
OpenFOAM generates a large number of files during run-time. In addition to the I/O load there is also the danger of using up available inodes.
Filesystems in excess of their allocation will cause any job trying to write there to crash.
There are a few ways to mitigate this
The nobackup directory has a significantly higher inode count and no disk space limits.
- ControlDict Settings
Using a high write interval reduce number of output files and I/O load.
Consider carefully an appropriate time-step, use adjustTimeStep if suitable.
Not applicable for many jobs, this keeps only the last n steps, e.g. purgeWrite 5 will keep the last 5 time-steps, with the directories being constantly overwritten.
When true, dictionaries will be re-read at the start of every time step. Setting this to false will decrease I/O load.
Setting this to binary as opposed to ascii will decrease disk use and I/O load.
- Monitor Filesystem
nn_storage_quotashould be used to track filesystem usage. There is a delay between making changes to a filesystem and seeing it on
Filesystem Available Used Use% Inodes IUsed IUse%
home_cwal219 20G 1.957G 9.79% 92160 21052 22.84%
project_nesi99999 2T 798G 38.96% 100000 66951 66.95%
nobackup_nesi99999 6.833T 10000000 2691383 26.91%
- Contact Support
If you are following the recommendations here yet are still concerned about indoes, open a support ticket and we can raise the limit for you.
You may find it useful to use environment variables in your dictionaries e.g.
Or create your variables to be set in your Slurm script.
This is essential when running parameter sweeps.
You can also directly edit your dictionaries with
sed -i "s/\(numberOfSubdomains \)[[:digit:]]*\(;\)/\1 $NSUBDOMAINS\2/g" system/controlDict
Generally, using 16 or less tasks will keep your job reasonably efficient. However this is highly dependant on the type of simulation and how the model was decomposed.
Adding custom solvers
Rather than installing custom solvers centrally, we encourage users to install under their user or project.
Generally your custom solver will be stored in a git repo. Make sure you have the same version as the OpenFOAM you plan to use, this may require changing branch.
If releases are available
Open the 'releases' tab, right click on the '.tar' or '.zip' of the version you want and select 'copy link address', then paste that link into your terminal after
wget. For example:
wget can also be used to fetch files from other sources.
If repo only
Use the command
git clone <path to repo>.git For example:
git clone https://github.com/vincentcasseau/hyStrath.git
If your source is a .zip file use the command
unzip <filename> if it is a .tar.gz use the command
tar -xvzf <filename>
From here steps will vary, it is best to check the README of the package you are installing.
wmake is an OpenFOAM tool used to compile code, based on
A more comprehensive description of the use of wmake can be found here.
A library/application named 'newApp' would have the structure.
To build `newApp` one would run:
module load OpenFOAM
However, most applications will involve building multiple libraries and solvers, and will generally come with a shell script (
.sh) that saves the user having to compile each item manually.
Some apps will try to place the new libraries in
$FOAM_LIBBIN and objects in
$FOAM_APPBIN, this will cause the build to fail as you will not have permission to write there. Make sure the
Make/options file specifies variables
User compiled libraries are kept in
$FOAM_USER_LIBBIN, by default this is set to
User compiled objects are kept in
$FOAM_USER_APPBIN, by default this is set to
You will need to change these locations if you want the rest of your project members to have access.
module load OpenFOAM
These variables need to be set to the same chosen paths before compiling and before running the solvers.
Make sure to
exportyour custom paths before
source $FOAM_BASHelse they will be reset to default.