OpenFOAM (Open Field Operation And Manipulation) is a open-source C++ toolbox maintained by the OpenFOAM foundation and ESI Group. Although primarily used for CFD (Computational Fluid Dynamics) OpenFOAM can be used in a wide range of fields from solid mechanics to chemistry.

The lack of licence limitations and native parallelisation makes OpenFOAM well suited for a HPC environment. OpenFOAM is an incredibly powerful tool, but does require a moderate degree of computer literacy to use effectively.

OpenFOAM can be loaded using;

module load OpenFOAM
source $FOAM_BASH

Example Script

#!/bin/bash -e
#=================================================#
#title          OpenFoamTest.sl
#description    Run simpleFoam in paralell.
#author         NeSI
#=================================================#
#SBATCH --time			    04:00:00
#SBATCH --job-name		    OF_16CORES
#SBATCH --output		    %x.output   #set output to job name
#SBATCH --ntasks		    16
#SBATCH --mem-per-cpu	            512MB
#=================================================#

#Working directory always needs to contain 'system', 'constant', and '0'
DIR_WORKING="/nesi/nobackup/nesi99999/OpenFOAM/testRun"

#Add this script to start of output
cat ${0}

cd ${DIR_WORKING}
module load OpenFOAM/v1712-gimkl-2017a
source ${FOAM_BASH}

decomposePar                   #Break domain into pieces for parallel execution.
srun simpleFoam -parallel       
reconstructPar -latestTime     #Collect 

Filesystem Limitations

OpenFOAM generates a large number of files during run-time. In addition to the I/O load there is also the danger of using up available inodes.

Filesystems in excess of their allocation will cause any job trying to write there to crash.

There are a few ways to mitigate this

• Use /nesi/nobackup
  The nobackup directory has a significantly higher inode count and no disk space limits.
• ControlDict Settings
  
  • WriteInterval
    Using a high write interval reduce number of output files and I/O load.
  • deltaT
    Consider carefully an appropriate time-step, use adjustTimeStep if suitable.
  • purgeWrite
    Not applicable for many jobs, this keeps only the last n steps, e.g. purgeWrite 5 will keep the last 5 time-steps, with the directories being constantly overwritten.
  • runTimeModifiable
    When true, dictionaries will be re-read at the start of every time step. Setting this to false will decrease I/O load.
  • writeFormat
    Setting this to binary as opposed to ascii will decrease disk use and I/O load.
• Monitor Filesystem 
  The command nn_storage_quota should be used to track filesystem usage. There is a delay between making changes to a filesystem and seeing it on nn_storage_quota.

  Filesystem         Available      Used     Use%     Inodes     IUsed     IUse%
  home_cwal219             20G    1.957G    9.79%      92160     21052    22.84%
  project_nesi99999         2T      798G   38.96%     100000     66951    66.95%
  nobackup_nesi99999              6.833T            10000000    2691383   26.91%
  

• Contact Support
  If you are following the recommendations here yet are still concerned about indoes, open a support ticket and we can raise the limit for you.

Environment Variables

You may find it useful to use environment variables in your dictionaries e.g.

numberOfSubdomains ${SLURM_NTASKS};

Or create your variables to be set in your Slurm script.

startFrom ${START_TIME};

 This is essential when running parameter sweeps.

You can also directly edit your dictionaries with sed, e.g.

NSUBDOMAINS=10
sed -i "s/\(numberOfSubdomains \)[[:digit:]]*\(;\)/\1 $NSUBDOMAINS\2/g" system/controlDict

Recommended Resources   

Generally, using 16 or less tasks will keep your job reasonably efficient. However this is highly dependant on the type of simulation and how the model was decomposed.

Adding custom solvers

Rather than installing custom solvers centrally, we encourage users to install under their user or project.

Downloading

Generally your custom solver will be stored in a git repo. Make sure you have the same version as the OpenFOAM you plan to use, this may require changing branch.

If releases are available

Open the 'releases' tab, right click on the '.tar' or '.zip' of the version you want and select 'copy link address', then paste that link into your terminal after wget. For example:

wget https://github.com/vincentcasseau/hyStrath/archive/Concordia.tar.gz

 wget can also be used to fetch files from other sources.

If repo only

Use the command git clone <path to repo>.git For example:

git clone https://github.com/vincentcasseau/hyStrath.git

Decompress 

If your source is a .zip file use the command unzip <filename> if it is a .tar.gz use the command tar -xvzf <filename>

From here steps will vary, it is best to check the README of the package you are installing.

wmake

wmake is an OpenFOAM tool used to compile code, based on make.

A more comprehensive description of the use of wmake can be found here.

A library/application named 'newApp' would have the structure.

 To build `newApp` one would run:

module load OpenFOAM
source $FOAM_BASH
wmake

However, most applications will involve building multiple libraries and solvers, and will generally come with a shell script (.sh) that saves the user having to compile each item manually.

Some apps will try to place the new libraries in $FOAM_LIBBIN and objects in $FOAM_APPBIN, this will cause the build to fail as you will not have permission to write there. Make sure the Make/options file specifies variables $FOAM_USER_LIBBIN or $FOAM_USER_APPBIN instead.

Location

User compiled libraries are kept in $FOAM_USER_LIBBIN, by default this is set to ~/$USER/OpenFOAM/$USER-<version>/platforms/linux64GccDPInt32Opt/lib 

User compiled objects are kept in $FOAM_USER_APPBIN, by default this is set to ~/$USER/OpenFOAM/$USER-<version>/platforms/linux64GccDPInt32Opt/bin

You will need to change these locations if you want the rest of your project members to have access.

For example

module load OpenFOAM
  
source $FOAM_BASH
  
export FOAM_USER_LIBBIN=/nesi/project/nesi99999/custom_OF/lib
export FOAM_USER_APPBIN=/nesi/project/nesi99999/custom_OF/bin

These variables need to be set to the same chosen paths before compiling and before running the solvers.

Warning

Make sure to export your custom paths before source $FOAM_BASH else they will be reset to default.