WRF

The Weather Research and Forecasting (WRF) Model is a next-generation mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting applications. It features two dynamical cores, a data assimilation system, and a software architecture supporting parallel computation and system extensibility. The model serves a wide range of meteorological applications across scales from tens of meters to thousands of kilometres.

WRF on Maui

This guide is based on WRF 4.1.1 and WPS 4.2 (it was also tested with Polar WRF 4.1.1 but for that, you would need to apply the Polar WRF and WPS patches before compiling).

Building WRF

Download WRF:

cd /nesi/project/<your_project_code>
wget https://github.com/wrf-model/WRF/archive/v4.1.1.tar.gz
tar xf v4.1.1.tar.gz
cd WRF-4.1.1

Load environment modules and set up the build environment (here we will build with the Intel compiler):

# load PrgEnv-intel (this assumes you currently have PrgEnv-cray loaded)
module switch PrgEnv-cray PrgEnv-intel

# load dependencies
module load cray-netcdf cray-hdf5 cray-parallel-netcdf

# the following two lines are optional and may improve performance
module switch craype-x86-skylake craype-broadwell
module load craype-hugepages2M

export NETCDF=$NETCDF_DIR
export HDF5=$HDF5_DIR
export PNETCDF=$PNETCDF_DIR

Apply patches for Polar WRF if required and then configure WRF:

./configure
# choose option 50 - INTEL (ftn/icc) Cray XC (dmpar)
# choose an appropriate nesting option or leave it at the default

The configure script takes some options, for example, to configure for a debug build (no optimisations and debug symbols enabled) you could run ./configure -d instead.

The configure script writes out a configure.wrf file which can also be edited manually if desired. For example, if you want an optimised build with debug symbols included, you could add -g to the FCDEBUG variable.

Next compile WRF:

./compile em_real >& log.compile

It is important to check the output in log.compile as the command will not exit with an error if it fails. Check the end of the log file for a message saying compilation was successful. You could also check the timestamps of the executables in the main subdirectory (main/wrf.exe, main/real.exe, etc).

Running WRF

An example Slurm script for running WRF on Maui, which can be submitted with sbatch name_of_script.sl:

#!/bin/bash -e
#SBATCH --job-name=wrf
#SBATCH --time=01:00:00
#SBATCH --ntasks=240
#SBATCH --hint=nomultithread

# important, required on Maui
export HDF5_USE_FILE_LOCKING=FALSE

# optional, may improve performance
module load craype-hugepages2M

# run real
srun --output=real.log ./real.exe

# run wrf
srun --output=wrf.log ./wrf.exe

Parallel netCDF

For some model configurations enabling parallel IO can be beneficial. By default WRF uses serial netCDF IO, which can be verified by looking in the namelist.input file for the io_form_* variables (these will mostly be in the time_control section of the namelist). If these variables are set to "2" then they will use serial netCDF. Setting them to 11 will use parallel netCDF instead. It is also recommended (possibly required) to set nocolons = .true. in the time_control section of the namelist when using parallel IO.

Building WPS

Download WPS:

wget https://github.com/wrf-model/WPS/archive/v4.2.tar.gz
tar xf v4.2.tar.gz
cd WPS-4.2

Load environment modules and set up the build environment (here we will build with the Intel compiler). We will build WPS using dynamic linking so that we can link against the system JasPer shared library.

# load PrgEnv-intel (this assumes you currently have PrgEnv-cray loaded)
module switch PrgEnv-cray PrgEnv-intel

# load dependencies
module load cray-netcdf cray-hdf5 cray-parallel-netcdf

# the following two lines are optional and may improve performance
module switch craype-x86-skylake craype-broadwell
module load craype-hugepages2M

export NETCDF=$NETCDF_DIR
export HDF5=$HDF5_DIR
export PNETCDF=$PARALLEL_NETCDF_DIR
export JASPERLIB=/usr/lib/libjasper.so.1
export JASPERINC=/usr/include/jasper/jasper.h
export WRF_DIR=../WRF-4.1.1
export CRAYPE_LINK_TYPE=dynamic

Configure WPS:

./configure
# choose option 39 - Cray XC Intel parallel build

This creates the configure.wps file, which can be edited in a similar way to configure.wrf above, if desired.

Now compile WPS:

./compile >& log.compile

Check that the executables were successfully created: geogrid.exe, ungrib.exe and metgrid.exe.

Running WPS

Both geogrid and metgrid are parallel applications and can be run on Maui compute nodes. However, ungrib is serial and should not be run on a compute node unless it is very quick to finish. Alternatively you could run ungrib on an interactive/login node if it will not take up many resources, or you could compile WRF and WPS on a Maui Ancillary node and run it there. 

Note that WPS does a lot of file IO and therefore probably won't scale up to as many processes as WRF.

This example shows running geogrid in a Slurm job:

#!/bin/bash -e
#SBATCH --job-name=wps
#SBATCH --time=01:00:00
#SBATCH --ntasks=40
#SBATCH --hint=nomultithread

# need to specify modules as we linked dynamically
module unload PrgEnv-cray PrgEnv-intel PrgEnv-gnu
module load PrgEnv-intel cray-netcdf cray-hdf5 cray-parallel-netcdf

# the following two lines are optional and may improve performance
module switch craype-x86-skylake craype-broadwell
module load craype-hugepages2M

# important, required on Maui
export HDF5_USE_FILE_LOCKING=FALSE

# run geogrid
srun --output=geogrid.log ./geogrid.exe