Service Status

Mahuika's new nodes are in an Early Access Programme (EAP) phase and not fully in production.

See Mahuika Extension Onboarding for more information about it.

This article describes a technique to build Apptainer containers using Mahuika Extension nodes, via a Slurm job. You can also build Singularity container using this technique.

Build Environment Variables

To build containers, you need to ensure that Apptainer has enough storage space to create intermediate files. It also requires a cache folder to save images pulled from a different location (e.g. DockerHub). By default both of these locations are set to /tmp which has limited space, large builds may exceed this limitation causing the builder to crash.

The environment variables APPTAINER_TMPDIR and APPTAINER_CACHEDIR environment can be used to overwrite the default location of these directories.

export APPTAINER_CACHEDIR=/nesi/nobackup/PROJECTID/apptainer_cache
export APPTAINER_TMPDIR=/nesi/nobackup/PROJECTID/apptainer_tmpdir
mkdir -p $APPTAINER_CACHEDIR $APPTAINER_TMPDIR
setfacl -b $APPTAINER_TMPDIR

where PROJECTID is your NeSI project ID.

Building container via Slurm

The new Mahuika Extension nodes can be used to build Apptainer containers using the fakeroot feature. This functionality is only available on these nodes at the moment due to their operating system version.

To illustrate this functionality, create an example container definition file my_container.def from a shell session on NeSI as follows:

cat << EOF > my_container.def
BootStrap: docker
From: ubuntu:20.04
%post
    apt-get -y update
    apt-get install -y wget
EOF

Then, from a Mahuika login node, you can build the container using the srun command as follows:

module purge
module load Apptainer/1.1.2
srun -p milan --time 0-00:30:00 --mem 4GB --cpus-per-task 2 apptainer build --fakeroot my_container.sif my_container.def

This command will start an interactive Slurm job for 30 minutes using 2 cores and 4 GB of memory to build the image. Make sure to set these resources correctly, some containers can take hours to build and require tens of GB of memory.

If you need more resources to build your container, please consider submitting your job using a Slurm submission script, for example:

#!/bin/bash -e
#SBATCH --job-name=apptainer_build
#SBATCH --partition=milan
#SBATCH --time=0-00:30:00
#SBATCH --mem=4GB
#SBATCH --cpus-per-task=2

module purge
module load Apptainer/1.1.2
apptainer build --fakeroot my_container.sif my_container.def

More information about how to submit a Slurm job is available in the Submitting your first job support page.

Known limitations

If your container uses RPM to install packages, i.e. is based on CentOS or Rocky Linux, you need to disable the APPTAINER_TMPDIR environment variable (use unset APPTAINER_TMPDIR) and request more memory for your Slurm job. Otherwise, RPM will crash due to an incompatibility with the nobackup filesystem.

If you encounter the following error when using a base Docker image in your Apptainer definition file

While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: unsupported image-specific operation on artifact with type "application/vnd.docker.container.image.v1+json"

it is likely due to an upstream issue (e.g. bad image on Dockerhub). In this case, try an older image version or a different base image.

Other limitations

This method, using fakeroot, is known to not work for all types of Apptainer/Singularity containers.

If you encounter an issue, please contact us at support@nesi.org.nz.