Apptainer

Apptainer is an open-source container platform designed specifically for high-performance computing (HPC) environments. While similar to Docker in many ways, Apptainer stands out by prioritizing integration over isolation. For example, containers in Apptainer run with the same user ID as the host system user (by default), and the user’s home directory is automatically mounted inside the container. This design simplifies user experience and avoids complex permission management.

Built from the ground up for HPC workloads, Apptainer offers native support for GPU acceleration and high-performance interconnects like InfiniBand and Omni-Path, making it ideal for running scientific and parallel applications. It also enables native execution of Docker containers, effectively serving as a Docker replacement on HPC systems without the need for root daemons or privilege escalation.

Key advantages of Apptainer include:

• Security: Users inside the container have the same privileges as on the host—no root access required or allowed by default.

• Simplicity: Containers are executed as regular binaries—no background services or daemons needed.

• Flexibility: Easily bind local file systems and access hardware devices.

• Performance: Seamlessly run MPI and GPU-based HPC workflows within containers.

In summary, Apptainer is a powerful container solution purpose-built for HPC, combining ease of use, performance, and security to support scientific and research applications at scale.

Build Your Container

We recommend working from your /scratch directory when first building your container.

Once the interactive session starts, set the Apptainer temporary directory:

export APPTAINER_TMPDIR=/scratch/$USER/apptainer/tmp
mkdir -p $APPTAINER_TMPDIR

A very common use case is to take an existing docker image and convert it into the singularity container format.Luckily this can be easily done by running the build command with e.g. the docker hub name of the image prefixed by docker:// and a name for the output image.

!ml av apptainer

1h=
----------------------------- /apps/modules/tools -----------------------------
   apptainer/1.1.5    apptainer/1.2.5 (D)

  Where:
   D:  Default Module

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".


?1l>

#### Using Public Containers from Docker Hub with Apptainer

Docker Hub is the most widely used registry for publicly available container images. Many open-source projects publish their containers there, making it a go-to source when you need to run or build containers.

With Apptainer, running a container directly from Docker Hub is straightforward. Simply prefix the image name with docker:// followed by the repository and tag.

For example, to run a container called python:3.11-slim hosted on Docker Hub, you can use:

!ml apptainer/1.2.5

!apptainer build python-3.13-slim.sif docker://python:3.13-slim

INFO:    Starting build...
Copying blob 2a47a8c4fd5c skipped: already exists  
Copying blob 2a47a8c4fd5c skipped: already exists  
Copying blob 47bbb0afa7fe skipped: already exists  
Copying blob 9c538fc35491 skipped: already exists  
Copying blob 8a628cdd7ccc skipped: already exists  
Copying blob 2a47a8c4fd5c skipped: already exists  
Copying blob 47bbb0afa7fe skipped: already exists  
Copying blob 9c538fc35491 skipped: already exists  
Copying blob 8a628cdd7ccc skipped: already exists  
Copying blob 2a47a8c4fd5c skipped: already exists  
Copying blob 47bbb0afa7fe skipped: already exists  
Copying blob 9c538fc35491 skipped: already exists  
Copying blob 8a628cdd7ccc skipped: already exists  
Copying config 0334b36fd0 [----------------------------] 0.0b / 5.2KiB | 0.0 b/s
Copying config 0334b36fd0 done   | 
Copying config 0334b36fd0 done   | 
Copying config 0334b36fd0 done   | 
Copying config 0334b36fd0 done   | 
Writing manifest to image destination
2025/04/21 15:10:37  info unpack layer: sha256:8a628cdd7ccc83e90e5a95888fcb0ec24b991141176c515ad101f12d6433eb96
2025/04/21 15:10:37  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/04/21 15:10:37  warn xattr{/tmp/build-temp-2851987064/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/04/21 15:10:37  info unpack layer: sha256:2a47a8c4fd5c358f98fb193471dc5d7a1e75e33d480fb7a25a7130840a8769d9
2025/04/21 15:10:37  warn xattr{var/cache/apt/archives/partial} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/04/21 15:10:37  warn xattr{/tmp/build-temp-2851987064/rootfs/var/cache/apt/archives/partial} destination filesystem does not support xattrs, further warnings will be suppressed
2025/04/21 15:10:37  info unpack layer: sha256:9c538fc354918cfe13231798fe17d7c4b463cda38500187c0ecedcc061d6129e
2025/04/21 15:10:38  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/04/21 15:10:38  warn xattr{/tmp/build-temp-2851987064/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/04/21 15:10:38  info unpack layer: sha256:47bbb0afa7fe5695aca89f358dc3e73f46655977e201f553a558b4590b951dac
INFO:    Creating SIF file...
INFO:    Build complete: python-3.11-slim.sif

$apptainer run python-3.11-slim.sif

This command will launch an Apptainer container and execute a runscript if one is defined for that container. The runscript is a metadata file within the container that contains shell commands.

!apptainer exec python-3.13-slim.sif python untitled.py

INFO:    Environment variable SINGULARITY_BINDPATH is set, but APPTAINER_BINDPATH is preferred
INFO:    Environment variable SINGULARITYENV_LD_PRELOAD is set, but APPTAINERENV_LD_PRELOAD is preferred
The current time is 15:12:14.47

This command runs the untitled.py script inside the container, using the Python interpreter provided by the python-3.11-slim.sif image. It ensures the script executes in a controlled, containerized environment without relying on the host system’s Python setup.

If you build an Apptainer container from the image docker://python:3.11 then when I run the container I can see that the PYTHON_VERSION variable is set in the container:

!apptainer exec python-3.13-slim.sif env | grep PYTHON_VERSION

INFO:    Environment variable SINGULARITY_BINDPATH is set, but APPTAINER_BINDPATH is preferred
INFO:    Environment variable SINGULARITYENV_LD_PRELOAD is set, but APPTAINERENV_LD_PRELOAD is preferred
PYTHON_VERSION=3.11.12

After the image has been built successfully, the default command of the container (aka the runscript if there is one) can be run by executing:

Build from definition files

An Apptainer Definition File (or .def file) serves as a blueprint for building custom container images. It specifies everything needed for the build process—such as the base operating system or Docker image to start from, required software packages, environment variables, host files to include, and metadata.

Whether you’re customizing an existing Docker image or creating a container from scratch, the definition file provides a clear and structured way to describe the contents and configuration of your final container image.

An Apptainer Definition File is structured in two main parts:

• Header – This section defines the base environment of the container, such as the Linux distribution, its version, and any core packages to include. It lays the foundation for the container’s operating system and configuration.

• Sections – These optional components, marked with a % followed by the section name (e.g., %post, %environment), contain scripts or data that define the build and runtime behavior. Build-time sections are executed with /bin/sh and can include shell options, while runtime sections generate scripts that configure the container when it’s used.

Preferred bootstrap agents:

• docker (images hosted on Docker Hub)

• oras (images from supporting OCI registries)

• library (images hosted on Library API Registries)

• scratch (a flexible option for building a container from scratch)

the definition file sections are executed follows the sequence outlined in the official documentation.

Let’s create an example definition file using the Miniconda3 Docker image as the base. Miniconda3 provides a minimal conda installation, which we’ll use to set up a Python environment inside the container. The following sections will be used in the definition file, listed in their order of execution:

• %files - a line by line list of source files that should be copied to a destination

• %post - section to install packages, run scripts etc

• %test - runs at the end of the build process and can be used to run scripts to validate that the build is successful

• %environment - lets the user define and set environment variables in the container

• %runscript - a script that is executed when the run command is used on the container

• %labels - can be used to add metadata to the /.singularity.d/labels.json

• %help - a help text to display when the help command is used with the container

%%writefile apptainer-demo-env.yaml
name: myenv
channels:
  - conda-forge
dependencies:
  - python=3.12.7
  - numpy=2.2.5

Overwriting apptainer-demo-env.yaml

%%writefile demo.def

Bootstrap: docker
From: continuumio/miniconda3  # or conda/miniconda3

%files
    apptainer-demo-env.yaml /opt/project/
    # add the test python script from tooling and its dependency
    untitled.py /opt/project/bin/
%post
    apt-get update -y
    apt-get clean

    # Give execution rights to script
    chmod +x /opt/project/bin/untitled.py
    # Create Conda environment from YAML
    conda env create -f /opt/project/apptainer-demo-env.yaml -p /opt/project/apptainer-demo-env

    # Activate conda env for container sessions
    echo "conda activate /opt/project/apptainer-demo-env" >> /environment
    
%environment
    export LC_ALL=C
    export PATH=/opt/project/bin:/usr/games:$PATH
    export PYTHONPATH=/opt/project/lib:$PYTHONPATH
    source activate /opt/project/apptainer-demo-env

%runscript
    echo "Container was created"

%test
    grep -q "NAME=\"Debian GNU/Linux\"" /etc/os-release
    if [ $? -eq 0 ]; then
        echo "Container base is Debian as expected."
    else
        echo "Container base is not Debian."
        exit 1
    fi

%labels
    Author python4hpc

%help
    This is a demo image based on Miniconda to illustrate the usage of def files and Conda environments.

Writing demo.def

After writing the definition file, you can build the container image by first loading the Apptainer module (if necessary) and then running:

!apptainer build apptainer-demo.sif apptainer-demo.def

INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    The %post section will be run under the fakeroot command
INFO:    Starting build...
Copying blob 94a9b8494f17 skipped: already exists  
Copying blob 3e5bad3d9203 skipped: already exists  
Copying blob ef8fe905bfbd skipped: already exists  
Copying blob 564874600cfe skipped: already exists  
Copying blob 6e909acdb790 skipped: already exists  
Copying blob b63682e94411 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 5485e84c0a6c skipped: already exists  
Copying blob af9342aa445b skipped: already exists  
Copying blob 94a9b8494f17 skipped: already exists  
Copying blob 3e5bad3d9203 skipped: already exists  
Copying blob ef8fe905bfbd skipped: already exists  
Copying blob 564874600cfe skipped: already exists  
Copying blob 6e909acdb790 skipped: already exists  
Copying blob b63682e94411 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 5485e84c0a6c skipped: already exists  
Copying blob af9342aa445b skipped: already exists  
Copying blob 94a9b8494f17 skipped: already exists  
Copying blob 3e5bad3d9203 skipped: already exists  
Copying blob ef8fe905bfbd skipped: already exists  
Copying blob 564874600cfe skipped: already exists  
Copying blob 6e909acdb790 skipped: already exists  
Copying blob b63682e94411 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 5485e84c0a6c skipped: already exists  
Copying blob af9342aa445b skipped: already exists  
Copying blob 1b41da903089 skipped: already exists  
Copying blob 94a9b8494f17 skipped: already exists  
Copying blob 3e5bad3d9203 skipped: already exists  
Copying blob ef8fe905bfbd skipped: already exists  
Copying blob 564874600cfe skipped: already exists  
Copying blob 6e909acdb790 skipped: already exists  
Copying blob b63682e94411 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 5485e84c0a6c skipped: already exists  
Copying blob af9342aa445b skipped: already exists  
Copying blob 1b41da903089 skipped: already exists  
Copying blob 94a9b8494f17 skipped: already exists  
Copying blob 3e5bad3d9203 skipped: already exists  
Copying blob ef8fe905bfbd skipped: already exists  
Copying blob 564874600cfe skipped: already exists  
Copying blob 6e909acdb790 skipped: already exists  
Copying blob b63682e94411 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 5485e84c0a6c skipped: already exists  
Copying blob af9342aa445b skipped: already exists  
Copying blob 1b41da903089 skipped: already exists  
Copying config 1dc5bd57d5 [----------------------------] 0.0b / 5.7KiB | 0.0 b/s
Copying config 1dc5bd57d5 done   | 
Copying config 1dc5bd57d5 done   | 
Copying config 1dc5bd57d5 done   | 
Copying config 1dc5bd57d5 done   | 
Writing manifest to image destination
2025/04/13 18:33:49  info unpack layer: sha256:6e909acdb790c5a1989d9cfc795fda5a246ad6664bb27b5c688e2b734b2c5fad
2025/04/13 18:33:49  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/04/13 18:33:49  warn xattr{/tmp/build-temp-3929990805/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/04/13 18:33:50  info unpack layer: sha256:b63682e94411d59329167bf7eb372b82fa73ecdd204d080f42056b3d19f93dc5
2025/04/13 18:33:50  info unpack layer: sha256:94a9b8494f17f9a8795a62e93948e586bae79c4270e659aa8423718aff924bb3
2025/04/13 18:33:50  info unpack layer: sha256:564874600cfe164cf3e825ccded4f36a733571e963838a5396c3428cf8c74894
2025/04/13 18:33:50  info unpack layer: sha256:3e5bad3d92038af495ae0e6cb99cd4bfc59d29d88330a19b7f579c7b63c4236c
2025/04/13 18:33:50  info unpack layer: sha256:ef8fe905bfbd4aec6c3867ba5a78d6bd7f0f1a9b6a1d4cfae81eae3ecfda77ae
2025/04/13 18:33:50  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/04/13 18:33:50  warn xattr{/tmp/build-temp-3929990805/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/04/13 18:33:50  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2025/04/13 18:33:50  info unpack layer: sha256:af9342aa445bba74ce00d25b76c8617e8a1048acf1515dc6f7dfd4f078f3c7b9
2025/04/13 18:33:50  info unpack layer: sha256:5485e84c0a6c2a12228792d1a58e7b3d0a0211de9e2e483a85610504694f3ca9
2025/04/13 18:33:50  info unpack layer: sha256:1b41da903089d32e39b516ff452d68a813ea895ec5aa7e094eb7a0e32f849a54
INFO:    Copying apptainer-demo-env.yaml to /opt/project/
INFO:    Copying calc_sum_test.py to /opt/project/bin/
INFO:    Copying tooling/introspection_tools.py to /opt/project/lib/
INFO:    Running post scriptlet
FATAL:   container creation failed: mount hook function failure: mount /apps/all/XALT/3.0.2->/apps/all/XALT/3.0.2 error: while mounting /apps/all/XALT/3.0.2: destination /apps/all/XALT/3.0.2 doesn't exist in container
INFO:    If error was from fakeroot, try --ignore-fakeroot-command and
INFO:      maybe use fakeroot inside the %post section as described at
INFO:      https://apptainer.org/docs/user/latest/fakeroot.html#fakeroot-inside-def
FATAL:   While performing build: while running engine: while running %post section: exit status 255

Build a container image named apptainer-demo.sif using the definition file apptainer-demo.def

!apptainer test python-3.13-slim.sif

INFO:    Environment variable SINGULARITY_BINDPATH is set, but APPTAINER_BINDPATH is preferred
INFO:    Environment variable SINGULARITYENV_LD_PRELOAD is set, but APPTAINERENV_LD_PRELOAD is preferred
INFO:    No test script found in container, exiting
No test found in container, executing /bin/sh -c true

The test command allows you to execute a testscript (if available) inside of a given container

To execute the test section manually you can run

!apptainer exec python-3.13-slim.sif python untitled.py

INFO:    Environment variable SINGULARITY_BINDPATH is set, but APPTAINER_BINDPATH is preferred
INFO:    Environment variable SINGULARITYENV_LD_PRELOAD is set, but APPTAINERENV_LD_PRELOAD is preferred
The current time is 16:58:55.05

note

The --fakeroot flag in Apptainer is used to simulate root privileges for the user during the build process or while running containers. This is especially useful when building containers that require root access, but you don’t actually have root privileges on the system (like on HPC clusters).

Developing sandbox containers

In Apptainer (formerly known as Singularity), a sandbox is a directory-based container — as opposed to an image file-based container (like .sif or .img). When you create or use a sandbox, you’re working with a container that exists as a normal directory on your file system, with all the container’s files and structure laid out in a readable and writable format.

Key Features of a Sandbox in Apptainer:

• Writable (optional): You can build and modify the container’s filesystem more easily (especially useful during development).

• Human-readable: You can cd into the directory and see/modify files like a normal folder.

• Used for building: When creating or modifying containers, it’s common to work in sandbox mode, then convert to .sif for distribution.

$module load apptainer/1.2.5

$ apptainer build --sandbox my_sandbox/ my_container.def

requires either ‘–fakeroot’ or ‘sudo’

$ apptainer shell --fakeroot --writable my_sandbox/

Once we are inside the container, we can just install the things we need:

Apptainer> apt update

Apptainer> apt install cowsay fortune

Apptainer> conda install geopandas

…

Apptainer> exit

When we are done with the setup an image can be created with:

$ apptainer build my_sandbox.sif my_sandbox/

$ apptainer exec apptainer-complex.sif python3 /home/gth/it4i_course/untitled.py

apptainer cache clean

The apptainer cache clean command is used to clear Apptainer’s local cache, which is typically stored at $HOME/.apptainer/cache unless the APPTAINER_CACHEDIR environment variable is set. This cache includes downloaded images, blobs, and other temporary data. By default, the entire cache is removed, but you can customize the behavior using options like --days to only delete items older than a certain number of days, or --type to target specific cache types such as library, oci, shub, etc.

If Apptainer is run as root, the cache is stored in /root/.apptainer/cache, and cleaning it requires root privileges or sudo.

# Clean the entire cache
$ apptainer cache clean

# Dry run (shows what would be deleted)
$ apptainer cache clean --dry-run

# Force clean without confirmation
$ apptainer cache clean --force

# Clean cache entries older than 30 days
$ apptainer cache clean --days 30

# Clean only library and oci cache types
$ apptainer cache clean --type=library,oci

### Understanding Bind Mounts and Environment Configuration in Apptainer

Bindmounts

Apptainer provides fine-grained control over which host directories are mounted into the container, which is useful when you want a more isolated or reproducible environment. By default, Apptainer mounts the user’s $HOME, the current working directory (CWD), and temporary directories like /tmp. However, this behavior can be modified using flags like --no-home, --no-mount, and --containall.

--no-home and --no-mount home These flags prevent your host’s $HOME directory from being mounted inside the container. This is especially useful if your container image includes its own $HOME contents, which would normally be hidden by the host’s $HOME overlay. Both flags are functionally equivalent:

use

$ apptainer shell --no-home my_container.sif

$ apptainer shell --no-mount home my_container.sif

There are many more mount options see the documentation.

Instances

An instance in Apptainer is a way to run a container in the background, often to host a persistent service like a web server or API server. This contrasts with typical container use where commands like run, exec, or shell start containers in the foreground.

Why Use Instances?

• Run long-lived services (e.g., NGINX, API servers).

• Keep containers running independently of your terminal.

• Access, monitor, or stop the service anytime.

• Support multiple parallel services from the same container image.

Basic Commands

• Pull an image:
  $ apptainer pull oras://ghcr.io/apptainer/alpine:latest

• Start an instance:
  $ apptainer instance start alpine_latest.sif myinstance

• List running instances:
  $ apptainer instance list

• Interact with an instance:
  $ apptainer exec instance://myinstance cat /etc/os-release
$ apptainer shell instance://myinstance
$ apptainer run instance://myinstance
  

• Stop an instance:
  $ apptainer instance stop myinstance
$ apptainer instance stop --all

Environment

You can control the environment inside an Apptainer container using multiple methods depending on your use case. For persistent, build-time settings, define variables in the %environment section of the definition file. For temporary or per-execution needs, use the --env flag at runtime, or load multiple variables at once using --env-file. These options allow flexibility in configuring containers for different environments, users, or workflows.

In %environment (Definition File) – Persistent:

%environment
export MY_VAR="Hello"

At runtime – Temporary:

$ apptainer exec --env MY_VAR=Hello image.sif env

From file:

$ apptainer exec --env-file myvars.env image.sif env

Singularity and MPI applications

MPI is widely used in HPC applications for communication across compute nodes. Singularity supports MPI, and the most common method for running MPI applications in containers is using the host’s MPI implementation—known as the Host or Hybrid model where both host and container MPI components work together.

Alternatively, the Bind model uses only the host’s MPI, with none installed in the container, requiring the host MPI to be mounted into the container at runtime.

Hybrid Model

The basic idea behind the Hybrid Approach is that when executing a container with MPI code, mpiexec (or another launcher) is used to initiate the Apptainer command itself. This means the MPI process on the host system coordinates with the MPI installation inside the container. Together, they enable seamless communication and parallel execution across nodes. This approach allows users to maintain consistent MPI environments while leveraging containerized applications in HPC settings.

Bind Model

Similar to the Hybrid Approach, the basic idea behind the Bind Approach is to start the MPI application by invoking the MPI launcher (e.g., mpirun) from the host system. However, the key difference lies in the container setup: the Bind Approach typically does not include any MPI implementation within the container itself. Instead, Apptainer binds the host’s MPI libraries and binaries directly into the container at runtime. This allows the containerized application to use the host’s MPI stack, ensuring compatibility with the host environment and reducing redundancy.

### Example

using an anaconda based container with MPI.

%%writefile apptainer-mpi-env.yaml
name: apptainer-mpi-env
channels:
  - conda-forge
dependencies:
  - python=3.12
  - numpy=1.26
  - threadpoolctl>=3
  - pyyaml>=6
  - mpi4py=3.1.6
  - openmpi=4.1.6=external_*

Overwriting apptainer-mpi-env.yaml

%%writefile apptainer-mpi.def
Bootstrap: docker
From: continuumio/miniconda3

%files
    apptainer-mpi-env.yaml /opt/project/

%post
    conda env create -f /opt/project/apptainer-mpi-env.yaml --prefix /opt/project/apptainer-mpi-env
    echo 'eval "$(conda shell.bash hook)"' >> $APPTAINER_ENVIRONMENT
#        /opt/conda/bin/conda init bash

    echo 'conda activate /opt/project/apptainer-mpi-env' >> $APPTAINER_ENVIRONMENT
    conda clean -afy

%environment
    # Override all Conda paths to point ONLY to the container
    export PATH="/opt/project/apptainer-mpi-env/bin:/opt/conda/bin:$PATH"
    export LD_LIBRARY_PATH="/opt/project/apptainer-mpi-env/lib:$LD_LIBRARY_PATH"
    export PYTHONPATH="/opt/project/apptainer-mpi-env/lib/python3.12/site-packages:$PYTHONPATH"
    
    # Isolate Conda completely
    unset CONDA_PREFIX
    unset CONDA_DEFAULT_ENV
    unset CONDA_PROMPT_MODIFIER

%help
    This is an MPI demo image for the python4hpc training

Overwriting apptainer-mpi.def