Apptainer

Overview

Apptainer, formerly known as Singularity, is a container system for shared environments that allows you to bundle your entire environment into a single file that can be run on almost any system (including all CSaW cluster systems). One of its great features is that it allows you run many Docker images that have become popular among many computing researchers who are used to working on their personal systems and want to move their environment to a different system.

This document will walk you through the basics of pulling or building a container, and running it in the CSaW cluster environment. For full Apptainer documentation, please see the Apptainer User Guide.

Note

All Apptainer commands will need to be run from an Execute Point (EP), also known as a compute node. You can get an interactive shell on an EP using condor_submit’s interactive option:

condor_submit -interactive request_cpus=4 request_memory=4GB getenv=True

Using Existing Images

The easiest way to get started with Apptainer is to leverage the work someone else has done. Many times you will be able to find an image that someone has built that will contain the tool(s) you need to run your research.

Apptainer has access to both a vast library of images built specifically for it, and also to the even bigger library at Docker Hub (a popular repository with hundreds of thousands of Docker images).

Images can be pulled down to the cluster environment from various sources using the apptainer pull command. Alternatively, you can skip the pull and just specify the URL to the build, exec, run, or shell apptainer commands to create an ephemeral container that will go away after the command is run.

Caution

Be careful where you run the apptainer command. If it needs to download an image, the image will be downloaded in your current directory. Container images can be larger than your quota allows, so it is suggested to store them somewhere in your shared research directory (/cluster/research-groups/${PI}) unless you’re sure you have enough space.

From Docker Hub

Docker Hub has over 100,000 pre-built images. Unfortunately, not all Docker images can be run in Apptainer, but most HPC/HTC and scientific research ones run just fine. See the notes on Docker compatibility section for additional details. From the Docker Hub web site you can search through thousands of pre-built images. Be mindful that not all containers can be trusted, and be sure to favor containers that are marked as “Trusted Content” either via the “Docker Image Official”, “Verified Publisher”, or “Sponsored OSS” badges.

Let’s pull a pre-built image from David Godlove (who is conveniently an Apptainer/Singularity developer!), “lolcow” which will display a fortune message in the terminal.

[USER@c-X-X ~]$ apptainer pull docker://godlovedc/lolcow
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Copying blob 8e860504ff1e done   |
Copying blob d010c8cf75d7 done   |
Copying blob 3b61febd4aef done   |
Copying blob 7fac07fb303e done   |
Copying blob 9fb6c798fa41 done   |
Copying blob 9d99b9777eb0 done   |
Copying config 38dc06177c done   |
Writing manifest to image destination
2024/08/05 11:44:19  info unpack layer: sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
2024/08/05 11:44:19  warn rootless{dev/agpgart} creating empty file in place of device 10:175
2024/08/05 11:44:19  warn rootless{dev/audio} creating empty file in place of device 14:4
... CUT FOR SPACE ...
024/08/05 11:44:19  warn rootless{dev/urandom} creating empty file in place of device 1:9
2024/08/05 11:44:19  warn rootless{dev/zero} creating empty file in place of device 1:5
2024/08/05 11:44:20  info unpack layer: sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
2024/08/05 11:44:20  info unpack layer: sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
2024/08/05 11:44:20  info unpack layer: sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
2024/08/05 11:44:20  info unpack layer: sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
2024/08/05 11:44:20  info unpack layer: sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
INFO:    Creating SIF file...

Here the docker:// prefix without a specific host following it is used to specify it should come from the default Docker Hub container registry. You could alternatively supply a hostname to tell it to use a different OCI compatible container registry, such as Quay.io or Github Container Registry if the project you’re using needs that instead.

Once you pull the image, it will create a new file in your current directory named lolcow_latest.sif. The image name is suffixed with _latest to denote which tag was used when creating it. In this case we did not specify a tag (tags can be used to denote specific versions), so it simply grabbed the latest.

Note

If reproducibility is important to your work, you can specify a tag to pull by ending the URL with a :tag such as python:3.9.21-alpine3.21, so that others can use the exact same image you used. Otherwise six-months from now latest could be updated and provide different results.

You might have also noticed all the warnings when it was converting the image to the Apptainer format. This is because Docker containers run with elevated privileges unlike in Apptainer.

Now that you have an image downloaded, lets look at options to run it before we deploy it via HTCondor.

Running Containers

There are three ways you can use an Apptainer container: run, exec, and shell.

The commands run and exec are similar, with one exception: run will run whichever command is defined to be run by the Apptainer image, while exec will let you specify which command to run.

The last command mentioned above, shell, will simply give you a shell into the container and let you interact with it. This is a good way for testing before submitting non-interactive jobs to the scheduler.

Run

Using the example image pulled above, we can view the difference between all three options for running a container. First the run command:

USER@c-X-X:~$ apptainer run lolcow_latest.sif
 ________________________
< Don't get to bragging. >
 ------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

One of the neat features about Apptainer is that it’s images are also executable! So you can just run the image like it was a normal executable for your system:

USER@c-X-X:~$ ./lolcow_latest.sif
 _________________________________________
< If you can read this, you're too close. >
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Note

The run command will also let you pass arguments to whatever program is going to run, but the lolcow container does not take any arguments. Here’s a small example of using it for future reference:

apptainer run ./my_container.sif -arg1 -arg2

Exec

The run command above ran the default program (or programs) when the container started, but sometimes you need to run something slightly differently in the same container. The previous run example was invoking the shell pipeline: fortune | cowsay | lolcat. This is not directly possible using exec, because we can only run one program, and not a pipeline. However, instead of using all of the pipeline we can run things individually, such as just fortune or cowsay with our own pre-determined messages:

USER@c-X-X:~$ apptainer exec lolcow_latest.sif cowsay "Hello, world!"
 _______________
< Hello, world! >
 ---------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Here we’ve run cowsay with our message “Hello, world!” (The "’s are important to pass the message as one argument to the program). We could also run fortune on it’s own:

USER@c-X-X:~$ apptainer exec lolcow_latest.sif fortune
You will triumph over your enemy.

exec let us run something from inside the container other than the pre-defined program, which can be very useful if you have additional testing tools or software in there. And for one final example, I’ll show how you can run a pipeline of commands using exec even though I previously said you couldn’t do it:

USER@c-X-X:~$ apptainer exec lolcow_latest.sif sh -c 'fortune | cowsay | lolcat'
 ______________________________________
< Stay away from flying saucers today. >
 --------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

So the command we ran was sh, which is a POSIX shell. Then we passed it two options: -c and 'fortune | cowsay | lolcat'. The -c says the next argument should be run in the shell, and the next argument was a pipeline of commands grouped together with quotes, so it’s only one argument.

Now that we’ve seen the difference between run and exec let’s look at using the container in an interactive manner, using the shell command.

Shell

Sometimes testing things interactively is the easiest way. Apptainer support this concept by allowing you to get a shell prompt inside the container. This lets you run multiple commands, look at available files, and see the environment as it exists when it’s run in order to troubleshoot any problems that occur.

The shell command will simply start the container, and give you a shell prompt into it. You’ll know you’re in your container when the prompt changes to Apptainer> instead of your USER@cluster-node$: prompt.

USER@c-X-X:~$ apptainer shell ./lolcow_latest.sif
Apptainer>

From here you can inspect, test, and interact the environment.

USER@c-X-X:~$ apptainer shell ./lolcow_latest.sif
Apptainer> whoami
USER
Apptainer> id
uid=######(USER) gid=######(USER) groups=######(USER),65534(nogroup) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Apptainer> pwd
/cluster/home/USER
Apptainer> date
Fri Oct  1 10:24:07 PDT 2021
Apptainer> fortune | cowsay | lolcat
 ________________________________________
/ If you learn one useless thing every   \
| day, in a single year you'll learn 365 |
\ useless things.                        /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Now that we can run things interactively, we see that we are our normal user, have our normal user privileges, and our home directory is mounted in the container. Apptainer has preserved (almost) everything for us so we can access our data, but we’re running in our special container environment.

Other Options

Mounting Other Directories

In the Apptainer Shell example I mentioned that Apptainer preserved almost everything from your normal environment. What did it miss? Well, it got your UID/GID, groups, some directories, and environment variables. But it didn’t get the entire filesystem. This becomes a problem if you need access to additional directory, since it’s not mounted into your container environment. Fortunately there’s an easy fix for this: the --bind argument.

Note

Here’s the complete list that comprises the “some directories” mentioned above according to the Apptainer documentation:

In the default configuration, the system default bind points are $HOME, /sys:/sys, /proc:/proc, /tmp:/tmp, /var/tmp:/var/tmp, /etc/resolv.conf:/etc/resolv.conf, /etc/passwd:/etc/passwd, and $PWD.

In addition to the defaults above, all nodes in the CSaW cluster are configured to also automatically mount the following: /cluster, /scratch, /scratch_memory_backed.

Because of the added mounts above, you will most likely not need to add additional bind mounts.

Using the --bind argument can be done two ways, first where you simply specify the path and it appears as specified, second where you specify the path on the host, and a different path inside the container. You may bind multiple directories by separating them with a ,. Here’s an example of both options:

apptainer run --bind "/projects" lolcow_latest.sif

This will create a /projects inside your container, and it’s contents will be that of /projects from the host that is running the container.

Apptainer run --bind "/projects:/research_data" lolcow_latest.sif

This second example takes the /projects from the host that runs the container, but puts it in as /research_data instead. This can be useful if your research application expects data to be available at a fixed location, and you can’t adjust it.

Using a GPU

The Apptainer commands run, exec, and shell can all accept the --nv argument that will automatically configure the container to be able to use the GPU(s). HTCondor will automatically set the appropriate environment variable ( CUDA_VISIBLE_DEVICES ) when you set request_gpus >= 1 in your submission file so that your program will use the correct device(s) on a multi-GPU system.

Submitting to HTCondor

Submitting a job using Apptainer is done in much the same way you submit any other job. You can use Apptainer in HTCondor’s Vanilla, Parallel, and Container universes. Using the Parallel universe will not be covered here, but is possible. Please see the Apptainer MPI documentation for an idea of the required work to begin running MPI inside of a container, and please reach out to support for additional help.

Using the Vanilla Universe

Submitting to the Vanilla universe is easy, and only requires a handful of lines to run the container:

Basic HTCondor Apptainer submission

universe   = vanilla

executable = apptainer
arguments  = run lolcow_latest.sif

output     = out.log
error      = err.log
log        = condor.log

queue

This basic example is enough to submit and run the lolcow image, but it could be even smaller because if you’re using the run command you can leverage the fact that the image itself is executable:

Even simpler HTCondor Apptainer submission

universe   = vanilla

executable = lolcow_latest.sif

output     = out.log
error      = err.log
log        = condor.log

queue

This is perhaps the smallest submission file possible that still has enough to get all of the things you need. Of course if you need to specify arguments to your container you can add back in the Arguments variable.

Because Apptainer is started by HTCondor it will have whatever resource limits you requested. The above examples will get a single core, and some small amount of memory space. It’s better to add the two lines in that request a reasonable amount of resources:

HTCondor Apptainer submission with proper resource allocation

universe       = vanilla

executable     = lolcow_latest.sif

output         = out.log
error          = err.log
log            = condor.log

request_cpus   = 1
request_memory = 1gb

queue

This will allow your container to use multiple cores, and request a reasonable amount of memory to use. You should of course adjust your requests based on your needs, but please remember that we’re in a shared environment. If HTCondor has allocated the resources to you and don’t use them, it’s potentially taking away resources from someone else who could use them.

Using a GPU in the Vanilla Universe

Building on the previous example above, we can pretend that our lolcow container needs a GPU to do it’s job. As mentioned above in the using a gpu we need to pass the --nv flag to tell Apptainer to allow the container access to a GPU, as well as request a GPU via HTCondor.

Basic HTCondor Apptainer GPU submission

universe       = vanilla

executable     = apptainer
arguments      = run --nv lolcow_latest.sif

output         = out.log
error          = err.log
log            = condor.log

request_cpus   = 8
request_memory = 32gb
request_gpus   = 1

queue

Using the Container Universe

HTCondor has a universe dedicated to running containers. This can make using them even easier than the vanilla universe, because HTCondor will do the heavy lift of ensuring that the container is present and Apptainer is available on the host before starting it.

Leveraging a Container Registry

The easiest way to leverage the container universe is to specify the container image as a URL to the OCI repository, or use the default, Docker Hub.

Basic HTCondor Container Universe submission

universe        = container
container_image = docker://godlovedc/lolcow

executable      = /bin/sh
arguments       = "-c 'fortune | cowsay | lolcat'"

output          = out.log
error           = err.log
log             = condor.log

request_cpus    = 1
request_memory  = 1gb

queue

Note the executable is a path inside the container. HTCondor will start the container, then start a process inside the container, much like when the “exec” functionality is shown above. We specify the full path to the program inside the container, and then the arguments to run. The odd double quotes and single quotes is because we need to tell HTCondor that the arguments are a string, but there’s special meaning to the way it handles spaces for our command. Depending on your program, you will probably not need any weird quoting like this example does.

Transferring the Container Image

If you’ve built your own image but haven’t put it in an OCI registry, you will need to transfer the image to the execute point. Fortunately, this is only two extra lines. One line to tell HTCondor we want to transfer the container, and one tell HTCondor we want to ensure that files are transferred. The second line is only required because HTCondor knows that we have shared file storage and by default doesn’t want to transfer files.

Basic HTCondor Container Universe submission with image transfer

universe              = container
container_image       = lolcow_latest.sif
transfer_container    = true
should_transfer_files = yes

executable            = /bin/sh
arguments             = "-c 'fortune | cowsay | lolcat'"

output                = out.log
error                 = err.log
log                   = condor.log

request_cpus          = 1
request_memory        = 1gb

queue

Using a GPU with the Container Universe

One of the nice things about the container universe is that because it invokes the Apptainer command for you, it also know when it needs to tell the container about the GPU you requested. This means there is no need to do anything differently, except request the GPU.

Basic HTCondor Container Universe submission

universe        = container
container_image = docker://godlovedc/lolcow

executable      = /bin/sh
arguments       = "-c 'fortune | cowsay | lolcat'"

output          = out.log
error           = err.log
log             = condor.log

request_cpus    = 8
request_memory  = 32gb
request_gpus    = 1

queue

The above example will of start the container with access to 1 GPU, 8 cpu threads, and 32GB of memory. The GPU that HTCondor has allocated for you will be set with CUDA_VISIBLE_GPUS, which is not the same as what is visible with nvidia-smi. Apptainer is working on a fix for this issue, but as of now it is not available.

And as above, please only request resource you intend to fully use. If your code will not use a GPU, please do not request one. They are far fewer GPUs available in the clusters than there are CPU cores.

Building Your Own containers

While there are hundreds of thousands of containers between Docker Hub and various registries, sometimes you can’t find one that does what you need it to do.

Writing an Apptainer Definition File (.def)

An Apptainer Definition File (.def) contains to parts: A header, and sections.

The header specifies what to use as the base of the image. This can be anything from a basic Linux environment, another .sif image, or a bare environment that you copy everything into.

The sections can be thought of a little scripts that define what to do when during the build process, as well as the runscript which defines what to do when the container is run.

The first line of the file will always contain Bootstrap: to define how to begin building the image. As to which bootstrap method to use, it depends on the use case. But it is perhaps easiest to use an image from the library, such that it’s one that you’re familiar with, and then modify it to suit your needs in the sections portion of the file. Please see the Header section of the Apptainer documentation to see all possible bootstrap options, as well as how to use them in their appendix sections.

Once the header is defined, the sections part will be one or more of the following sections:

To use the same example we’ve been using throughout this document, the lolcow image is built from the following:

lolcow.def example

BootStrap: docker
From: ubuntu:16.04

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD

The header is the two lines:

lolcow.def header part

BootStrap: docker
From: ubuntu:16.04

This will use the existing Ubuntu 16.04 from Docker Hub as a base to work with. (Note: The Ubuntu 16.04 image is very old, and should not be used for anything modern. It’s kept here only as an example)

The sections then becomes the rest:

lolcow.def sections part

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD

The %post section is the area where you can run commands to install software inside the image while building it. Here we use Ubuntu’s apt utility to update it’s cache, and install our three tools: fortune, cowsay, and lolcat.

The %environment section sets environment variables for when the container is run. While Apptainer will inherit your current environment values, these will set and/or ensure that they are set to their intended values. Here we ensure that our locale is defined as the C locale, and that the PATH is updated to include the directory that has the commands we intend to run.

The %runscript section defines what commands to run when you invoke the container. Either with apptainer run or ./lolcow.sif.

And finally, the %labels is where the David Godlove gives himself credit for creating this great example image.

There are many additional sections and things you can do in your images. Please see the full Apptainer Documentation for Definition Files to learn more. If you need help writing your .def files, please reach out to support for assistance.

Building the .sif File

The apptainer build command supports the fakeroot feature, which allows you to build container images without needed special privileges. We’ll re-use the above example to build our own lolcow.sif.

lolcow.def example

BootStrap: docker
From: ubuntu:16.04

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD

Saving the above as lolcow.def, we can then issue the remote build command, see the build status, and pull the built image.

apptainer build lolcow.sif lolcow.def

USER@c-X-X:~$ apptainer build lolcow.sif lolcow.def
INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    The %post section will be run under fakeroot
INFO:    Starting build...
Copying blob 3713021b0277 done   |
Copying config 8a3cdc4d1a done   |
Writing manifest to image destination
2024/08/05 16:17:51  info unpack layer: sha256:3713021b02770a720dea9b54c03d0ed83e03a2ef5dce2898c56a327fee9a8bca
INFO:    Running post scriptlet
+ apt-get -y update
... CUT FOR SPACE ...
+ apt-get -y install fortune cowsay lolcat
... CUT FOR SPACE ...
done.
INFO:    Adding labels
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: lolcow.sif

This uploaded our lolcow.def file to their build server, showed us the build output (almost all of which was cut to keep the display small), and then downloaded the image to the requested lolcow.sif name.

Now we can run the image as we have the past:

USER@c-X-X:~$ apptainer run lolcow.sif
 ______________________________________
< Today is what happened to yesterday. >
 --------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

There’s many more things you can do with remote builds, but that covers the basics. The full Apptainer documentation pages cover many of the additional options, but please contact support if you need additional help.

Building With a Personal System

While out of the scope of this document, it’s worth mentioning that due to the nature of Apptainer containing everything it needs in one file, you are able to build your .sif and upload it to be run on the cluster environment.

Unsupported Features

Apptainer supports many more features than are mentioned on this page. Unfortunately, some features can’t be used due to security concerns when operating in a shared environment because they require elevated privileges.

Fakeroot Containers

If you need to a run a process as root inside your container, it’s possible to map root’s id inside the container back to that of your user outside the container. The process believes it is being run as root, and in reality it’s running as you. Depending on what the process needs the root privileges for this may or may not work as intended. The use of this feature requires specialized map files to be generated for each user in the environment, on each host that the container will run on. This can be done, but due to the number of users it will only be done on a case-by-case basis. If you think you need this feature, please reach out to support and we’ll see if we can help you get your code running.

Notes on Docker Compatibility

Because Docker initially runs as the root user (when a non-root user is granted access to the Docker group they have root access), there are certain features that it allows that are not compatible with Apptainer. One of these features is dropping privileges to another user. In a well crafted Docker file you may encounter the USER directive to become a different user instead of root. Because Apptainer runs as you, you can’t become someone else, only root can do that. Docker containers that depend on this functionality will not work without modification. Fortunately this should be a rare occurrence with research containers, and is typically found in service containers instead.

For a full list of things that function differently between Docker and Apptainer please see the best practices and troubleshooting section of the Apptainer user documentation.

Additional Notes / Tips

Apptainer Cache

The cluster environments have by default set your Apptainer cache directory to be inside the /scratch directory, which does not travel with you from node to node. All files in that directory will be deleted if they have not been accessed for more than a day.

Setting your cache path to a scratch directory helps ensure you do not fill up your home directory with parts of an Apptainer image.

Clearing Out Your Cache

If you’ve downloaded bunch of Apptainer images and want to cleanup the cache before the system cleans them up, you can run the apptainer cache clean command. Please see the linked documentation for additional information.