Singularity

Overview

Singularity is a container system for shared environments that allows you to bundle your entire environment into a single file that can be run on almost any system (including all CSaW cluster systems). One if it’s great features is that it allows you run many Docker images that have become popular among many computing researchers who are used to working on their personal systems and want to move their environment to a different system.

This document will walk you through the basics of pulling or building a container, and running it in the CSaW cluster environment. For full Singularity documentation, please see the Singularity User Guide.

Using Existing Images

The easiest way to get started with Singularity is to leverage the work someone else has done. Many times you will be able to find an image that someone has built that will contain the tool(s) you need to run your research.

Singularity has access to both a vast library of images built specificaly for it, and also to the even bigger library at DockerHub (a popular repository with hundreds of thousands of Docker images).

Images can be pulled down to the cluster environment from various sources using the singularity pull command. Alternatively, you can skip the pull and just specify the URL to the build, exec, run, or shell singularity commands to create an ephemeral container that will go away after the command is run.

Caution

Be careful where you run the singularity command. If it needs to download an image, the image will be downloaded in your current directory. Container images can be larger than your quota allows, so it is suggested to store them somewhere in your shared research directory (/cluster/research-groups/${PI}) unless you’re sure you have enough space.

From Sylabs

Let’s pull a pre-built image “lolcow” which will display a fortune in the terminal.

USER@cluster-head:~$ singularity pull library://lolcow
INFO:    Downloading library image
77.7MiB / 77.7MiB [===================================] 100 % 3.4 MiB/s 0s

Here the library:// prefix is used to specify it should come from the default library (in this case it’s Sylabs’ Cloud Library). From the Sylabs Cloud Library you can search through thousands of pre-built images, or you can also search from the command line using singularity search.

Once you pull the image, it will create a new file in your current directory named lolcow_latest.sif. The image name is suffixed with _latest to denote which tag was used when creating it. In this case we did not specify a tag (tags can be used to denote specific versions), so it simply grabbed the latest.

Note

If reproducibility is important to your work, you can specify a tag to pull by ending the URL with a :tag such as singularity pull library://lolcow:my_version_tag, so that others can use the exact same image you used. Otherwise six-months from now latest could be updated and provide different results.

From DockerHub

While the Sylabs’ Cloud Library has thousands of pre-built images for you to use, DockerHub has over 100,000 pre-built images. Unfortunately, not all Docker images can be run in Singularity, but most HPC/HTC and scientific research ones run just fine. See the notes on Docker compatibility section for additional details.

If you previously downloaded the lolcow image above, you can skip this step. This is simply an example of how to pull an image from DockerHub instead of the Sylabs Cloud Library. If you haven’t, lets grab (more or less) the same image, built by someone on DockerHub.

Pulling an image from DockerHub is done with the docker:// prefix instead of library://. For this example we will pull what is an (almost) identical container to the lolcow from Sylabs, except that it will be downloaded from DockerHub and converted to a Singularity file instead of it’s original OCI format.

USER@cluster-head:~$ singularity pull docker://godlovedc/lolcow
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 9fb6c798fa41 done
Copying blob 3b61febd4aef done
Copying blob 9d99b9777eb0 done
Copying blob d010c8cf75d7 done
Copying blob 7fac07fb303e done
Copying blob 8e860504ff1e done
Copying config 73d5b1025f done
Writing manifest to image destination
Storing signatures
2021/09/29 10:08:02  info unpack layer: sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
2021/09/29 10:08:02  warn rootless{dev/agpgart} creating empty file in place of device 10:175
2021/09/29 10:08:02  warn rootless{dev/audio} creating empty file in place of device 14:4
2021/09/29 10:08:02  warn rootless{dev/audio1} creating empty file in place of device 14:20
** TRIMMED TO SAVE SPACE FOR EXAMPLE ***
2021/09/29 10:08:02  warn rootless{dev/tty9} creating empty file in place of device 4:9
2021/09/29 10:08:02  warn rootless{dev/urandom} creating empty file in place of device 1:9
2021/09/29 10:08:02  warn rootless{dev/zero} creating empty file in place of device 1:5
2021/09/29 10:08:04  info unpack layer: sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
2021/09/29 10:08:04  info unpack layer: sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
2021/09/29 10:08:04  info unpack layer: sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
2021/09/29 10:08:04  info unpack layer: sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
2021/09/29 10:08:04  info unpack layer: sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
INFO:    Creating SIF file...

Once you pull the image, it will create a new file in your current directory named lolcow_latest.sif. The image name is suffixed with _latest to denote which tag was used when creating it. In this case we did not specify a tag (tags can be used to denote specific versions), so it simply grabbed the latest.

Note

If reproducibility is important to your work, you can specify a tag to pull by ending the URL with a :tag such as singularity pull docker://godlovedc/lolcow:my_version_tag, so that others can use the exact same image you used. Otherwise six-months from now latest could be updated and provide different results.

You might have also noticed all the warnings when it was converting the image to the Singularity format. This is because Docker containers run with elevated privileges unlike in Singularity.

Now that you have an image downloaded, lets look at options to run it before we deploy it via HTCondor.

Running Containers

There are three ways you can use a Singularity container: run, exec, and shell.

The commands run and exec are similar, with one exception: run will run whichever command is defined to be run by the Singularity image, while exec will let you specify which command to run.

The last command mentioned above, shell, will simply give you a shell into the container and let you interact with it. This is a good way for testing before submitting non-interactive jobs to the scheduler.

Run

Using the example image pulled above (The Sylabs image will print the date, the DockerHub image will print a small fortune), we can view the difference between all three options for running a container. First the run command:

USER@cluster-head:~$ singularity run lolcow_latest.sif
 ________________________
< Don't get to bragging. >
 ------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

One of the neat features about Singularity is that it’s images are also executable! So you can just run the image like it was a normal executable for your system:

USER@cluster-head:~$ ./lolcow_latest.sif
 _________________________________________
< If you can read this, you're too close. >
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Note

The run command will also let you pass arguments to whatever program is going to run, but the lolcow container does not take any arguments. Here’s a small example of using it for future reference:

singularity run ./my_container.sif -arg1 -arg2

Exec

The run command above ran the default program (or programs) when the container started, but sometimes you need to run something slightly differently in the same container. The previous run example was invoking the shell pipeline: fortune | cowsay | lolcat. This is not directly possible using exec, because we can only run one program, and not a pipeline. However, instead of using all of the pipeline we can run things individually, such as just fortune or cowsay with our own pre-determined messages:

USER@cluster-head:~$ singularity exec lolcow_latest.sif cowsay "Hello, world!"
 _______________
< Hello, world! >
 ---------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Here we’ve run cowsay with our message “Hello, world!” (The "’s are important to pass the message as one argument to the program). We could also run fortune on it’s own:

USER@cluster-head:~$ singularity exec lolcow_latest.sif fortune
You will triumph over your enemy.

exec let us run something from inside the container other than the pre-defined program, which can be very useful if you have additional testing tools or software in there. And for one final example, I’ll show how you can run a pipeline of commands using exec even though I previously said you couldn’t do it:

USER@cluster-head:~$ singularity exec lolcow_latest.sif sh -c 'fortune | cowsay | lolcat'
 ______________________________________
< Stay away from flying saucers today. >
 --------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

So the command we ran was sh, which is a POSIX shell. Then we passed it two options: -c and 'fortune | cowsay | lolcat'. The -c says the next argument should be run in the shell, and the next argument was a pipeline of commands grouped together with quotes, so it’s only one argument.

Now that we’ve seen the difference between run and exec let’s look at using the container in an interactive manner, using the shell command.

Shell

Sometimes testing things interactively is the easiest way. Singularity support this concept by allowing you to get a shell prompt inside the container. This lets you run multiple commands, look at available files, and see the environment as it exists when it’s run in order to troubleshoot any problems that occur.

The shell command will simply start the container, and give you a shell prompt into it. You’ll know you’re in your container when the prompt changes to Singularity> instead of your USER@cluster-node$: prompt.

USER@cluster-head:~$ singularity shell ./lolcow_latest.sif
Singularity>

From here you can inspect, test, and interact the environment.

USER@cluster-head:~$ singularity shell ./lolcow_latest.sif
Singularity> whoami
USER
Singularity> id
uid=1000(USER) gid=1000(USER) groups=1000(USER),4(adm),24(cdrom),30(dip),46(plugdev)
Singularity> pwd
/cluster/home/USER
Singularity> date
Fri Oct  1 10:24:07 PDT 2021
Singularity> fortune | cowsay | lolcat
 ________________________________________
/ If you learn one useless thing every   \
| day, in a single year you'll learn 365 |
\ useless things.                        /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Now that we can run things interactively, we see that we are our normal user, have our normal user privileges, and our home directory is mounted in the container. Singularity has preserved (almost) everything for us so we can access our data, but we’re running in our special container environment.

Other Options

Mounting Other Directories

In the Singularity Shell example I mentioned that Singularity preserved almost everything from your normal environment. What did it miss? Well, it got your UID/GID, groups, some directories, and environment variables. But it didn’t get the entire filesystem. This becomes a problem if you need access to additional directory, since it’s not mounted into your container environment. Fortunately there’s an easy fix for this: the --bind argument.

Note

Here’s the complete list that comprises the “some directories” mentioned above according to the Singularity documentation:

In the default configuration, the system default bind points are $HOME, /sys:/sys, /proc:/proc, /tmp:/tmp, /var/tmp:/var/tmp, /etc/resolv.conf:/etc/resolv.conf, /etc/passwd:/etc/passwd, and $PWD.

In addition to the defaults above, all nodes in the CSaW cluster are configured to also automatically mount the following: /cluster, /scratch, /scratch_memory_backed.

Because of the added mounts above, you will most likely not need to add additional bind mounts.

Using the --bind argument can be done two ways, first where you simply specify the path and it appears as specified, second where you specify the path on the host, and a different path inside the container. You may bind multiple directories by separating them with a ,. Here’s an example of both options:

singularity run --bind "/projects" lolcow_latest.sif

This will create a /projects inside your container, and it’s contents will be that of /projects from the host that is running the container.

Singularity run --bind "/projects:/research_data" lolcow_latest.sif

This second example takes the /projects from the host that runs the container, but puts it in as /research_data instead. This can be useful if your research application expects data to be available at a fixed location, and you can’t adjust it.

Using a GPU

The Singularity commands run, exec, and shell can all accept the --nv argument that will automatically configure the container to be able to use the GPU(s). HTCondor will automatically set the appropriate environment variable ( CUDA_VISIBLE_DEVICES ) when you set request_gpus >= 1 in your submission file so that your program will use the correct device(s) on a multi-GPU system.

Submitting to HTCondor

Submitting a job using Singularity is done in much the same way you submit any other job. You can use singularity in both of HTCondor’s Vanilla and Parallel universes. Using the Parallel universe will not be covered here, but is possible. Please see the Singularity MPI documentation for an idea of the required work to begin running MPI inside of a container, and please reach out to support for additional help.

Submitting to the Vanilla universe is easy, and only requires a handful of lines to run the container:

Basic HTCondor Singularity submission

Universe   = vanilla
Executable = singularity
Arguments  = run lolcow_latest.sif
Output     = out.log
Error      = err.log
Log        = condor.log
Queue

This basic example is enough to submit and run the lolcow image, but it could be even smaller because if you’re using the run command you can leverage the fact that the image itself is executable:

Even simpler HTCondor Singularity submission

Universe   = vanilla
Executable = lolcow_latest.sif
Output     = out.log
Error      = err.log
Log        = condor.log
Queue

This is perhaps the smallest submission file possible that still has enough to get all of the things you need. Of course if you need to specify arguments to your container you can add back in the Arguments variable.

Because Singularity is started by HTCondor it will have whatever resource limits you requested. The above examples will get a single core, and some small amount of memory space. It’s better to add the two lines in that request a reasonable amount of resources:

HTCondor Singularity submission with proper resource allocation

Universe   = vanilla
Executable = lolcow_latest.sif
Output     = out.log
Error      = err.log
Log        = condor.log
Request_CPUs   = 4
Request_Memory = 8GB
Queue

This will allow your container to use multiple cores, and request a reasonable amount of memory to use. You should of course adjust your requests based on your needs, but please remember that we’re in a shared environment. If HTCondor has allocated the resources to you and don’t use them, it’s potentially taking away resources from someone else who could use them.

Using a GPU

Building on the previous example above, we can pretend that our lolcow container needs a GPU to do it’s job. As mentioned above in the using a gpu we need to pass the --nv flag to tell Singularity to allow the container access to a GPU, as well as request a GPU via HTCondor.

Basic HTCondor Singularity submission

Universe   = vanilla
Executable = singularity
Arguments  = run --nv lolcow_latest.sif
Output     = out.log
Error      = err.log
Log        = condor.log
Request_CPUs   = 8
Request_Memory = 32GB
Request_GPUs   = 1
Queue

The above example will of start the container with access to 1 GPU, 8 cpu threads, and 32GB of memory. The GPU that HTCondor has allocated for you will be set with CUDA_VISIBLE_GPUS, which is not the same as what is visible with nvidia-smi. Singularity is working on a fix for this issue, but as of now it is not available.

And as above, please only request resource you intend to fully use. If your code will not use a GPU, please do not request one. They are far fewer GPUs available in the clusters than there are CPU cores.

Building Your Own containers

While there are hundreds of thousands of containers between the Sylabs Cloud Library and DockerHub, sometimes you can’t find one that does what you need it to do.

Writing a Singularity Definition File (.def)

A Singularity Definition File (.def) contains to parts: A header, and sections.

The header specifies what to use as the base of the image. This can be anything from a basic Linux environment, another .sif image, or a bare environment that you copy everything into.

The sections can be thought of a little scripts that define what to do when during the build process, as well as the runscript which defines what to do when the container is run.

The first line of the file will always contain Bootstrap: to define how to begin building the image. As to which bootstrap method to use, it depends on the use case. But it is perhaps easiest to use an image from the library, such that it’s one that you’re familiar with, and then modify it to suit your needs in the sections portion of the file. Please see the Header section of the Singularity documentation to see all possible bootstrap options, as well as how to use them in their appendix sections.

Once the header is defined, the sections part will be one or more of the following sections:

• %setup

• %filse

• %app*

• %post

• %test

• %environment

• %startscript

• %runscript

• %labels

• %help

To use the same example we’ve been using throughout this document, the lolcow image is built from the following:

lolcow.def example

BootStrap: library
From: ubuntu:16.04

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD

The header is the two lines:

lolcow.def header part

BootStrap: library
From: ubuntu:16.04

This will use the existing Ubuntu 16.04 from the Sylabs Cloud Library as a base to work with.

The sections then becomes the rest:

lolcow.def sections part

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD

The %post section is the area where you can run commands to install software inside the image while building it. Here we use Ubuntu’s apt utility to update it’s cache, and install our three tools: fortune, cowsay, and lolcat.

The %environment section sets environment variables for when the container is run. While Singularity will inherit your current environment values, these will set and/or ensure that they are set to their intended values. Here we ensure that our locale is defined as the C locale, and that the PATH is updated to include the directory that has the commands we intend to run.

The %runscript section defines what commands to run when you invoke the container. Either with singularity run or ./lolcow.sif.

And finally, the %labels is where the Dan Godlove gives himself credit for creating this great example image.

There are many additional sections and things you can do in your images. Please see the full Singularity Documentation for Definition Files to learn more. If you need help writing your .def files, please reach out to support for assistance.

Building With Sylabs Cloud Builder

The singularity build command accepts the --remote option to upload your .def file to Sylab’s build server, allowing you to build an image without needing escalated privileges in the cluster environment. Sylabs even as a nice web interface to monitor you build progress, and let you know about any errors. Once the new image is build you can pull it back down from their cloud library and run it.

Create / Login to your Sylabs Cloud account

In order to use the Sylabs Cloud Builder, you will need a Sylabs account, but this account is accessed by another account from an external source.

Note

You will need an account with an external source:

• Google

• GitHub

• GitLab (Only GitLab.com accounts, departmental GitLab accounts will not work)

• Microsoft (While your @wwu.edu account is backed by Microsoft, it is not by default a Microsoft Account)

Once you have an account at one of the external sources, you can proceed to the Sylabs Cloud Builder site and begin to create your Sylabs account or login once it’s created.

Click the Sign in button in the top right hand corner of the page, which will present the option to sign in with one of the aforementioned accounts.

Once you have authenticated with that account, the Sylabs Cloud Account page will prompt you to enter your Sylabs Cloud username. It does not need to match your external account name or WWU username. This will be the username you use when pulling down the cloud image after it’s built.

After selecting a username, you will be presented with the option to create a new token. It is suggested to enter a name for where the token will be used, such as “csaw” or “cluster” in case you use other tokens on your personal systems. After entering your token name, click the Create New Access Token button.

Once the next page loads you will be presented with your token for use in the cluster environment. Click the Copy token to Clipboard button, and then head back over to your shell.

The command you will want to enter is singularity remote login.

USER@cluster-head:~$ singularity remote login
Generate an access token at https://cloud.sylabs.io/auth/tokens, and paste it here.
Token entered will be hidden for security.
Access Token:
INFO:    Access Token Verified!
INFO:    Token stored in /cluster/home/USER/.singularity/remote.yaml

The command prints out “paste it here”. To paste in your terminal, it varies from OS to OS. Here’s some common ones:

Ways to paste in your terminal

• Windows

Ctrl + V

• macOS

Command + V

Right-Click mouse, select Paste

• Linux / UNIX

Ctrl + Shift + V

Ctrl + Shift + Insert

Right-Click mouse, select Paste

The singularity remote login will confirm that your token works. After which you can begin remotely building your .def file. Let’s use the Singularity provided example file:

Doing the Remote Build

lolcow.def example

BootStrap: library
From: ubuntu:16.04

%post
    apt-get -y update
    apt-get -y install fortune cowsay lolcat

%environment
    export LC_ALL=C
    export PATH=/usr/games:$PATH

%runscript
    fortune | cowsay | lolcat

%labels
    Author GodloveD

Saving the above as lolcow.def, we can then issue the remote build command, see the build status, and pull the built image.

singularity build --remote lolcow.sif lolcow.def

USER@cluster-head:~$ singularity build --remote lolcow.sif lolcow.def
INFO:    Access Token Verified!
INFO:    Token stored in /root/.singularity/remote.yaml
INFO:    Remote "default" now in use.
INFO:    Starting build...
INFO:    Downloading library image
…
INFO:    Verifying bootstrap image /root/.singularity/cache/library/sha256.21678b2846063434e2009dee535d3e28ded35774ca72722e1d62b37077de30da
WARNING: integrity: signature not found for object group 1
WARNING: Bootstrap image could not be verified, but build will continue.
INFO:    Running post scriptlet
+ apt-get -y update
*** CUT FOR SPACE ***
+ apt-get -y install fortune cowsay lolcat
*** CUT FOR SPACE ***
done.
INFO:    Adding labels
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: /tmp/image-2207812356
WARNING: Skipping container verification
93.7MiB / 93.7MiB [========================================] 100 % 66.0 MiB/s 0s

Library storage: using 187.41 MiB out of 11.00 GiB quota (1.7% used)
Container URL: https://cloud.sylabs.io/library/USER/remote-builds/rb-6160c0793c3b09d4a37d789e
INFO:    Build complete: lolcow.sif

This uploaded our lolcow.def file to their build server, showed us the build output (almost all of which was cut to keep the display small), and then downloaded the image to the requested lolcow.sif name.

Now we can run the image as we have the past:

USER@cluster-head:~$ singularity run lolcow.sif
 ______________________________________
< Today is what happened to yesterday. >
 --------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

There’s many more things you can do with remote builds, but that covers the basics. The full Singularity documentation pages cover many of the additional options, but please contact support if you need additional help. It should also be noted that while it’s not documented in the official documentation, the %files section will not be uploaded to the remote build server if you’re building with the --remote flag.

Building With a Personal System

While out of the scope of this document, it’s worth mentioning that due to the nature of Singularity containing everything it needs in one file, you are able to build your .sif and upload it to be run on the cluster environment.

Unsupported Features

Singularity supports many more features than are mentioned on this page. Unfortunately, some features can’t be used due to security concerns when operating in a shared environment because they require elevated privileges.

Building an Image Locally

Building an image requires elevated privileges. Because of this it is not possible to do in our shared environment. However, two options are available: building with the Sylabs Cloud Builder, and building with your personal system.

Building/Running Sandbox Images

Singularity has the ability to allow you to build and run a container out of a writable directory (known as a sandbox), and then save the contents of the directory to a .sif file when you’re finished. This is sometimes possible to use without elevated permissions, but to ensure that it works correctly it requires them. As such, they can not be reliably used in our environment.

Fakeroot Containers

If you need to a run a process as root inside your container, it’s possible to map root’s id inside the container back to that of your user outside the container. The process believes it is being run as root, and in reality it’s running as you. Depending on what the process needs the root privileges for this may or may not work as intended. The use of this feature requires specialized map files to be generated for each user in the environment, on each host that the container will run on. This can be done, but due to the number of users it will only be done on a case-by-case basis. If you think you need this feature, please reach out to support and we’ll see if we can help you get your code running.

Notes on Docker Compatibility

Because Docker initially runs as the root user (when a non-root user is granted access to the Docker group they have root access), there are certain features that it allows that are not compatible with Singularity. One of these features is dropping privileges to another user. In a well crafted Docker file you may encounter the USER directive to become a different user instead of root. Because Singularity runs as you, you can’t become someone else, only root can do that. Docker containers that depend on this functionality will not work without modification. Fortunately this should be a rare occurrence with research containers, and is typically found in service containers instead.

For a full list of things that function differently between Docker and Singularity please see the best practices and troubleshooting section of the Singularity user documentation.

Additional Notes / Tips

Singularity Cache

The cluster environments have by default set your Singularity cache directory to be inside the /scratch directory, which does not travel with you from node to node. All files in that directory will be deleted if they have not been accessed for more than a day.

Setting your cache path to a scratch directory helps ensure you do not fill up your home directory with parts of a Singularity image.

Clearing Out Your Cache

If you’ve downloaded bunch of Singularity images and want to cleanup the cache before the system cleans them up, you can run the singularity cache clean command. Please see the linked documentation for additional information.