Singularity

Overview

Singularity (http://singularity.lbl.gov/) is an open-source software developed to add containers and reproducibility to the scientific high performance computing.

This page briefly overviews Singularity software, its main goals and example use cases.

Singularity is developing to provide container technologies on HPC systems. It gives users an easy way to use different HPC systems with different OSs there as simple as possible just simply deploy and run there their containers with established user environments and do not provide a pathway for privilege
escalation on HPC systems. For this reason Singularity can be separated into two logical parts: User Endpoint and Running Endpoint, see Fig.1 for the Singularity usage workflow.

Fig 1. Singularity usage workflow

 

  • In User Endpoint users just develop Singularity container on their own resource (workstation, laptop, virtual machine, etc.) on the full control. It means users need the root access to create, develop, modify & update  Singularity containers.
  • In Running Endpoint on the left side users run their singularity images as any stand-alone programs. They also can have access to the data and files inside of containers as easily as on the original system.To make changes in the containers users have to return to the endpoint system with root access.

Supported formats

In additional to the default Singularity container image format Singularity also supports numerous other formats. Suffix is necessary to identify the image type.

FormatDescription
tar.gz Zlib compressed tar archive
tar.bz2 Bzip2 compressed tar archive
tar Just uncompressed tar archive
cpio Uncompressed CPIO archive
cpio.gz Zlib compressed CPIO archive
directory Unix directories containing a root container image

To obtain the Singularity images users can use a standard URI format:

http:// https:// docker:// shub:// - (the Singularity image registry hub)

Examples of Singularity commands

Next table summarizes Singularity commands (based on version 2.3.1). For more information see Singularity User Guide at http://singularity.lbl.gov/user-guide

General commands
help show additional help for a command
Usage commands
exec Execute a command within container
run Launch a runscript within container
shell Run a Bourne shell within container
test Execute any test code defined within container
Usage options for commands
singularity help <command> additional options for chosen <command>
Global options
-h --help display usage summary
-q --quiet only print errors
--version singularity application version
-v --verbose increase verbosity
-d --debug print debugging information
-x --sh-debug print shell wrapper debugging information
Management commands (requires root)
bootstrap bootstrap a new image
copy copy files from host into the container
create create new container image
export export the contents of a container via a tar pipe
import import/add container contents via a tar pipe
mount mount a container image

Structure of Singularity container

At runtime singularity directly interacts with several files inside of container. They are:

  • /singularity - this file contains a user-specified scripts to be run at container direct execution or through the 'singularity run' command
  • /.env/ - this directory contains an arbitrary number of files or scripts to be sourced on runtime
  • Entrypoint - scripts below are executed  when corresponding singularity command is called
    • /.exec - sources the /environment file and executes user commands
    • /.run - sources the /environment file and executes the runscript located at /singularity
    • /.shell - sources the /environment file and executes /bin/sh
  • Header - an image header unique for each singularity image

One can include into container image a base OS's applications and libraries, scientific or custom programs, data, scripts and pipelines. Thereby container images are portable between Linux distributions as long as binary format (ELF x86_64) is compatible. Singularity can run on host Linux distributions from RHEL 6 and contained images have been tested as far back as Linux 2.2 (~ 14 years old). Because singularity images treat as standard single files this simplifies access control & management  via POSIX based file permissions. If user is either owner of image or has read access to that image, he can start a shell inside the image. Limiting of access to a shared image can be done by changing the permission ACLs on that files.

Work with Singularity

To create a container file with a particular size on User Endpoint:

$ singularity create --size 2000 ubuntu.img

Empty file waiting to be filled is created. Next step is "bootstrapping" - the process of filling the container. This can be done by using a bootstrap definition recipe which install OS and necessary applications http://singularity.lbl.gov/bootstrap-image:

$ singularity bootstrap ubuntu.img ubuntu.def

To run shell withing a container image (Running Endpoint):

$ singularity shell ubuntu.img

With --writable option one can write to the container (User Endpoint):

$ singularity shell --writable ubuntu.img

To run a container to perform a specific task described in a runscript called singularity at the root of the image file system:

$ singularity run ubuntu.img --input arg1 --input arg2

In example above runscript is expected to know how to parse inputs and arguments. Container can be also run by treating the image as an executable:

$ ./ubuntu.img --input arg1 --input arg2

Singularity launches the containers in read-only mode by default

To execute a container by sending a custom command:

$ singularity exec ubuntu.img cat /etc/os-release

Command run inside the container and output printed to the terminal on the local machine

To share/mount volumes between host OS and container:

$ singularity run --bind /tmp:/scratch ubuntu.img

here we bind host /tmp to /scratch inside of the container

For work-flows, pipe and IO compatibility one can run programs not only inside a container but also redirect IO, arguments, files, pipes, shell redirects and sockets directly to the applications inside the container:

  • run the image's xterm:

$ singularity exec ubuntu.img xterm

  • run script.py on local machine using the python inside the container:

$ singularity exec ubuntu.img python script.py

  • the same as above but by way of a pipe:

$ singularity exec ubuntu.img python < /path/to/sctipt.py

 

Singularity at MPCDF HPC systems

To have access to Singularity software on general purpose Draco HPC cluster at MPCDF please send us via the MPCDF helpdesk a short description of your project/problem which you are going to solve using Singularity.  Your feedback and use cases will be added on this page as an Singularity use case example.

Document Actions