-
Notifications
You must be signed in to change notification settings - Fork 0
Home
This wiki covers the basics of how to use a singularity container. What's singularity (in a sense of the subject software)? What's a container? Those questions are where we start below :) For full documentation, read official documentation. This wiki page is meant to be shorter and more efficient to learn our use in DeepLearnPhysics.
If you don't care concepts/ideas at all, you can skip reading the rest of this front page. Go read the requirements and getting started. Assuming your goal is to run a program (python script, a compiled executable or something), those sections should be enough for you to just do that. Soon or later you probably want to start building your container, and we discuss that here.
If you google it, likely you find its meaning as the no-return point where A.I. surpass human (yay!), a point where the math requires you to take a college senior level real analysis course. In this context, it means a particular container software technology started by a team of developers at the Lawrence Berkeley National Lab (LBNL). You can read the wikipedia page about it. We don't dig any further here. If you are familiar with a container and wonder "what's different from docker?", I answer at the end.
I give my descriptions later. But if you want to learn from a better source, here are a few suggestions:
- Brett Virren's blog post on WireCell software
- Docker documentation (docker is a yet another container software)
- Singularity documentation ... in the order of easiness to read (with my personal bias).
Lengthy description: a container is a unit of software designed to be portable onto various computing platforms. When we write a program, it depends on more basic (and often many) libraries. In order to execute this program, all dependent libraries need to be provided together. Otherwise this program cannot be run on a different platform. A container addresses this issue by defining all collections of basic libraries as a portable unit.
In short: a container ships/provides a whole environment to execute a program, which is essentially a whole operating system (except for the kernel to execute your program, see below for this point).
The last point is a key that differs a container from a virtual machine. While running a virtual machine (VM) requires running an independent operating system's kernel per instance, a container can use the host platform's kernel. This reduces much of resource usage per a running container instance (and hence a burden on the host platform). The idea is that, "hey, to run your python script, I don't need to be running kernel..." ;)
Docker is the most popular container software out there. Singularity is similar to docker except it is rootless executable. When you run a singularity container, you can execute with your own user account and also use parts of the host's file system with access writes unchanged. This removes extra burdens for admins to manage environments for users to run a program, and/or removes burdens from the users for being in a root environment when running a program or developing software inside a container.
To be more specific with examples, Docker is great for running daemon jobs like hosting a web server or a database service on a cloud server machine (like AWS, Azure, gcloud, etc.) which are often preferred to be run by the system and not by your "user account". Singularity is suited for running your personal code (such as analysis python script for your research) on a high performance computing (HPC) cluster which would have a security concern for applications run by root (in case of using a Docker image). The key difference is in the need for a run-environment. In the former, you typically want to avoid using a user account. In the latter, it is natural to use his/her user account as is for scientists (at least in the way scientific computing evolved so far).
Hopefully this made it clear that Singularity is not in competition against Docker. In fact it works nicely with docker: you can build a singularity container based on a docker container. So, as a singularity user, you can take benefits from both type of container images, plus HPC admins would love you.
Finally, a rootless container is one of major on-going discussion and development effort in their field. Singularity needs root access when you are building an image, so not a truly rootless container technology. You can google more about rootless container if you are interested int.