software environment -containers
TRANSCRIPT
SOFTWARE ENVIRONMENT - CONTAINERSJEREMY SMITH
WHAT IS A CONTAINER?
Source: https://www.containersfirst.com.au/wp-content/uploads/2013/08/40ft-HC-A-Grade-Shipping-Container-BLUE-copy.jpg
WHAT IS A CONTAINER?
¡ Containers are encapsulations of system environments – Greg Kurtzer (Singularity CEO)
¡ A software package that contains everything required to run an application/workflow, including files, environmental variables, libraries and dependencies
¡ Isolation occurs at the kernel level, requires no guest operating system
https://www.docker.com/resources/what-containerhttps://www.hpcwire.com/solution_content/ibm/cross-industry/containers-move-upscale/
WHAT IS A CONTAINER?
¡ Enables custom requirements and workflows
¡ Allows for interchangeable environments and configurations at runtime
¡ Allows portability between systems
https://www.docker.com/resources/what-containerhttps://www.hpcwire.com/solution_content/ibm/cross-industry/containers-move-upscale/
CONTAINER VS VIRTUAL MACHINE
Containers
¡ Abstraction at the application layer (abstracts the software)
¡ Use host OS kernel – achieves almost native application speeds
¡ Efficient, fast, lightweight
¡ (usually) Small footprint (10MBs)
VMs
¡ Abstraction of the physical hardware layer (abstracts the hardware)
¡ EachVM has full copy of OS
¡ Can be slow to boot
¡ Larger footprint (GBs)
Combined powers, containers and VMs provide great flexibility!
https://www.electronicdesign.com/dev-tools/what-s-difference-between-containers-and-virtual-machines
CONTAINER VS VIRTUAL MACHINE
Source: https://www.bmc.com/blogs/containers-vs-virtual-machines/
SINGULARITY CONTAINERS
¡ Launched in 2016
¡ Designed for science/academia and HPC
¡ Maintains user’s privileges - privileges inside containers same as outside
¡ Files on the host are reachable from within the container
¡ Integration with SLURM, MPI and GPU
https://singularity.lbl.gov/abouthttps://www.sylabs.io/about-us/history/
REPRODUCIBLE SCIENCE
¡ No layered building of container images
¡ Replicate and share system environment
¡ Keep a record/reference of the environment for reproducibility (def files, singularityHub)
https://singularity.lbl.gov/abouthttps://www.sylabs.io/about-us/history/
USING A CONTAINER
¡ Shell
USING A CONTAINER
¡ Exec
¡ Run
USING A CONTAINER
¡ MPI with SLURM example using MPICASA
SLURM batch file called my-casa-job.sh:
Terminal on SLURM cluster:
Frank, B. CASA @ HPC+IDIA. [pdf document] IDIA, UCT, Cape Town
INTEGRATION WITH JUPYTER NOTEBOOKS
Containers used as kernels forJupyter Notebook
CONTAINERS CURRENTLY SUPPORTED
casa-stable jupyter-casa python-2.7 (3.6) kern source-finding
• Access: Terminal (SSH)
• The most recent, stable build of CASA, and can only be used on the terminal (e.g., by SSH’ing into the machine)
• Includes MPI support
• Access: Jupyter (https)
• A working development version of CASA which is controlled through the Jupyter Notebook. Open a new notebook with this kernel to run casa as a python module in a jupyternotebook
• Access: Terminal (SSH) and Jupyter (https)
• Basic Python 2.7 (3.6) stack
• Access: Terminal (SSH) and Jupyter (https)
• A comprehensive radio astronomy software environment that has all the packages listed on http://kernsuite.info/packages
• Access: Terminal (SSH) and Jupyter (https)
• Includes commonly used tools for source-finding and data inspection including pyBDSF, AEGEAN, SciServer, DS9, image mosaicing software, etc
CREATING A CONTAINER
¡ Build from Recipe
¡ Build from Singularity
¡ Build from Docker
Example definition file:
https://www.sylabs.io/guides/3.0/user-guide.pdfhttp://singularity.lbl.gov/
