Docker 101 - all about Docker containers
Post on 16-Apr-2017
Docker 101WATCH WEBINAR ON DEMAND
What are Docker containers - relative to physical machines, VMs and other containers?Who is responsible for Docker containers? Why and when were Docker containers created?What is the container ecosystem?Where is use of containers appropriate and not appropriate? HPC applications? Big Data Analytics? Specifically, Spark-based applications? On premise and in the cloud?Is running Docker different in HPC versus microservice-based applications?How can I make use of Docker containers? How can I containerize my application?How can I create, or make use of, a Docker image?How can I run Docker containers as I do other types of workloads?Getting Started and Next StepsAgendawww.univa.com2
Benefits of CGROUPS supportAddresses long-standing issues for which classical Unix resource control means (rlimit) provide no complete solutionAllows for well controlled concurrent usage of servers by multiple jobs with no unmanaged conflictsResource isolationAllows for dependable and complete termination of jobsAvoids that complex jobs consisting of process hierarchies or parallel tasks grow out of boundsAllows for soft limits dynamically responding to resource usage of other workloads on same serversAllows for run-time adjustments of resource limitsProvides more robust means for suspending workloads3Univa Grid Engine 8.2, August 2014Source: Advanced Administrative Training Course
www.univa.com4Docker and Software ContainersWhat is Docker?Docker is a tool that packages an application, filesystem, and all other dependencies into an easily distributable software package that can be installed and run on any modern Linux Server.
What is a Software Container?Similar to a Virtual Machine but a single Operating System is sharedLess overhead and generally faster than Virtual MachinesYou can run more Software Containers on a physical machine than VMsApplications more portable from Dev through deployment than VMs
Not a new concept, Sun Microsystems has Solaris Zones
Why is Docker different?
www.univa.comVMs vs. Containers5
6Docker on Google TrendsRapid growth since the end of 2013 continues
Key: Blue = Germany, Red = US, Orange = India, Green = UK & Purple = Japan; China dominates all others
7Docker on Google TrendsRapid growth globally since the end of 2013 continues Key: Blue = France, Red = Germany, Orange = US, Green = India & Purple = Japan; China dominates all others
www.univa.comDocker Linux Interfaces8
WorkflowDEIS, OpenShiftSchedulingNavops Command, MarathonOrchestrationKubernetes, Mesos, SwarmContainer EngineDocker, RocketOperating SystemRHEL, CoreOSVirtual InfrastructureGCE, AWS, vSpherePhysical InfrastructureCompute, Network, Storage
Simplified Technology Stack9
10The Container Landscape
ContainerHYPERVISORSKVM, ESX, HyperV, VMWare, FusionCLOUD PROVIDERSAzure, Amazon, Google, VMWare,BARE METALOPERATING SYSTEMSCoreOS, Red Hat Atomic, Ubuntu Snappy Core, VMWare Photon, Rancher OSCONTAINER HOST RUNTIMEDocker Engine, RunC, Cloud Foundry Garden, CoreOS RocketPaaS WORKFLOW MANAGEMENTEngine Yard DEIS, Red Hat OpenShiftCONTAINER NETWORKINGCoreOS Flannel, Open vSwitch, Docker NetworkingCONTAINER CLUSTERINGDocker Swarm, CoreOS Fleet, Kubernetes, Mesosphere DCOS, Rancher SchedulingAPPLICATION SCHEDULINGCloud Foundry Diego, Kubernetes, Mesosphere Marathon, NAVOPS CommandCONTAINER IMAGE REGISTRYDocker Registry, CoreOS Registry CONTAINER IMAGE TRUST AND VERIFICATIONDocker Notary, Hashicorp Vault, Intel Clear ContainersCONFIGURATION MANAGEMENT AND AUTOMATIONAnsible, Puppet, Chef, SaltCONTINUOUS INTEGRATION AND DELIVERYUrbanCode, Jenkins, TeamCityDISTRIBUTED SERVICE DISCOVERY AND CONFIGURATION STOREEtcd, Hashicorp Consul & Serf, ZookeeperContainerContainerContainerSTORAGEGluster, CephCLUSTER PROVISIONINGNAVOPS Launch, Terraform, Kraken, GKEREPORTING, MONITORING and ALERTINGDataDog, SysDig, New Relic, CAdvisorAPPLICATIONSWordpress, CouchDB, Hadoop, Spark, NGINX,ADMINISTRATION CONSOLEDocker Universal Control Plane, Shipyard
What are Docker containers - relative to physical machines, VMs and other containers?Who is responsible for Docker containers? Why and when were Docker containers created?What is the container ecosystem?Where is use of containers appropriate and not appropriate? HPC applications? Big Data Analytics? Specifically, Spark-based applications? On premise and in the cloud?Is running Docker different in HPC versus microservice-based applications?How can I make use of Docker containers? How can I containerize my application?How can I create, or make use of, a Docker image?How can I run Docker containers as I do other types of workloads?Getting Started and Next StepsAgendawww.univa.com11
Web AppUse Caseswww.univa.com12https://docs.docker.com/engine/userguide/containers/usingdocker/
Big Data AnalyticsUse Caseswww.univa.com13
Spark Use Case
The app-server tier illustrating Apache Spark and its support for various data sources.
www.univa.com15ThunderWritten in Spark's Python API (Pyspark)Makes use of scipy, numpy, and scikit-learnJupyter Notebook serves as interactive GUI Runs in a Web browserNotebooks can include text and graphics Secure, remote access to an in-cluster IPython Notebook server Includes modular functions for time-series analysisCan interface with C/C++ from Pythonhttp://thunder-project.org/
www.univa.comContainerized Spark Environment 16
www.univa.comContainerized PySpark Example 17
www.univa.com18Customizing ContainerUpdate and commit Build via Dockerfile
20Use Case DescriptionScientific data analytics for genome sequence discoveryMassive data analysis large clusterLife-science analysis standardized on Grid EngineCluster is a shared resourceMany usersAdvanced policies, e.g. fair-sharing, back-filling and dependable resource controlsAdvanced job types, e.g. array jobsDetailed accounting and billing for resource consumption
21Challenges and SolutionChallengesSandboxing maintain many production environments for a long timeAt minimal or no performance impact:From running applications in a containerFrom network and shared file system access from within a containerFrom starting the same containers over and over on nodesAvoid to reload imagesSolutionCRG Nextflow workflow managementIntegrated with Univa Grid EngineAnd integrated with DockerMake Univa Grid Engine Docker-awareEnable Docker jobsContainer image cache-aware scheduling
22Results4% increase of Docker application run-time vs native run-time with cached images12.5% increase with container bootstrapping, i.e. downloading from image repositoryImage-cache aware scheduling has solid benefit on utilization and throughputCost is considered low vs benefit by CRGUse case requirements really can't be satisfied without containershttp://www.univa.com/resources/wp-crg.php
23UGE Container Edition: Architecture
www.univa.com24Univa Grid Engine Container Edition (1)Launch Docker Container on best machine in clusterReduces time wasted (it can be minutes or longer) Attempting to launch on an improperly serviced execution host.Waiting for the Docker image to download from the Docker registry. Ensures container runs faster increasing throughput in the cluster.Run Docker Containers in a Univa Grid Engine ClusterBusiness Critical containers are prioritized over other containers. Increases efficiency of the overall organization.Containers can be orchestrated alongside other critical workloads such as batch jobs and frameworks. $ qsub -o /home/jdoe -j y -xdv "/home:/home" -l docker,docker_images="*centos:latest* my_job.sh
www.univa.com25Univa Grid Engine Container Edition (2)Job Control and Limits for Docker ContainersProvides user and administrator control over containers running on Grid Engine Hosts.Accounting for Docker ContainersKeeps track of containers. Share policies require accounting.Data file Management for Docker ContainersTransparent access to input, output and error files. Simplifies the management of input and output files for Docker Containers and ensures any output or error files are moved to a location where the user can access them.Interactive Docker ContainersGood for debugging when containers dont work correctly!Parallel jobs in Docker ContainersMessage-passing parallel jobs can each run a set of tasks in a container on a machine.
HPC as a Containerized Cloud Based Service
Ian Lumb (IL) - ValidateCloud Native Computing Foundation (CNCF)For current applications and services Uptake of cloud computing remains an afterthought from a systems-architecture perspectiveCNCF aims to introduce a cloud-native paradigm shift that emphasizes:ContainerizationDynamic schedulingOrientation around micro servicesMaking use of Kubernetes as a seed technology#1 priority: Integrate the orchestration layer of the container ecosystemUniva is a Founding MemberAlong with Google, IBM, Intel, Red Hat and numerous others ... Prototype implementations becoming available
Univa Container Solutions
Easy installation, preconfigured solution including pre-integration with cloud services. Build a container cluster on premise or in the cloud.The fastest way to build a container cluster!!
Respond Quickly: Easy to resize, adapt, dynamic provisioningOrchestrate and Optimize: Best use of resources and keep track of containersThe most advanced container orchestration!!
www.univa.com29Webinar April 28, 2016 @ 1 pm EDTGoing Cloud-Native with Navops Launch and Docker
Discussion topics to include:The promise of containers in the enterpriseHow to address the complexity of building a Kubernetes-based clusterHow to install and configure Navops Launch in minutes - a Kubernetes-based container clusterHow to build a hybrid container cluster - one that spans and bursts from your on-premise environment into the cloud (this is cool!)A cloud-native use case that makes use of Google Compute Engine via Navops LaunchRegister via http://navops.io/onlinemeetups.html
SummaryEarly adopters report easier replication, faster deployment and lower configuration and operating costs of workflows involving Docker containersDocker containers can be managed in the same way as other types of workloads and workflowsMacro services can be supported without a need for refactoring applications/code/Alongside Kubernetes, Docker containers key to cloud-native applications
www.univa.comTHANK YOUIan LumbSolutions Architect
+1 630 email@example.comWATCH WEBINAR ON DEMAND
https://github.com/NVIDIA/nvidia-docker Docker and GPUs
An open-source authorization service and user interface for the next generation Docker RegistryDeveloped by SUSE engineers during a hackweek Manage users with teams plus images with secure namespacesViewers can only pull imagesContributors can push and pull imagesOwners can push and pull images plus manage users UI with viewing and searching capabilities respective of authorization levels Audit trail that logs events Compatible with Univa Grid Engine Container Edition and NAVOPS