docker vs. mesos unified container
Post on 13-Feb-2017
255 Views
Preview:
TRANSCRIPT
Unified ContainerApr 2016
Guang Ya Liu (liugya@cn.ibm.com)
Qian Zhang (zhangqxa@cn.ibm.com)
Yong Feng (yongfeng@ca.ibm.com)
IBM Platform Computing
What is a Container
Loosely defined
A lightweight “VM”
To Mesos
A per-task/executor isolated execution
Container History
LXC (2010)
Cgroups (2012)
Linux namespaces (2013)
Docker (2014)
Unified Container (2016)
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 2
Two Major Containerizers in Mesos
Mesos Containerizer (Default)
Implements unified container.
Provides lightweight containerization and
resource isolation using Linux-specific
functionality such as cgroups and namespaces.
It is composable so operators can selectively
enable different resource isolators.
Docker Containerizer
Docker containerizer delegates container
management to the Docker engine.
Dynamically chosen
Based on ContainerInfo
o ContainerInfo::MESOS
o ContainerInfo::DOCKER
If both are specified, based on the sequence
configured in agent flag of --
containerizers=docke
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 3
Agent
Mesos Containerizer Docker Containerizer
Docker Daemon
Docker Executor
Isolator
Isolator
Isolator
Isolator
Command/Customer
Executor
Why Mesos Implements Unified Container
Unified container (Mesos Containerizer) will be the long term solution in Mesos for container
management. Docker Containerizer will exist for a long time but sunset eventually.
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 4
In the middle of 2015, Mesos started to implement its own unified container with Mesos
Containerizer and tried to remove the dependency on Docker daemon, the motivations behind this
initiative are: Docker daemon is not stable and buggy (e.g. crash) in a large scale environment (reported by twitter)
Docker is trying to foster its own ecosystem which is not that open for 3rd part container orchestrators to
integrate with. E.g., Docker relies on CNM/libnetwork for network integration, but the design of libnetwork and its drivers is specific to Docker
which is not easy for integration without depending on Docker daemon. That’s one of the reasons why Mesos and Kubernetes
have chosen CNI for container network integration rather than Docker CNM. CNI follows the UNIX philosophy of doing one
thing well, and it doesn't require daemons, and is cross-platform (i.e., work the same across runtimes).
Docker Containerizer has gaps and hard to resolve.
o Requires Docker installation and maintenance.
o Tasks die with Docker daemon upgrade, etc.
o Cannot compose with Mesos isolators (disk quota, port mapping, volume, CNI).
Mesos is target to support all the major container image specs in a generic way rather than sticking to Docker.o Docker, AppC and OCI.
o Maintain one containerizer for each supported image spec will be hard.
o Introducing a new feature may need to update all containerizers.
Unified Container Architecture
Launcher
Fork the executor process in the
containerized context.
Provisioner
Pull, cache and create rootfs for the
container when launching container
Support AppC and Docker Image format
Universal provisioner for different
container spec
Isolator
Runtime Isolator: Isolate image entry
point, cmd, env, working dir.
CNI Isolator: Manage (add and delete)
network for container with CNI command
line
Docker Volume Isolator: Enable container
to use external storage
…
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 5
Docker vs. Mesos Unified Container – Architecture
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 6
Docker Daemon
Image Store
Create and
access images
Layer Store
Create container
FS via driver
Puller
Pull images from
v1/v2 registry
libcontainerd
Talk to containerd
for container
runtime and
resource isolation
OverlayAufs
. . .VFS
Volume Store
Volume mngt
via driver
FlockerLocal
. . .Convoy
libnetwork
Network mngt
via driver
ipvlanBridge
. . .Overlay
containerdDaemon to control runC
Container Container Container
Mesos Containerizer
Provisioner
LauncherFork executor in the
containerized context
via namespace
DVD Isolator
Container Volume
management via
dvdcli
CNI Isolator
Container network
mngt via CNI
pluginipvlanBridge
. . .Flannel
Container
(Executor)
Container
(Executor)
Container
(Executor)
Image Store
Pull and store Docker/Appc
images
Backend
Create container FS
Copy / bind / overlay
Cgroups IsolatorResource isolation
via Cgroups
Flocker Convoy
Rex-Ray ...
Provisioner Deep Dive
Image Fetch and Store
Vendor specific store which does discover, fetching and processing.
Cache Image for fast provisioning
Force pull image to bypass cache
Provision Backend
Copy
o Small Image less than 1 G
o request IO when copying image
Bind
o Only one layer image, design for large images (multi-GB), request almost
zero IO.
o The rootfs is read-only, need external storage if want to customized some
RW volume.
Overlay
o Support both one layer and multi layer images, request almost zero IO.
o The rootfs is writable.
Roadmap: Smart backend selection based on image size.
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 7
File System Isolator
Provisioner
Backend Store
Copy
Overlay
Bind
Docker
AppC
OCI
Docker Runtime Isolator
Runtime configuration
Set up environment variables.
Set up working directory.
Set up command to be executed.
If user specifies a command in
CommandInfo, that will override the
default Entrypoint/Cmd in the Docker
image.
Otherwise, the container will use the
default Entrypoint/Cmd and append
arguments specified in CommandInfo
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 8
CNI (Container Network Interface) is a proposed standard developed by CoreOS for configuring
network interfaces for Linux containers. CNI is adopted by Kubernetes and is bening considered as
an standard by CNCF. Many CNI plugins (e.g., bridge, flannel, calico, etc.) have already been
developed.
CNI isolator invokes CNI plugins to achieve container network management.
Add a container to CNI networks.
Remove a container from CNI networks.
Report the allocated IP addresses of container to framework.
Support both single-host and multi-hosts network.
Containers created with different image specs (e.g., Docker, Appc) in the same CNI network can communicate
smoothly.
Containers created by different container orchestrators (e.g., Mesos, Kubernetes) in the same CNI network can
communicate smoothly.
Cannot work with network/port_mapping isolator as they are in different level layer 3 (IP) Vs layer 4 (TCP/UDP)
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 9
CNI isolator
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 10
CNI isolator in Network Management of Mesos Container
• IP Address Management (IPAM) Server
• assigns IPs on demand
• recycles IPs once they have been released
• (optionally) can tag IPs with a given string/id.
• IPAM client
• tightly coupled with a particular IPAM server
• acts as a bridge between the “Network Isolator Module”
and the IPAM server
• communicates with the server to request/release IPs
• Network Isolator Module such as CNI Isolator:
• looks at TaskInfos to detect the IP requirements for the
tasks
• communicates with the IPAM client to request/release
IPs
• communicates with an external network
virtualizer/isolator to enable network isolation
• Cleanup Module:
• responsible for doing a cleanup (releasing IPs, etc.)
during an Agent lost event, dormant otherwise
Docker Volume Isolator
External Storage Integration
Leverage DVD (Docker Volume Driver)
Leverage dvdcli to call DVD API
o Get Volume Path
o Mount Volume
o UnMount Volume
It does not handle the life-cycle of volume
management. User has to call Docker API/CLI
(after Docker 1.9) or DVD API/CLI to create
and manage the life-cycle of volume.
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 11
Docker vs. Mesos Unified Container - Functionality
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 12
Provision Image
Mesos: Support both Docker and Appc image specs and will support OCI in future.
Docker: Can only support Docker image spec.
Network Management
Mesos: Depend on CNI which is a generic container network solution and it can support containers created from
any image specs (Docker, Appc, etc.).
Docker: Depend on libnetwork which is specific to Docker container.
Volume Management
Mesos: Depend on dvdcli to leverage Docker Volume Driver Plugin Proto to integrate with 3rd party storage. The
dvdcli will be created as a Mesos C++ library in Mesos roadmap.
Docker daemon: Rely on Docker Volume Driver Plugin to integrate with 3rd party storage.
Docker vs. Mesos Unified Container - Functionality
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 13
Recovery
Mesos: For the frameworks which enable checkpoint, their container will continue running uninterrupted when
Mesos agent is down, and once Mesos agent is started again, it will take over those containers again.
Docker: All containers will die when Docker daemon is down. Docker may enhance this in the nearly future.
Extensibility
Mesos: Easy to be extended with isolator which can be implemented as a separate module (.so), e.g., if there is
a new type of resource need to be isolated between containers, we can just implement a new isolator for it and no
need to touch any code in Mesos core path.
Docker: Hard to extend and customize isolation for container because there is no way to inject logics during the
lifecycle of container.
Docker vs. Mesos Unified Container - Integration
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 14
Interface to End User / Framework
Mesos: Provides HTTP API to framework for offer/task management and also provides HTTP endpoints to
operator for operating the Mesos cluster (e.g., set weight/quota for role, perform agent host maintenance, etc.)
Docker: Provides CLI and API to end user for container lifecycle management.
Network Integration
Mesos: Integrate with different network technologies (e.g., bridge, IPVLAN, MACVLAN, etc.) with different CNI
plugins.
Docker: Integrate with different network technologies (e.g., bridge, IPVLAN, MACVLAN, etc.) with different
Docker network drivers.
Storage Integration
Both Docker and Mesos integrate with different storage technologies (e.g., Amazon EBS) with different Docker
volume driver plugins.
Docker vs. Mesos Unified Container - interoperability
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 15
Network
Mesos: Containers created by Mesos can smoothly communicate with containers created by other container
runtimes as long as they are in the same CNI network.
Docker: It is not easy for Docker container to communicate with containers created by other container runtimes,
there may be some potential issues, e.g., libnetwork has its own way to allocate IP which is not exposed outside,
so it may cause IP conflict if a container created by another runtime wants to join a Docker network.
There is a POC work to make Docker container communicate with container created by Kubernetes, but it
is kind of hacky since it relies on a special label “cni.network” when executing “docker run” so that the
Docker container will be attached to the specified CNI network.
Storage
Since both Mesos and Docker rely on Docker volume driver plugin to do storage integration, it is possible that a
container created by Mesos mounts and uses a volume created by Docker.
Agent Configuration
--isolation
docker/runtime: enable docker runtime isolator
network/cni: enable network cni isolator
docker/dvd: enable docker volume isolator
--image_providers
Enable container image support in Mesos containerizer.
Support docker and appc currently
--appc_simple_discovery_uri_prefix or --
docker_registry
URI prefix to be used for simple discovery of appc images or
Docker images
--appc_store_dir or --docker_store_dir
Directory the appc or Docker provisioner will store images in.
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 16
Example
$ sudo mesos-slave \
--containerizers=mesos \
--image_providers=appc,docker \
--isolation=filesystem/linux,docker/runtime
Future Work
AppC Enhancement
Need support runtime configuration like docker runtime isolator
Support meta image discovery
o Only simple image discovery
o appc_simple_discovery_uri_prefix specify image prefix
o Search Image via appc_simple_discovery_uri_prefix/{name}-{version}-{os}-{arch}.{ext}
OCI Support
Need support OCI Spec, however OCI Spec in not finalized yet.
Security
Currently launches all tasks without any control on the privilege levels under which they will be run. This makes
the execution environment insecure.
Need enforcing control on the available privileges on the launched task
Mesos CLI
Need some functions like “docker ps”, “docker inspect” to simplify some query operations
IBM GCG STG Lab
© 2015 IBM Corporation | IBM Confidential 17
Thank You
top related