docker and cloud - enables for devops - by aca-it
TRANSCRIPT
ACA IT-SOLUTIONS | © 2015
Stijn Wijndaele
[email protected] @stijnwijndaele
Introduction?
Your Hosts For Today
Business Development Manager Cloud & Mobile Solutions
ACA IT-SOLUTIONS | © 2016
Stijn Van den Enden
[email protected] @stieno
CTO
Wants to be the greatest IT service provider
by being fanatic about
maximising customer value
'DOCKER' & CLOUD: ENABLERS For DEVOPS
DevOps
DevOps is gericht op het tot stand brengen van een cultuur binnen organisaties waardoor het
ontwikkelen, valideren en releasen van software sneller, meer betrouwbaar en frequenter kan
verlopen
CLOUD ?
Why the Cloud ?
• Elastic Capacity • Business Agility • CapEx becomes OpEx • Enabler for DevOps
Why the Cloud ?
• Elastic Capacity • Business Agility • CapEx becomes OpEx • Enabler for DevOps
Elastic Capacity
Traditional: Rigid Cloud: Elastic
Capacity
Demand
Capacity
Excess Capacity Wasted $
Demand
Un-met Demand
Why the Cloud ?
• Elastic Capacity • Business Agility • CapEx becomes OpEx • Enabler for DevOps
Business Agility
Add New Dev EnvironmentAdd New Prod EnvironmentAdd New Environment in JapanAdd 1,000 ServersRemove 1,000 serversDeploy 2 PB Data warehouseShut down 2 PB Data warehouse
IAAS/PAAS: Infrastructure in Minutes
Old World: Infrastructure in Weeks
Everything changes with this kind of agility
+ weeks
Service Request
Why the Cloud ?
• Elastic Capacity • Business Agility • CapEx becomes OpEx • Enabler for DevOps
Capex becomes Opex
• no initial investments • no termination fees • no (real) commitments • clear pricing model
Why the Cloud ?
• Elastic Capacity • Business Agility • CapEx becomes OpEx • Enabler for DevOps
Enabler for devops
• Automation - Less repetitive tasks • Better management tools • no need to build from scratch - reuse • Focus on what matters - your business
Netflix in 2015
• 69,2 million users • 42,5 billion streaming hours • 45GB bandwidth user/month
Source : DMR January 2016
Netflix in 2015
• 10x Customers • 100x Traffic • 5x Devs • 2x Cost of IT infrastructure
Source : DMR January 2016
in 2008
Infrastructure as CodeFrom API accessible xAAS services to automated
deployment
Tools can enable change in behavior and eventually change culture ‘
[patrick debois]
AMI, VMX, OVF, …
Cloud Provider
Continuous Integration
Build, test and integrate applications
Configuration Management
Configure and setup an existing machine
Orchestration Tools
Provision Infrastructure and services based on a configuration template
heat
base image
additional packages
application
code-artefact-765
'DOCKER' & CLOUD: ENABLERS For DEVOPS
Docker
• What is docker ? • What’s more ?
Docker
• What is docker ? • What’s more ?
DockerIT works on my System
Supply Matrix - From Hell!
IT Matrix From Hell!
One Size - Fits All
Virtual Machine ?
Compute, Storage, Network
Host OS
Hypervisor
VM1
MicroService
Guest OS
JVM
VM2
MicroService
Guest OS
JVM
Compute, Storage, Network
Host OS
container1
container2
container3
container4
JVM JVM JVM
MicroService MicroService MicroService
JVM
MicroService
Containers have own isolated resources
Performance Comparison: http://ibm.co/V55Otq
VM’s abstract underlying hardware, but limits resource utilisation
Ship without worrying
Docker - Layering
• Container = Writeable layer
• Image = Static layers
• Containers = Share the kernel
docker build -t test/a .
FROM aca—base-ubuntu:1.1
RUN apt-get update
RUN apt-get install -y apache2
RUN touch /opt/a.txt
Let’s build Webserver A!
Our Dockerfile
docker build -t test/a .
FROM aca—base-apache:2.1
RUN touch /opt/a.txt
Or even better….
docker history test/a
IMAGE CREATED CREATED BY SIZE
4dc359259700 About a minute ago /bin/sh -c touch /opt/a.txt 8 B
9977b78fbad7 About a minute ago /bin/sh -c apt-get install -y apache2 54.17 MB
e83b3bf07b42 2 minutes ago /bin/sh -c apt-get update 20.67 MB
9cd978db300e 3 months ago /bin/sh -c #(nop) ADD precise.tar.xz in / 204.4 MB
6170bb7b0ad1 3 months ago /bin/sh -c #(nop) MAINTAINER Tianon Gravi <ad 0 B
511136ea3c5a 10 months ago 0 B
What is in Webserver A?
docker build -t test/b .
Let’s build Webserver B!
FROM aca—base-ubuntu:1.1
RUN apt-get update
RUN apt-get install -y apache2
RUN touch /opt/b.txt
Our Dockerfile
docker history test/b
IMAGE CREATED CREATED BY SIZE
c0daf4bw2ed4 5 seconds ago /bin/sh -c touch /opt/b.txt 8 B
9977b78fbad7 About a minute ago /bin/sh -c apt-get install -y apache2 54.17 MB
e83b3bf07b42 2 minutes ago /bin/sh -c apt-get update 20.67 MB
9cd978db300e 3 months ago /bin/sh -c #(nop) ADD precise.tar.xz in / 204.4 MB
6170bb7b0ad1 3 months ago /bin/sh -c #(nop) MAINTAINER Tianon Gravi <ad 0 B
511136ea3c5a 10 months ago 0 B
What is in Webserver B?
Containers are here to stay!
The Open Container Initiative (OCI) is a collaborative project hosted under the Linux Foundation designed to establish common standards for containers.
Docker
• What is docker ? • What’s more ?
Docker Hub - Public Library
Docker Registry - Private Library
Docker Registry - Private Library
• Tightly control where your images are being stored • Fully own your images distribution pipeline • Integrate image storage and distribution
tightly into your in-house development workflow • Collaborate with your colleagues
Docker Hub
• Docker HUB = Docker Registry - As A Service • Zero Maintenance • Additional Functionality to increase collaboration • Alternative Providers : Google / AWS / CoreOS
/preference-service
Repository
DockerFile
Continuous Integration Infrastructure
Container Image Repository
Compute, Storage, Network
Host OS
daemon
container1
JVM
MicroService
pull
push
build
provision
container1
JVM
MicroService
Source Control System
The real value of Docker is not technology
It’s getting people to agree on something
Shipping Applications Beyond a single container
Container Image
code-artefact-765
Minimal OS
Docker
Virtual MachineHardware
kubelet
Pod
Container
proxy
NodeMaster
scheduler
controller manager(replication controller, etc.)
API endpoint
distributed configurationstorage
kubectl(client command line tool)
K/V
K/V
K/V
K/V
K/V
/etc distributed
raft - leader election
//Adding a value $ curl http://127.0.0.1:2379/v2/keys/message -XPUT -d value="Hello world”
//Quering $ curl http://127.0.0.1:2379/v2/keys/message { "action": "get", "node": { "createdIndex": 2, "key": "/message", "modifiedIndex": 2, "value": "Hello world" } }
//Delete $ curl http://127.0.0.1:2379/v2/keys/message -XDELETE
Operations
apiVersion: v1kind: ReplicationControllermetadata: name: es-data labels: component: elasticsearch role: data visualize: "true"spec: replicas: 2 selector: component: elasticsearch role: data template: metadata: labels: name: es-data component: elasticsearch role: data visualize: "true" spec: serviceAccount: elasticsearch containers: - name: es-data securityContext: capabilities: add: - IPC_LOCK image: quay.io/pires/docker-elasticsearch-kubernetes:1.7.2 env:
es-data-rc.yaml
Pod
Container
es-data-rc.yaml
metadata: labels: name: es-data component: elasticsearch role: data visualize: "true" spec: serviceAccount: elasticsearch containers: - name: es-data securityContext: capabilities: add: - IPC_LOCK image: quay.io/pires/docker-elasticsearch-kubernetes:1.7.2 env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace
… ports: - containerPort: 9300 name: transport protocol: TCP volumeMounts: - mountPath: /data name: storage
volumes: - name: storage
persistentVolumeClaim: claimName: elasticsearch-storage-claim
kind: PersistentVolumeClaimapiVersion: v1metadata: name: elasticsearch-storage-claimspec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi
es-claim.yaml
Pod
Container
Minimal OS
Docker
Virtual MachineHardware
kubelet
proxy
Node
Pod
Container
kind: PersistentVolumeClaimapiVersion: v1metadata: name: elasticsearch-storage-claimspec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi
es-claim.yaml
apiVersion: "v1"kind: "PersistentVolume"metadata: name: “ebs-volume-1“ spec: capacity: storage: "100Gi" accessModes: - "ReadWriteOnce" awsElasticBlockStore: fsType: "ext4" volumeID: "vol-f331a05cc"
ebs-volume-1.yaml
/data
scheduler
Amazon EBS
serviceapiVersion: v1kind: Servicemetadata: name: elasticsearch labels: component: elasticsearch role: client visualize: "true"spec: type: LoadBalancer selector: component: elasticsearch role: client ports: - name: rest port: 9200 protocol: TCP - name: transport port: 9300 protocol: TCP
es-svc.yaml
Minimal OS
Docker
Virtual MachineHardware
kubelet
Pod
Container
proxy
Node
(replication controller, etc.)
Minimal OS
Docker
Virtual MachineHardware
kubelet
proxy
NodeMaster
scheduler
controller manager(replication controller, etc.)
API endpoint
distributed configurationstorage
kubectl(client command line tool)
Pod
Container
• cluster topology design • operational cluster management • cluster upgrade and maintenance
• application • declarative resource specification
DevOps• container image standardisation • deployment processes
Elastic LoadBalancer
Node
service
Minimal OS
Docker
Virtual MachineHardware
kubelet
proxy
Master
scheduler
controller manager(replication controller, etc.)
API endpoint
distributed configurationstorage
kubectl(client command line tool)
Pod
Container
ServerGroup
Open Source Commercially supported
kubernetes, what else?
Docker Swarm
Minimal OS
Docker
Virtual MachineHardware
Manager Node
Containerswarm
manager
swarmagent
swarmagent
distributed configurationstorage
docker remote API
overlay network
Amazon ECS
Docker Task
Container Instance
Amazon ECS
Container
ECS Agent
ELB
Internet
ELB
User / Scheduler
API
Cluster Management Engine
Task Container
Docker Task
Container Instance
Container
ECS Agent
Task Container
Docker Task
Container Instance
Container
ECS Agent
Task Container
AZ 1 AZ 2
Key/Value Store
Agent Communication Service
* slide from Deepak Singh, General Manager, Amazon EC2 Container Service
Slave
Master
framework
Virtual Pool of Resources (CPU, RAM, …)
• Set up in seconds, integrates in the AWS stack (ELB, CloudWatch, ECR) and yes, specific to Amazon WebServices
• Basic capabilities out of the box (no Discovery, …) but extensible • Ideal for simple containerised workloads
• Opinionated declarative cluster management solution • Runs on a single machine (in Docker) and on a large datacenter setup • Rich API for cluster management • Support for Secrets, Quota, Volumes, … • Provider agnostic (docker, rkt, amazon, gce, …)• Abstracts a cluster behind the Docker Remote API • Networking and scheduling support • Open plugin points for discovery, networking, … • Docker only • Lacks (being out of the box) service load balancing support• Essentially a resource abstraction • Scales to +10K nodes • Job agnostic (more than containers alone) • Support Kubernetes as one of the many frameworks
+
Container
declarative specification of operational needs
Cluster
continuous deployment pipeline includes quality gates: automated tests ops compliance
security tests manual steps
…
Patterns for Continuous Deployment
Container Image
preference-service-artefact-765
Blue Green
Content Based Router
Blue/Green deployments
Container Image
preference-service-artefact-765
Container Image
preference-service-artefact-123
production traffictest traffic
Container Image
preference-service-artefact-765
Stage 1 Stage 2 Stage 3
Content Based Router
Canary staged deployment
2 Container ecosystem is maturing rapidly and cAAS provides the abstraction sweet spot
1 Cloud services are a true enabler for business agility
3 Leverage Continuous Deployment as competitive advantage (as long as it lasts)
ACA IT-SOLUTIONS | © 2015
Stijn Wijndaele
[email protected] @stijnwijndaele
Questions?
Your Hosts For Today
Business Development Manager Cloud & Mobile Solutions
ACA IT-SOLUTIONS | © 2016
Stijn Van den Enden
[email protected] @stieno
CTO