Fits Docker into Devops

Evans Ye2014.10.6

#TrendInsight

Let’s discuss about Devops first

PRE-Devops• Developer builds and tests on dev servers• Devloper hands off code to operation• likely deployed manually• Sometimes it broken in Production• Need to be fixed, Now

4

Devops• Getting changes to users quickly, reliably, and

securely• Quick release cycle from code to Production• More confidence due to automated testing• reproducibility

– bugs are not always 100% reproducible

6

Fast Iterationbacked by automated testing

"Automated tests transform fear into boredom." -- Eran Messeri, Google

Google Devops (2013)• 75 million test cases are run daily

Continuous Integration• Same environment in Dev, QA and Production• Auto deploy

– deploy reliability and consistency

• Automated tests– unit test– smoke test (integration test)

How do you test hadoop features?• Manually test on dev cluster

– Hey folks, don’t move, I’m testing my own features.

• Pig unit– adhoc-query? dev environment only

• HBase mini-cluster– does not support Windows?

• Still not exactly the same environment as Production• what if written in python, ruby, php?

10

What if we can do Dev and Test in the same way, efficiently?

Docker• lightweight, portable Linux containers (like VMs)

– fast creation– system isolated– resource constrained

• Build, ship, run containers everywhere– containers as dev and test environment

Containers offer faster automation

#TrendInsight

Introduction to Docker

15

The Docker Engine runs containers

Architecture

VMDocker

17

The Docker Hub stores and ships

images

Docker image• $ docker runs <IMAGE>

to create a container• base image stores /bin,

/etc, /usr (Linux distro.)• add you app(diff) on top

of base image• commit containers• push, pull images

Docker offers you• Deploy efficiently• Quick start• Lightweight

Limitation of Docker• Any Linux with kernel > 3.8.x• Linux only (Linux container, after all)

Linux

21

What about Windows and Macbook…?

Vagrant• A open source command line VM(or IaaS)

provisioning toolProviders

Vagrantfile

VM

Solution for OS X / Windows

OS X / Windows with vagrant and

virtualbox installed

boot2docker(2XMB, tiny core Linux)

ContainerContaine

r

$ vagrant up DONE!

docker platform auto provisioned by vagrant

docker containers

25

Hadoop Integration

04/09/2023

Develop and test hadoop related features

service(hadoop-client)

data(hadoop-client)

hadoop fs –put …

link

#TrendInsight

Q & A