atlascamp 2015 docker continuous integration training

Continuous Integration with Docker and Bamboo

STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE

https://bitbucket.org/ssmith/atlascamp2015-docker-ci

© http://www.amigosdosbichos.org/

http://www.amigosdosbichos.org/

Warning:

Bad analogies coming up…

VMs vs Containers

Huffington Post

Virtual Machine

Virtual HW

BIOS/EFI

Guest FS / Kernel

Applications

Physical Hardware

Host FS / Kernel

VM Hypervisor

BIOS/EFI

Virtual HW

BIOS/EFI

Guest FS / Kernel

Applications

Containers / Docker

Physical Hardware

Host FS / Kernel

DockerEngine

BIOS/EFI

Guest FS

Application

Guest FS

Application Virtual HW

BIOS/EFI

Guest FS / Kernel

Applications

Physical Hardware

Host FS / Kernel

VM Hypervisor

BIOS/EFI

Virtual HW

BIOS/EFI

Guest FS / Kernel

Applications

Using features of the Linux kernel

Docker

libcontainer systemdnspawn

libvirt

cgroups

CPU accounting

Namespaces

Posix capabilities

btrfs

OverlayFS

LXC

Unix: chroot

Containers take this further…

Network Abstraction

Process Isolation

Filesystem Isolation

Resource Limits

Docker takes this to the next level…

Build Tools Packaging

Lifecycle Communication Dev (boot2docker)

Sharing

boot2docker

Virtual Hardware

Host FS / Kernel

BIOS/EFI

Guest FS

Application

Guest FS

Application

Quick Demo

Peter Kastner

Docker Concepts

Carlsberg

Docker containers are like OO classes…

InheritanceSourcefile

(Dockerfile) Compile

Dep. Injection Declarative Deps

FROM ubuntu:trusty

RUN apt-get update && \ apt-get install -y nginx && \ apt-get clean

EXPOSE 80 EXPOSE 443

CMD nginx -g daemon off;

Single inheritance

Add member

Override parent behaviour

Exposed linking points

Dockerfile ~= Sourcefile

Inheritance resolved at runtime

Ubuntu

Add nginx

Add run command

(sorta like JVM class assembly)

Union Filesystem - AUFS - BTRFS - Devicemapper - OverlayFS

‘run’ instantiates a new container

Ubuntu

Add nginx

Add run command

Writable

Immutable

(kinda like ‘new’)

Runtime layer

[ssmith] $ docker build -q --tag=mynginx docker/mynginx/ Sending build context to Docker daemon 12.72 MB Sending build context to Docker daemon Step 0 : FROM ubuntu:trusty ---> 8eaa4ff06b53 Step 1 : RUN apt-get update && apt-get install -y nginx && apt-get clea ---> Running in f05a4a034fe3 ---> 1e84ac75dd9a Removing intermediate container f05a4a034fe3 Step 2 : EXPOSE 80 ---> Running in 77602b8eb9ec ---> 4076f56233ff Removing intermediate container 77602b8eb9ec Step 3 : EXPOSE 443 ---> Running in b627b8c1d328 ---> f5a3ff140964 Removing intermediate container b627b8c1d328 Step 4 : CMD nginx -g daemon off; ---> Running in 132573fe4fc9 ---> c1df0408cf70 Removing intermediate container 132573fe4fc9

‘build’ compiles to binary

Inspecting the stack

[ssmith] $ docker history mynginx IMAGE CREATED CREATED BY 606f0f611cb2 6 days ago /bin/sh -c #(nop) CMD [nginx -g daemon off;] 27d0fad8620d 6 days ago /bin/sh -c #(nop) EXPOSE map[443/tcp:{}] dc26b50e9806 6 days ago /bin/sh -c #(nop) EXPOSE map[80/tcp:{}] 7a04b014a0a7 6 days ago /bin/sh -c apt-get update && apt-get install b39b81afc8ca 9 days ago /bin/sh -c #(nop) CMD [/bin/bash] 615c102e2290 9 days ago /bin/sh -c sed -i 's/^#\s*$deb.*universe$$/ 837339b91538 9 days ago /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic 53f858aaaf03 9 days ago /bin/sh -c #(nop) ADD file:ca5b63647c6b7a419e 511136ea3c5a 19 months ago

Linked Containers(dependency injection; sort-of…)

Docker host

postgresql client

--link postgresql:pg

can refer to hostname pg in commands

Volumes

Docker host

/var/volume1

DATA

/var/volume2

/var/volume1

client1

/var/volume2

--volumes-from DATA

(haven’t got a class analogy for this one)

Discussion

Testing with Docker

The example project

?

https://bitbucket.org/ssmith/atlascamp2015-docker-ci

The example project

Transfer

Trigger / Async

Data

Data

(See http://bit.do/postgres-es for details)

http://bit.do/postgres-es

How to test…

Vagrant? Dependencies everywhere? Docker?

Docker testing

Reuse images Startup speed Idempotent tests

Dev / Test match Deploy to Docker

Creating our images

Elasticsearch DockerfileFROM java:openjdk-7-jre

ENV ES_VER 1.3.4 ENV ES_URL https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ES_VER}.tar.gz

WORKDIR /opt/

RUN adduser --system elasticsearch

RUN curl ${ES_URL} -o elasticsearch.tgz && \ tar xzf elasticsearch.tgz && \ ln -s elasticsearch-${ES_VER} elasticsearch && \ mkdir /opt/elasticsearch/data/ && chown elasticsearch /opt/elasticsearch/data/ && \ mkdir /opt/elasticsearch/logs/ && chown elasticsearch /opt/elasticsearch/logs/ && \ rm elasticsearch.tgz

EXPOSE 9200 EXPOSE 9300

USER elasticsearch CMD /opt/elasticsearch/bin/elasticsearch -f

Elasticsearch build[ssmith] docker build -q --tag=elasticsearch docker/elasticsearch/ Sending build context to Docker daemon 2.56 kB Sending build context to Docker daemon Step 0 : FROM java:openjdk-7-jre ---> b797b3ba124d Step 1 : ENV ES_VER 1.3.4 ---> Running in 4842001c48d0 ---> 2717f23208b7 Removing intermediate container 4842001c48d0 Step 2 : ENV ES_URL https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ES_VER}.tar.gz ---> Running in b2f574153a18 ---> a292b608b854 Removing intermediate container b2f574153a18 Step 3 : WORKDIR /opt/ ---> Running in 0f138382e071 ---> aa2ebf2e8061 Removing intermediate container 0f138382e071 Step 4 : RUN adduser --system elasticsearch ---> Running in 0800878ae8c0 ---> 2b0c2619f328 Removing intermediate container 0800878ae8c0 Step 5 : RUN curl ${ES_URL} -o elasticsearch.tgz && tar xzf elasticsearch.tgz && ln -s elasticsearch-${ES_VER} elasticsearch && mkdir /opt/elasticsearch/data/ && chown elasticsearch /opt/elasticsearch/data/ && mkdir /opt/elasticsearch/logs/ && chown elasticsearch /opt/elasticsearch/logs/ && rm elasticsearch.tgz

Postgresql DockerfileFROM postgres:9.2

ENV POSTGRES_USER customer

COPY testschema.sql / COPY init-test-db.sh /docker-entrypoint-initdb.d/

Postgresql init file#!/bin/bash

echo "#### Create test DB ####"

gosu postgres postgres --single <<EOF CREATE USER customer PASSWORD 'customer'; CREATE DATABASE customer OWNER customer; GRANT ALL PRIVILEGES ON DATABASE customer TO customer; EOF

echo "#### Initialise test DB ####"

gosu postgres postgres --single customer -j < /testschema.sql

echo echo "#### Test DB initialised ####"

Postgresql build[ssmith] docker build -q --tag=postgres docker/postgres/ Sending build context to Docker daemon 5.12 kB Sending build context to Docker daemon Step 0 : FROM postgres:9.2 ---> 898ba6de4072 Step 1 : ENV POSTGRES_USER customer ---> Running in 932957e34d5e ---> 91b679d67fd9 Removing intermediate container 932957e34d5e Step 2 : COPY testschema.sql / ---> 65b67419fe54 Removing intermediate container 54298baaa088 Step 3 : COPY init-test-db.sh /docker-entrypoint-initdb.d/ ---> b6d84236c1f7 Removing intermediate container f9000db9da92 Successfully built b6d84236c1f7

Transfer DockerfileFROM java:openjdk-7-jre

# Install netcat for waitport RUN apt-get update && \ apt-get install -y netcat-openbsd && \ apt-get clean

COPY docker/waitport /usr/local/bin/

RUN curl https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -o /usr/local/bin/lein && \ chmod a+rx /usr/local/bin/lein

RUN adduser --disabled-password --gecos '' transfer ADD . /code RUN chown -R transfer.transfer /code

USER transfer WORKDIR /code

RUN lein deps

CMD waitport elasticsearch 9200 && \ waitport postgres 5432 && \ lein test-out junit

Running Tests Manually

Manual run[ssmith] docker run -d —name=postgres postgres 181470ed78836c9285664c18f0d5b4c6a9069d28c4f71b9d621d4c24762ad938

[ssmith] docker run -d --name=elasticsearch elasticsearch 5718e1ba9b7caf0e25f53d803a128d2a3d4d518ae8d747b843cc2cbdd78620ce

[smith] docker run --link postgres:postgres --link elasticsearch:elasticsearch transfer Waiting for TCP connection to elasticsearch:9200...OK Waiting for TCP connection to postgres:5432...OK Received notification for accountupdate for 55d3290b-5baa-4ca6-b353-cacf5b104c60 Performing update of 55d3290b-5baa-4ca6-b353-cacf5b104c60 Indexing row 55d3290b-5baa-4ca6-b353-cacf5b104c60 Mr. 55d3290b 55d3290b Esq. Dispatch complete

Manual cleanup[ssmith] docker ps CONTAINER ID IMAGE COMMAND CREATED … 5718e1ba9b7c elasticsearch:latest "/bin/sh -c '/opt/el 5 minutes ago … 181470ed7883 postgres:latest "/docker-entrypoint. 6 minutes ago …

[ssmith] docker kill elasticsearch postgres elasticsearch postgres

[ssmith] docker ps CONTAINER ID IMAGE COMMAND CREATED …

[ssmith] docker ps -a CONTAINER ID IMAGE COMMAND CREATED ff6e5409d8ba transfer:latest "/bin/sh -c 'waitpor 11 minutes ago 5718e1ba9b7c elasticsearch:latest "/bin/sh -c '/opt/el 11 minutes ago 181470ed7883 postgres:latest "/docker-entrypoint. 11 minutes ago

[smith] docker rm transfer postgres elasticsearch transfer postgres elasticsearch

Simplifying with Compose

feedyourskull.com

Compose automates Docker

Compose

Declarative Deps Fetch Images Build Images

Link Containers

Our docker-compose.ymltransfer: build: . links: - postgres - elasticsearch

postgres: build: docker/postgres

elasticsearch: build: docker/elasticsearch

Compose Run[ssmith] docker-compose up Creating devweek15code_postgres_1... Creating devweek15code_elasticsearch_1... Creating devweek15code_transfer_1... Attaching to devweek15code_postgres_1, devweek15code_elasticsearch_1, devweek15code_transfer_1 postgres_1 | The files belonging to this database system will be owned by user "postgres". postgres_1 | This user must also own the server process.

<SNIP> <SNIP> <SNIP> <SNIP> <SNIP>

transfer_1 | Waiting for TCP connection to postgres:5432...OK transfer_1 | Received notification for accountupdate for dcf286bf-7011-40f5-abb0-153d94acde transfer_1 | Performing update of dcf286bf-7011-40f5-abb0-153d94acde69 transfer_1 | Indexing row dcf286bf-7011-40f5-abb0-153d94acde69 Mr. dcf286bf dcf286bf Esq. elasticsearch_1 | creating index, cause [auto(index api)], shards [5]/[1], mappings [] elasticsearch_1 | update_mapping [account] (dynamic) transfer_1 | Dispatch complete devweek15code_transfer_1 exited with code 0 Gracefully stopping... (press Ctrl+C again to force) Stopping devweek15code_elasticsearch_1... Stopping devweek15code_postgres_1...

Compose Cleanup[ssmith] docker-compose ps Name Command State Ports -------------------------------------------------------------------------------- devweek15code_elasticsearch_1 /bin/sh -c /opt/elasticsea ... Exit -1 devweek15code_postgres_1 /docker-entrypoint.sh postgres Exit 0 devweek15code_transfer_1 /bin/sh -c waitport elasti ... Exit 0

[ssmith] docker-compose rm —force Going to remove devweek15code_transfer_1, devweek15code_elasticsearch_1, devweek15code_postgres_1 Removing devweek15code_postgres_1... Removing devweek15code_elasticsearch_1... Removing devweek15code_transfer_1…

[ssmith] docker-compose ps Name Command State Ports -------------------------------------------------------------------------------- [ssmith]

Discussion

Rafael Neff

Running With Bamboo

Plan

Stage

Job

Tasks QA

Staging

Prod

Artifacts

Deployment Environments

Git Repo

Bamboo Live

https://atlascamp2015.atlassian.net/

All New! Bamboo Docker Task!

AWS cloud agents with Docker

With Docker

Or create your own

Require Docker support

Bamboo will start on-demand

Building

Running

Linking

Compose with BambooSouth Florida Classical Review

No Compose? No problem…

Running Compose

Cleaning up

Extracting ResultsAtlantic Sentinel

Fetching container files[ssmith] docker-compose ps Name Command State Ports -------------------------------------------------------------------------------- devweek15code_elasticsearch_1 /bin/sh -c /opt/elasticsea ... Exit -1 devweek15code_postgres_1 /docker-entrypoint.sh postgres Exit 0 devweek15code_transfer_1 /bin/sh -c waitport elasti ... Exit 0

[ssmith] docker-compose ps -q transfer fd7e728e3e9a361d96c253f5aeadab1a3506538d1b0e19d27c82848d9bf48bf8

[ssmith] ID=`docker-compose ps -q transfer`

[ssmith] docker cp $ID:/code/testreports.xml .

[ssmith] file testreports.xml testreports.xml: XML document text

Fetching in Bamboo

Using JUnit Parser

Using JUnit results


Plan

Stage

Job

Tasks QA

Staging

Prod

Artifacts


Git Repo

Deployment EnvironmentsQA

Staging

Prod

Artifacts


Sharing Artifacts

Configuring the Environment

2

3

Useful Links

https://www.docker.com/subscribe_newsletter/

1 https://www.docker.com/

4 http://www.nkode.io/2014/08/24/valuable-docker-links.html

http://boot2docker.io/

https://www.docker.com/subscribe_newsletter/

https://www.docker.com/

http://www.nkode.io/2014/08/24/valuable-docker-links.html

http://boot2docker.io/

Thank you!

STEVE SMITH • DEVOPS ADVOCATE • ATLASSIAN • @TARKASTEVE

atlascamp 2015 docker continuous integration training

Technology

docker containers

binsh c apt

docker daemon step

docker build q

binsh c echo

y nginx apt

docker host postgresql

docker concepts carlsberg