docker - diving in

Diving InSee what happens when a Java developer

tries to do DevOps

Copper @ an Aussie Insurance Company@awilmore

Disclaimer

Me: DevOps n00b

I’m really just here to learn (but Stephen’s making me talk!)

Overview

1. How Docker was introduced to the Copper team

2. The Copper build agents - Doing it Wrong!

3. New service opportunity - Doing it Right!

4. The challenges ahead

A short detour…

What is Copper?

The Copper Team• We are our company’s rapid software delivery team

• We are cross-functional,“innovative early adopters of technology”

• Grails + AngularJS, plus many other technologies

• Provide a faster speed-to-market for customers

• Provide an alternative path to production for our business

Leo, Pete, Mike: DevOps expert consultants from Odecee

Their research project: Docker

Meanwhile…

Build Engineering Team had a problem:

Need more servers, for more Bamboo build agents!

So… In the beginning…

Docker-based Build Agents!

Solution…

Went from approx. 10 to approx. 50 build agents!

Some Copper devs were still not happy…

“Too much job queuing in Bamboo!”

So I forked the Bamboo build agents and created…

Grails-Dedicated Build Agents

• Manually built, manually managed

• 3.5 GB on disk (way too big)

• Rebuilds mean re-download-the-Internet

• Too much hands on

Docker: Doing it Wrong!

…and six months on…

Docker: Doing it Wrong!

• Containers simply do too much!

• Stateful and Stateless concerns mixed together

• Poor use of Dockerfiles and Image Layering

• No orchestration

First Problem

Containers Do Too Much

Containers Do Too Much

… which violates…

Single Responsibility Principle

Single Responsibility Principle

One-Concern-Per-Container Principle

Docker refers to it in places as the:

So, the build agents…

Grails Build Agent:

Grails

Maven

Gradle

Selenium

Firefox

NPM

NodeJS

Grunt

Ruby

Compass

Bamboo Remote Agent Service

Grails Build Agent:Grails

Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass


Doing Too Much





Java Build Agent:

Maven

Gradle


Selenium Build Agent:

SeleniumFirefox


Node Build Agent:NPM

NodeJS

Grunt

Ruby

Compass



NodeJS

Grunt

Ruby

Compass











NodeJS

Grunt

Ruby

Compass



NodeJS

Grunt

Ruby

Compass


Better


Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass



Maven

Gradle

SeleniumFirefox

NPM

NodeJS

Grunt

Ruby

Compass


Doing Too Much

Doing It Wrong

Mixing Stateful and Stateless Concerns

Doing It Wrong

The next problem…


Grails Build Agent:

Grails Cache


Grails Build Agent:

Grails Cache

Grails Build Agent:

Grails Cache

Grails Build Agent:

Grails Cache

Grails Build Agent:

Grails Cache

File System

Data Volume Data Volume

Better

Grails Build Agent:

Grails Cache

Grails Build Agent:

Grails Cache

Better


Grails Build Agent:

Grails Cache

Grails Build Agent:

Grails Cache

Not so good

Not so good

The next problem…

Poor use of Dockerfiles and Image Layering


https://registry.hub.docker.com/u/maxexcloo/postgresql/dockerfile/

Simple Dockerfile example:




Simple Dockerfile example:


The Grails Build Agent Project

Dockerfile

Shell ScriptShell Script

Puppet Stuff

Shell Script

Shell Script

Puppet Stuff Shell Script

Grails Build Agent:

Grails

Maven

Gradle

Selenium

Firefox

NPM

NodeJS

Grunt

Ruby

Compass


The Grails Build Agent Project

Dockerfile

Shell Script

Shell Script Puppet Stuff

Shell Script

Shell Script

Puppet Stuff Shell Script Lots of moving parts (ie. code)

for one image

Dockerfile

Base OS

Image Inheritance

Java7 Image

Grails Image

Dockerfile

Dockerfile

Dockerfile

Base OS

Better

Java7 Image

Grails Image

Dockerfile

Dockerfile

Dockerfile

Shell ScriptShell Script

Puppet Stuff

Shell Script

Shell Script

Puppet Stuff Shell Script

Grails Build Agent:

Grails

Maven

Gradle

Selenium

Firefox

NPM

NodeJS

Grunt

Ruby

Compass



Not so good

Not so good

The next problem…

No orchestration

No orchestration

[adam@laptop ~] ssh adam@build-server

adam@build-server:~ >> Last login: Tue Oct 28 22:13:27 2014 from 10.126.176.99

cd git/buildeng-puppet

adam@build-server:~/git/buildeng-puppet >> git checkout grails-agents-branch

adam@build-server:~/git/buildeng-puppet >> docker build -t=“agent-11” .Sending build context to Docker daemon Step 0 : FROM ubuntu:14.04 ---> e360b5673aef Step 1 : MAINTAINER Adam Wilmore "[email protected]" ---> Using cache ... ... ...(a long time later...)

When I need some new agents…

Too much hands on!

Too much hands on!

It would be better to have…… and more: http://www.mindmeister.com/389671722/docker-ecosystem

…you get the point.

But doing this…

[adam@laptop ~] ssh adam@build-server

adam@build-server:~ >> Last login: Tue Oct 28 22:13:27 2014 from 10.126.176.99

cd git/buildeng-puppet

adam@build-server:~/git/buildeng-puppet >> git checkout grails-agents-branch

adam@build-server:~/git/buildeng-puppet >> docker build -t=“agent-11” .Sending build context to Docker daemon Step 0 : FROM ubuntu:14.04 ---> e360b5673aef Step 1 : MAINTAINER Adam Wilmore "[email protected]" ---> Using cache ... ... ...(a long time later...)

Doing It Wrong

• Containers simply do too much!

• Stateful and Stateless concerns mixed together

• Poor use of Dockerfiles and Image Layering

• No orchestration

Doing It Wrong

Recap:

Next time, it’ll be perfect…

So how about

Doing It Right?

The Next Project

• Build new Artifactory* Service using Docker

• Find out a bit more about Docker in the process

• Take the learnings so far, build a bit, learn a bit more, repeat…

* Artifactory from JFrog is an artefact management tool, similar to Nexus

“Hang on a minute! How do you know that Docker is the right tool

for this type of service?”

Good question.

A short detour…

Why Copper Needs Docker

• Portable infrastructure - changing licensed platform

• Major infrastructure pressures within Copper - scaling, availability and limited resources

• Increasing demand for disposable infrastructure

• What the Copper team needs is…


The Copper PaaS!

And we need to start somewhere…

• Artifactory service comprised:

• Java app server (Jetty)

• Postgres backend

• Web-based file share

• Linux all the way down

• Performance and Availability are critical - can’t be worse than previous artefact manager

Project Requirements

Some Principles to Work With

• Use Puppet with Docker for “image creation”; and

• Favour Puppet over Docker for “doing work” *

• Stick to One Concern Per Container!

• Dive in and learn!

* Why? For reuse, “might move to EC2 one day”, already familiar with Puppet…?

“Hang on a minute! What about service discovery, orchestration, logging,

monitoring and backups?”

<crickets…>

So how did it turn out?

Container Structure

Image Inheritance

The Challenges Ahead

The State of Ops• How do we scale and load-balance for performance and

availability?

• How do we backup all critical data?

• How do we do upgrades?

• How do we keep operational configuration changes under source-control?

• Can we restore everything in the event of a disaster?

• What about service discovery, orchestration, monitoring, all that stuff to help with some of these questions?

The State of Dev• Have we correctly designed containers for extensibility?

• Are we using the right tools and libraries to build the containers?

• How do we version and release our code changes?

• Is our code set up for optimal reuse?

• How do we test our builds?

• How do we collaborate most effectively as developers writing infrastructure software?

One last detour…

…on the question about using tools properly…

Like Docker vs Puppet…

Is it…

(Puppet installs Docker)

Or is it…

(Docker installs Puppet)

The answer appears to be:

“It’s both, and it depends.”

The Copper PaaS

And so it appears there’s still lots of work to do

Which is great because…

Next time, it’ll be perfect!

Next time, it’ll be perfect!

Copper @ an Aussie Insurance Company@awilmore

docker - diving in

Technology