Best Practices for Developing & Deploying Java Applications with DockerEric Smalling - Solution Architect, Docker Inc.@ericsmalling

JavaOne 2017 | CON7957

2

Who Am I?● Eric Smalling

○ Solution ArchitectDocker Customer Success Team

● ~25 years in software development, architecture, version control admin, etc…

● ~10 years in build & test automation

● Docker user since pre-1.0 days

● Java developer since 1.1.x days

Agenda

● Docker 101

● Running a simple Java web application in Docker

● Services, stacks & deploying to clusters

● Application management & troubleshooting

● Application Configuration

● Q & A

Docker 101A quick overview of Docker

Some Docker vocabulary

Docker Image

The basis of a Docker container. Represents a full application

Docker Container

The standard unit in which the application service resides and executes

Docker Engine

Creates, ships and runs Docker containers deployable on a physical or virtual, host

locally, in a datacenter or cloud service provider

Registry Service (Docker Hub or Docker Trusted Registry)

Cloud or server based storage and distribution service for your images

Docker File SystemImages, Layers & Containers

● Logical file system by grouping different file system primitives into branches (directories,

file systems, subvolumes, snapshots)

● Each branch represents a layer in a Docker image

● Allows images to be constructed / deconstructed as needed vs. a huge monolithic image

(ala traditional virtual machines)

● When a container is started a writeable layer is added to the “top” of the file system

Docker File SystemContainers & Copy on Write

● Super efficient:

Sub second instantiation times for containers

New container can take <1 Mb of space

● Containers appears to be a copy of the original image

But, it is really just a link to the original shared image

● If someone writes a change to the file system, a copy of the affected file/directory is

“copied up”

Docker File SystemWhat about data persistence?

● Volumes allow you to specify a directory in the container that exists outside of the docker

file system structure

● Can be used to share (and persist) data between containers

● Directory persists after the container is deleted

Unless you explicitly delete it

● Can be created in a Dockerfile or via CLI

Dockerfile - Linux + Java Example: Initial state

Image Layers

Kernel

Ubuntu Linux 16:04

Update apt catalogs

Install JDK and curl

Download Tomcat

Install Tomcat

Copy webapp

Start tomcat

Initial State

Building the image

The docker client command

“build” = build an image

“-t” = apply a name and optional build

Image name and optional tag

Path to build context and Dockerfile

Running the image in a container

The docker client command

“run” = start a container

“--rm” = delete container when it exits

“-t” = run with a tty (for console i/o)“-i” = run in interactive modeThese often are used in combination like this

Image name and optional tag

DemoBuild and run demonstration

Dockerfile - Linux + Java Example: Optimization step 1

Image LayersOptimization Step 1

Kernel

Ubuntu Linux 16:04

Update apt catalogs, install JDK and curl, clean up

Download Tomcat

Install Tomcat

Copy webapp

Start tomcat

Dockerfile - Linux + Java Example: Optimization step 2

Image LayersOptimization Step 2

Kernel

OpenJDK:8-alpine

Update apk catalogs, install curl

Download Tomcat

Install Tomcat

Copy webapp

Start tomcat

Dockerfile - Linux + Java Example: Fully Optimized

Image LayersFully optimized

Kernel

tomcat:8.5-alpine

Copy webapp

Deploying to ClustersServices, Stacks and Swarms

More terminology

● Swarm

○ A group of docker hosts, connected and running as a cluster

○ 1-n managers

○ 1-n workers

● Service

○ An application (or part of an application) that provides a specific function

(catalog lookup, web front end, payment processing)

● Stack

○ A way of representing multi-service applications

○ Made up of 1-n services

Stack deploy demo

Simple J2EE application deployment with 2 containers:

● React based front end

● Java based back end

Application ManagementMonitoring & Troubleshooting

Health ChecksHelping Docker help you

● HEALTHCHECK instruction in DockerFile

● Tells Docker how to test a container to check that it is still working

● New status added to container lists

● Adds “(healthy)” to Status column in a “docker ps response”

Health ChecksHelping Docker help you

● Examples:

○ HEALTHCHECK CMD curl --fail http://localhost || exit 1

○ HEALTHCHECK --interval=12s --timeout=12s --start-period=30s \

CMD node /healthcheck.js

● References:

○ Documentation: https://docs.docker.com/engine/reference/builder/#healthcheck

○ Elton Stoneman blog about not using curl/iwr: https://t.co/Zgdd1lyzhk

JVM MemoryTips and tricks

● Always explicitly specify JVM heap size with “-Xmx” arguments

○ By default, J2SE 5.0+ will use up to 25% of the host machine’s RAM or 1GB (whichever is smaller)

○ Container memory limits (enforced via cgroups) are ignored* (*cgroup awareness is planned for Java 9)

○ It’s just a good practice to specify it anyway

● Do use Docker cpu and memory reservations and limits to avoid over-subscribing your host machines

○ --memory

○ --memory-reservation

○ --cpus

○ etc…

● If limiting cpu, be sure to update GC Thread limiter in JVM

○ -XX:ParallelGCThreads

LoggingDealing with application logs

● Docker EE Reference Architecture document about this: http://dockr.ly/logging

● Do not output logs into the container’s RW layer

○ slow

○ have to exec or cp out of the container to see them

● Option 1: send logs to stdout (see logging drivers below)

○ Visible via “docker logs” command

○ Visible via Docker UCP web console

● Option 2: send logs to volume

○ Many use a centralized NAS/SAN volume for this

● Option 3: Docker logging drivers

Docker Log DriversLog drivers available (as of 9/4/17)Latest always available at: https://docs.docker.com/engine/admin/logging/overview/#supported-logging-drivers

Application Log DriversConsider the following when selecting application log drivers:

● syslog and splunk:

○ Good options if log data is highly sensitive since they can be configured to use TLS for

transporting logs.

● journald:

○ great for retaining the usage of docker logs as well as logging Docker daemon logs

○ allows for easier troubleshooting and log portability at the same time

○ logs write first locally, so that there is less reliance on logging infrastructure.

● awslogs or gcplogs:

○ Only if cluster exist solely on a single cloud provider

Application Log Drivers (continued)

Consider the following when selecting application log drivers:

● gelf and fluentd:

○ good choice if there's a NoSQL database somewhere in the environment where the logs can

be stored.

Again, see http://dockr.ly/logging for much more detail on logging.

TroubleshootingHow to use Java tools with container based JVMs

● JVM command line tools via docker exec

○ GC Stats: jstat --gcutil

○ Heap dumps/histograms: jmap

● Expose JMX ports for jconsole or other utilities

● Intelligent health checks

○ More than just “port 8080 is listening”

● Check third party monitoring tools for updated to be “container aware”

○ i.e. Licensing issues with older monitoring tools because each container appears as a new

host

● Also, docker specific commands/tools:

○ docker stats

○ ctop

Application ConfigurationManaging multi-environment config’s

Application ConfigurationDeploying to disparate environments with identical images

● Build artifacts are your Docker images, not .war files or similar

● Build images in CI, store in registry, deploy same images everywhere

● Patterns and tools to deal with configuration differences

○ Separate Stack yaml files

○ Docker secrets

○ Application configuration via volume mounts

○ Third party configuration tools such as Consul and/or Vault

■ consul-template

■ Joyent Containerpilot

■ Roll-your-own

Environment specific Stacks

● Different environment variable values

● Services that mock production endpoints

○ db

○ web service

prod.yml

dev.yml

Docker Secrets

● Stored encrypted in swam

● Exposed only to nodes that run services that need them

● Presented in container via RAM only tmpfs files

○ never persisted to disk in encrypted format

○ when container stops, secret is no longer present

● All communications between swam nodes via TLS, so secret never in the clear on the wire either

● Different secret values per environment using tags

● UCP can manage who/where secrets are available

Application configuration in volume mounts

● Use volumes that are only available in physical environment they apply to

● Contain environment-specific application configuration properties

● DO NOT store secrets in these (use Docker Secrets or other secure mechanism)

● You can bind mount files (doesn’t have to be full directory structures)

ResourcesSo much to talk about, so little time to do so!

ResourcesSo much to talk about, so little time to do so!

● Docker Resources: https://www.docker.com/products/resources

○ Logging Reference Architecture: http://dockr.ly/logging

○ Training: https://training.docker.com

■ Instructor led

■ Self paced with “Play With Docker”

○ Containerizing legacy applications?

■ https://docker.com/MTA

● SquareSpace Blog: Understanding Linux Container Scheduling (with JVMs)

https://engineering.squarespace.com/blog/2017/understanding-linux-container-scheduling

THANK YOU :)@ericsmalling