best practices for managing docker versions as presented at javaone 2017
TRANSCRIPT
@jbaruch #oraclecode
About me
»@jbaruch
»Developer Advocate @JFrog
»We might still have some t-shirts left!
@jbaruch #oraclecode
Poll Time!
»Heard about Docker
»Can do the tutorial
»PoCing, playing etc.
»Production, baby!
@jbaruch #oraclecode
»There are 2 hard problems in computer science:
cache invalidation,
naming things,
and off-by-1 errors.
@jbaruch #oraclecode
Why Do We Care about versions?
»Pipeline Automation/Orchestration
»Traceability/Communication
»Maintainability/Storage Management
@jbaruch #oraclecode
Those are old news,
we have been using versions for years
»Docker is different !!
»Versions are Tags
@jbaruch #oraclecode
Docker Manifest and Tags
ac-image:1.0 ac-image:1.0-beta ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
Application: SHA2: 3
Layers
Application: SHA2: 4
Tags
Manife
st
@jbaruch #oraclecode
So let’s talk about tags in detail
»Mutable and Dynamic in nature
»Example: my-image:5.0
»And Latest is not latest
»Confusion !!
@jbaruch #oraclecode
my-image:5.0
OS layer 1.0
Framework layer 2.0
Application layer 2.0
OS layer 1.1
Framework layer 2.1
Application layer 2.1
Yesterday Today
@jbaruch #oraclecode
Promotion
ac-image:1.0-
devac-image:1.0-qa
ac-image:1.0-
release
Manifest
sha256:252564..
OS: SHA2
Framework: SHA2
Application: SHA2: 3
L
a
y
e
r
s
T
a
g
s
@jbaruch #oraclecode
Traceability!
»We always know what this image is
»Trace it back to CI
»Trace it back to source
@jbaruch #oraclecode
Traceability: Classic Approaches
»Version Names based On Git Hashes
»Version Names based on Jenkins Builds
»Version Names based on the packaged software
version number
»Version names with a unique date
(the build timestamp)
@jbaruch #oraclecode
Static vs. Dynamic Tags
»Static Tags:⋄Assigned at image CREATION
⋄Reflect metadata that traces the tag to its build and/or contents
»Dynamic Tags:⋄Reflect an image you should use, the current known-good version
⋄Classic example is ubuntu:trusty
⋄Remember, ‘latest’ isn’t actually latest, just one tagged as such!
@jbaruch #oraclecode
Love and Hate the dynamic versions
Pros Cons
Versions express promotion state Consistency
No version – trash! Traceability
@jbaruch #oraclecode
Double Tag
»Push every image with a static tag
»Create a second, dynamic tag for the same
image as appropriate⋄Just a like a sym-link
⋄Tag name is not traceable by itself, because the “dynamic”
tag may have moved since client pulled.
⋄Docker Inspect, Search, Find image for traceability
»Cleanup static tags based on some policy
@jbaruch #oraclecode
Static and Dynamic Tags
ac-image:1.0-
2017-04-01-
111
ac-image:latest1-
betaac-image:latest1
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.0-
2017-04-03-
222
OS: SHA2
Framework: SHA2
Application: SHA2: 3
L
a
y
e
r
s Application: SHA2: 4
T
a
g
s
ac-
image:latest1
@jbaruch #oraclecode
Another problem!
»So, if we change tags during promotion, what
happens to not promoted images?
»They loose their version!!!
(WTF, what does that mean?)
@jbaruch #oraclecode
The unlucky ones
»In Java we call it SNAPSHOT-s.
»In Docker – images without versions.
»They are huge!
@jbaruch #oraclecode
Docker Distribution and DTR
»Checksum based storage ⋄Multiple tags with the same manifest does not use up additional disk
space
»Deleting a tag does NOT delete the layers from
storage⋄Actual Delete based on SHA2 reference ONLY
⋄Tag is effectively deleted if you re-push with a same tag
⋄Old Manifest remains, and still referenceable by SHA2
@jbaruch #oraclecode
Deleting a tag does NOT delete
the layers from storage
ac-image:1.0 ac-image:1.0-beta ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
Application: SHA2: 3
L
a
y
e
r
s Application: SHA2: 4
T
a
g
s
@jbaruch #oraclecode
Actual DELETE is based on SHA2
reference ONLY
ac-image:1.0 ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
Application: SHA2: 3
L
a
y
e
r
s Application: SHA2: 4
T
a
g
s
Delete via
SHA2
@jbaruch #oraclecode
Docker GC
»Required to delete layers with no manifests
»Required to clear up disk space
»Stop the world
@jbaruch #oraclecode
Docker Distribution, Best Practice
»Delete layers without tags up front.⋄HEAD Tag get SHA2 of manifest
⋄Delete SHA2
⋄After deleting reference, run GC to clean up
@jbaruch #oraclecode
Docker Distribution, Best Practice
»(Not OOB) To clean up a registry with manifests
that have no tag⋄Command used to find these layers:
comm -23 <(find . -type f -name "link" | grep
"_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/
'{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep
"_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g';
echo; done | sort) | wc –l
@jbaruch #oraclecode
JFrog Artifactory
»We delete any layer that is not referenced by a
tag immediately
»We delete manifests that is not referenced by a
tag
»We have an API to copy/move a docker image or
change its tag without using the docker client to
pull/push a second time
»Configure the max number of tags per image
@jbaruch #oraclecode
Built-in cleanup
ac-image:1.0 ac-image:1.0-qa
Manifest
sha256:252564..Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
Application: SHA2: 3
L
a
y
e
r
s Application: SHA2: 4
T
a
g
s
Delete via
SHA2
Manifest
sha256:462564..
ac-image:1.2
Application: SHA2: 5
Delete Tag
@jbaruch #oraclecode
Recap
»Dynamic versions are good for promotion
»Static versions are good for traceability
»Retagging should be done in the registry
»All unreferenced objects should be collected