Modern Container Orchestration
Kubernetes, CoreOS, and more
@coreoslinux@brandonphilips
Brandon PhilipsCTO, CoreOS
github.com/philips
Easy CoreOS+Kubernetes Setupvagrant, aws, bare metal, etc
coreos.com/kubernetes/docs/latest/
Demo Instructionsgithub.com/philips/hacks
2015-all-things-open
The smartest way to run your container infrastructure.
tectonic.com @tectonic
QUAYSecure hosting for private Docker repositories
quay.io @quayio
Why build CoreOS?
you as a sw engineer
your
with Ada.Text_IO;
procedure Hello_World is
use Ada.Text_IO;
begin
Put_Line("Hello, world!");
end;
#include <stdio.h>
int main()
{
printf("Hello, world!\n");
}
package main
import "fmt"
func main() {
fmt.Println("Hello, world!")
}
your containerimage
your /bin/java/opt/app.jar/lib/libc
your /bin/python/opt/app.py/lib/libc
your com.example.app
d474e8c57737625c
your d474e8c57737625c
Signed By: Alice
you as an ops engineer
your
com.example.webappx3
your
com.example.webappx3
your
???
com.example.webappx3
reduce API contracts
minimal
kernelsystemdrktsshdocker
pythonjavanginxmysqlopenssl
app
dis
tro
dist
ro d
istr
o di
stro
dis
tro
dist
ro
pythonjavanginxmysqlopenssl
app d
istr
o di
stro
dis
tro
dist
ro d
istr
o di
stro
kernelsystemdrktsshdocker
pythonopenssl-A app1
dis
tro
dist
ro d
istr
o di
stro
dis
tro
dist
ro
javaopenssl-B app2
javaopenssl-B app3
kernelsystemdrktsshdocker
CoreOS
container
dis
tro
dist
ro d
istr
o di
stro
dis
tro
dist
ro
container
container
updates
OS operations
automatic updates
automatic updates
atomic update with rollback
CoreOS Updates
machine configuration
OS operations
get into the cluster
machine config
[Service]
ExecStart=/usr/bin/kubelet --
api_servers=https://172.17.4.101 --
register-node=true --hostname-
override=172.17.4.201 --cluster_dns=10.
3.0.10 --cluster_domain=cluster.local
--tls-cert-file=worker.pem --tls-
private-key-file=worker-key.pem
[Service]
ExecStart=/usr/bin/kubelet --
api_servers=https://172.17.4.101 --
register-node=true --hostname-
override=172.17.4.201 --cluster_dns=10.
3.0.10 --cluster_domain=cluster.local
--tls-cert-file=worker.pem --tls-
private-key-file=worker-key.pem
[Service]
ExecStart=/usr/bin/kubelet --
api_servers=https://172.17.4.101 --
register-node=true --hostname-
override=172.17.4.201 --cluster_dns=10.
3.0.10 --cluster_domain=cluster.local
--tls-cert-file=worker.pem --tls-
private-key-file=worker-key.pem
[Service]
ExecStart=/usr/bin/kubelet --
api_servers=https://172.17.4.101 --
register-node=true --hostname-
override=172.17.4.201 --cluster_dns=10.
3.0.10 --cluster_domain=cluster.local
--tls-cert-file=worker.pem --tls-
private-key-file=worker-key.pem
distributed configuration
cluster operations
AvailableLeader
Follower
AvailableLeader
Follower
AvailableLeader
Follower
UnavailableLeader
Follower
AvailableLeader
Follower
AvailableLeader
Follower
Temporarily Unavailable
Leader
Follower
Available
Leader
Follower
Unavailable
Leader
Follower
what should run
cluster operations
k8s/mesos/etc scheduler
scheduling
getting work to servers
scheduling
$ scp app host:/opt$ ssh host systemd-run /opt/app
$ scp app host:/opt$ ssh host systemd-run /opt/app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy deploy:collector-app
$ fab lowest-loadaverage
$ fab lowest-loadaveragehost1
$ fab lowest-loadaveragehost1$ fab -H host1 deploy:job
You
Scheduler API
Scheduler
Machine(s)
while true { todo = diff(desState, curState) schedule(todo)}
while true { todo = diff(desState, curState) schedule(todo)}
while true { todo = diff(desState, curState) schedule(todo)}
while true { todo = diff(desState, curState) schedule(todo)}
$ kubectl run host-info--image=quay.io/philips/host-info--replicas=1
$ kubectl get podsPOD IPhost-info-97wt8 10.2.29.4
$ kubectl scale rc host-info--replicas=2
$ kubectl get podsPOD IPhost-info-97wt8 10.2.29.4host-info-f839d 10.2.29.8
podenv=prodapp=web
podenv=prodapp=web
podenv=prodapp=web
rc web-prodselect(env=prod,app=web)count=1
podenv=prodapp=web
podenv=prodapp=web
podenv=prodapp=web
rc web-prodselect(env=prod,app=web)count=1
podenv=prodapp=web
rc web-prodselect(env=prod,app=web)count=1
podenv=prodapp=web
rc web-prodselect(env=prod,app=web)count=5
podenv=prodapp=web
podenv=prodapp=web
podenv=prodapp=web
podenv=prodapp=web
podenv=prodapp=web
rc web-prodselect(env=prod,app=web)count=5
where is it running
cluster operations
dns, LBs, k8s labels
services
flexible service discovery
k8s labels
podenv=devapp=web
podenv=testapp=web
podenv=prodapp=web
podenv=devapp=web
podenv=testapp=web
podenv=prodapp=web
service test.example.comselect(env=dev,app=web)
service beta.example.comselect(env=test,app=web)
ORselect(env=prod,app=web)
service example.comselect(env=prod,app=web)
podenv=testapp=web
podenv=prodapp=web
podenv=prodapp=web
podenv=devapp=web
podenv=testapp=web
podenv=prodapp=web
service test.example.comselect(env=dev,app=web)
service beta.example.comselect(env=test,app=web)
ORselect(env=prod,app=web)
service example.comselect(env=prod,app=web)
podapp=foo,version=1
service foo.cluster.localselect(app=foo)
podapp=foo,version=1
podapp=foo,version=2
service foo.cluster.localselect(app=foo)
podapp=foo,version=1
podapp=foo,version=2
service foo.cluster.localselect(app=foo)
$ kubectl expose rc host-info --port=80 --target-port=5483 --type=NodePort
$ curl http://172.17.4.202:32430/
architecture in practice
cluster operations
workerkubelet
workerkubelet
workerkubelet
scheduler& API
workerkubelet
workerkubelet
workerkubelet
workerkubelet
workerkubelet
workerkubelet
scheduler& API
worker &API
works on 1 node too
Easy CoreOS+Kubernetes Setupvagrant, aws, bare metal, etc
coreos.com/kubernetes/docs/latest/
coreos.com/careers
work with us
@coreoslinux@tectonicstack
@brandonphilips
thank you
continuous delivery of the OS
- Linux Kernel API promise
- Containers are required
- Consistency of configuration
$ date -d "-674 days"Mon Jul 1 2013
Alpha Beta Stable
α β S
$ cd coreos/manifest
$ cd coreos/manifest$ git tag | wc -l
$ cd coreos/manifest$ git tag | wc -l329
$ uname -r3.8.0
$ uname -r4.0.0
$ init --versionsystemd 207
$ init --versionsystemd 207
$ init --versionsystemd 219
github.com/coreos/etcd
github.com/coreos/etcd
containernetworking
github.com/appc/cni
- Defining external plugins e.g. ipvlan, bridge, etc
- Used in rkt today for setting up network namespaces
- Collaborating with folks from Red Hat, Cisco, and others
192.168.1.10192.168.1.40
192.168.1.10192.168.1.40
10.0.0.3
10.0.0.8 10.0.1.10
10.0.1.20
192.168.1.10192.168.1.40
192.168.1.10192.168.1.40
10.0.0.0/24 10.0.1.0/24
routes to192.168.1.40
192.168.1.10192.168.1.40
10.0.0.0/24 10.0.1.0/24
192.168.1.40
10.0.1.0/24
192.168.1.10
routes to192.168.1.10