@RossKukulinski
Building a SaaS with CoreOS and etcd
CoreOS Fest May 4, 2015
@RossKukulinski
Ross Kukulinski Yodlr Founder / CEO
BayNode Co-Organizer
iojs/nodejs Evangelist
Soccer Fanatic
@RossKukulinski
The internal tool that wasn’t internal anymore
@RossKukulinski
Our Goals• Reduce application complexity
• A/B test multiple versions of apps
• Consistent app from dev → test → staging → prod
• Scalable
• Fault tolerant
• Minimize time spent doing ‘devops’
@RossKukulinski
Docker
@RossKukulinski
Our Goals• Reduce application complexity
• A/B test multiple versions of apps
• Consistent app from dev → test → staging → prod
• Scalable
• Fault tolerant
• Minimize time spent doing ‘devops’
@RossKukulinski
How do you ship Docker containers?
@RossKukulinski
Linux for Massive Server Deployments
@RossKukulinski
Goals• Reduce application complexity
• Run multiple versions of the same app
• Consistent app from dev → test → staging → prod
• Scalable
• Fault tolerant
• Minimize time spent doing ‘devops’
@RossKukulinski
After further deliberation, we’re going big and switching to CoreOS/Docker entirely. We feel its the right direction long-term for us and application development in general.
- Me, internal memo September 2014
Lightbulb! by Matthew WynnMonitor photo by Don Fulano
@RossKukulinski
1. etcd is a databaseSo treat it like one
@RossKukulinski
2. etcd is a databaseFor your real-time infrastructure state
@RossKukulinski
Service Availability
REST API microservice Sidekick
Sidekick Pattern: 1. Get ip:port of microservice (docker inspect) 2. Health checks (containerized integration tests) 3. Publish service instance details to etcd
etcd1 & 2 3
@RossKukulinski
Service Discovery
haproxy confd
Confd: 1. Query etcd for ip:port of upstream services 2. Update configuration from template 3. Reload load balancer process
etcd
github.com/kelseyhightower/confd
2 & 3 1
@RossKukulinski
Dynamic external services
DNS & Cloud Load Balancers
lb-config
lb-config: 1. Query etcd for domain/ip:port of internal lbs 2. Update DNS & external load balancers
etcd12
github.com/yodlr/rax-lb-config
@RossKukulinski
3.
@RossKukulinski
3. etcd is a databaseWith super important data: Protect it!
PSA: $private_ipv4 is probably not what you think it is
@RossKukulinski
4. Sometimes you just need a GUI(Docker & CoreOS do have awesome CLI tools though)
github.com/yodlr/CoreGI
Also check out purpleworks/fleet-ui
@RossKukulinski
5. Fleet unit templates aren’t enough
[email protected] rest-api@{1..3}.service —> 3 instances of the service
@RossKukulinski
rest-api-{{env}}-{{tag}}@.service
github.com/yodlr/fleet-templater
@RossKukulinski
App deploys before
Kubernetes
@RossKukulinski
Rolling Deploys as a Service
github.com/yodlr/fleet-rolling-deploys
@RossKukulinski
Closing thoughts• Containers are awesome
• We <3 CoreOS
• You can get really far with fleet & etcd
• Kubernetes is so close to primetime
• Remember: etcd is a database, treat it like one