finagle linkerd and apache mesos twitter style microservices at scale
TRANSCRIPT
![Page 1: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/1.jpg)
Finagle, linkerd, and MesosTwitter-style microservices at scaleoliver gouldcto, buoyant
ApacheCon North America, May 2016
from
![Page 2: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/2.jpg)
oliver gould • cto @ buoyantopen-source microservice infrastructure
• previously, tech lead @ twitter:observability, traffic
• core contributor: finagle
• creator: linkerd
• loves: dogs@olix0r [email protected]
![Page 3: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/3.jpg)
overview• 2010: A Failwhale Odyssey
• Automating the Datacenter
• Microservices: A Silver Bullet
• Finagle: The Once and Future Layer 5
• Introducing linkerd
• Demo
• Q&A
![Page 4: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/4.jpg)
2010A FAILWHALE ODYSSEY
![Page 5: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/5.jpg)
Twitter, 2010107 users
107 tweets/day
102 engineers
101 services
101 deploys/week
102 hosts
0 datacenters
101 user-facing outages/weekhttps://blog.twitter.com/2010/measuring-tweets
![Page 6: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/6.jpg)
![Page 7: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/7.jpg)
![Page 8: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/8.jpg)
The Monorail, 2010
103 of RPS
102 of RPS/host
101 of RPS/process
hardware lb
the monorail
mysql memcache kestrel
![Page 9: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/9.jpg)
Problems with the MonorailRuby performance
MySQL scaling
Memcache operability
Deploys
![Page 10: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/10.jpg)
Events
https://blog.twitter.com/2013/new-tweets-per-second-record-and-how
![Page 11: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/11.jpg)
Asymmetry
Photo by Troy Holden
![Page 12: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/12.jpg)
Provisioning
![Page 13: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/13.jpg)
automating the datacenter
![Page 14: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/14.jpg)
mesos.apache.orgUC Berkeley, 2010
Twitter, 2011
Apache, 2012
Abstracts compute resources
Promise: don’t worry about the hosts
![Page 15: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/15.jpg)
aurora.apache.orgTwitter, 2011
Apache, 2013
Schedules processes on Mesos
Promise: no more puppet, monit, etc
![Page 16: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/16.jpg)
timelines
Aurora (or Marathon, or …)
host
Mesos
host host host host host
users notifications
x800 x300 x1000
![Page 17: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/17.jpg)
microservicesA SILVER BULLET
![Page 18: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/18.jpg)
scaling teams
growing software
![Page 19: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/19.jpg)
flexibility
![Page 20: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/20.jpg)
performance correctness monitoring debugging efficiency security
resilience
![Page 21: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/21.jpg)
not a silver bullet.(sorry.)
![Page 22: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/22.jpg)
Resilience is an imperative: our software runs on the truly dismal computers we call datacenters. Besides being heinouslycomplex… they are unreliable and prone to operator error.
Marius Eriksen @mariusRPC Redux
![Page 23: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/23.jpg)
resilience in microservicessoftware you didn’t write
hardware you can’t touch
network you can’t configure
break in new and surprising ways
and your customers shouldn’t notice
![Page 24: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/24.jpg)
resilient microservices means
resilient communication
![Page 25: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/25.jpg)
datacenter
[1] physical
[2] link
[3] network
[4] transport aurora, marathon, … mesos canal, weave, …
aws, azure, digitalocean, gce, …
business languages, libraries[7] application
rpc[5] session
[6] presentation json, protobuf, thrift, …
http/2, mux, …
![Page 26: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/26.jpg)
layer 5 dispatches requests onto layer 4 connections
![Page 27: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/27.jpg)
finagleTHE ONCE AND FUTURE LAYER 5
![Page 28: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/28.jpg)
github.com/twitter/finagleRPC library (JVM)
asynchronous
built on Netty
scala
functional
strongly typed
first commit: Oct 2010
![Page 29: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/29.jpg)
used by…
![Page 30: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/30.jpg)
programming finagleval users = Thrift.newIface[UserSvc](“/s/users”) val timelines = Thrift.newIface[TimelineSvc](“/s/timeline”)
Http.serve(“:8080”, Service.mk[Request, Response] { req => for { user <- users.get(userReq(req)) timeline <- timelines.get(user) } yield renderHTML(user, timeline) })
![Page 31: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/31.jpg)
operating finagletransport security
service discovery
circuit breaking
backpressure
deadlines
retries
tracing
metrics
keep-alive
multiplexing
load balancing
per-request routing
service-level objectives
Obs
erve
Sess
ion
timeo
ut
Ret
ries
Req
uest
dra
inin
g
Load balancer
Mon
itor
Obs
erve
Trac
e
Failu
re a
ccru
al
Req
uest
tim
eout
Pool
Fail
fast
Expiration Dispatcher
![Page 32: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/32.jpg)
“It’s slow”is the hardest problem you’ll ever debug.
Jeff Hodges @jmhodgesNotes on Distributed Systems for Young Bloods
![Page 33: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/33.jpg)
the more components you deploy, the more problems you have
![Page 34: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/34.jpg)
the more components you deploy, the more problems you have
😩
![Page 35: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/35.jpg)
the more components you deploy, the more problems you have
😩
😩
😩
😩
😩
😩
![Page 36: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/36.jpg)
lb algorithms: • round-robin • fewest connections • queue depth • exponentially-weighted
moving average (ewma) • aperture
load balancing at layer 5
![Page 37: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/37.jpg)
timeouts & retries
timelines
users
web
db
timeout=400ms retries=3
timeout=400ms retries=2
timeout=200ms retries=3
timelines
users
web
db
![Page 38: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/38.jpg)
deadlines
timelines
users
web
db
timeout=400ms
deadline=223ms
deadline=10ms
177ms elapsed
213ms elapsed
![Page 39: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/39.jpg)
retry budget
typical:
retries=3 worst-case: 300% more load!!!
better: retryBudget=20% worst-case: 20% more load
![Page 40: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/40.jpg)
tracing
![Page 41: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/41.jpg)
tracing
![Page 42: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/42.jpg)
😎
tracing
![Page 43: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/43.jpg)
layer 5 routing
![Page 44: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/44.jpg)
layer 5 routingapplications refer to logical names
requests are bound to concrete names
delegations express routing
/s/users
/io.l5d.zk/prod/users
/s => /io.l5d.zk/prod/http
![Page 45: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/45.jpg)
per-request routing: staging
GET / HTTP/1.1 Host: mysite.comDtab-local: /s/B => /s/B2
![Page 46: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/46.jpg)
per-request routing: debug proxy
GET / HTTP/1.1Host: mysite.comDtab-local: /s/E => /s/P/s/E
![Page 47: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/47.jpg)
so all i have to do is rewrite my app in scala?
![Page 48: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/48.jpg)
linkerd
![Page 49: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/49.jpg)
github.com/buoyantio/linkerdmicroservice rpc proxy
layer-5 router
aka l5d
built on finagle & netty
pluggable
http, thrift, …
etcd, consul, kubernetes, marathon, zookeeper, …
…
![Page 50: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/50.jpg)
magic resiliency sprinklestransport security
service discovery
circuit breaking
backpressure
deadlines
retries
tracing
metrics
keep-alive
multiplexing
load balancing
per-request routing
service-level objectives
Service Binstance
linkerd
Service Cinstance
linkerd
Service Ainstance
linkerd
![Page 51: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/51.jpg)
namerdreleased in March
centralized routing policy
delegates logical names to service discovery
pluggable
etcd
kubernetes
zookeeper
…
![Page 52: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/52.jpg)
namerd
![Page 53: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/53.jpg)
demo: gob’s microservice
![Page 54: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/54.jpg)
web
wordgen
l5d
l5dl5d
![Page 55: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/55.jpg)
web
wordgen
gen-v2
l5d
l5dl5d
l5d
![Page 56: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/56.jpg)
web
wordgen
gen-v2
l5d
l5dl5d
l5d
namerd
![Page 57: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/57.jpg)
master
dc/os marathon zookeeper
node nodepublic node node
…
ELB ELB
![Page 58: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/58.jpg)
master
dc/os marathon zookeeper
node nodepublic node node
…
linkerd linkerd linkerd linkerd
ELB ELB
namerd
![Page 59: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/59.jpg)
master
dc/os marathon zookeeper
node nodepublic node node
…
linkerd linkerd linkerd linkerd
ELB ELB
namerd
web (x1) gen (x3)
word (x3)
word-growthhack (x3)
![Page 60: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/60.jpg)
github.com/buoyantio/linkerd-examples
![Page 61: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/61.jpg)
linkerd roadmap• Netty4.1
• HTTP/2+gRPC linkerd#174 • TLS client certs • Richer routing policies • Announcers • More configurable everything
![Page 62: Finagle linkerd and Apache Mesos Twitter style microservices at scale](https://reader035.vdocuments.site/reader035/viewer/2022062317/58a1aabd1a28abf0458ba239/html5/thumbnails/62.jpg)
more at linkerd.io
slack: slack.linkerd.io
email: [email protected]
twitter:
• @olix0r
• @linkerd
thanks!