The Twitter StackFinagle, Ostrich, Zipkin,…

Dominik Gruber, @the_dom Vienna Scala User Group – Aug. 27, 2014

Dominik Gruber • @the_domTwitter Stack

Agenda• Introduction

• Prominent Adopters

• Finagle

• Ostrich

• Zipkin

• Iago

• Finatra

Dominik Gruber • @the_domTwitter Stack

Introduction

• Twitter started to move off Ruby on Rails in 2010

• Currently still on Scala 2.9

Dominik Gruber • @the_domTwitter Stack

Prominent Adopters• Box

• eBay

• Etsy

• Foursquare

• Gigya

• Nest

• Pinterest

• Rdio

• SoundCloud

• Tumblr

Dominik Gruber • @the_domTwitter Stack

Dominik Gruber • @the_domTwitter Stack

Finagle“Finagle is an extensible RPC system for the

JVM, used to construct high-concurrency servers. Finagle implements uniform client

and server APIs for several protocols, and is designed for high performance and

concurrency. Most of Finagle’s code is protocol agnostic, simplifying the

implementation of new protocols.”

Dominik Gruber • @the_domTwitter Stack

Finagle provides…• Connection pools

• Failure detection

• Failover strategies

• Load balancers

• Back-pressure techniques

Dominik Gruber • @the_domTwitter Stack

Finagle…

• publishes standard statistics, logs, and exception reports

• supports distributed tracing across protocols

• optionally uses ZooKeeper for cluster management

• supports common sharding strategies

Dominik Gruber • @the_domTwitter Stack

Modules (Examples)• finagle-http

• finagle-memcached

• finagle-mysql

• finagle-protobuf

• finagle-redis

• finagle-thrift

Dominik Gruber • @the_domTwitter Stack

Basic abstractions

• Future

• Service

• Filter

Dominik Gruber • @the_domTwitter Stack

Ostrich

Dominik Gruber • @the_domTwitter Stack

Ostrich• A stats collector & reporter

• Collect runtime statistics (counters, gauges, metrics, and labels)

• Report those statistics through a simple web interface or into log files

• Similar to Metrics https://github.com/dropwizard/metrics

Dominik Gruber • @the_domTwitter Stack

Dominik Gruber • @the_domTwitter Stack

Zipkin

“Zipkin is a distributed tracing system that helps us gather timing data for all the

disparate services at Twitter. It manages both the collection and lookup of this data through

a Collector and a Query service.”

Dominik Gruber • @the_domTwitter Stack

Zipkin

Dominik Gruber • @the_domTwitter Stack

Zipkin’s Architecture

Dominik Gruber • @the_domTwitter Stack

Zipkin

• “Scribe” to transport traces to Zipkin and Hadoop

• Data stored in either Cassandra, Redis, HBase, MySQL, PostgreSQL, SQLite, or H2

Dominik Gruber • @the_domTwitter Stack

Iago

Dominik Gruber • @the_domTwitter Stack

Iago

• A load generator that replays production or synthetic traffic

• Iago supports arbitrarily high rates of traffic via built-in support for creating cluster

Dominik Gruber • @the_domTwitter Stack

Protocols• HTTP

• Thrift

• Memcache

• Kestrel (A queueing system from Twitter)

• UDP

Dominik Gruber • @the_domTwitter Stack

Finatra

Dominik Gruber • @the_domTwitter Stack

Dominik Gruber • @the_domTwitter Stack

Features• Fully Async using Futures

• Mustache Templates

• Detailed Metrics (p99,p999, etc)

• Profile/Debug over HTTP

• SSL/TLS Support

• Deploys with a single jar

Dominik Gruber • @the_domTwitter Stack

There’s more…

•

• Summingbird

• https://github.com/twitter

Dominik Gruber • @the_domTwitter Stack

Q & A

Dominik Gruber • @the_domTwitter Stack

Source• http://blog.oskarsson.nu/post/40196324612/the-

twitter-stack https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rpc-system

• https://www.youtube.com/watch?v=whuT_-cFb5Ihttps://www.youtube.com/watch?v=QdER-Zl_YCM

• https://github.com/twitter https://github.com/finagle