samza tech talk_2015 - strata

Stream Processing @Scale in LinkedIn

Yi PanData Infrastructure

Samza Team @LinkedIn

Databus

• What is Stream Processing?• What is Samza?• Stream Processing @LinkedIn• Upcoming features

Overview

• What’s stream processing– Input: an unbounded sequence of events

• E.g. web server logs, user activity tracking events, database changelogs, etc.

– Latency: near real-time• From milliseconds to minutes, instead of hours to

days– Output: an unbounded sequence of changes to

the derived dataset• The derived dataset is usually the final or partial

analytic results that can either be in another stream, or a serving data store

Stream Processing

Response latency

Stream

Processing

Milliseconds to minutes

RPC

Synchronous Later. Possibly much later.

0 ms

Stream Processing

• What are the application requirements?– Scalable, fast, stateful stream processing– What scale should we operate at?

• Traffic Volume: 1.4 Trillion events/day• Intermediate State Size: multi TB / colo (*)

– Why is it expensive to run stream processing at scale?

• Intermediate data set needs to be stored to allow low latency processing

• Large volume of data needs to be pulled and pushed via network

Stream Processing

• What is Stream Processing?• What is Samza?• Stream Processing @LinkedIn• Upcoming features

Overview

• Samza is a distributed Turing machine– Single Task Samza Job is a stateful

Turing machine

What’s Samza

Samza TaskInput stream Output stream

Statechangelogch

eckp

oint

– Scaling a Samza job: partition the streams

What’s SamzaIn

put s

trea

m A

partition 0

partition 1

partition 2

partition 3

partition n

– Scaling a Samza job: partition the streams

What’s SamzaIn

put s

trea

m B

partition 0

partition 1

partition 2

partition 3

partition n

– Scaling a Samza job: replicating the state machine

What’s Samza

shared checkpoint

Job

• Samza Execution in Yarn

What’s Samza

Host 1 Host 2 Host 3

Application Master

Samza container Samza container

Samza container

Deploy Samza job

• States in Samza– Checkpoints

• Offsets per input stream partitions– State Stores

• In-memory or on-disk (RocksDB) derived data set

What’s Samza

Samza TaskOutput stream partitions

State changelog partitionsch

eckp

oint

Host 1

• States in Samza– Checkpoints and local state stores are backed

by distributed logs

What’s Samza

Samza TaskOutput stream partitions

State changelog partitionsch

eckp

oint

Host 1

• What is Stream Processing?• What is Samza?• Stream Processing @LinkedIn• Upcoming features

Overview

Stream Processing @ LinkedIn

WebServersWebServers

WebServersWebServers

WebServersWebServers

WebServersMonitorServers

Oracle

Espresso

Kafka Databus

Trackingevents

Metrics

changelog

changelog

Samza JobsSamza

JobsSamza JobsSamza

Jobs

bootstrap

bootstrap

VoldemortDerivedData

DerivedData

Stream Processing @ LinkedIn

• Tracking aggregate/analysis (ACG)

Stream Processing @ LinkedIn

• Content standardization w/ adjunct data setMember

Profile DBBootstrap

JobDatabus

Kafka

Content Standardization

Kafka

Kafka

Stream Processing @ LinkedIn

• Kafka Deployment– 1.1 Trillion messages / day

• Databus Deployment– 300 Billion messages / day

• Samza Deployment– multiple colos– 10+ Yarn clusters– 200+ nodes– 100+ Jobs in production

• What is Stream Processing?• What’s Samza• Stream Processing @LinkedIn• Upcoming features

Overview

• New features– Local state store improvements

• RocksDB TTL support• Fast recovery

– Dynamic configuration– Easier deployment w/ standalone jobs– High-level query language for faster

development

Upcoming Features

Contact Us / Get Involved• Open Source

–Documentation: samza.apache.org–Mailing list:

dev@samza.apache.org– JIRA: https

://issues.apache.org/jira/browse/SAMZA