unified batch and stream processing with flink @ big data beers berlin may 2015

Robert MetzgerFlink committer

@rmetzger_

ApacheFlink

1 year of Flink - code

April 2014 April 2015

Community growth

3

Aug-10 Feb-11 Sep-11 Apr-12 Oct-12 May-13 Nov-13 Jun-14 Dec-14 Jul-150

20

40

60

80

100

120

#unique contributors by git commits

What is Flink?

4

Gelly

Table

ML

SA

MO

A

DataSet (Java/Scala/Python)DataStream (Java/Scala)

Hadoop M

/R

Local Remote Yarn Tez Embedded

Data

flow

Data

flow

(W

iP)

MR

QL

Table

Casc

adin

g

(WiP

)

Streaming dataflow runtime

Program compilation

5

case class Path (from: Long, to: Long)val tc = edges.iterate(10) { paths: DataSet[Path] => val next = paths .join(edges) .where("to") .equalTo("from") { (path, edge) => Path(path.from, edge.to) } .union(paths) .distinct() next }

Optimizer

Type extraction

stack

Task schedulin

g

Dataflow metadata

Pre-flight (Client)

MasterWorkers

DataSource

orders.tbl

Filter

MapDataSour

celineitem.tbl

JoinHybrid Hash

buildHT

probe

hash-part [0] hash-part [0]

GroupRed

sort

forward

Program

DataflowGraph

deployoperators

trackintermediate

results

Native workload support

6

Flink

Streaming topologies

Long batchpipelines

Machine Learning at scale

How can an engine natively support all these workloads?

And what does "native" mean?

Graph Analysis

E.g.: Non-native iterations

7

Step Step Step Step Step

Client

for (int i = 0; i < maxIterations; i++) {// Execute MapReduce job

}

E.g.: Non-native streaming

8

streamdiscretizer

Job Job Job Jobwhile (true) { // get next few records // issue batch job}

Native workload support

9

Flink

Streaming topologies

Heavy batch jobs

Machine Learning at scale

How can an engine natively support all these workloads?

And what does native mean?

Flink Engine

1. Execute everything as streams

2. Allow some iterative (cyclic) dataflows

3. Allow some mutable state

4. Operate on managed memory

10

Flink by Use Case

11

Data Streaming Analysisstreaming dataflows

12

3 Parts of a Streaming Infrastructure

13

Gathering

Broker Analysis

Sensors

Transactionlogs …

Server Logs

3 Parts of a Streaming Infrastructure

14

Gathering

Broker Analysis

Sensors

Transactionlogs …

Server Logs

Result may be fed back to the broker

Cornerstones of Flink Streaming

Pipelined stream processor (low latency)

Expressive APIs

Flexible operator state, streaming windows

Efficient fault tolerance for streams and state.

15

Pipelined stream processor

16

StreamingShuffle!

17

Expressive APIs

case class Word (word: String, frequency: Int)

val lines: DataStream[String] = env.fromSocketStream(...)

lines.flatMap {line => line.split(" ") .map(word => Word(word,1))} .window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS)) .groupBy("word").sum("frequency") .print()

val lines: DataSet[String] = env.readTextFile(...)

lines.flatMap {line => line.split(" ") .map(word => Word(word,1))} .groupBy("word").sum("frequency") .print()

DataSet API (batch):

DataStream API (streaming):

Checkpointing / Recovery

18

Chandy-Lamport Algorithm for consistent asynchronous distributed snapshots

Pushes checkpoint barriersthrough the data flow

Operator checkpointstarting

Checkpoint done

Data Stream

barrier

Before barrier =part of the snapshot

After barrier =Not in snapshot

Checkpoint done

checkpoint in progress

(backup till next snapshot)

Long batch pipelinesBatch on Streaming

19

20

Batch Pipelines

Batch on Streaming

Batch programs are a special kind of streaming program

21

Infinite Streams Finite Streams

Stream Windows Global View

PipelinedData Exchange

Pipelined or Blocking Exchange

Streaming Programs Batch Programs

22

Batch Pipelines

Data exchange (shuffle / broadcast)is mostly streamed

Some operators block (e.g. sorts / hash tables)

23

Operators Execution Overlaps

24

Memory Management

Memory Management

25

Smooth out-of-core performance

26More at: http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html

Blue bars are in-memory, orange bars (partially) out-of-core

Table API

27

val customers = envreadCsvFile(…).as('id, 'mktSegment) .filter("mktSegment = AUTOMOBILE")

val orders = env.readCsvFile(…) .filter( o => dateFormat.parse(o.orderDate).before(date) ) .as("orderId, custId, orderDate, shipPrio")

val items = orders .join(customers).where("custId = id") .join(lineitems).where("orderId = id") .select("orderId, orderDate, shipPrio, extdPrice * (Literal(1.0f) – discount) as revenue")

val result = items .groupBy("orderId, orderDate, shipPrio") .select('orderId, revenue.sum, orderDate, shipPrio")

Machine Learning Algorithms

Iterative data flows

28

29

Iterate by looping

for/while loop in client submits one job per iteration step

Data reuse by caching in memory and/or disk

Step Step Step Step Step

Client

30

Iterate in the Dataflow

Example: Matrix Factorization

31

Factorizing a matrix with28 billion ratings forrecommendations

More at: http://data-artisans.com/computing-recommendations-with-flink.html

Graph AnalysisStateful Iterations

32

33

Iterate natively with state/deltas

Effect of delta iterations…#

of

ele

men

ts u

pd

ate

d

iteration

35

… fast graph analysis

More at: http://data-artisans.com/data-analysis-with-flink.html

Closing

36

37

Flink Roadmap for 2015

Some examples: More flexible state and state backends in

streaming

Master Failover

Improved monitoring

Integration with other Apache projects• SAMOA, Zeppelin, Ignite

More additions to the libraries

Flink Forward registration & call for abstracts is open now

flink.apache.org 38

• 12. and 13. October 2015• Kulturbrauerei Berlin• With Flink Workshops/Training!

39

flink.apache.org@ApacheFlink

41

42

43

Examples of optimization

Task chaining• Coalesce map/filter/etc tasks

Join optimizations• Broadcast/partition, build/probe side, hash or sort-

merge

Interesting properties• Re-use partitioning and sorting for later operations

Automatic caching• E.g., for iterations