apache flink - a stream processing engine

15
1 Aljoscha Krettek @aljoscha December 8, 2016 Google Singapore Apache Flink™ A Stream Processing Engine

Upload: aljoscha-krettek

Post on 21-Apr-2017

94 views

Category:

Software


2 download

TRANSCRIPT

Page 1: Apache Flink - A Stream Processing Engine

1

Aljoscha Krettek@aljoscha

December 8, 2016Google Singapore

Apache Flink™A Stream Processing Engine

Page 2: Apache Flink - A Stream Processing Engine

2

Original creators of Apache Flink®

Providers of the dA Platform, a supported

Flink distribution

Page 3: Apache Flink - A Stream Processing Engine

What is Apache Flink?

3

Distributed Deployment

Stateful Stream Processing

DataStream API, Windows

Table API

Building Block

Core API

Declarative DSL

Basis

Page 4: Apache Flink - A Stream Processing Engine

Stream Processing

4

Data Stream Data Stream

Your code

Page 5: Apache Flink - A Stream Processing Engine

Stateful Stream Processing

5

State

Data Stream Data Stream

Your code

Page 6: Apache Flink - A Stream Processing Engine

Stateful Steam Processing

6

Source Filter /Transform

Stateread/write Sink

Page 7: Apache Flink - A Stream Processing Engine

Stateful Steam Processing

7

Scalable embedded state Access at memory speed &scales with parallel operators

Page 8: Apache Flink - A Stream Processing Engine

Stateful Steam Processing

8

Re-load state

Reset positionsin input streams

Rolling back computationRe-processing

Page 9: Apache Flink - A Stream Processing Engine

Stateful Steam Processing

9

Restore to differentprograms

Bugfixes, Upgrades, A/B testing, etc

Page 10: Apache Flink - A Stream Processing Engine

Beam/Flink Interactions

10

Dataflow/MillWheel influenced

runs on

Page 11: Apache Flink - A Stream Processing Engine

Code Example!

11

Based on http://data-artisans.com/robust-stream-processing-flink-walkthrough/

Page 12: Apache Flink - A Stream Processing Engine

12

Thank you!@aljoscha@ApacheFlink @dataArtisanshttp://flink.apache.org

Page 13: Apache Flink - A Stream Processing Engine

13

Appendix

Page 14: Apache Flink - A Stream Processing Engine

We are hiring!

data-artisans.com/careers

Page 15: Apache Flink - A Stream Processing Engine

15

Flink Forward 2017

San Francisco Berlin• April 10-11, 2017• The first Flink Forward

event outside of Berlin• Register at sf.flink

-forward.org/

• September 11-13• Over 350 attendees at

the last event• Registration opening

soon!