apache beam (incubating)

Apache Beam (incubating)

Kenneth [email protected]@KennKnowles Apache Apex Meetup, 2016-06-

27

https://goo.gl/LTLjKt

mailto:[email protected]

http://twitter.com/KennKnowles

https://goo.gl/LTLjKt

Motivation

Beam Model

Beam Project / Technical Vision

Agenda1

2

3

2

3

Motivation1

https://commons.wikimedia.org/wiki/File:Globe_centered_in_the_Atlantic_Ocean_(green_and_grey_globe_scheme).svg4

5

Unbounded, delayed, out of order

9:008:00 14:00

13:00

12:00

11:00

10:00

2:001:00 7:006:005:004:003:00

5

8:00

8:008:00

Incoming!

Score per

user?

6

Organizing the stream

7

8:00

8:00

8:00

Completeness

Latency Cost

$$$

Data Processing Tradeoffs

8

What is important for your application?

Completeness Low Latency Low Cost

Important

Not Important

$$$9

Monthly Billing


Important

Not Important

$$$10

Billing estimate


Important

Not Important

$$$11

Abuse Detection


Important

Not Important

$$$12

13

The Beam Model

2

The Beam Model

Pipeline

14

PTransform

PCollection

The Beam Vision (for users)

Sum Per Key

15

input.apply( Sum.integersPerKey())

Java

input | Sum.PerKey()

Python

Apache Flink

Apache Spark

Cloud Dataflow

⋮ ⋮

Apache Apex

Apache Gearpump

(incubating)

Pipeline p = Pipeline.create(options);

p.apply(TextIO.Read.from("gs://dataflow-samples/shakespeare/*"))

.apply(FlatMapElements.via(line → Arrays.asList(line.split("[^a-zA-Z']+"))))

.apply(Filter.byPredicate(word → !word.isEmpty()))

.apply(Count.perElement())

.apply(MapElements.via(count → count.getKey() + ": " + count.getValue())

.apply(TextIO.Write.to("gs://..."));

p.run();

What your (Java) Code Looks Like

16

The Beam Model: Asking the Right Questions What are you computing?

Where in event time?

When in processing time are results produced?

How do refinements relate?

17





18

Aggregations, transformations, ...

The Beam Model: What are you computing?

Sum Per User

19

The Beam Model: What are you computing?

Sum Per Key

20

input.apply(Sum.integersPerKey()) .apply(BigQueryIO.Write.to(...));

Java

input | Sum.PerKey() | Write(BigQuerySink(...))

Python

http://beam.apache.org/blog/2016/05/27/where-is-my-pcollection-dot-map.html

http://beam.apache.org/blog/2016/05/27/where-is-my-pcollection-dot-map.html





21

Event time windowing

22

The Beam Model: Where in Event Time?8:00

8:00

8:00

Processing Time vs Event Time

Event Time = Processing Time ??

23


24

Proc

essi

ng T

ime

Proc

essi

ng T

ime


Realtime

25

This is not possible


26

Processing DelayPr

oces

sing

Tim

e

Processing Time vs Event TimeVery delayed

27

Proc

essi

ng T

ime

Event Time

Processing Time windows(probably are not what you want)

Proc

essi

ng T

ime

Event Time 28

Event Time Windows

29

Proc

essi

ng T

ime

Event Time

Proc

essi

ng T

ime

Event Time

Event Time Windows

30

(implementing processing time windows)

Just throw away your data's timestamps and replace them with "now()"

input |

WindowInto(FixedWindows(3600) | Sum.PerKey() | Write(BigQuerySink(...))

Python

The Beam Model: Where in Event Time?

Sum Per Key

Window Into

31

input.apply(

Window.into( FixedWindows.of( Duration.standardHours(1))) .apply(Sum.integersPerKey()) .apply(BigQueryIO.Write.to(...))

Java

So that's what and where...

32





33

Watermarks &

Triggers

Event time windowsPr

oces

sing

Tim

e

34

Event Time

Fixed cutoff (we can do better)Pr

oces

sing

Tim

e

Event Time35

Allowed delay

Concurrent windows

Perfect watermarkPr

oces

sing

Tim

e

36

Event Time

Check out Slava's slides from Strata London 2016 talk on watermarks:https://goo.gl/K4FnqQ

https://goo.gl/K4FnqQ

Heuristic WatermarkPr

oces

sing

Tim

e

37

Event Time


oces

sing

Tim

e

38

Current processing time

Event Time


oces

sing

Tim

e

39


Event Time


oces

sing

Tim

e

40


Late data

Event Time

Watermarks measure completeness

41

$$$

$$$

$$$

? Running Total

✔ Monthly billing

? Abuse Detection

The Beam Model: When in Processing Time?

Sum Per Key

Window Into

42

input .apply(Window.into(FixedWindows.of(...))

.triggering( AfterWatermark.pastEndOfWindow())) .apply(Sum.integersPerKey()) .apply(BigQueryIO.Write.to(...))

Java

input | WindowInto(FixedWindows(3600),

trigger=AfterWatermark()) | Sum.PerKey() | Write(BigQuerySink(...))

Python

Trigger after end of window

Proc

essi

ng T

ime

Event Time

AfterWatermark.pastEndOfWindow()

43


Proc

essi

ng T

ime

Event Time44


Proc

essi

ng T

ime

Event Time

Late data

45



Proc

essi

ng T

ime

Event Time46

High completeness

Potentially high latency

Low cost


$$$

Proc

essi

ng T

ime

Event Time

Repeatedly.forever( AfterPane.elementCountAtLeast(2))

47

Proc

essi

ng T

ime

Event Time48




Proc

essi

ng T

ime

Event Time49


Proc

essi

ng T

ime

Event Time50




Proc

essi

ng T

ime

Event Time51


Proc

essi

ng T

ime

Event Time52


Low completeness

Low latency

Cost driven by input

$$$

Build a finely tuned trigger for your use caseAfterWatermark.pastEndOfWindow()

.withEarlyFirings( AfterProcessingTime .pastFirstElementInPane() .plusDuration(Duration.standardMinutes(1))

.withLateFirings(AfterPane.elementCountAtLeast(1)) 53

Bill at end of month

Near real-time estimates

Immediate corrections

Proc

essi

ng T

ime

Event Time54

.withEarlyFirings(after 1 minute)

.withLateFirings(ASAP after each element)

Proc

essi

ng T

ime

Event Time55




Proc

essi

ng T

ime

Event Time56


Low completeness

Low latency

Low cost, driven by time

$$$




Proc

essi

ng T

ime

Event Time57




Proc

essi

ng T

ime

Event Time

Late output

58



Proc

essi

ng T

ime

Event Time

Late output

59



Trigger CatalogueComposite TriggersBasic Triggers

60

AfterEndOfWindow()

AfterCount(n)

AfterProcessingTimeDelay(Δ)

AfterEndOfWindow() .withEarlyFirings(A) .withLateFirings(B)

AfterAny(A, B)AfterAll(A, B)Repeat(A)Sequence(A, B)





61

Accumulation Mode

The Beam Model: How do refinements relate?

62

input

.apply(Window.into(...).triggering(...).discardingFiredPanes()) .apply(Sum.integersPerKey()) .apply(BigQueryIO.Write.to(...))

vs

1

3 7

4

10

5

1

3 7

4

10

15

discarding accumulating





63

64

Beam Project / Technical Vision

3

1. End users: who want to write pipelines in a language that’s familiar.

2. SDK writers: who want to make Beam concepts available in new languages.

3. Runner writers: who have a distributed processing environment and want to run Beam pipelines

Beam Fn API: Invoke user-definable functions

Apache Flink

Apache Spark

Beam Runner API: Build and submit a piepline

OtherLanguagesBeam Java Beam

Python

Execution Execution

Cloud Dataflo

w

Execution

The Beam Vision

Apache Apex

Apache Gearpump (incubatin

g)

Project Setup (vision meets code)GoogleCloudPlatform/DataflowJavaSDK cloudera/spark-dataflow dataArtisans/flink-dataflow

apache/incubator-beam

Direct (on your laptop)Google Cloud DataflowFlinkSparkIn pull request: Apex, Gearpump

Integration tests

Runners

Examples

I/O Connectors

sharing

HDFSKafkaBigQueryGoogle Cloud Storage, Pubsub, Bigtable, DatastoreIn pull request: JMS, CassandraProposed: Sqoop, Parquet, JDBC, SocketStream, ...

SDKs

https://github.com/GoogleCloudPlatform/DataflowJavaSDK

https://github.com/cloudera/spark-dataflow/

https://github.com/dataArtisans/flink-dataflow

https://github.com/apache/incubator-beam

https://issues.apache.org/jira/browse/BEAM-261

https://github.com/apache/incubator-beam/pull/323







Committers from Google, Data Artisans, Cloudera, Talend, Paypal● ~40 commits/week● Rigorous code review for every commit

Contributors [with GitHub badges] from: Spotify, Intel, Twitter, Capital One, DataTorrent, …, <your name here>

● Improvements to existing I/O connectors● Improvements to Spark runner● Utility classes for users● Documentation fixes● Bug diagnoses● New I/O connectors● Gearpump runner PoC● Apex runner PoC!

… and it has been awesomeapache/incubator-beam

https://github.com/apache/incubator-beam

Java SDK: Transition from Dataflow

Dataflow Java 1.x

Apache Beam Java 0.x

Apache Beam Java 2.xBug Fix

Feature

Breaking Change

We are here

Feb 201

6

Late 2016

Understanding: Capability Matrix

http://beam.incubator.apache.org/capability-matrix/

http://beam.incubator.apache.org/capability-matrix/

Why Apache Beam?Unified - One model handles batch and streaming use cases.

Portable - Pipelines can be executed on multiple execution environments, avoiding lock-in.

Extensible - Supports user and community driven SDKs, Runners, transformation libraries, and IO connectors.

Why Apache Beam?http://data-artisans.com/why-apache-beam/

"We firmly believe that the Beam model is the correct programming model for streaming and batch data processing."

- Kostas Tzoumas (Data Artisans)

https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective

"We hope it will lead to a healthy ecosystem of sophisticated runners that compete by making users happy, not [via] API lock in."

- Tyler Akidau (Google)

http://data-artisans.com/why-apache-beam/



72

Creating an Apache Beam CommunityCollaborate - Beam is becoming a community-driven effort with participation from many organizations and contributors.

Grow - We want to grow the Beam ecosystem and community with active, open involvement so Beam is a part of the larger OSS ecosystem.

We love contributions. Join us!

Apache Beam

http://beam.incubator.apache.org/Why Apache Beam? (from Data Artisans)Why Apache Beam? (from Google)

Programming Model Overviews

Streaming 101Streaming 102The Dataflow Beam Model

Join the community!User discussions - [email protected] discussions - [email protected] @ApacheBeam on Twitter

Learn More!

73

http://beam.incubator.apache.org/

http://data-artisans.com/why-apache-beam/


http://oreilly.com/ideas/the-world-beyond-batch-streaming-101

http://oreilly.com/ideas/the-world-beyond-batch-streaming-102

http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf%20





END

74

apache beam (incubating)

Technology