alexander kolb – flink. yet another streaming framework?

Alexander Kolb, Otto Group BI Hamburg, Germany, 2015

Flink, Yet another Streaming Framework?

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Alexander Kolb otto group BI

@lofifnc

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Introduction

Eval

uatio

n

Usability

Functionality

Architecture

Support

Non-Functional-Requirements

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Approach

5

Rating based on: - Research - Hands-on

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Use-case

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Use-case

7

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Frameworks

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Frameworks

9

SQLStream

Pulsar

SPQR

Apache Spark

Apache Flink

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

SQLStream

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

SQLStream

11

Architecture

source: sqlstream.com

SQLS

trea

m

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

SQLStream

13

Window Aggregation

1 SELECT STREAM 2 B.pagetype,3 B.ecid,4 productid,5 SUM(QUANTITY) AS "views"6 FROM VIEWS AS B7 GROUP BY FLOOR((B.ROWTIME - TIMESTAMP '1970-01-01 00:00:00')8 MINUTE / 5 to MINUTE),9 PRODUCTID, PAGETYPE, ECID;

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Pulsar

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Pulsar

15

Architecture

source: github.com/pulsarIO

Puls

ar

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Pulsar

17

Window Aggregation

1 create context MCContext start @now end after 60 seconds; 2 3 context MCContext 4 insert into ViewAgg select count(*) as views, prid 5 from PageView group by prid output snapshot when terminated;

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

SPQR

SPQ

R

source: github.com/ottogroup/SPQR

SPQ

R

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

SPQR

21

Window Aggregation

1 select productid, ecid, sum(quantity)2 from views.win:time_batch(5 min) 3 group by productid, ecid

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Apache Spark

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Apache Spark

23

Architecture

source: spark.apache.org

Apa

che

Spar

k

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Apache Spark

25

Aggregation

1 val aggViews = views.reduceByKeyAndWindow({ 2 case ((pageType, ecid, sum, price),(_,_,quant,_)) => 3 (pageType, ecid, sum + quant, price) 4 }, Minutes(5), Minutes(5))

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Apache Flink

26

Architecture

source: spark.apache.org

source: flink.apache.org

Apa

che

Flin

k

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Apache Flink

28

Window Aggregation

1 val aggViews = input.window(Time.of(5, TimeUnit.MINUTES)) 2 .groupBy(“productId”).sum(“quantity”);

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Result Evaluation

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Summary

30

Use-case

Topic UnitFramework

Pulsar.io SQLStream SPQR Flink Spark

Time for building the stream hours 40 35+

(POC)8+

(POC) 13 4

Time for adding missing connector

hours 3 8 1 3 0.5

Points 3.14 2.06 3.44 4.16 4.45

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

List of Rating Aspects

31

DSL/DDL/UI for creating Pipelines / Required know-how to define new Pipelines / Project documentation /Workflow / Testing Workflows / hot deploying / redeploying of pipelines / dynamic topology changes / Monitoring / Deployment / Dashboard for data visualization / Ease of defining udf's / Merge / Sum / Count / Min/max/avg / Aggregate / Transform / Parsing (xml/json/csv) / Group-by / Join / Ease of defining new connectors / Kafka / WebSocket / JDBC / JMS / HDFS / File / Effort for cluster deployment / Configuration effort / Supports YARN / Supports Mesos / Scalability / Resilience /Predefined communication framework / Dependencies / Flexibility / Expandability / Buffering/Pressure handling / Partitioning/Parallelism / Strategy for Partitioning/ Parallelism? / Ordering / Guarantees / State-Management / Fault tolerance / Licensing model / Professional support available / Community Activity / License / Maturity / Manageable code-base / Community Size

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Summary

32

Topic Framework weightSQLStream Pulsar.io SPQR Spark Flink

Usability 2.6 3 2.2 3.6 3.9 15Flexibility 2.5 4 3 1.5 1.5 8User-Interface 3.3 1.8 1 3 3.3 6Operators 4.8 4.3 4.3 3.9 4.7 10Connectors 4.7 1.9 1.9 2.4 2.6 6Deployment 2.4 3.2 2.8 3.8 3.7 10Architecture/ Concepts 2 3.2 3.3 4 4 12

Functional Requirements 2 3.2 2.4 4 4 14

Costs 0 5 5 5 5 5Service/ Support 2.5 0.5 1 4.5 3.5 8Project 1.8 3.3 2.5 4.3 3.8 6

Sum 2.58 3.11 2.74 3.74 3.80 100

Sum

mar

yUsability

Flexibility

User-Interface

Operators

Connectors

Deployment

Architecture/ Concepts

Functional Requirements

SQLStreamPulsar.ioSPQRSparkFlink

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

Final Scores

34

SQLStream Pulsar.io SPQR Spark Flink

Evaluation 2.58 3.11 2.74 3.74 3.8

Use-case 2.06 3.14 3.44 4.45 4.16

Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015

ottogroup.comWE ARE HIRING!