real-time analytics as a service at king

© King.com Ltd 2016 – Commercially confidential

Real-Time Analytics as a Service at KingGyula Fóra

Data Warehouse Engineer

Apache Flink PMC


We make awesome mobile games

463 million monthly active users

30 billion events per day

And a lot of data…

About King


From streaming perspective…

DB

30 billion events / day

Analytics/Processing applications

Terabytes of state

DB

DB


This is awesome, but…

End-users are often not Java/Scala developers

Writing streaming applications is pretty hard

Large state and windowing doesn’t help either

Seems to work in my IDE, what next?

We need a “turnkey” solution


The RBea platform

Powered by Apache Flink

Scripting on the live streams

Window aggregates

Stateful computations

Scalable + fault tolerant


RBea architecture

Events Output

REST API

RBEA web frontend

Libraries

http://hpc-asia.com/wp-content/uploads/2015/09/World-Class-Consultancy-Seeking-Data-Scientist-CA-Hobson-Associates-Matthew-Abel-Recruiter.jpg

Data Scientists


RBea backend implementation

One stateful Flink job / game

Stream events and scripts

Events are partitioned by user id

Scripts are broadcasted

Output/Aggregation happens downstream

S1 S2

S3S4

S5

Add/Remove scripts

Event stream Loop over deployed scripts and process

CoFlatMap

Output based on API calls


Dissecting the DSL

@ProcessEvent(semanticClass=SCPurchase.class)def process(SCPurchase purchase,

Output out,Aggregators agg) {

long amount = purchase.getAmount() String curr = purchase.getCurrency() out.writeToKafka("purchases", curr + "\t" + amount)

Counter numPurchases = agg.getCounter("PurchaseCount", MINUTES_10) numPurchases.increment()

}


Dissecting the DSL





}

Processing methods by annotation

Event filter conditions

Flexible argument list

Code-generate Java classes

=> void processEvent(Event e, Context ctx);


Dissecting the DSL





}

Output calls create Output events

Output(KAFKA, “purchases”, “…” )

These events are filtered downstream and sent to a Kafka sink


Dissecting the DSL





}

Aggregator calls create Aggregate events

Aggr (MYSQL, 60000, “PurchaseCount”, 1)

Flink window operators do the aggregation


Aggregators

long size = aggregate.getWindowSize();long start = timestamp - (timestamp % size);long end = start + size;TimeWindow tw = new TimeWindow(start, end);

Event time windowsWindow size / aggregator

Script1

Script2

Window 1Window 2 NumGames

Revenue

W1: 8999W2: 9001

W1: 200W2: 300

MyAggregator

W1: 10W2: 5

Dynamic window assignment


RBea physical plan


How do we run Flink

Standalone => YARN

Few heavy streaming jobs => more and more

RocksDB state backend

Custom deployment/monitoring tools


Monitoring our jobs


King Streaming SDK (sneak preview)

Goal: Bridge the gap between RBea and Flink

Build data pipelines from RBea processors

Strict event format, limited set of operations

Easy ”stream joins” and pattern matching

Thin wrapper around Flink


King Streaming SDK (sneak preview)

Last<GameStart> lastGS = Last.semanticClass(GameStart.class);

readFromKafka("event.myevents.log", "gyula").keyByCoreUserID().join(lastGS).process((event, context) -> {

context.getJoined(lastGS).ifPresent(lastGameStart -> {

context.getAggregators().getCounter("Purchases", MINUTES_10).setDimensions(lastGameStart.getLevel()).increment();

});});


Closing

RBea makes streaming accessible to every data scientist at King

We leverage Flink’s stateful and windowed processing capabilities

People love it because it’s simple and powerful

Thank you!

real-time analytics as a service at king

Data & Analytics