dancing with stream processing

Dancing with Stream

Processing

Y.S. [email protected]

mailto:[email protected]

07/10/16 MSc. Distributed Systems 2

● Motivation● Event Stream Processing

– Pub/sub, CEP, “”Buzzwords”– Stream processing Engines

● Spark Streaming, Storm, Etc.

● Graph Stream Processing– Theory... {“sketching”, “spanners”, “sparsifiers”}– Challenges

● Discussion !!

Lightning Talk


Motivation


The Streaming Era

● Today, most data is continuously produced- user activity logs, web logs, sensors, database

transactions, …● The common approach to analyze such data so far

- Record data stream to stable storage (DBMS, HDFS, …)- Periodically analyze data with batch processing engine (DBMS, MapReduce, …)

● Streaming processing engines analyze data while it arrives


The Streaming Era (Contd.)

● Decreases the overall latency to obtain results- No need to persist data in stable storage- No periodic batch analysis jobs

● Simplifies the data infrastructure- Fewer moving parts to be maintained and coordinated

● Makes time dimension of data explicit- Each event has a timestamp- Data can be processed based on timestamps


Event Streams[Immutable]

Web Page Event

Wikipedia Page Update Event

LinkedIn User Update Event


Middleware


Direct coupling Strict Identity Time coupling

Not good for volatile environment

Not a good way to communicate with several participants

Space uncoupling Anonymity Time uncoupling

Independent lifetimes between parties

Through persistent communication channel

Point-to-point communication

Indirect communication


Taxonomy

Indirect Communication

Communication based

Group communication

Message Queues

Publish/subscribe

State based

Tuple spaces

Distributed Shared Memory


Pub/Sub Messaging Pattern

Topic-based- Each event belongs to a

number of topics (e.g. “music”, “sport”)

- Users subscribe to topics and receive all relevant events

Content-based - Users subscribe to the

actual content of the events/ a structured summary of it

- More expressive


Pub/Sub Activities

Subscription processing Indexing and storing subscriptions.

Event Stream Processing (ESP) Pub/sub approach: upon arrival of events, access

subscription index and identify all matched subscriptions.

Event delivery deliver event to clients with matched subscriptions.


Event Stream Processing (ESP)

Wikipedia


Today's world...

Pub/sub ≈ ESP ≈


“Buzzwords”

https://martin.kleppmann.com/2015/01/29/stream-processing-event-sourcing-reactive-cep.html


Complex Event Processing (CEP)

● A set of event processing principles

● Match patterns of events– Comparable to SQL queries– High-level query language

● Cloud of causally related events– POSET (Partially Ordered Set of Events)



● Some CEP Examples:– When 2 transactions happen on an account from

radically different geographic locations within a certain time window then report as potential fraud.

– When a gold customer's trouble ticket is not resolved within 1 hour, then escalate.

– When a team meeting request overlaps with my lunch break, then deny the team meeting and demote the meeting organizer.


ESP and CEP[Timeline]

2002

AuroraAurora

2003

Medusa

2005

Borealis

STREAM

TelegraphCQ

<20001989 - 1995

Rapide

Esper Apama

StreamBase

SQLStream

WSO2 CEP

2016


ESP vs. CEP

http://www.slideshare.net/TimBassCEP/mythbusters-event-stream-processing-v-complex-event-processing-presentation


Today's world...

ESP ≈ CEP ≈


Laundry of “Buzzwords”

● Actor Frameworks– Better mechanism to handle concurrency– E.g. Akka, Orleans and Erlang OTP

● “Reactive”– Language semantics for bringing event streams to the user

interface– Responsive, Resilient, Elastic and Message Driven– E.g. Data flow languages, Functional reactive programming

● Event Sourcing● Change Data Capture (CDC)


Analytics ≈ Stream Transformations



Target

● Better Scalability● High Throughput● Low latency● Powerful semantics● Easy integration

via Low Level Stream Processing

Frameworks !!


Spark Streaming

● General purpose computing engine to run batch, interactive and streaming jobs

● Based on Resilient Distributed Datasets (RDD)– Restricted form of distributed shared memory– Immutable– Can only be built through deterministic

transformations● Efficient fault recovery using lineage graph

– Recompute lost partitions on failure– No cost if nothing fails


Spark Streaming (Contd.)[Key concepts]

● DStream – sequence of RDDs representing a stream of data– HDFS, Kafka, Flume, ZeroMQ, Akka Actor, TCP sockets

● Transformations – modify data from one DStream to another– Standard RDD operations – map, countByValue, reduce,

join, …– Stateful operations – window, countByValueAndWindow, …

● Output Operations – send data to external entity– saveAsHadoopFiles – saves to HDFS– foreach – do anything with each batch of results


Spark Streaming (Contd.)

● Run a streaming computation as a series of very small, deterministic batch jobs– Chop up the live stream into batches of X seconds– Spark treats each batch of data as RDDs and processes them using RDD

operations– Finally, the processed results of the RDD operations are returned in

batches


Berkeley Data Stack


Spark 2.0 is coming !!


Apache Storm[Key concepts] ● Tuple

– Core Unit of Data– Immutable Set of Key/Value Pairs

● Spouts– Source of Streams– Wraps a streaming data source and emits Tuples

● Bolts– Core functions of a streaming computation– Receive tuples and do stuff– Optionally emit additional tuples


Apache Storm[Key concepts]

● Topology– DAG of Spouts and

Bolts– Data Flow

Representation– Streaming

Computation


Apache Storm[Physical View]


Twitter introduces Heron !!

[Storm's successor]


Stream Processing Engines

Many More !!!


Hidden computation paradigm

via pipelining !!


Pipelining ≈ Task Execution

https://martin.kleppmann.com/unix


Let's build the concept again...


Linux pipelining in modern middle-ware...



Spark, Storm, Samza, Flink Etc.



Pub/sub pitch



Streaming Machine Learning

● By using a programing abstraction for distributed streaming– Apache SAMOA


Graph Stream ProcessingReferred Author: Vasia Kalavri, KTHhttps://berlinbuzzwords.de/sites/berlinbuzzwords.de/files/media/documents/buzzwords-kalavri.pdf


Static Graph Processing

● Load: read the graph from disk and partition it in memory

● Compute: read and mutate the graph state

● Store: write the final graph state back to disk


Static Graph Processing[Drawbacks]

● It is slow– wait until the computation is over before you see

any result– pre-processing and partitioning

● It is expensive– lots of memory and CPU required in order to scale

● It requires re-computation for graph changes– no efficient way to deal with updates


Streaming Graph Processing

We consume events in real-time● Get results faster

– No need to wait for the job to finish– Sometimes, early approximations are better

than late exact answers

● Get results continuously– Process unbounded number of events


Real-world scenarios

● Targeted Advertisement– Finding Strongly Connected Components in a

social network graph– Targeted chain of advertisement on detected

communities

Jane Joeknows

#Tesla

postslikes

Self driving carsAds

Peter Taphousechecks-in

John

subscribesDinner Offer

Ads


Streaming Graph Processing[Challenges]● Maintain the graph structure

– How to apply state updates efficiently?

● Result updates– Re-run the analysis for each event?– Design an incremental algorithm?– Run separate instances on multiple snapshots?

● How to preserve graph properties?– Natural behavior?


Streaming Graph Processing[Current Research]

Each event is an edge addition

Jane Joeknows

Jane #Teslalikes

Joe #Teslaposts

Peter TapHousechecks-in


Dynamic Graph Processing

● Instead of analyzing the whole graph– Analyze it's properties by preserving them

continuously● Connectivity or Distance (spanners)● Graph cut estimation (sparsifiers)● Neighborhood or homomorphic properties (sketches)


Dynamic Graph Processing (Contd.)

Jane Joeknows

#Tesla

postslikes


Peter Taphousechecks-in

John

subscribesDinner Offer

Ads

Peter Janeloves

loves



Stream Connected Components

● State: a disjoint set data structure for the components

● Computation: For each edge– if seen for the 1st time, create a component with ID

the min of the vertex IDs– if in different components, merge them and update

the component ID to the min of the component IDs– if only one of the endpoints belongs to a

component, add the other one to the same component


Distributed Stream Connected Components


Streaming Graph Processing[Current Work]● We're working with Gelly-Streams on

– Preserving natural properties in large scale real-world evolving graphs

– Joining multiple streams for detects graph causality/ bipartite

– Efficient graph partitioning mechanisms to on-board with popular data-stores like Cassandra, HDFS

– Producing a platform to benchmark NPC problems in real-world graphs


Discussion !!


Thank you !!

[email protected]

mailto:[email protected]

dancing with stream processing

Data & Analytics