MULTIPLEMULTIPLEMULTIPLEMU

THREADSTHREADSTHREADSTHRMULTIPLETHREADS

EVENT STREAM PROCESSING

with

Sylvain HalléRaphael KhourySébastien Gaboury

Université du Québec a Chicoutimi

The Problem

input event stream

The Problem

Pcomputation

input event stream

The Problem

output event streamP

computation

input event stream

The Problem

Find ways to split P into parts thatcan be processed in parallel

1

output event streamP

computation

input event stream

The Problem

genericFind ways to split P into parts thatcan be processed in parallel

1

output event streamP

computation

input event stream

The Problem

Input events arriveone by one2

genericFind ways to split P into parts thatcan be processed in parallel

1

output event streamP

computation

input event stream

The Problem

Input events arriveone by one2

generic

Output events areproduced one by one3

Find ways to split P into parts thatcan be processed in parallel

1

output event streamP

computation

input event stream

The Problem

The Problem

Single-threaded

The Problem

Single-threaded

Multi-threaded

The Problem

The System

The System

Based on thecomposition (piping)of simple computingunits called

processors

The System

The System

→

Fork f = new Fork(2);

The System

→

→

→

→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor( );

f

The System

→

→

→

+

→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor(Addition instance);

f

The System

→

→

→

+

→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor(Addition.instance);

CountDecimate decimate = new CountDecimate(n);

f

n

The System

→

→

→

+

→→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor(Addition.instance);

CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT)

f

n

The System

→

→

→

+

→→

→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor(Addition.instance);

CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT). connect(fork, RIGHT, decimate, INPUT)

f

n

The System

→

→

→

+

→→

→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor(Addition.instance);

CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT). connect(fork, RIGHT, decimate, INPUT). connect(decimate, OUTPUT, sum, RIGHT);

f

n

The System

→

→

→

+

→→

→

Fork f = new Fork(2);FunctionProcessor sum =new FunctionProcessor(Addition.instance);

CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT). connect(fork, RIGHT, decimate, INPUT). connect(decimate, OUTPUT, sum, RIGHT);Pullable p = sum.getOutputPullable(OUTPUT);while (p.hasNext() != NextStatus.NO) {Object o = p.next();

. ..}

f

n

The System

The System

FunctionApplies a function toevery input event

CumulativeComputes theprogressive "sum" ofall input events

TrimRemoves the first ninput events

DecimateOutputs everyn-th event

GroupEncloses a group ofconnected processors

ForkDuplicates a streaminto multiple copies

SliceSplits a stram into multiplesub-streams

WindowApplies a function toa sliding window ofn events

f

Σ.f

n

FilterA first, Boolean stream, decides if each event of a second stream should be output

{f

The System

X

G F

U

1

<Σ f

⊥

<

Σ f⊥

F

G

=

= =

= Σ f

?

♣

LTL PALETTE

Existing processors should be changed !4 not

P P PPPP PPP

The Problem

Existing processors should be changed !4 not

P P PPPP PPP

Enclose them within new, processors

Solution:

thread-aware

P

The Problem

Thread Manager

Responsible for a poolof N threads M

Runnable r = ...ManagedThread t = m.tryNewThread(r);if (t != null) t.start(); else r.run();

Returns a new thread that can be started, or null ifall N threads are busy

If t is null, the desired processing must be donein the current thread

Thread Manager

Responsible for a poolof N threads M

ψφψ.pull

Sequential Pulling

TIM

E

ψφψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

φ.pull

ψ.pull

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

φ.pull

ψ.pull

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

e2

φ.pull

ψ.pull

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

e2

φ.pull

ψ.pull

ψ.pull

Sequential Pulling

TIM

E

ψ

φ.pull

φ

e1

e2

φ.pull

ψ.pull

ψ.pull

Sequential Pulling

TIM

E

ψφ W

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ WW.pull ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ WW.pull ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

W.pull

e1

ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

φ.pull

W.pull

e1

ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

φ.pull

W.pull

e1

W.pull

e2

ψ.pull

ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

φ.pulle3

W.pull

e1

W.pull

e2

ψ.pull

ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

φ.pulle3

W.pull

e1

W.pull

e2

ψ.pull

ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

φ.pulle3

W.pull

e1

W.pull

e2

W.pull

e3

ψ.pull

ψ.pull

ψ.pull

Pre-emptive Pulling

TIM

E

ψ

φ.pull

φ W

φ.pull

e1

e2

φ.pulle3

W.pull

e1

W.pull

e2

W.pull

e3

ψ.pull

ψ.pull

ψ.pull

Pre-emptive Pulling

TIM

E

ψφ

S

Pull Pipeline

TIM

E

ψφ.pull

φ

S

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ1

φe1

S

e'1

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2

S

push(e2)

e'1e'2

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

push(e2)

e'1e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

push(e2)

e'1e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

push(e2)

e'1

e'1

e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

push(e2)

e'1

e'1

e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

φ.pull

push(e2)

e'1

e'1

e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

φ.pull

push(e2)

e'1

e'1

e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

φ.pull

push(e2)

e'1

e'1

e'2

e'2

φ2

push(e3)

e'3

Pull Pipeline

TIM

E

ψ

push(e1)

φ.pull

φ2φ1

φe1e2e3

S

φ.pull

push(e2)

e'1

e'1

e'2

e'2

φ2

push(e3)

e'3

e'3φ.pull

Pull Pipeline

TIM

E

Window ψ

Blocking Push

TIM

E

Window ψ

push(e)

Blocking Push

TIM

E

Window ψ

push(e)

φ1.push(e)

Blocking Push

TIM

E

Window ψ

push(e)

φ1.push(e)

φ2.push(e)

Blocking Push

TIM

E

Window ψ

push(e)

φ1.push(e)

φn-1.push(e)

. . .

φ2.push(e)

Blocking Push

TIM

E

Window ψ

push(e)

φ1.push(e)

φn-1.push(e)

. . .

collect e'shiftcopy

φ2.push(e)

Blocking Push

TIM

E

Window ψ

push(e)

φ1.push(e)

φn-1.push(e)

. . .

collect e'shiftcopy

push(e')

φ2.push(e)

Blocking Push

TIM

E

Window ψ

push(e)

φ1.push(e)

φn-1.push(e)

. . .

collect e'shiftcopy

push(e')

φ2.push(e)Event e is pushed to the n copiesof φ sequentially. Each call topush is blocking.

Blocking Push

TIM

E

Window ψ

Non-blocking Push

TIM

E

Window ψ

push(e)

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)

φ0

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e) φ1.push(e)

φ0 φ1

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)φn-1.push(e)

φ1.push(e)

φ0 φ1 φn-1. . .

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)φn-1.push(e)

φ1.push(e)

φ0

φ0.waitFor

φ1 φn-1. . .

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)φn-1.push(e)

φ1.push(e)

φ0

φ0.waitForφn-1.waitFor

φ1 φn-1. . .

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)φn-1.push(e)

collect e'shiftcopy

φ1.push(e)

φ0

φ0.waitForφn-1.waitFor

φ1 φn-1. . .

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)φn-1.push(e)

collect e'shiftcopy push(e')

φ1.push(e)

φ0

φ0.waitForφn-1.waitFor

φ1 φn-1. . .

Non-blocking Push

TIM

E

Window ψ

push(e) φ0.push(e)φn-1.push(e)

collect e'shiftcopy push(e')

φ1.push(e)

φ0

φ0.waitForφn-1.waitFor

φ1 φn-1. . .

Calls to push are non-blocking;each runs in a separate thread.

Non-blocking Push

TIM

E

3 new thread-aware processors

Non-blockingpush

Pullpipeline

Pre-emptivepull

Implemented in a separateindependent from the rest of the code

palette

→

→ NP → PP → PE

Multi-threading in a Query

→

→ {

f

→→

+

Σ.f

10

FunctionProcessor sum = new FunctionProcessor( new CumulativeFunction(Addition.instance));

WindowProcessor win = new WindowProcessor(sum, 10);

ThreadManager m = new ThreadManager(4);FunctionProcessor sum = new FunctionProcessor( new CumulativeFunction(Addition.instance));NonBlockingProcessor nbp = new NonBlockingProcessor(sum, m);WindowProcessor win = new WindowProcessor(nbp, 10);

→

→ {

f 10

Multi-threading in a Query

M→

→ →

→

+

Σ.f4

Experimental Results

Auction biddingCandidate selectionEndless bashingSpontaneous Pingu creationTurn around

1.074.441.451.051.86

Benchmark (CRV 2016) Speedup

A single thread-aware processor inserted in the query

Experimental Results

Turn around

→

A/a/b//character[status=Walker]/id/text()

→ p1

→

A

→ p2

→ →→

→

/a/b

//character[status=Blocker]/id/text()

→

→→

3

→

→

→

<?

→

→

→

→

→

→

→

→

f1

f2

→

→ →

→

→

→

→

M4

Multi-threaded

Experimental Results

Turn around

→

A/a/b//character[status=Walker]/id/text()

→ p1

→

A

→ p2

→ →→

→

/a/b

//character[status=Blocker]/id/text()

→

→→

3

→

→

→

<?

→

→

→

→

→

→

→

→

f1

f2

→

→ →

→

→

→

→

Take-home Points

Multi-threading capabilities added to BeepBeep 3

Existing processors do not need to be aware of the presence of threads

Minimally intrusive: less than 5 lines to add multi-threading to part of a query

5% - 400% speedup on a small sample of queries

Requires manual tuning and intuition

Not all queries can gain from parallelism!

https://liflab.github.io/beepbeep-3

THEENDTHEEND ...but for how long ?

https://www.researchgate.net/publication/318337325