c-mr: continuously executing mapreduce workflows on multi-core processors

C-MR: Continuously Executing MapReduce Workflows on Multi-

Core Processors

Speaker: LIN Qianhttp://www.comp.nus.edu.sg/~linqian

2

Problem

• Stream applications are often time-critical• Enabling stream support for MapReduce

jobs– Simple for the Map operations– Hard for the Reduce operations

• Continuously executing MapReduce workflows requires a great deal of coordination

3

C-MR Workflow

• Windows: temporal subdivisions of a stream described by– size (the amount of the stream spanning)– slide (the interval between windows)

C-MR Programming Interface

• Map/Reduce operations

C-MR Programming Interface (cont.1)

• Input/Output streams

C-MR Programming Interface (cont.2)

• Create workflows of continuous MapReduce jobs

7

C-MR vs. MapReduce

• MapReduce computing nodes receive a set of Map or Reduce tasks and each node must wait for all other nodes to complete their tasks before being allocated additional tasks.

• C-MR uses pull-based data acquisition allowing computing nodes to execute any Map or Reduce workload as they are able. Thus, straggling nodes will not hinder the progress of the other nodes if there is data available to process elsewhere in the workflow.

8

C-MR Architecture

Stream and Window Management

• The merged output streams are not guaranteed to retain their original orderings.

• Solution: Replicating window-bounding punctuations

10

Stream and Window Management (cont.1)

A node consumes the punctuation from the sorted input stream-buffer

Stream and Window Management (cont.2)

Replicate that punctuation to the other nodes

Stream and Window Management (cont.3)

After all replicas are received at the intermediate buffer, collect data whose timestamps fall into the applicable interval and materialize them as a window

Operator Scheduling

• Scheduling framework– Execute multiple policies simultaneously– Transition between policies based on

resource availability

• Scheduling policies

Incremental Computation

Output1 = d1 + d2 + d3 + ... + dn

Output2 = d2 + d3 + d4 + ... + dn+1

Output3 = d3 + d4 + d5 + ... + dn+2

Output4 = d4 + d5 + d6 + ... + dn+3

Share the common data subset of computation

15

Evaluation

• Continuously executing a MapReduce job– Compare with Phoenix++

16

Evaluation (cont.1)

• Operator scheduling– Oldest data first (ODF)– Best memory trade-off (MEM)– Hybrid utilization of both policies

17

Evaluation (cont.2)

• Workflow optimization

18

Evaluation (cont.3)

• Workflow optimization– Latency and throughput

Thank you

19

20

Two Properties of Streams

• Unbounded• Accessed sequentially

Hard to be handled using traditional DBMS

21

Query Operators

• Unbounded stateful operators– maintain state with no upper bound in size

run out of memory

• Blocking operators– read an entire input before emitting a

single output

might never produce a result

• Never use them, or• Use them under a refactoring

22

Punctuations

• Mark the end of substreams – allowing us to view an infinite stream as a

mixture of finite streams