JetStream:Cluster-scaleParallelizationofInformationFlowQueries

AndrewQuinn,DavidDevecsery,PeterChenandJasonFlinn

2

• DIFT instruments execution to track causality• Also known as “Taint-Tracking”

Dynamic Information Flow Tracking

Sources (Inputs)

Sinks (Outputs)

3

o1

DIFT for Debugging

o1Server

4

DIFT for Debugging

o1

Outputs

Inputs

Server

5

DIFT for Debugging

o1

Outputs

Inputs

Server

6

DIFT for Debugging

o1

Outputs

Inputs

Server

7

DIFT for Debugging

o1

Outputs

Inputs

Server

8

DIFT – limitations

Arnold ‘14 Overheads ~100x

X-Ray ‘12 Long queries

TaintDroid ‘10 No native code

Backtracker ‘03 Coarse-grained causality

9

Parallelize DIFT

10

Parallelizing DIFT is HARD

A = read()B = read()C = A + BD = X + Y E = CB = 0Z = A[D]F = E write(F)

SequentialDependencies

11

Parallelizing DIFT is HARD

A = read()B = read()C = A + BD = X + Y E = CB = 0Z = A[D]F = E write(F)

SequentialDependencies

Speck (ASPLOS ‘08)

Parallel Lifeguards (ASPLOS ‘08)

12

Parallelizing DIFT is HARD

A = read()B = read()C = A + BD = X + Y E = CB = 0Z = A[D]F = E write(F)

SequentialDependencies

Speck (ASPLOS ‘08)

Parallel Lifeguards (APSLOS ‘08)

“EmbarrassinglySequential”- Ruwase etal.

13

JetStream

Aggregation – pipeline parallelism

Local DIFT – epoch parallelism 2xà21x

Fasterthanoriginalexecution!

14

• Design of JetStream• Local DIFT• Aggregation

• Evaluation

Outline

15

Debugging Query

Outputs

Inputs

16

Local DIFT

Outputs

Inputs

Time slice execution into Epochs

17

Local DIFT

Outputs

Inputs

Leverage Record and Replay to calculate DIFT in parallel

18

Local DIFT

Outputs

Inputs

Track mapping between all intermediate locations

19

Local DIFT

Outputs

Inputs

Mapping is too expensive to calculate:• log operations• defer calculating relationships until aggregation

20

DIFT

D = B + C C = A[D]

• Fast Forward: replay execution until start of epoch• Analysis: log operations using a graph

CAFast Forward

Analysis

B D

CA B D

21

DIFT

D = B + C C = A[D]

• Fast Forward: replay execution until start of epoch• Analysis: log operations using a graph

CAFast Forward

Analysis

B D

CA B D

22

DIFT

D = B + C C = A[D]

• Fast Forward: replay execution until start of epoch• Analysis: log operations using a graph

CAFast Forward

Analysis

B D

CA B D

23

Local DIFT output

Outputs

Inputs

24

JetStreamLocal DIFT – epoch parallelism

Aggregation – pipeline parallelism

25

Aggregation

Outputs

Inputs

Calculate paths between source and sinks• Many nodes are not on path between source and sink• Use sequential information to prune work

26

Forward Pass

Outputs

Inputs

Pass locations which are derived from a source

derivedlocations

27

Forward Pass

Outputs

Inputs

Pass locations which are derived from a source

derivedlocations

28

Backward Pass

Outputs

Inputs

usedlocations

Pass locations which are used by a sink

29

Backward Pass

Outputs

Inputs

usedlocations

Pass locations which are used by a sink

30

In the paper:• Insights about why naïve approaches fail• Partitioning – challenging to predict the local DIFT time • Pre-pruning – garbage collection of the graph

JetStream

31

• Evaluation

Outline

32

• CloudLab cluster of 32 machines, 1–128 cores

Experimental Setup

Benchmark Sequential DIFT Time (Minutes)

Sources(millions)

Sinks(millions)

Ghostscript 1.3 3 0.2Gzip 1.8 64 488Evince 3.9 10 104Nginx 3.3 10 35Mongodb 5.2 9 117OpenOffice 7.0 10 32Firefox 30.6 0.9 2

33

Benchmark Sequential DIFT Time (Minutes)

Sources(millions)

Sinks(millions)

Ghostscript 1.3 3 0.2Gzip 1.8 64 488Evince 3.9 10 104Nginx 3.3 10 35Mongodb 5.2 9 117OpenOffice 7.0 10 32Firefox 30.6 0.9 2

• CloudLab cluster of 32 machines, 1–128 cores

Experimental Setup

sources:Cookiessinks:suspiciousconnections

sources:homedirectorysinks:all

34

• Unexpected analysis:• prioritize low record overhead

• Expected analysis• periodic checkpoint• gather partitioning stats

Two Different Scenarios

35

1

10

100

1 10 100

Normalize

dSped

up

NumberofCoresGzip Ghostscript Evince MongodbNginx OpenOffice Firefox Ideal

mean:13x

Scalability of Unexpected Analysis

36

1

10

100

1 10 100

Normalize

dSpeedu

p

NumberofCoresGzip Ghostscript Evince MongodbNginx OpenOffice Firefox Ideal

mean:21x

Scalability of Expected Analysis

37

JetStream

Aggregation – pipeline parallelism

Local DIFT – epoch parallelism 2xà21x

Fasterthanoriginalexecution!

38

Questions

39

Related Work

Epoch Parallelism• Wallace and Hazelwood, “SuperPin: Parallelizing

Dynamic Instrumentation for Real-Time Performance”

Local DIFT• Ruwase et al. “Parallelizing Dynamic Information Flow

Tracking”• Nightengale et al. “Parallelizing security checks on

commodity hardware”

40

Benchmarks

Benchmark Replay Time (seconds)

JetStream Time(seconds)

Ghostscript 1.0 5.6Gzip 3.0 2.3Evince 13.5 19.5Nginx 4.8 5.6Mongodb 22.8 13.8OpenOffice 7.6 25.0Firefox 67.4 94.4

41

Aggregation Results

0

2

4

6

8

10

12

BackwardsPass(seconds)

BothPassesTime(seconds)

42

FastForward Instrumentation Analysis Pre-pruneForwardPass Prune BackwardPass

Gzip

0

0.5

1

1.5

2

2.5

3

3.5

4

0 20 40 60 80 100 120

GzipFirstQuery

0

0.5

1

1.5

2

2.5

3

3.5

4

0 20 40 60 80 100 120

GzipSecondQuery

43

FastForward Instrumentation Analysis Pre-pruneForwardPass Prune BackwardPass

OpenOffice

0

5

10

15

20

25

30

35

40

0 20 40 60 80 100 120

OpenOfficeFirstQuery

0

5

10

15

20

25

30

35

40

0 20 40 60 80 100 120

OpenOfficeSecondQuery