JetStream:Cluster-scaleParallelizationofInformationFlowQueries
AndrewQuinn,DavidDevecsery,PeterChenandJasonFlinn
2
• DIFT instruments execution to track causality• Also known as “Taint-Tracking”
Dynamic Information Flow Tracking
Sources (Inputs)
Sinks (Outputs)
3
o1
DIFT for Debugging
o1Server
4
DIFT for Debugging
o1
Outputs
Inputs
Server
5
DIFT for Debugging
o1
Outputs
Inputs
Server
6
DIFT for Debugging
o1
Outputs
Inputs
Server
7
DIFT for Debugging
o1
Outputs
Inputs
Server
8
DIFT – limitations
Arnold ‘14 Overheads ~100x
X-Ray ‘12 Long queries
TaintDroid ‘10 No native code
Backtracker ‘03 Coarse-grained causality
9
Parallelize DIFT
10
Parallelizing DIFT is HARD
A = read()B = read()C = A + BD = X + Y E = CB = 0Z = A[D]F = E write(F)
SequentialDependencies
11
Parallelizing DIFT is HARD
A = read()B = read()C = A + BD = X + Y E = CB = 0Z = A[D]F = E write(F)
SequentialDependencies
Speck (ASPLOS ‘08)
Parallel Lifeguards (ASPLOS ‘08)
12
Parallelizing DIFT is HARD
A = read()B = read()C = A + BD = X + Y E = CB = 0Z = A[D]F = E write(F)
SequentialDependencies
Speck (ASPLOS ‘08)
Parallel Lifeguards (APSLOS ‘08)
“EmbarrassinglySequential”- Ruwase etal.
13
JetStream
Aggregation – pipeline parallelism
Local DIFT – epoch parallelism 2xà21x
Fasterthanoriginalexecution!
14
• Design of JetStream• Local DIFT• Aggregation
• Evaluation
Outline
15
Debugging Query
Outputs
Inputs
16
Local DIFT
Outputs
Inputs
Time slice execution into Epochs
17
Local DIFT
Outputs
Inputs
Leverage Record and Replay to calculate DIFT in parallel
18
Local DIFT
Outputs
Inputs
Track mapping between all intermediate locations
19
Local DIFT
Outputs
Inputs
Mapping is too expensive to calculate:• log operations• defer calculating relationships until aggregation
20
DIFT
D = B + C C = A[D]
• Fast Forward: replay execution until start of epoch• Analysis: log operations using a graph
CAFast Forward
Analysis
B D
CA B D
21
DIFT
D = B + C C = A[D]
• Fast Forward: replay execution until start of epoch• Analysis: log operations using a graph
CAFast Forward
Analysis
B D
CA B D
22
DIFT
D = B + C C = A[D]
• Fast Forward: replay execution until start of epoch• Analysis: log operations using a graph
CAFast Forward
Analysis
B D
CA B D
23
Local DIFT output
Outputs
Inputs
24
JetStreamLocal DIFT – epoch parallelism
Aggregation – pipeline parallelism
25
Aggregation
Outputs
Inputs
Calculate paths between source and sinks• Many nodes are not on path between source and sink• Use sequential information to prune work
26
Forward Pass
Outputs
Inputs
Pass locations which are derived from a source
derivedlocations
27
Forward Pass
Outputs
Inputs
Pass locations which are derived from a source
derivedlocations
28
Backward Pass
Outputs
Inputs
usedlocations
Pass locations which are used by a sink
29
Backward Pass
Outputs
Inputs
usedlocations
Pass locations which are used by a sink
30
In the paper:• Insights about why naïve approaches fail• Partitioning – challenging to predict the local DIFT time • Pre-pruning – garbage collection of the graph
JetStream
31
• Evaluation
Outline
32
• CloudLab cluster of 32 machines, 1–128 cores
Experimental Setup
Benchmark Sequential DIFT Time (Minutes)
Sources(millions)
Sinks(millions)
Ghostscript 1.3 3 0.2Gzip 1.8 64 488Evince 3.9 10 104Nginx 3.3 10 35Mongodb 5.2 9 117OpenOffice 7.0 10 32Firefox 30.6 0.9 2
33
Benchmark Sequential DIFT Time (Minutes)
Sources(millions)
Sinks(millions)
Ghostscript 1.3 3 0.2Gzip 1.8 64 488Evince 3.9 10 104Nginx 3.3 10 35Mongodb 5.2 9 117OpenOffice 7.0 10 32Firefox 30.6 0.9 2
• CloudLab cluster of 32 machines, 1–128 cores
Experimental Setup
sources:Cookiessinks:suspiciousconnections
sources:homedirectorysinks:all
34
• Unexpected analysis:• prioritize low record overhead
• Expected analysis• periodic checkpoint• gather partitioning stats
Two Different Scenarios
35
1
10
100
1 10 100
Normalize
dSped
up
NumberofCoresGzip Ghostscript Evince MongodbNginx OpenOffice Firefox Ideal
mean:13x
Scalability of Unexpected Analysis
36
1
10
100
1 10 100
Normalize
dSpeedu
p
NumberofCoresGzip Ghostscript Evince MongodbNginx OpenOffice Firefox Ideal
mean:21x
Scalability of Expected Analysis
37
JetStream
Aggregation – pipeline parallelism
Local DIFT – epoch parallelism 2xà21x
Fasterthanoriginalexecution!
38
Questions
39
Related Work
Epoch Parallelism• Wallace and Hazelwood, “SuperPin: Parallelizing
Dynamic Instrumentation for Real-Time Performance”
Local DIFT• Ruwase et al. “Parallelizing Dynamic Information Flow
Tracking”• Nightengale et al. “Parallelizing security checks on
commodity hardware”
40
Benchmarks
Benchmark Replay Time (seconds)
JetStream Time(seconds)
Ghostscript 1.0 5.6Gzip 3.0 2.3Evince 13.5 19.5Nginx 4.8 5.6Mongodb 22.8 13.8OpenOffice 7.6 25.0Firefox 67.4 94.4
41
Aggregation Results
0
2
4
6
8
10
12
BackwardsPass(seconds)
BothPassesTime(seconds)
42
FastForward Instrumentation Analysis Pre-pruneForwardPass Prune BackwardPass
Gzip
0
0.5
1
1.5
2
2.5
3
3.5
4
0 20 40 60 80 100 120
GzipFirstQuery
0
0.5
1
1.5
2
2.5
3
3.5
4
0 20 40 60 80 100 120
GzipSecondQuery
43
FastForward Instrumentation Analysis Pre-pruneForwardPass Prune BackwardPass
OpenOffice
0
5
10
15
20
25
30
35
40
0 20 40 60 80 100 120
OpenOfficeFirstQuery
0
5
10
15
20
25
30
35
40
0 20 40 60 80 100 120
OpenOfficeSecondQuery