![Page 1: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/1.jpg)
Introduction to Data Management CSE 344
Unit 5: Parallel Data Processing
Parallel RDBMSMapReduce
Spark
(4 lectures)
![Page 2: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/2.jpg)
Introduction to Data ManagementCSE 344
Spark
CSE 344 - 2018au 2
![Page 3: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/3.jpg)
Announcement
• HW6 posted– We use Amazon Web Services (AWS)– Urgent: please sign up for AWS credits (see
instructions on the homework)
• No classes on Monday (Veterans’ Day)
CSE 344 - 2018au 3
![Page 4: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/4.jpg)
Class Overview
• Unit 1: Intro• Unit 2: Relational Data Models and Query Languages• Unit 3: Non-relational data• Unit 4: RDMBS internals and query optimization• Unit 5: Parallel query processing
– Spark, Hadoop, parallel databases
• Unit 6: DBMS usability, conceptual design• Unit 7: Transactions• Unit 8: Advanced topics (time permitting)
4
![Page 5: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/5.jpg)
Parallelism is of Increasing Importance
• Multi-cores:– Most processors have multiple cores– This trend will likely increase in the future
• Big data: too large to fit in main memory– Distributed query processing on 100x-1000x
servers– Widely available now using cloud services
CSE 344 - 2018au 5
![Page 6: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/6.jpg)
Performance Metrics for Parallel DBMSs
Nodes = processors, computers
• Speedup: – More nodes, same data è higher speed
• Scaleup:– More nodes, more data è same speed
CSE 344 - 2018au 6
![Page 7: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/7.jpg)
Linear v.s. Non-linear Speedup
CSE 344 - 2018au 7
# nodes (=P)
Speedup
×1 ×5 ×10 ×15
Ideal
![Page 8: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/8.jpg)
Linear v.s. Non-linear Scaleup
CSE 344 - 2018au 8# nodes (=P) AND data size
BatchScaleup
×1 ×5 ×10 ×15
Ideal
![Page 9: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/9.jpg)
Why Sub-linear?
• Startup cost– Cost of starting an operation on many nodes
• Interference– Contention for resources between nodes
• Skew– Slowest node becomes the bottleneck
CSE 344 - 2018au 9
![Page 10: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/10.jpg)
SparkA Case Study of the MapReduce
Programming Paradigm
CSE 344 - 2018au 10
![Page 11: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/11.jpg)
Parallel Data Processing @ 2010
CSE 344 - 2018au 11
![Page 12: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/12.jpg)
Spark
• Open source system from UC Berkeley
• Distributed processing over HDFS
• Differences from MapReduce (CSE322):
– Multiple steps, including iterations
– Stores intermediate results in main memory
– Closer to relational algebra (familiar to you)
• Details:
http://spark.apache.org/examples.html
![Page 13: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/13.jpg)
Spark
• Spark supports interfaces in Java, Scala, and
Python
– Scala: extension of Java with functions/closures
• We will illustrate use the Spark Java interface in
this class
• Spark also supports a SQL interface
(SparkSQL), and compiles SQL to its native
Java interface
CSE 344 - 2018au 13
![Page 14: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/14.jpg)
Programming in Spark• A Spark program consists of:
– Transformations (map, reduce, join…). Lazy– Actions (count, reduce, save...). Eager
• Eager: operators are executed immediately
• Lazy: operators are not executed immediately– A operator tree is constructed in memory instead– Similar to a relational algebra tree
![Page 15: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/15.jpg)
Collections in Spark
• RDD<T> = an RDD collection of type T– Distributed on many servers, not nested– Operations are done in parallel– Recoverable via lineage; more later
• Seq<T> = a sequence– Local to one server, may be nested– Operations are done sequentially
![Page 16: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/16.jpg)
ExampleGiven a large log file hdfs://logfile.logretrieve all lines that:• Start with “ERROR”• Contain the string “sqlite”
s = SparkSession.builder()...getOrCreate();
lines = s.read().textFile(“hdfs://logfile.log”);
errors = lines.filter(l -> l.startsWith(“ERROR”));
sqlerrors = errors.filter(l -> l.contains(“sqlite”));
sqlerrors.collect();
![Page 17: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/17.jpg)
ExampleGiven a large log file hdfs://logfile.logretrieve all lines that:• Start with “ERROR”• Contain the string “sqlite”
s = SparkSession.builder()...getOrCreate();
lines = s.read().textFile(“hdfs://logfile.log”);
errors = lines.filter(l -> l.startsWith(“ERROR”));
sqlerrors = errors.filter(l -> l.contains(“sqlite”));
sqlerrors.collect();
s has type JavaRDD<String>
![Page 18: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/18.jpg)
s = SparkSession.builder()...getOrCreate();
lines = s.read().textFile(“hdfs://logfile.log”);
errors = lines.filter(l -> l.startsWith(“ERROR”));
sqlerrors = errors.filter(l -> l.contains(“sqlite”));
sqlerrors.collect();
TransformationsNot executed yet…TransformationsNot executed yet…Transformation:Not executed yet…
Action:triggers executionof entire program
Given a large log file hdfs://logfile.logretrieve all lines that:• Start with “ERROR”• Contain the string “sqlite”
Example
s has type JavaRDD<String>
![Page 19: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/19.jpg)
errors = lines.filter(l -> l.startsWith(“ERROR”));
Recall: anonymous functions (lambda expressions) starting in Java 8
Example
class FilterFn implements Function<Row, Boolean>{ Boolean call (Row r) { return l.startsWith(“ERROR”); }
}
errors = lines.filter(new FilterFn());
is the same as:
![Page 20: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/20.jpg)
s = SparkSession.builder()...getOrCreate();
sqlerrors = s.read().textFile(“hdfs://logfile.log”).filter(l -> l.startsWith(“ERROR”)).filter(l -> l.contains(“sqlite”)).collect();
Given a large log file hdfs://logfile.logretrieve all lines that:• Start with “ERROR”• Contain the string “sqlite”
Example
“Call chaining” style
![Page 21: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/21.jpg)
Example
CSE 344 - 2018au 21
s = SparkSession.builder()...getOrCreate();
sqlerrors = s.read().textFile(“hdfs://logfile.log”).filter(l -> l.startsWith(“ERROR”)).filter(l -> l.contains(“sqlite”)).collect();
Error… Warning… Warning… Error… Abort… Abort… Error… Error… Warning… Error…
The RDD s:
![Page 22: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/22.jpg)
Example
CSE 344 - 2018au 22
s = SparkSession.builder()...getOrCreate();
sqlerrors = s.read().textFile(“hdfs://logfile.log”).filter(l -> l.startsWith(“ERROR”)).filter(l -> l.contains(“sqlite”)).collect();
Error… Warning… Warning… Error… Abort… Abort… Error… Error… Warning… Error…
filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”)
The RDD s: Parallel step 1
![Page 23: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/23.jpg)
Example
CSE 344 - 2018au 23
s = SparkSession.builder()...getOrCreate();
sqlerrors = s.read().textFile(“hdfs://logfile.log”).filter(l -> l.startsWith(“ERROR”)).filter(l -> l.contains(“sqlite”)).collect();
Error… Warning… Warning… Error… Abort… Abort… Error… Error… Warning… Error…
Error… Error… Error… Error… Error…
filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”)
The RDD s: Parallel step 1
![Page 24: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/24.jpg)
Example
CSE 344 - 2018au 24
s = SparkSession.builder()...getOrCreate();
sqlerrors = s.read().textFile(“hdfs://logfile.log”).filter(l -> l.startsWith(“ERROR”)).filter(l -> l.contains(“sqlite”)).collect();
Error… Warning… Warning… Error… Abort… Abort… Error… Error… Warning… Error…
Error… Error… Error… Error… Error…
filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”) filter(“ERROR”)
filter(“sqlite”) filter(“sqlite”) filter(“sqlite”) filter(“sqlite”) filter(“sqlite”)
The RDD s: Parallel step 1
Parallel step 2
![Page 25: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/25.jpg)
Fault Tolerance• When a job is executed on x100 or x1000
servers, the probability of a failure is high
• Example: if a server fails once/year, then a job with 10000 servers fails once/hour
• Different solutions:– Parallel database systems: restart. Expensive.– MapReduce: write everything to disk, redo. Slow.– Spark: redo only what is needed. Efficient.
25
![Page 26: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/26.jpg)
Resilient Distributed Datasets• RDD = Resilient Distributed Dataset
– Distributed, immutable and records its lineage– Lineage = expression that says how that relation
was computed = a relational algebra plan• Spark stores intermediate results as RDD• If a server crashes, its RDD in main memory
is lost. However, the driver (=master node) knows the lineage, and will simply recomputethe lost partition of the RDD
CSE 344 - 2018au 26
![Page 27: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/27.jpg)
Persistencelines = s.read().textFile(“hdfs://logfile.log”);errors = lines.filter(l->l.startsWith(“ERROR”));sqlerrors = errors.filter(l->l.contains(“sqlite”));sqlerrors.collect();
If any server fails before the end, then Spark must restart
![Page 28: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/28.jpg)
lines = s.read().textFile(“hdfs://logfile.log”);errors = lines.filter(l->l.startsWith(“ERROR”));sqlerrors = errors.filter(l->l.contains(“sqlite”));sqlerrors.collect();
Persistencehdfs://logfile.log
result
filter(...startsWith(“ERROR”)filter(...contains(“sqlite”)
RDD:
If any server fails before the end, then Spark must restart
![Page 29: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/29.jpg)
Persistence
If any server fails before the end, then Spark must restart
hdfs://logfile.log
result
filter(...startsWith(“ERROR”)filter(...contains(“sqlite”)
RDD:
lines = s.read().textFile(“hdfs://logfile.log”);errors = lines.filter(l->l.startsWith(“ERROR”));sqlerrors = errors.filter(l->l.contains(“sqlite”));sqlerrors.collect();
lines = s.read().textFile(“hdfs://logfile.log”);errors = lines.filter(l->l.startsWith(“ERROR”));errors.persist();sqlerrors = errors.filter(l->l.contains(“sqlite”));sqlerrors.collect()
New RDD
Spark can recompute the result from errors
![Page 30: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/30.jpg)
Persistence
If any server fails before the end, then Spark must restart
hdfs://logfile.log
result
Spark can recompute the result from errors
hdfs://logfile.log
errors
filter(..startsWith(“ERROR”)
result
filter(...contains(“sqlite”)
RDD:
filter(...startsWith(“ERROR”)filter(...contains(“sqlite”)
lines = s.read().textFile(“hdfs://logfile.log”);errors = lines.filter(l->l.startsWith(“ERROR”));errors.persist();sqlerrors = errors.filter(l->l.contains(“sqlite”));sqlerrors.collect()
New RDD
lines = s.read().textFile(“hdfs://logfile.log”);errors = lines.filter(l->l.startsWith(“ERROR”));sqlerrors = errors.filter(l->l.contains(“sqlite”));sqlerrors.collect();
![Page 31: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/31.jpg)
Example
31
SELECT count(*) FROM R, SWHERE R.B > 200 and S.C < 100 and R.A = S.A
R(A,B)S(A,C)
R = strm.read().textFile(“R.csv”).map(parseRecord).persist();S = strm.read().textFile(“S.csv”).map(parseRecord).persist();
Parses each line into an object
persisting on disk
![Page 32: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/32.jpg)
Example
32
SELECT count(*) FROM R, SWHERE R.B > 200 and S.C < 100 and R.A = S.A
R(A,B)S(A,C)
R = strm.read().textFile(“R.csv”).map(parseRecord).persist();S = strm.read().textFile(“S.csv”).map(parseRecord).persist();RB = R.filter(t -> t.b > 200).persist();SC = S.filter(t -> t.c < 100).persist();J = RB.join(SC).persist();J.count();
R
RB
filter((a,b)->b>200)
S
SC
filter((b,c)->c<100)
J
join
action
transformationstransformations
![Page 33: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/33.jpg)
Recap: Programming in Spark
• A Spark/Scala program consists of:– Transformations (map, reduce, join…). Lazy– Actions (count, reduce, save...). Eager
• RDD<T> = an RDD collection of type T– Partitioned, recoverable (through lineage), not
nested
• Seq<T> = a sequence– Local to a server, may be nested
![Page 34: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/34.jpg)
Transformations:map(f : T -> U): RDD<T> -> RDD<U>
flatMap(f: T -> Seq(U)): RDD<T> -> RDD<U>
filter(f:T->Bool): RDD<T> -> RDD<T>
groupByKey(): RDD<(K,V)> -> RDD<(K,Seq[V])>
reduceByKey(F:(V,V)-> V): RDD<(K,V)> -> RDD<(K,V)>
union(): (RDD<T>,RDD<T>) -> RDD<T>
join(): (RDD<(K,V)>,RDD<(K,W)>) -> RDD<(K,(V,W))>
cogroup(): (RDD<(K,V)>,RDD<(K,W)>)-> RDD<(K,(Seq<V>,Seq<W>))>
crossProduct(): (RDD<T>,RDD<U>) -> RDD<(T,U)>
Actions:count(): RDD<T> -> Long
collect(): RDD<T> -> Seq<T>
reduce(f:(T,T)->T): RDD<T> -> T
save(path:String): Outputs RDD to a storage system e.g., HDFS
![Page 35: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/35.jpg)
Spark 2.0
The DataFrame and Dataset Interfaces
![Page 36: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/36.jpg)
DataFrames• Like RDD, also an immutable distributed
collection of data
• Organized into named columns rather than individual objects– Just like a relation– Elements are untyped objects called Row’s
• Similar API as RDDs with additional methods– people = spark.read().textFile(…);
ageCol = people.col(“age”);ageCol.plus(10); // creates a new DataFrame
![Page 37: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/37.jpg)
Datasets• Similar to DataFrames, except that elements must be typed
objects
• E.g.: Dataset<People> rather than Dataset<Row>
• Can detect errors during compilation time
• DataFrames are aliased as Dataset<Row> (as of Spark 2.0)
• You will use both Datasets and RDD APIs in HW6
![Page 38: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/38.jpg)
Datasets API: Sample Methods• Functional API
– agg(Column expr, Column... exprs)Aggregates on the entire Dataset without groups.
– groupBy(String col1, String... cols)Groups the Dataset using the specified columns, so that we can run aggregation on them.
– join(Dataset<?> right)Join with another DataFrame.
– orderBy(Column... sortExprs)Returns a new Dataset sorted by the given expressions.
– select(Column... cols)Selects a set of column based expressions.
• “SQL” API– SparkSession.sql(“select * from R”);
• Look familiar?
![Page 39: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/39.jpg)
Introduction to Data ManagementCSE 344
Parallel Databases
CSE 344 - 2018au 39
![Page 40: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/40.jpg)
Architectures for Parallel Databases
• Shared memory
• Shared disk
• Shared nothing
CSE 344 - 2018au 40
![Page 41: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/41.jpg)
Shared Memory• Nodes share both RAM and
disk• Dozens to hundreds of
processors
Example: SQL Server runs on a single machine and can leverage many threads to speed up a query• check your HW3 query plans
• Easy to use and program• Expensive to scale
CSE 344 - 2018au 41
Interconnection Network
P P P
Global Shared Memory
D D D
![Page 42: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/42.jpg)
Shared Disk• All nodes access the same disks• Found in the largest "single-box"
(non-cluster) multiprocessors
Example: Oracle
• No more memory contention
• Harder to program• Still hard to scale: existing
deployments typically have fewer than 10 machines
CSE 344 - 2018au 42
Interconnection Network
P P P
D D D
M M M
![Page 43: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/43.jpg)
Shared Nothing• Cluster of commodity machines on
high-speed network• Called "clusters" or "blade servers”• Each machine has its own memory
and disk: lowest contention.
Example: Spark
Because all machines today have many cores and many disks, shared-nothing systems typically run many "nodes” on a single physical machine.
• Easy to maintain and scale• Most difficult to administer and tune.
CSE 344 - 2018au 43We discuss only Shared Nothing in class
Interconnection Network
P P P
D D D
M M M
![Page 44: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/44.jpg)
Purchase
pid=pid
cid=cid
Customer
ProductPurchase
pid=pid
cid=cid
Customer
Product
Approaches toParallel Query Evaluation
• Inter-query parallelism– Transaction per node– Good for transactional workloads
• Inter-operator parallelism– Operator per node– Good for analytical workloads
• Intra-operator parallelism– Operator on multiple nodes– Good for both?
CSE 344 - 2018au 44We study only intra-operator parallelism: most scalable
Purchase
pid=pid
cid=cid
Customer
Product
Purchase
pid=pid
cid=cid
Customer
Product
Purchase
pid=pid
cid=cid
Customer
Product
![Page 45: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/45.jpg)
Single Node Query Processing (Review)
Given relations R(A,B) and S(B, C), no indexes:
• Selection: σA=123(R)– Scan file R, select records with A=123
• Group-by: γA,sum(B)(R)– Scan file R, insert into a hash table using A as key– When a new key is equal to an existing one, add B to the value
• Join: R ⋈ S– Scan file S, insert into a hash table using B as key– Scan file R, probe the hash table using B
CSE 344 - 2018au 45
![Page 46: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/46.jpg)
Distributed Query Processing
• Data is horizontally partitioned on servers
• Operators may require data reshuffling
CSE 344 - 2018au 46
![Page 47: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/47.jpg)
Horizontal Data Partitioning
CSE 344 - 2018au 47
1 2 P . . .
Data: Servers:
K A B… …
![Page 48: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/48.jpg)
Horizontal Data Partitioning
CSE 344 - 2018au 48
K A B… …
1 2 P . . .
Data: Servers:
K A B
… …
K A B
… …
K A B
… …
Which tuplesgo to what server?
![Page 49: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/49.jpg)
Horizontal Data Partitioning
• Block Partition: – Partition tuples arbitrarily s.t. size(R1)≈ … ≈ size(RP)
• Hash partitioned on attribute A:– Tuple t goes to chunk i, where i = h(t.A) mod P + 1– Recall: calling hash fn’s is free in this class
• Range partitioned on attribute A:– Partition the range of A into -∞ = v0 < v1 < … < vP = ∞– Tuple t goes to chunk i, if vi-1 < t.A < vi
CSE 344 - 2018au 49
![Page 50: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/50.jpg)
Uniform Data v.s. Skewed Data
• Let R(K,A,B,C); which of the following partition methods may result in skewedpartitions?
• Block partition
• Hash-partition– On the key K
– On the attribute A
CSE 344 - 2018au 50
Uniform
Uniform
May be skewed
Assuming goodhash function
E.g. when all recordshave the same valueof the attribute A, thenall records end up in thesame partition
Keep this in mind in the next few slides
![Page 51: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/51.jpg)
Parallel Execution of RA Operators:Grouping
Data: R(K,A,B,C)Query: γA,sum(C)(R)
How to compute group by if:
• R is hash-partitioned on A ?
• R is block-partitioned ?
• R is hash-partitioned on K ?
CSE 344 - 2018au 51
![Page 52: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/52.jpg)
Parallel Execution of RA Operators:Grouping
Data: R(K,A,B,C)Query: γA,sum(C)(R)• R is block-partitioned or hash-partitioned on K
CSE 344 - 2018au 52
R1 R2 RP . . .
R1’ R2’ RP’. . .
Reshuffle Ron attribute A
Run grouping on reshuffled
partitions
![Page 53: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/53.jpg)
Speedup and Scaleup
• Consider:
– Query: γA,sum(C)(R)
– Runtime: only consider I/O costs
• If we double the number of nodes P, what is
the new running time?
– Half (each server holds ½ as many records)
• If we double both P and the size of R, what is
the new running time?
– Same (each server holds the same # of records)
CSE 344 - 2018au 53But only if the data is without skew!
![Page 54: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/54.jpg)
Skewed Data
• R(K,A,B,C)• Informally: we say that the data is skewed if
one server holds much more data that the average
• E.g. we hash-partition on A, and some value of A occurs very many times (“Justin Bieber”)
• Then the server holding that value will be skewed
CSE 344 - 2018au 54
![Page 55: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/55.jpg)
Parallel Execution of RA Operators:Partitioned Hash-Join
• Data: R(K1, A, B), S(K2, B, C)
• Query: R(K1, A, B) ⋈ S(K2, B, C)
– Initially, both R and S are partitioned on K1 and K2
CSE 344 - 2018au 55
R1, S1 R2, S2 RP, SP . . .
R’1, S’1 R’2, S’2 R’P, S’P . . .
Reshuffle R on R.Band S on S.B
Each server computesthe join locally
![Page 56: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/56.jpg)
Parallel Join IllustrationData: R(K1,A, B), S(K2, B, C)Query: R(K1,A,B) ⋈ S(K2,B,C)
CSE 344 - 2018au 56
K1 B1 202 50
K2 B101 50102 50
K1 B3 204 20
K2 B201 20202 50
R1 S1 R2 S2
K1 B1 203 204 20
K2 B201 20
K1 B2 50
K2 B101 50102 50202 50
R1’ S1’ R2’ S2’
M1 M2
M1 M2
Shuffle on B
⋈ ⋈
Partition
Local Join
![Page 57: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/57.jpg)
Broadcast Join
CSE 344 - 2018au 57
Data: R(A, B), S(C, D)Query: R(A,B) ⋈B=C S(C,D)
R1 R2 RP. . .
R’1, S R’2, S R’P, S. . .
Reshuffle R on R.B
Broadcast S
S
Why would you want to do this?
![Page 58: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/58.jpg)
Putting it Together:Example Parallel Query Plan
CSE 344 - 2018au 58
SELECT * FROM Order o, Line iWHERE o.item = i.itemAND o.date = today()
join
select
scan scan
date = today()
o.item = i.item
Order oItem i
Find all orders from today, along with the items ordered
Order(oid, item, date), Line(item, …)
![Page 59: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/59.jpg)
Example Parallel
Query Plan
CSE 344 - 2018au 59
Node 1 Node 2 Node 3
select
date=today()
select
date=today()
select
date=today()
scan
Order oscan
Order oscan
Order o
hash
h(o.item)
hash
h(o.item)
hash
h(o.item)
Node 1 Node 2 Node 3
join
select
scan
date = today()
o.item = i.item
Order o
Order(oid, item, date), Line(item, …)
![Page 60: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/60.jpg)
Example Parallel
Query Plan
CSE 344 - 2018au 60
Node 1 Node 2 Node 3
scan
Item i
Node 1 Node 2 Node 3
hash
h(i.item)
scan
Item i
hash
h(i.item)
scan
Item i
hash
h(i.item)
join
scan
date = today()
o.item = i.item
Order o
Item i
Order(oid, item, date), Line(item, …)
![Page 61: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/61.jpg)
Example Parallel Query Plan
CSE 344 - 2018au 61
Node 1 Node 2 Node 3
join join join
o.item = i.item o.item = i.item o.item = i.item
contains all orders and all
lines where hash(item) = 1
contains all orders and all
lines where hash(item) = 2
contains all orders and all
lines where hash(item) = 3
Order(oid, item, date), Line(item, …)
![Page 62: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/62.jpg)
A Challenge
• Have P number of servers (say P=27 or P=1000)
• How do we compute this Datalog query in one step?
• Q(x,y,z) :- R(x,y), S(y,z), T(z,x)
CSE 344 - 2018au 62
![Page 63: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/63.jpg)
A Challenge
• Have P number of servers (say P=27 or P=1000)
• How do we compute this Datalog query in one step?
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
• Organize the P servers into a cube with side P⅓
– Thus, each server is uniquely identified by (i,j,k), i,j,k≤P⅓
63
i
jk
(i,j,k)
P⅓1
![Page 64: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/64.jpg)
HyperCube Join
• Have P number of servers (say P=27 or P=1000)
• How do we compute this Datalog query in one step?
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
• Organize the P servers into a cube with side P⅓
– Thus, each server is uniquely identified by (i,j,k), i,j,k≤P⅓
• Step 1:
– Each server sends R(x,y) to all servers (h(x),h(y),*)
– Each server sends S(y,z) to all servers (*,h(y),h(z))
– Each server sends T(x,z) to all servers (h(x),*,h(z))
CSE 344 - 2018au 64
i
j
R(x,y)
![Page 65: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/65.jpg)
HyperCube Join
• Have P number of servers (say P=27 or P=1000)
• How do we compute this Datalog query in one step?
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
• Organize the P servers into a cube with side P⅓
– Thus, each server is uniquely identified by (i,j,k), i,j,k≤P⅓
• Step 1:
– Each server sends R(x,y) to all servers (h(x),h(y),*)
– Each server sends S(y,z) to all servers (*,h(y),h(z))
– Each server sends T(x,z) to all servers (h(x),*,h(z))
• Final output:
– Each server (i,j,k) computes the query R(x,y),S(y,z),T(z,x) locally
CSE 344 - 2018au 65
i
j
![Page 66: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/66.jpg)
HyperCube Join
• Have P number of servers (say P=27 or P=1000)
• How do we compute this Datalog query in one step?
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
• Organize the P servers into a cube with side P⅓
– Thus, each server is uniquely identified by (i,j,k), i,j,k≤P⅓
• Step 1:
– Each server sends R(x,y) to all servers (h(x),h(y),*)
– Each server sends S(y,z) to all servers (*,h(y),h(z))
– Each server sends T(x,z) to all servers (h(x),*,h(z))
• Final output:
– Each server (i,j,k) computes the query R(x,y),S(y,z),T(z,x) locally
• Analysis: each tuple R(x,y) is replicated at most P⅓
times
CSE 344 - 2018au 66
i
j
![Page 67: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/67.jpg)
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
x y1 23 2
y z4 74 9
R1 S1
P1
Shuffle
Partition
Local Join
z x1 13 3
T1x y5 47 6
y z2 32 9
R2 S2 T2x y8 69 6
y z6 76 9
R3 S3 T3
P2 P3
x y1 2
y z2 7
R1’ S1’
P1: (1, 2, 7)
x y1 2
y z2 3
R2’ S2’x y3 2
y z2 3
R3’ S3’
P2: (1, 2, 3) P3: (3, 2, 3)
z x7 1
T1z x3 1
T2 T3
Hypercube join
z x9 53 1
z x7 13 1
z x3 3
![Page 68: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/68.jpg)
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
x y
1 2
3 2
y z
4 7
4 9
R1 S1
P1
Shuffle
Partitionz x
1 1
3 3
T1x y
5 4
7 6
y z
2 3
2 9
R2 S2 T2x y
8 6
9 6
y z
6 7
6 9
R3 S3 T3
P2 P3
Hypercube join
z x
9 5
3 1
z x
7 1
3 1
What ifh(x): h(1) = h(3)
CSE 344 - 2018au 68
![Page 69: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/69.jpg)
Q(x,y,z) = R(x,y),S(y,z),T(z,x)
x y1 23 2
y z4 74 9
R1 S1
P1
Shuffle
Partition
Local Join
z x1 13 3
T1x y5 47 6
y z2 32 9
R2 S2 T2x y8 69 6
y z6 76 9
R3 S3 T3
P2 P3
x y1 23 2
y z2 7
R1’ S1’
P1: (1, 2, 7)
x y1 2
y z2 3
R2’ S2’x y3 21 2
y z2 3
R3’ S3’
P2: (1, 2, 3) P3: (3, 2, 3)
z x7 1
T1z x3 1
T2z x3 3
T3
Hypercube join
z x9 53 1
z x7 13 1
What ifh(x): h(1) = h(3)
![Page 70: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/70.jpg)
Introduction to Data ManagementCSE 344
MapReduce
CSE 344 - 2018au 70
![Page 71: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/71.jpg)
Announcements
• HW6 due tomorrow• HW7 to be posted tomorrow; HW8 next week• Makeup lectures:
– Tuesday, Nov. 27, Room EEB 105– Tuesday, Dec. 4, Room EEB 105
• Canceled lectures:– Wednesday, Dec. 5– Friday, Dec. 7
CSE 344 - 2018au 71Check the calendar!
![Page 72: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/72.jpg)
Parallel Data Processing @ 2000
CSE 344 - 2018au 72
![Page 73: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/73.jpg)
Optional Reading
• Original paper:https://www.usenix.org/legacy/events/osdi04/tech/dean.html
• Rebuttal to a comparison with parallel DBs:http://dl.acm.org/citation.cfm?doid=1629175.1629198
• Chapter 2 (Sections 1,2,3 only) of Mining of Massive Datasets, by Rajaraman and Ullmanhttp://i.stanford.edu/~ullman/mmds.html
CSE 344 - 2018au 73
![Page 74: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/74.jpg)
Motivation• We learned how to parallelize relational database
systems
• While useful, it might incur too much overhead if our query plans consist of simple operations
• MapReduce is a programming model for such computation
• First, let’s study how data is stored in such systems
CSE 344 - 2018au 74
![Page 75: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/75.jpg)
Distributed File System (DFS)
• For very large files: TBs, PBs• Each file is partitioned into chunks, typically
64MB• Each chunk is replicated several times (≥3),
on different racks, for fault tolerance• Implementations:
– Google’s DFS: GFS, proprietary– Hadoop’s DFS: HDFS, open source
CSE 344 - 2018au 75
![Page 76: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/76.jpg)
MapReduce
• Google: paper published 2004• Free variant: Hadoop
• MapReduce = high-level programming model and implementation for large-scale parallel data processing
CSE 344 - 2018au 76
![Page 77: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/77.jpg)
Typical Problems Solved by MR
• Read a lot of data• Map: extract something you care about from each
record• Shuffle and Sort• Reduce: aggregate, summarize, filter, transform• Write the results
CSE 344 - 2018au 77
Paradigm stays the same,change map and reduce functions for different problems
slide source: Jeff Dean
![Page 78: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/78.jpg)
Data ModelFiles!
A file = a bag of (key, value) pairs
A MapReduce program:• Input: a bag of (inputkey, value) pairs• Output: a bag of (outputkey, value) pairs
CSE 344 - 2018au 78
![Page 79: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/79.jpg)
Step 1: the MAP Phase
User provides the MAP-function:• Input: (input key, value)• Ouput: bag of (intermediate key, value)
System applies the map function in parallel to all (input key, value) pairs in the input file
CSE 344 - 2018au 79
![Page 80: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/80.jpg)
Step 2: the REDUCE Phase
User provides the REDUCE function:• Input: (intermediate key, bag of values)• Output: bag of output (values)
System groups all pairs with the same intermediate key, and passes the bag of values to the REDUCE function
CSE 344 - 2018au 80
![Page 81: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/81.jpg)
Example
• Counting the number of occurrences of each word in a large collection of documents
• Each Document– The key = document id (did)– The value = set of words (word)
81
map(String key, String value):// key: document name// value: document contentsfor each word w in value:
EmitIntermediate(w, “1”);
reduce(String key, Iterator values):// key: a word// values: a list of countsint result = 0;for each v in values:
result += ParseInt(v);Emit(AsString(result));
![Page 82: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/82.jpg)
MAP REDUCE
(w1,1)
(w2,1)
(w3,1)
…
(w1,1)
(w2,1)
…
(did1,v1)
(did2,v2)
(did3,v3)
. . . .
(w1, (1,1,1,…,1))
(w2, (1,1,…))
(w3,(1…))
…
…
…
…
(w1, 25)
(w2, 77)
(w3, 12)
…
…
…
…
Shuffle
CSE 344 - 2018au 82
![Page 83: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/83.jpg)
Comparison to Spark
MapReduce in Spark:
• col.flatMap(f) applies in parallel the predicate f to all elements x of the partitioned collection; for each x, f(x) is a list; then flatMap returns their concatenation
• col.reduceByKey(g) applies in parallel the function g to all elements with a common key
83
![Page 84: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/84.jpg)
Jobs v.s. Tasks
• A MapReduce Job– One single “query”, e.g. count the words in all docs– More complex queries may consists of multiple jobs
• A Map Task, or a Reduce Task– A group of instantiations of the map-, or reduce-
function, which are scheduled on a single worker
CSE 344 - 2018au 84
![Page 85: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/85.jpg)
Workers
• A worker is a process that executes one task at a time
• Typically there is one worker per processor, hence 4 or 8 per node
CSE 344 - 2018au 85
![Page 86: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/86.jpg)
Fault Tolerance
MapReduce handles fault tolerance by writing
intermediate files to disk:
• Mappers write file to local disk
• Reducers read the files (=reshuffling); if the
server fails, the reduce task is restarted on
another server
CSE 344 - 2018au 86
![Page 87: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/87.jpg)
MAP Tasks (M) REDUCE Tasks (R)
(w1,1)
(w2,1)
(w3,1)
…
(w1,1)
(w2,1)
…
(did1,v1)
(did2,v2)
(did3,v3)
. . . .
(w1, (1,1,1,…,1))
(w2, (1,1,…))
(w3,(1…))
…
…
…
…
(w1, 25)
(w2, 77)
(w3, 12)
…
…
…
…
Shuffle
CSE 344 - 2018au 87
![Page 88: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/88.jpg)
MapReduce Execution Details
CSE 344 - 2018au 88
Map
(Shuffle)
Reduce
Data not necessarily local
Intermediate datagoes to local disk:M × R files (why?)
Output to disk, replicated in cluster
File system: GFS or HDFS
Task
Task
![Page 89: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/89.jpg)
Local storage`
MapReduce Phases
CSE 344 - 2018au 89
![Page 90: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/90.jpg)
Implementation
• There is one master node
• Master partitions input file into M splits, by key
• Master assigns workers (=servers) to the M map tasks, keeps track of their progress
• Workers write their output to local disk, partition
into R regions• Master assigns workers to the R reduce tasks• Reduce workers read regions from the map
workers’ local disks
CSE 344 - 2018au 90
![Page 91: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/91.jpg)
Interesting Implementation Details
Worker failure:
• Master pings workers periodically,
• If down then reassigns the task to another worker
CSE 344 - 2018au 91
![Page 92: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/92.jpg)
Interesting Implementation Details
Backup tasks:• Straggler = a machine that takes unusually long
time to complete one of the last tasks. E.g.:– Bad disk forces frequent correctable errors (30MB/s à
1MB/s)– The cluster scheduler has scheduled other tasks on
that machine• Stragglers are a main reason for slowdown• Solution: pre-emptive backup execution of the
last few remaining in-progress tasks
CSE 344 - 2018au 92
![Page 93: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/93.jpg)
Straggler Example
CSE 344 - 2018au 93
time
Worker 3
Worker 2
Worker 1
Straggler
Backup execution
Killed
Killed
![Page 94: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/94.jpg)
Using MapReduce in Practice:
Implementing RA Operators in MR
![Page 95: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/95.jpg)
Relational Operators in MapReduce
Given relations R(A,B) and S(B, C) compute:
• Selection: σA=123(R)
• Group-by: γA,sum(B)(R)
• Join: R ⋈ S
CSE 344 - 2018au 95
![Page 96: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/96.jpg)
Selection σA=123(R)
96
map(Tuple t):if t.A = 123:
EmitIntermediate(t.A, t);
reduce(String A, Iterator values):for each v in values:
Emit(v);
At1 23t2 123t3 123t4 42
(123, [ t2, t3 ] )
( t2, t3 )
![Page 97: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/97.jpg)
Selection σA=123(R)
97
map(String value):if value.A = 123:
EmitIntermediate(value.key, value);
reduce(String k, Iterator values):for each v in values:
Emit(v);No need for reduce.But need system hacking in Hadoopto remove reduce from MapReduce
![Page 98: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/98.jpg)
Group By γA,sum(B)(R)
98
map(Tuple t):EmitIntermediate(t.A, t.B);
reduce(String A, Iterator values):s = 0for each v in values:
s = s + vEmit(A, s);
A Bt1 23 10t2 123 21t3 123 4t4 42 6
(23, [ t1 ] )(42, [ t4 ] )(123, [ t2, t3 ] )
( 23, 10 ), ( 42, 6 ), (123, 25)
![Page 99: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/99.jpg)
Join
Two simple parallel join algorithms:
• Partitioned hash-join (we saw it, will recap)
• Broadcast join
CSE 344 - 2018au 99
![Page 100: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/100.jpg)
Partitioned Hash-Join
CSE 344 - 2018au 100
R1, S1 R2, S2 RP, SP . . .
R’1, S’1 R’2, S’2 R’P, S’P . . .
Reshuffle R on R.Band S on S.B
Each server computesthe join locally
Initially, both R and S are horizontally partitioned
R(A,B) ⋈B=C S(C,D)
![Page 101: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/101.jpg)
Partitioned Hash-Join
102
map(Tuple t):case t.relationName of
‘R’: EmitIntermediate(t.B, t);‘S’: EmitIntermediate(t.C, t);
reduce(String k, Iterator values):R = empty; S = empty;for each v in values:
case v.relationName of:‘R’: R.insert(v)‘S’: S.insert(v);
for v1 in R, for v2 in SEmit(v1,v2);
R(A,B) ⋈B=C S(C,D)
actual tuple
A Bt1 10 5t2 20 6t3 20 5
C Dt4 5 2t5 6 10
R S
(5, [ t1, t3, t4 ] )(6, [ t2, t5 ] )
(t1, t4), (t3, t4)(t2, t5)
![Page 102: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/102.jpg)
Broadcast Join
CSE 344 - 2018au 103
R1 R2 RP. . .
R’1, S R’2, S R’P, S. . .
Reshuffle R on R.B
Broadcast S
S
R(A,B) ⋈B=C S(C,D)
![Page 103: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/103.jpg)
Broadcast Join
104
map(String value):open(S); /* over the network */hashTbl = new()for each w in S:
hashTbl.insert(w.C, w)close(S);
for each v in value:for each w in hashTbl.find(v.B)
Emit(v,w); reduce(…):/* empty: map-side only */
map should readseveral records of R:value = some group
of records
Read entire table S,build a Hash Table
R(A,B) ⋈B=C S(C,D)
![Page 104: Introduction to Data Management CSE 344 · •E.g.: Datasetrather than Dataset •Can detect errors during compilation time •DataFramesare aliased as Dataset(as](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed3826162e058372d439c07/html5/thumbnails/104.jpg)
Summary
• MapReduce: simple abstraction, distribution, fault tolerance
• Load balancing (needed for speedup): by dynamic task allocation; skew still possible
• Writing intermediate results to disk is necessary for fault tolerance, but very slow.
• Spark replaces this with “Resilient Distributed Datasets” = main memory + lineage