parallel databases - new mexico state university · 2016. 4. 5. · database management systems ii,...

Database Management Systems II, Huiping Cao! 1!

Parallel Databases!

2!Database Management Systems II, Huiping Cao!

DBMS: The || Success Story!

q  Why parallel access to data"q  Divide a big problem to many smaller ones to be solved in parallel"

q  DBMSs are the most (only?) successful application of parallelism."q  Every major DBMS vendor has some parallel server"q  Workstation manufacturers now depend on parallel DB server sales."

q  Reasons for success"q  Bulk-processing (= partition parallelism)"q  Natural pipelining"q  Inexpensive hardware can do the trick"q  Users/app-programmers do not need to think in parallel"


Parallel DBMS: Intro!

q  Parallelism is natural to DBMS processing"q  Pipeline parallelism: many machines each doing one step in

a multi-step process. "q  Partition parallelism: many machines doing the same thing to

different pieces of data."q  Both are natural in DBMS!!

Pipeline

Partition

Any Sequential Program


Sequential Sequential Sequential

Sequential



outputs split N ways, inputs merge M ways


Some Parallel Terminology!

q  Speed-Up"q  More resources means

proportionally less time for given amount of data."

"q  Scale-Up"

q  If resources increased in proportion to increase in data size, time is constant."

degree of ||-ism (# of CPUs)"Fixed DB size"

Xact

/sec

."(th

roug

hput

)"

Ideal!

degree of ||-ism "(# of CPUs, DB size)"

sec.

/Xac

t"(re

spon

se ti

me)"

Ideal!


Architecture Issue: Shared What?!Shared Memory

(SMP) Shared Disk Shared Nothing

(network)

CLIENTS CLIENTS CLIENTS

Memory Processors

Easy to program Expensive to build Difficult to scaleup

Hard to program Cheap to build Easy to scaleup

Sequent, SGI, Sun VMScluster, Sysplex Tandem, Teradata, SP2


Different Types of DBMS ||-ism!

q  Intra-operator parallelism"q  get all machines working to compute a given operation

(scan, sort, join)"q  Inter-operator parallelism"

q  each operator may run concurrently on a different site (exploits pipelining)"

q  Inter-query parallelism"q  different queries run on different sites"

q  We focus on intra-operator ||-ism "


Automatic Data Partitioning!

Partitioning a table: Range Hash Round Robin

Shared disk and memory less sensitive to partitioning, Shared nothing benefits from "good" partitioning

A...E F...J K...N O...S T...Z A...E F...J K...N O...S T...Z A...E F...J K...N O...S T...Z

Good for equijoins, range queries group-by

Good for equijoins Good to spread load


Parallel operators!

q  Scan"q  Sorting"q  Joins"


Parallel Scans!

q  Scan in parallel, and merge."q  Selection may not require all sites for range or hash partitioning."


Parallel Sorting!

q  Idea: "q  Scan in parallel, and range-partition as you go."q  As tuples come in, begin “local” sorting on each"q  Resulting data is sorted, and range-partitioned."q  Problem: skew!!q  Solution: “sample” the data at start to determine partition points. !


Parallel Joins!

q  Nested loop:"q  Each outer tuple must be compared with each inner tuple

that might join."q  Easy for range partitioning on join cols, hard otherwise!"

q  Sort-Merge (or plain Merge-Join):"q  Sorting gives range-partitioning."

! But what about handling 2 skews?"q  Merging partitioned tables is local."


Parallel Hash Join!

q  In first phase, partitions get distributed to different sites:"q  A good hash function automatically distributes work evenly!"

q  Almost always the winner for equi-join."q  Range partition à parallel sort-merge join "q  The size of each partition is still large: further join Ai and Bi in

parallel"

Original Relations (R then S)

OUTPUT

2

B main memory buffers Disk Disk

INPUT 1

hash function

h B-1

Partitions

1 2

B-1 . . .!

Phas

e 1"


Dataflow Network for Parallel Join!

q  Good use of split/merge makes it easier to build parallel versions of sequential join code."


Complex Parallel Query Plans !

q  Complex Queries: Inter-Operator parallelism"q  Pipelining between operators: "

! note that sort and phase 1 of hash-join block the pipeline!!"q  Bushy Trees"

A" B" R" S"

Sites 1-4"Sites 5-8"

Sites 1-8"


N×M-way Parallelism!

A...E F...J K...N O...S T...Z

Merge

Join

Sort

Join

Sort

Join

Sort

Join

Sort

Join

Sort

Merge Merge

N inputs, M outputs, no bottlenecks. Partitioned Data Partitioned and Pipelined Data Flows


Observations!

q  It is hard to write a robust and world-class parallel query optimizer."q  There are many tricks."q  One quickly hits the complexity barrier."q  Still open research!"


Parallel Query Optimization!

q  Common approach: 2 phases"q  Pick best sequential plan (System R algorithm)"q  Pick degree of parallelism based on current system parameters."

q  “Bind” operators to processors"q  Take query tree, “decorate” as in previous picture."


q  Best serial plan != Best || plan! Why?"q  Trivial counter-example:"

q  Table partitioned with local secondary index at two nodes"q  Range query: all of node 1 and 1% of node 2."q  Node 1 should do a scan of its partition."q  Node 2 should use secondary index."

What is Wrong With That?!

N..Z

Table Scan

A..M

Index Scan


Parallel DBMS Summary!

q  ||-ism natural to query processing:"q  Both pipeline and partition ||-ism!"

q  Shared-Nothing vs. Shared-Memory"q  Shared-disk too, but less standard"q  Shared-memory easy, costly. Does not scaleup."q  Shared-nothing cheap, scales well, harder to implement."

q  Intra-op, Inter-op, & Inter-query ||-ism all possible."


|| DBMS Summary, cont.!

q  Data layout choices important!"q  Most DB operations can be done partition-||"

q  Sort."q  Sort-merge join, hash-join."

q  Complex plans. "q  Allow for pipeline-||ism, but sorts, hashes block the pipeline."q  Partition ||-ism achieved via bushy trees."


|| DBMS Summary, cont.!

q  Hardest part of the equation: optimization."q  2-phase optimization simplest, but can be ineffective."q  More complex schemes still at the research stage."

q  We have not said anything about Xacts, logging."q  Easy in shared-memory architecture."q  Takes some care in shared-nothing."


What Systems Work This Way!

Shared Nothing Teradata: 400 nodes Tandem: 110 nodes IBM / SP2 / DB2: 128 nodes Informix/SP2 48 nodes ATT & Sybase ? nodes

Shared Disk Oracle 170 nodes DEC Rdb 24 nodes

Shared Memory Informix 9 nodes RedBrick ? nodes

CLIENTS

MemoryProcessors

CLIENTS

CLIENTS

(as of 9/1995)"


Jim Gray & Gordon Bell: VLDB 95 Parallel Database Systems Survey

Parallel Aggregates!

A...E F...J K...N O...S T...Z

A Table

Count Count Count Count Count

Count

q  For each aggregate function, need a decomposition:"q  count(S) = Σ count(s(i)), differs for sum()"q  avg(S) = (Σ sum(s(i))) / Σ count(s(i))"q  and so on..."

q  For groups:"q  Sub-aggregate groups close to the source."q  Pass each sub-aggregate to its group’s site."

! Chosen via a hash function."

parallel databases - new mexico state university · 2016. 4. 5. · database management systems ii,...

Documents