chapter 12: query processing - · pdf fileremember the architecture of a b+ tree index height...

Database System Concepts, 6th Ed.

©Silberschatz, Korth and Sudarshan

See www.db-book.com for conditions on re-use

Chapter 12: Query Processing

http://www.db-book.com/

©Silberschatz, Korth and Sudarshan12.2Database System Concepts - 6th Edition

Basic Steps in Query Processing

1. Parsing and translation

2. Optimization

3. Evaluation


Basic Steps in Query Processing

1. Parsing and translation

• translate the query into its internal form. This is then

translated into relational algebra.

• Parser checks syntax, verifies names, relations, functions, etc.

3. Evaluation

• The query-execution engine takes a query-evaluation plan,

executes that plan, and returns the answers to the query.


Basic Steps in Query Processing :

Optimization

An SQL query can be translated into multiple relational algebra

expressions, e.g.

salary75000(salary(instructor))

is equivalent to

salary(salary75000(instructor))

Furthermore, each relational algebra operation can be evaluated

using one of several different algorithms!


Optimization

Example:

We can use an index on salary to find instructors with salary <

75000,

or we can perform a complete (serial) relation scan and discard

instructors with salary 75000

The sequence of annotated relational algebra expressions is called

an evaluation plan.

An annotated relational algebra

expression specifying the

detailed evaluation strategy is

called an evaluation primitive


The goal of Query Optimization: From all equivalent

evaluation plans, choose the one with the lowest cost.

How do we measure cost?


12.2 Measures of Query Cost

How do we measure cost?

• Number of tuples in each relation

• Size of tuples

• Number of disk access operations

• CPU time

• RAM space needed

• Time needed for data transfer over a network (LAN, SAN)

• Etc.


12.2 Measures of Query Cost

Time cost = total elapsed time for answering the query

Many factors contribute to time cost …

Typically, disk access is the predominant time cost (also relatively easy

to estimate!).

Measured by taking into account

Number of seeks * average-seek-cost

Number of blocks read * average-block-read-cost

Number of blocks written * average-block-write-cost

Cost to write a block is greater than cost to read a block b/c data is read

back after being written to ensure that the write was free of errors!

Example on next slide


Testing a 32-GB flash drive


Measures of Query Cost

Simplification: we just use the number of block transfers

from disk and the number of seeks as the cost measures

tT – time to transfer one block

tS – time for one seek

Cost for b block transfers plus S seeks

b * tT + S * tS

Simplification: we ignore CPU costs (time)

Real systems do take CPU cost into account

Simplification: we do not include cost to writing the output

to disk. Why? All execution plans we must choose from

end up with the same data …

4 ms0.1 ms


QUIZ: Measures of Query Cost


b * tT + S * tS

If the size of a block is 8 KB, and the transfer rate is

100 MB/s, calculate the block transfer time.

≈ 4 ms≈ 0.1 ms

Today’s HDDs are capable of getting burst rates of

~150MB/s and average read/writes of 60-80MB/s.




b * tT + S * tS

If the size of a block is 8 KB, and the transfer rate is

200 MB/s, calculate the block transfer time.

8 KB / 100 MB/s = 8/(100*1024) s = 0.0000195 s =

= 0.0781 ms ≈ 0.1 ms

≈ 4 ms≈ 0.1 ms




b * tT + S * tS

Evaluation of a query requires 3 disk seeks, each

followed by transfer of 50 blocks.

What is the total time cost?

≈ 4 ms≈ 0.1 ms




b * tT + S * tS

Evaluation of a query requires 3 disk seeks, each

followed by transfer of 50 blocks.

What is the total time cost?

3*50* 0.1 ms + 3*4 ms = 15 ms + 12 ms = 27 ms

≈ 4 ms≈ 0.1 ms


Perspective: Transfer Rates

Theoretical maximum for SATA: 600 MB/s (SATA III), but in

practice ~100-200 MB/s

SATA Express announced in 2014: 2 GB/s (theoretical)

PCI Express (PCIe) 3.0 bus: ~1 GB/s per lane, with typical 16x

configurations :

SSDs connected through PCI have surpassed 1 GB/s

SATA = Serial ATA (Advanced Technology Attachment)

PCI = Periferal Component Interconnect

http://www.tested.com/tech/457440-theoretical-vs-actual-bandwidth-pci-express-and-thunderbolt/

http://www.tested.com/tech/457440-theoretical-vs-actual-bandwidth-pci-express-and-thunderbolt/


12.3 Selection Operation

Selection w/o an index is a file scan

Algorithm A1 (linear search). Scan each file block and test all records to see whether they satisfy the selection condition.

• Cost: tS + br block transfers

br denotes number of blocks containing records from relation r

If selection is on a key attribute, can stop on finding that record

• Cost = tS + (br /2) block transfers

Linear search can be applied regardless of

selection condition, or

ordering of records in the file, or

availability of indicesaverage


Extra-credit


QUIZ: A1 Linear search

A table has 50,000 tuples, each 100 Bytes in length.

What is the average time cost using linear search?

State your assumptions!


Solution

A table has 50,000 tuples, each 100 Bytes in length.

What is the average time cost using linear search?

State your assumptions!

The table has a total of 50,000*100 = 5 million Bytes

5 million B / 4096 B/block = 1,220.7 →1,221 blocks

Assuming the section is on a key attribute (unique):

1,221 / 2 = 610.5 → 611 blocks on average

611 * 0.1 ms + 4 ms = 65.1 ms


Selection Operation

Why not use binary search?

Generally, it does not make sense, because:

Data is not stored in sorted order of the search key

• except when it’s the primary key, or there is an index available

Binary search requires more seeks than index search


Remember the architecture of a B+ tree

index

Height of index

tree, denoted hion the following

slides

Shouldn’t it be hi+1 instead?

Yes, but see the skipped section

11.4 B+ Tree File Organization!


Selections Using Indices

Index scan – search algorithms that use an index

Selection condition must be on search-key of index!

A2 (primary B+ tree index, equality on key). Retrieve a single record

that satisfies the corresponding equality condition

• Cost = (hi + 1) * (tT + tS)

A3 (primary B+ tree index, equality on nonkey) Retrieve multiple

records (duplicates).

• Records will be in consecutive blocks (Why?)

Let b = number of blocks containing matching records

• Cost = hi * (tT + tS) + tS + tT * b

Typo: text formula is missing this seek time!


Extra-credit


QUIZ: A3

A table has 50,000 tuples, each 100 Bytes in length. The

index is a B+ tree with n = 100. In our table, there are no

more than 1000 duplicates for any nonkey value.

What is the average time cost using A3 (primary B+ tree

index, equality on nonkey)?

Hint: Find the height of the tree first!


Selections Using Indices

A4 (secondary B+ tree index, equality on nonkey).

• Retrieve a single record if search-key is candidate key

Cost = (hi + 1) * (tT + tS) (Same as A2)

• Retrieve multiple records if search-key is not candidate key

Each of n matching records may be on a different block

Cost = (hi + n) * (tT + tS)

– Can be very expensive, since it may require in the

worst case an I/O for each of the n records!

– Linear file scan may be cheaper!

Don’t confuse with the

number of blocks b from

the previous fmlas.!


Selections Involving Comparisons

Can implement selections of the form AV (r) or A V(r) by using

• a linear file scan,

• or by using indices in the following ways:

A5 (primary B+ tree index, comparison - Relation is sorted on the attribute A)

• For A V(r) use index to find first tuple v and scan relation sequentially from there

• For AV (r) just scan relation sequentially until first tuple > v; do not use index

• Cost = hi * (tT + tS) + tS + tT * b (same as A3)


Selections Involving Comparisons

A6 (secondary B+ tree index, comparison).

• For A V(r) use index to find first index entry v and scan index sequentially from there, to find pointers to records.

• For AV (r) just scan leaf pages of index finding pointers to records, until first entry > v

• In either case, retrieve records that are pointed to

• Cost = (hi + n) * (tT + tS) (same as A4)

– Requires an I/O for each of the n records!

– Linear file scan may be cheaper!


Food for thought:

Try to derive similar cost formulas for

hash indices!


QUIZ: Query Processing

What are the 3 main steps in query processing?

What is an evaluation primitive?

What is a query evaluation plan?


QUIZ: Find the best algorithm!

The query optimizer receives the query:

SELECT ID, name FROM student

WHERE tot_credit > 50;

The following facts are known about the relation student:

It has 100,000 records, each 200 Bytes in length.

Its file is not sorted according to tot_credit.

The tot_credit values are uniformly distributed between 10 and

200.

Hint: What fraction of the records are returned by the query (on

average)?

There is a B+ tree index on tot_credit, with n = 90.

Estimate the execution time for linear scan and index-based search.

(Hint: Which algorithm?)

Which one will the query optimizer choose?


QUIZ: Find the best algorithm!

Answers:

Nr. of blocks for entire relation is b = 2,442

Assuming block size 8 KB

Height of B+ tree is hi = 4

Nr. of records is n = 100,000∙200−50

200−10= 78,948

Index algorithm is A6 (secondary B+ index, comparison)

Cost estimates:

248.2 ms for linear scan

10.028 s for B+ tree index


SKIP section

12.3.3 Implementation of Complex

Selections

EOL 1


12.4 Sorting

Why sort?

Instead, we may build an B+ tree index on the relation, and

then use the index to read the relation in sorted order.

Unfortunately, this may lead in the worst case to one disk

block access for each tuple:

• Cost = (hi + n) * (tT + tS) (same as A4 and A6)

Requires an I/O for each of the n records!

Linear file scan may be cheaper!


12.4 Sorting

For relations that fit in memory, algorithms like quicksort

can be used.

What is the “Big-Oh” of Quicksort?

What operations are we counting in the formula above?

If the relation doesn’t fit in memory, sorting is known as

external sorting

External sort-merge is a good algorithm.


External Sort-Merge

1. Create sorted runs. Let i be 0 initially.

Repeatedly do the following till the end of the relation:

(a) Read M blocks of relation into memory

(b) Sort the in-memory blocks

(c) Write sorted data to run Ri; increment i.

Let the final value of i be N

2. Merge the runs (next slide)…..

Let M denote the memory buffer size (in blocks).


External Sort-Merge (Cont.)

2. Merge the runs (N-way merge). We assume for now that N < M.

Use N blocks of memory to buffer input runs, and 1 block to buffer

output. Read the first block of each run into its buffer blocks.

repeat

• Select the first record (in sort order) among all buffer blocks

• Write the record to the output buffer. If the output buffer is

full write it to disk.

• Delete the record from its input buffer block.

If the buffer block becomes empty then read into the buffer

the next block (if any) of the run.

until all input buffer blocks are empty


External Sort-Merge (Cont.)

3. If N M, several merge passes are required.

In each pass, contiguous groups of M - 1 runs are

merged.

A pass reduces the number of runs by a factor of M -1,

and creates runs longer by the same factor.

E.g. If M=11, and there are 90 runs, one pass

reduces the number of runs to 9, each 10 times the

size of the initial runs

Repeated passes are performed till all runs have been

merged into one.


Example: External Sorting Using Sort-Merge

g

a

d 31

c 33

b 14

e 16

r 16

d 21

m 3

p 2

d 7

a 14

a 14

a 19

b 14

c 33

d 7

d 21

d 31

e 16

g 24

m 3

p 2

r 16

a 19

b 14

c 33

d 31

e 16

g 24

a 14

d 7

d 21

m 3

p 2

r 16

a 19

d 31

g 24

b 14

c 33

e 16

d 21

m 3

r 16

a 14

d 7

p 2initial

relationcreateruns

mergepass–1

mergepass–2

runs runssortedoutput

24

19

Fo

r sim

plic

ity,

we

assu

me

th

at th

e

entire

blo

ck o

nly

hold

s o

ne tuple

.

M =

3, N

= 4


Cost analysis of Merge Sort

Total number of block transfers (reads + writes):

br ( 2 log M–1 (br / M) + 1)

Total number of seeks:

2 br / M + br / bb · (2 logM-1–1(br / M) -1)

br is the nr. of blocks of relation r (on disk)

M is the nr. of blocks of the memory buffer

bb is the nr. of blocks allocated to each run in the merge phase

FYI: Read the derivation on pp.548-9 of text.


QUIZ:

Verify the formula for the

nr. of block transfers in

this example

g

a

d 31

c 33

b 14

e 16

r 16

d 21

m 3

p 2

d 7

a 14

a 14

a 19

b 14

c 33

d 7

d 21

d 31

e 16

g 24

m 3

p 2

r 16

a 19

b 14

c 33

d 31

e 16

g 24

a 14

d 7

d 21

m 3

p 2

r 16

a 19

d 31

g 24

b 14

c 33

e 16

d 21

m 3

r 16

a 14

d 7

p 2initial

relationcreateruns

mergepass–1

mergepass–2


24

19


Solution:

3x8 + 6x4 + 12 = 60

Remember that we’re not

counting the final 12

writes!

g

a

d 31

c 33

b 14

e 16

r 16

d 21

m 3

p 2

d 7

a 14

a 14

a 19

b 14

c 33

d 7

d 21

d 31

e 16

g 24

m 3

p 2

r 16

a 19

b 14

c 33

d 31

e 16

g 24

a 14

d 7

d 21

m 3

p 2

r 16

a 19

d 31

g 24

b 14

c 33

e 16

d 21

m 3

r 16

a 14

d 7

p 2initial

relationcreateruns

mergepass–1

mergepass–2


24

19


QUIZ:

With the usual

assumptions, what is the

time needed for the

Mergesort?

g

a

d 31

c 33

b 14

e 16

r 16

d 21

m 3

p 2

d 7

a 14

a 14

a 19

b 14

c 33

d 7

d 21

d 31

e 16

g 24

m 3

p 2

r 16

a 19

b 14

c 33

d 31

e 16

g 24

a 14

d 7

d 21

m 3

p 2

r 16

a 19

d 31

g 24

b 14

c 33

e 16

d 21

m 3

r 16

a 14

d 7

p 2initial

relationcreateruns

mergepass–1

mergepass–2


24

19


12.5 Join Operation

There are several different algorithms to implement joins:

Nested-loop join

Block nested-loop join

Indexed nested-loop join

Merge-join

Hash-join

The choice is made by the query optimizer based on cost

estimates of the candidates.

Examples on next slides use the following information:

Number of records: student: 5,000 takes: 10,000

Number of blocks: student: 100 takes: 400


Nested-Loop Join

To compute the theta join r s, do this:

r is called the outer relation and s the inner relation of the join.

Requires no indices and can be used with any kind of join condition.

Expensive, since it examines every pair of tuples in the two relations.

Do not confuse with outer/inner join!

Concatenation of tuples


Cost estimates for Nested-Loop Join

Worst case: there is enough memory to hold only one

block of each relation

Estimated cost is:

nr bs + br block transfers, plus

nr + br seeks

Calculate the time needed to carry out the join with either

relation as the outer one!


Assuming worst case memory availability, the cost estimate is

with student as outer relation:

5000 400 + 100 = 2,000,100 block transfers,

5000 + 100 = 5100 seeks

with takes as the outer relation

10000 100 + 400 = 1,000,400 block transfers,

10000 + 400 = 10,400 seeks

220.41 s

101.64 s


Cost estimates for Nested-Loop Join

If the smaller relation fits entirely in memory, use that as the

inner relation s. This reduces cost to

br + bs block transfers + 2 seeks

Calculate the time needed to carry out the join!


If the smaller relation fits entirely in memory, use that as the

inner relation s. This reduces cost to

br + bs block transfers + 2 seeks

Cost estimate:

100 + 400 = 500 block transfers.

2 seeks58 ms


Extra-credit


Block Nested-Loop Join

It is a variant of nested-loop join in which every block of the

inner relation is paired with every block of the outer relation.

for each block Br of r do begin

for each block Bs of s do begin

for each tuple tr in Br do begin

for each tuple ts in Bs do begin

Check if (tr,ts) satisfy the join condition

if they do, add tr • ts to the result.

end

end

end

end

Improvement over

Nested-Loop:

Each block in the

inner relation is

read only once for

each block in the

outer relation!


Cost estimates for Block Nested-Loop Join

Worst case estimate:

br bs + br block transfers, plus

2 * br seeks

Each block in the inner relation s is read once for each block in the outer relation!

Best case estimate:

br + bs block transfers, plus

2 seeks


QUIZ: Apply the estimates to the example

Worst case estimate:

br bs + br block transfers, plus

2 * br seeks

Each block in the inner relation s is read once for each block in the outer relation!

Best case estimate:

br + bs block transfers, plus

2 seeks


SKIP

Improvements to nested loop and block nested loop algorithms


Indexed Nested-Loop Join

Index lookups can replace file scans if

join is an equi-join or natural join and

an index is available on the inner relation’s join attribute

Can construct an index just to compute a join.

For each tuple tr in the outer relation r, use the index to look up tuples in s that satisfy the join condition with tuple tr.

Worst case: buffer has space for only one page of r, and, for each tuple in r, we perform an index lookup on s.

Cost of the join: br (tT + tS) + nr c

Where c is the cost of traversing index and fetching all matching stuples for one tuple or r

c can be estimated as cost of a single selection on s using the join condition.

If indices are available on join attributes of both r and s,use the relation with fewer tuples as the outer relation.


Example of Nested-Loop Join Costs

Compute student takes, with student as the outer relation.

Let takes have a primary B+-tree index on the attribute ID, which

contains 20 entries in each index node.

Since takes has 10,000 tuples, the height of the tree is 4, and one

more access is needed to find the actual data

student has 5000 tuples

Cost of block nested loops join

400*100 + 100 = 40,100 block transfers + 2 * 100 = 200 seeks

assuming worst case memory

may be significantly less with more memory

Cost of indexed nested loops join

100 + 5000 * 5 = 25,100 block transfers and seeks.

CPU cost likely to be less than that for block nested loops join


Merge-Join

1. Sort both relations on their join attribute (if not already sorted on the join attributes).

2. Merge the sorted relations to join them

1. Join step is similar to the merge stage of the sort-merge algorithm.

2. Main difference is handling of duplicate values in join attribute — every pair with same value on join attribute must be matched

3. Detailed algorithm in book


Cost estimate for Merge-Join

Can be used only for equi-joins and natural joins

Each block needs to be read only once (assuming all

tuples for any given value of the join attributes fit in

memory

Thus the cost of merge join is:

br + bs block transfers + br / bb + bs / bb seeks

plus the cost of sorting if relations are unsorted.


Hybrid Merge-Join

If one relation is sorted, and the other has a secondary B+-tree

index on the join attribute:

Merge the sorted relation with the leaf entries of the B+-tree .

Sort the result on the addresses of the unsorted relation’s

tuples

Scan the unsorted relation in physical address order and

merge with previous result, to replace addresses by the

actual tuples

• Sequential scan is more efficient than random lookup!

Food for thought: Derive a cost estimate for the hybrid

merge-join (state assumptions!)


SKIP the remainder of this chapter,

starting with

12.5.5 Hash-Join


No homework assigned

Practice all examples from text and

slides!

chapter 12: query processing - · pdf fileremember the architecture of a b+ tree index height...

Documents