project part ii starting point: existing db application (java + sql) ideally, your system from...
TRANSCRIPT
![Page 1: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/1.jpg)
Project Part II Starting Point: Existing DB Application (Java + SQL)
ideally, your system from Part I of the projectotherwise, the student enrollment demo app (+ queries)
Task Implement a DB library: relational algebra on file systemReplace all JDBC statements with calls to your library
GoalUnderstand the internals of a database systemUnderstand that there is always an alternative
Brief glimpse on the „Information Systems“ lecture1
![Page 2: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/2.jpg)
The Data Management Universe
Project Part I Project Part 2
2
![Page 3: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/3.jpg)
Logs Indexes DB Catalogue
Storage Manager
TA ManagementRecovery
Runtime Schema
Query Optimizer DBMS
DML-Compiler DDL-Compiler
Application Ad-hoc QueryManagement
toolsCompiler
„Naive“User
ExpertUser
App-Developer
DB-admin
External Storage (e.g., disks)
Components of a Database System
3
![Page 4: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/4.jpg)
Application
Query Processor
Data Manager (Indexes, Records)
Storage Manager (Pages)
SQL {tuples}
get/put block
Tra
nsac
tions
(Loc
king
, Lo
ggin
g)
Met
adat
a M
gmt
(Sch
ema,
Sta
ts)
Server (Network, App Buffers, Security)
Storage System (disks, SSDs, SAN, …)
4
![Page 5: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/5.jpg)
Why use a DBMS? Avoid redundancy and inconsistency
Query Processor, Transaction Manager, Catalog
Rich (declarative) access to the dataQuery Processor
Synchronize concurrent data accessTransaction Manager
Recovery after system failuresTransaction Manager, Storage Layer
Security and privacyServer, Catalog 5
![Page 6: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/6.jpg)
Application
Query Processor
Indexes, Records
Manager, Pages
SQL / rel. algebra {tuples}
get/put block
Tra
nsac
tions
(Loc
king
, Lo
ggin
g)
Met
adat
a M
gmt
(Sch
ema,
Sta
ts)
Server (Network, App Buffers, Security)
Storage System (disks, SSDs, SAN, …)
Query Processor
Data Manager
Storage Manager
Project Part II 6
![Page 7: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/7.jpg)
What does a Database System do? Input: SQL statement Output: {tuples}1. Translate SQL into a set of get/put req. to backend
storage2. Extract, process, transform tuples from blocks
Tons of optimizations Efficient algorithms for SQL operators (hashing, sorting) Layout of data on backend storage (clustering, free space) Ordering of operators (small intermediate results) Semantic rewritings of queries Buffer management and caching Parallel execution and concurrency Outsmart the OS Partitioning and Replication in distributed system Indexing and Materialization Load and admission control
+ Security + Durability + Concurrency Control + Tools
7
![Page 8: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/8.jpg)
Database Optimizations Query Processor (based on statistics)
Efficient algorithms for SQL operators (hashing, sorting) Ordering of operators (small intermediate results) Semantic rewritings of queries Parallel execution and concurrency
Storage Manager Load and admission control Layout of data on backend storage (clustering, free space) Buffer management and caching Outsmart the OS
Transaction Manager Load and admission control
Tools (based on statistics) Partitioning and Replication in distributed system Indexing and Materialization
8
![Page 9: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/9.jpg)
DBMS vs. OS Optimizations
Many DBMS tasks are also carried out by OSLoad controlBuffer managementAccess to external storageScheduling of processes…
What is the difference?DBMS has intimate knowledge of workloadDBMS can predict and shape access pattern of a queryDBMS knows the mix of queries (all pre-compiled)DBMS knows the contention between queriesOS does generic optimizations
Problem: OS overrides DBMS optimizations!9
![Page 10: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/10.jpg)
Application
Query Processor
Indexes, Records
Manager, Pages
SQL / rel. algebra {tuples}
get/put block
Tra
nsac
tions
(Loc
king
, Lo
ggin
g)
Met
adat
a M
gmt
(Sch
ema,
Sta
ts)
Server (Network, App Buffers, Security)
Storage System (disks, SSDs, SAN, …)
Query Processor
Data Manager
Storage Manager
10
![Page 11: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/11.jpg)
Query ProcessorSQL
Parser
Rewrite
Optimizer
QGM
QGM
Co
mp
iler
CodeGen
QGM++
InterpreterPlan
{tuples}
Ru
ntim
e System
11
![Page 12: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/12.jpg)
SQL -> Relational Algebra
SQL
select A1, ..., An
from R1, ..., Rk
where P;
A1, ..., An(P (R1 x ... x Rk )) A1, ..., An
P
x
x Rk
x R3
R2R1
Relational Algebra
12
![Page 13: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/13.jpg)
Runtime System Three approaches
A. Compile query into machine codeB. Compile query into relational algebra and interpret
thatC. Hybrid: e.g., compile predicates into machine code
What to do?A: better performanceB: easier debugging, better portabilityProject: use Approach B
Query Interpreterprovide implementation for each algebra operatordefine interface between operators 13
![Page 14: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/14.jpg)
Algorithms for Relational Algebra Table Access
scan (load each page at a time) index scan (if index available)
SortingTwo-phase external sorting
Joins (Block) nested-loops Index nested-loopsSort-MergeHashing (many variants)
Group-by (~ self-join)SortingHashing
14
![Page 15: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/15.jpg)
Two-phase External Sorting Phase I: Create Runs
1. Load allocated buffer space with tuples2. Sort tuples in buffer pool3. Write sorted tuples (run) to disk4. Goto Step 1 (create next run) until all tuples
processed Phase II: Merge Runs
Use priority heap to merge tuples from runs
Special cases buffer >= N: no merge needed buffer < sqrt(N): multiple merge phases necessary (N size of the input in pages)
15
![Page 16: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/16.jpg)
16
External Sort
971735
27162
9913
![Page 17: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/17.jpg)
17
External Sort
971735
27162
9913
97
17
3
load
![Page 18: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/18.jpg)
18
External Sort
971735
27162
9913
3
17
97
sort
![Page 19: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/19.jpg)
19
External Sort
971735
27162
9913
33
1797
17
97
writerun
![Page 20: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/20.jpg)
20
External Sort
971735
27162
9913
53
1797
27
16
load
![Page 21: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/21.jpg)
21
External Sort
971735
27162
9913
53
17975
1627
16
27
sort & write
![Page 22: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/22.jpg)
22
External Sort
971735
27162
9913
23
17975
1627
99
13
load
![Page 23: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/23.jpg)
23
External Sort
971735
27162
9913
2
317975
16272
1399
13
99
End of Phase 1
![Page 24: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/24.jpg)
24
External Sort
3
317975
16272
1399
5
2
merge
![Page 25: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/25.jpg)
25
External Sort
2
3
317975
16272
1399
5
2
merge
![Page 26: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/26.jpg)
26
External Sort
23 3
317975
16272
1399
5
13
merge
![Page 27: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/27.jpg)
27
External Sort
235
17
317975
16272
1399
5
13
merge
![Page 28: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/28.jpg)
28
External Sort
235
17
317975
16272
1399
16
13
merge
![Page 29: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/29.jpg)
29
External Sort
235
13
17
317975
16272
1399
16
13
![Page 30: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/30.jpg)
Multi-way Merge (N = 13; M = 2)
30
![Page 31: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/31.jpg)
Analysis N: size of table in pages M: size of (available) main memory in pages IO Cost
O(N): if M >= sqrt(N)2 * N: if M >= N4 * N: if N > M >= sqrt(N)
O(N logM N): if M < sqrt(N)Base of logarithm: in O notation not relevant, but constants
matter
CPU Cost (M >= sqrt(N))Phase 1 (create N/M runs of length M): O(N * log2 M)Phase 2 (merge N tuples with heap): O(N * log2 N/M) Exercise: Do CPU cost increase/decrease with M?
31
![Page 32: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/32.jpg)
Sorting Summary Complexity: N * log(N) theory is right, but
DB people care about CPU and IO complexityConstants matter!Buffer allocation matters! Many concurrent queries?More main memory can hurt performance!
Main memory is large. Do two-way sort because…Parallelize sorting on different machinesOr many concurrent sorts on same machineBut more than 2-ways very rare in practice
Knuth suggests Replacement Selection Increases length of runsBut, higher constant for CPU usage Typically, not used in practice
32
![Page 33: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/33.jpg)
(Grace) Hash Join
33
![Page 34: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/34.jpg)
Grace Hash Join
34
![Page 35: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/35.jpg)
Sorting vs. Hashing Both techniques can be used for joins, group-by, …
binary and unary matching problems
Same asymptotic complexity: O(N log N) In both IO and CPUHashing has lower constants for CPU complexity IO behavior is almost identical
Merging (Sort) vs. Partitioning (Hash)Merging done afterwards; Partitioning done beforePartitioning depends on good statistics to get right
Sorting more robust. Hashing better in average case! 35
![Page 36: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/36.jpg)
Iterator Model Plan contains many operators
Implement each operator indepentlyDefine generic interface for each operatorEach operator implemented by an iterator
Three methods implemented by each iteratoropen(): initialize the internal state (e.g., allocate
buffer)char* next(): produce the next result tupleclose(): clean-up (e.g., release buffer)
N.B. Modern DBMS use a Vector Modelnext() returns a set of tuplesWhy is that better? 36
![Page 37: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/37.jpg)
Iterator Model at Work
37
R S T
scan scan scan
NLJoin
NLJoin
Application
![Page 38: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/38.jpg)
Iterator Model at Work
38
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: execute()
![Page 39: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/39.jpg)
Iterator Model at Work
39
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: execute()
open()
![Page 40: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/40.jpg)
Iterator Model at Work
40
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: execute()
open()
open()
open()
![Page 41: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/41.jpg)
Iterator Model at Work
41
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: execute()
open()
open()
open/close foreach R tuple
![Page 42: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/42.jpg)
Iterator Model at Work
42
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: execute()
open()
open/close foreach R,S tuple
![Page 43: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/43.jpg)
Iterator Model at Work
43
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()
next()
![Page 44: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/44.jpg)
Iterator Model at Work
44
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()
r1
![Page 45: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/45.jpg)
Iterator Model at Work
45
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()r1
open()
![Page 46: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/46.jpg)
Iterator Model at Work
46
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()r1
next()
![Page 47: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/47.jpg)
Iterator Model at Work
47
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()r1
s1
![Page 48: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/48.jpg)
Iterator Model at Work
48
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()r1
next()
![Page 49: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/49.jpg)
Iterator Model at Work
49
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
next()r1
s2
![Page 50: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/50.jpg)
Iterator Model at Work
50
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
r1, s2
![Page 51: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/51.jpg)
Iterator Model at Work
51
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
r1, s2
open()
![Page 52: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/52.jpg)
Iterator Model at Work
52
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
r1, s2
next()
![Page 53: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/53.jpg)
Iterator Model at Work
53
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
next()
r1, s2
t1
![Page 54: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/54.jpg)
Iterator Model at Work
54
R S T
scan scan scan
NLJoin
NLJoin
ApplicationJDBC: next()
r1, s2, t1
… r2, r3, … … s3, s4, … … t2, t3, …
![Page 55: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/55.jpg)
Iterators Summary: Easy & Costly Principle
data flows bottom up in a plan (i.e. operator tree)control flows top down in a plan
Advantagesgeneric interface for all operators: great information hidingeasy to implement iterators (clear what to do in any phase)works well with JDBC and embedded SQLsupports DBmin and other buffer management strategiesno overheads in terms of main memorysupports pipelining: great if only subset of results consumedsupports parallelism and distribution: add special iterators
Disadvantageshigh overhead of method callspoor instruction cache locality
55
![Page 56: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/56.jpg)
Query ProcessorSQL
Parser
Rewrite
Optimizer
QGM
QGM
Co
mp
iler
CodeGen
QGM++
InterpreterPlan
{tuples}
Ru
ntim
e System
56
![Page 57: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/57.jpg)
SQL -> Relational Algebra
SQL
select A1, ..., An
from R1, ..., Rk
where P;
A1, ..., An(P (R1 x ... x Rk )) A1, ..., An
P
x
x Rk
x R3
R2R1
Relational Algebra
57
![Page 58: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/58.jpg)
SQL -> QGM
SQL
select a
from R
where a in (select b
from S);
QGM
R
a
b
S
in
58
![Page 59: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/59.jpg)
Parser Generates rel. alg. tree for each sub-query
constructs graph of trees: Query Graph Model (QGM)nodes are subqueriesedges represent relationships between subqueries
Extended rel. algebra because SQL more than RAGROUP BY: operatorORDER BY: sort operatorDISTINCT: can be implemented with operator
Parser needs schema informationWhy? Give examples.
Why can`t a query be compiled into one tree?59
![Page 60: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/60.jpg)
SQL -> Relational Algebra
SQL
select A1, ..., An
from R1, ..., Rk
where P;
A1, ..., An(P (R1 x ... x Rk )) A1, ..., An
P
x
x Rk
x R3
R2R1
Relational Algebra
60
![Page 61: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/61.jpg)
61
Example: SQL -> Relational Algebraselect Title
from Professor, Lecture
where Name = ´Popper´ and
PersNr = Reader
Professor Lecture
Name = ´Popper´ and PersNr=Reader
Title
Title (Name = ´Popper´ and PersNr=Reader (Professor Lecture))
![Page 62: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/62.jpg)
62
First Optimization: Push-down select Title
from Professor, Lecture
where Name = ´Popper´ and
PersNr = Reader
Professor
Lecture
PersNr=Reader
Title
Title (PersNr=Reader ((Name = ´Popper´ Professor) Lecture))
Name = ´Popper´
![Page 63: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/63.jpg)
63
Second Optimization: Push-down select Title
from Professor, Lecture
where Name = ´Popper´ and
PersNr = Reader
Professor
Lecture
PersNr=Reader
Title
Name = ´Popper´
PersNr Title,Reader
![Page 64: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/64.jpg)
Correctness: Push-down Title (PersNr=Reader ((Name = ´Popper´ Professor) Lecture))
(composition of projections)
Title (Title,PersNr,Reader (… ((…Professor) Lecture)))
(commutativity of and )
Title (… (Title,PersNr,Reader ((…Professor) Lecture)))
(commutativity of and )
Title (… (PersNr (…Professor) Title,Reader (Lecture)))
64
![Page 65: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/65.jpg)
Second Optimization: Push down Correctness (see previous slide – example
generalizes)
Why is it good? ( almost same reason as for reduces size of intermediate resultsbut: only makes sense if results are materialized; e.g.
sortdoes not make sense if pointers are passed around in
iterators
65
![Page 66: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/66.jpg)
66
Third Optimization: + x = A select Title
from Professor, Lecture
where Name = ´Popper´ and
PersNr = Reader
Professor
Lecture
A
Title
Name = ´Popper´
PersNr Title,Reader
![Page 67: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/67.jpg)
Third Optimization: + x = A Correctness by definition of A operator Why is this good?
x always done using nested-loops algorithmA can also be carried out using hashing, sorting, index supportchoice of better algorithm may result in huge wins
x produces large intermediate resultsresults in a huge number of „next()“ calls in iterator modelmethod calls are expensive
Selection, projection push-down are no-brainersmake sense whenever applicabledo not need a cost model to decide how to apply them (exception: expensive selections, projections with UDF)done in a phase called query rewrite, based on rules
More complex query rewrite rules…67
![Page 68: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/68.jpg)
Unnesting of Views Example: Unnesting of Views
select A.xfrom Awhere y in
(select y from B)
Example: Unnesting of Viewsselect A.xfrom Awhere exists
(select * from B where A.y = B-y)
Is this correct? Why is this better? (not trivial at all!!!)
select A.xfrom A, Bwhere A.y = B.y
68
select A.xfrom A, Bwhere A.y = B.y
![Page 69: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/69.jpg)
Query Rewrite Example: Predicate Augmentation
select *from A, B, Cwhere A.x = B.x
and B.x = C.x
select *from A, B, Cwhere A.x = B.x
and B.x = C.xand A.x = C.x
Why is that useful?
69
![Page 70: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/70.jpg)
Pred. Augmentation: Why useful?
… x
… 1
… 3
… 5
… …
70
A (odd numbers)
… x
… 1
… 2
… 3
… …
B (all numbers)
… x
… 2
… 4
… 6
… …
C (even numbers)
• Cost((A A C) A B) < Cost((A A B) A C)• get second join for free
• Query Rewrite does not know that, …• but it knows that it might happen and hopes for optimizer…
• Codegen gets rid of unnecessary predicates (e.g., A.x = B.x)
![Page 71: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/71.jpg)
Query Optimization Two tasks
Determine order of operatorsDetermine algorithm for each operator (hashing,
sorting, …)
Components of a query optimizerSearch spaceCost modelEnumeration algorithm
Working principleEnumerate alternative plansApply cost model to alternative plansSelect plan with lowest expected cost 71
![Page 72: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/72.jpg)
Query Optimization: Does it matter? A x B x C
size(A) = 10,000size(B) = 100size(C) = 1cost(X x Y) = size(X) + size(Y)
cost( (A x B) x C) = 1,010,101cost(A x B) = 10,100cost(X x C) = 1,000,001 with X = A x B
cost ( A x (B x C)) = 10,201cost(B x C) = 101cost(A x X) = 10,100 with X = B x C
72
![Page 73: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/73.jpg)
Query Opt.: Does it matter? A x B x C
size(A) = 1000size(B) = 1size(C) = 1cost(X x Y) = size(X) * size(Y)
cost( (A x B) x C) = 2000cost(A x B) = 1000cost(X x C) = 1000 with X = A x B
cost ( A x (B x C)) = 1001cost(B x C) = 1cost(A x X) = 1000 with X = B x C
73
![Page 74: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/74.jpg)
Search Space: Relational Algebra Associativity of joins:
(A A B) A C = A A (B A C)
Commutativity of joins: A A B = B A A
Many more rulessee Kemper/Eickler or Garcia-Molina text books
What is better: A A B or B A A?it dependsneed cost model to make decision
74
![Page 75: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/75.jpg)
Search Space: Group Bys
SELECT … FROM R, S WHERE R.a = S.a GROUP BY R.a, S.b;
R.a, S.b(R A S)
S.b(R.a(R) A S)
Often, many possible ways to split & move group-bys
again, need cost model to make right decisions
75
![Page 76: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/76.jpg)
Cost Model Cost Metrics
Response Time (consider parallelism)Resource Consumption: CPU, IO, network$ (often equivalent to resource consumption)
PrincipleUnderstand algorithm used by each operator (sort,
hash, …)estimate available main memory buffersestimate the size of inputs, intermediate results
Combine cost of operators: sum for resource consumptionmax for response time (but keep track of bottlenecks)
Uncertaintiesestimates of buffers, interference with other operatorsestimates of intermediate result size (histograms) 76
![Page 77: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/77.jpg)
Equi-Width Histogram
77
SELECT * FROM person WHERE 25 < age < 40;
![Page 78: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/78.jpg)
Equi-Depth Histogram
0
10
20
30
40
50
60
20 bis 42 42 bis 48 48 bis 53 53 bis 59 59 bis 70
78
SELECT * FROM person WHERE 25 < age < 40;
![Page 79: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/79.jpg)
Multi-Dimensional Histogram
0
10
20
30
40
50
60
20 bis 30 30 bis 40 40 bis 50 50 bis 60 60 bis 70
70-100
100-150
150-250
79
SELECT * FROM person WHERE 25 < age < 40 AND salary > 200;
;
![Page 80: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/80.jpg)
Enumeration Algorithms Query Optimization is NP hard
even ordering of Cartesian products is NP hard in general impossible to predict complexity for given
query Overview of Algorithms
Dynamic Programming (good plans, exp. complexity)Greedy heuristics (e.g., highest selectivity join first)Randomized Algorithms (iterative improvement, Sim.
An., …)Other heuristics (e.g., rely on hints by programmer)Smaller search space (e.g., deep plans, limited group-
bys) Products
Dynamic Programming used by many systemsSome systems also use greedy heuristics in addition
80
![Page 81: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/81.jpg)
Dynamic Programming
access_plans: enumerate all ways to scan a table (indexes, …)
join_plans: enumerate all ways to join 2 tables (algos, commut.)
prune_plans: discard sub-plans that are inferior (cost & order)
81
![Page 82: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/82.jpg)
Access Plans SELECT * FROM R, S, T WHERE R.a = S.a AND R.b = T.b
ORDER BY R.c;
Assume Indexes on R.a, R.b, R.c, R.d
scan(R): cost = 100; order = none idx(R.a): cost = 100; order = R.a idx(R.b): cost = 1000; order = R.b idx(R.c): cost = 1000; order = R.c idx(R.d): cost = 1000; order = none
Keep blue plans only. Why?And how can all that be? (Whole lecture on all this.)
82
![Page 83: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/83.jpg)
Access Plans for S SELECT * FROM R, S, T WHERE R.a = S.a AND R.b = T.b
ORDER BY R.c;
Assume Indexes on S.b, S.c, S.d
scan(S): cost = 1000; order = none idx(S.b): cost = 10000; order = none idx(S.c): cost = 10000; order = none idx(S.d): cost = 10000; order = none
83
![Page 84: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/84.jpg)
Access Plans for T SELECT * FROM R, S, T WHERE R.a = S.a AND R.b = T.b
ORDER BY R.c;
Assume Indexes on T.a, T.b
scan(T): cost = 10; order = none idx(T.a): cost = 100; order = none idx(T.b): cost = 100; order = T.b
84
![Page 85: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/85.jpg)
Join Plans for R join S SELECT * FROM R, S, T WHERE R.a = S.a AND R.b = T.b
ORDER BY R.c;
Consider all combinations of (blue) access plans Consider all join algorithms (NL, IdxNL, SMJ, GHJ, …) Consider all orders: R x S, S x R Prune based on cost estimates, interesting orders Some examples:
scan(R) NLJ scan(S): cost = 100; order = nonescan(S) IdxNL Idx(R.a): cost = 1000; order = none idx(R.b) GHJ scan(S): cost = 150; order = R.b idx(R.b) NLJ scan(S): cost = 250; order = R.b
85
![Page 86: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/86.jpg)
Join Plans for three-way (+) joins SELECT * FROM R, S, T WHERE R.a = S.a AND R.b = T.b
ORDER BY R.c; Consider all combinations of joins (assoc., commut.)
e.g., (R A S) A T, S A (T A R), ….sometimes even enumerate Cartesian products
Use (pruned) plans of prev. steps as building blocksconsider all combinations
Prune based on cost estimates, interesting orders interesting orders for the special optimality principle heregets more complicated in distributed systems
Exercise: Space and Time complexity of DP for DBMS. 86
![Page 87: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/87.jpg)
Application
Query Processor
Indexes, Records
Manager, Pages
SQL / rel. algebra {tuples}
get/put block
Tra
nsac
tions
(Loc
king
, Lo
ggin
g)
Met
adat
a M
gmt
(Sch
ema,
Sta
ts)
Server (Network, App Buffers, Security)
Storage System (disks, SSDs, SAN, …)
Query Processor
Data Manager
Storage Manager
87
![Page 88: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/88.jpg)
Storage System Basics Storage is organized in a hierarchy
combine different media to mimic one ideal storage Storage systems are distributed
disks organized in arrayscloud computing: DHT over 1000s of servers (e.g.,
S3)advantages of distributed storage systems
cost: use cheap hardware performance: parallel access and increased bandwidth fault tolerance: replicate data on many machines
Storage access is non uniformmulti-core machines with varying distance to bankssequential vs. random on disk and SSDsplace hot data in the middle of disk 88
![Page 89: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/89.jpg)
Idea (1min)
Building (10min)
City (1.5h)
Pluto (2 years)
Andromeda
(2000 years)89
Storage Hierarchy
<1ns
register
<10ns
L1/L2 Cache
100ns
Main Memory
3-10 ms
Disk
secs
Tape
Factor 105
![Page 90: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/90.jpg)
Why a Storage Hierarchy? Mimics ideal storage: speed of register at cost
of tapeunlimited capacity // tapezero cost // tapePersistent // tapezero latency for read + write // register infinte bandwidth // register
How does it work?Higher layer „buffers“ data of lower layerExploit spatial and temporal locality of applications
90
![Page 91: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/91.jpg)
Disks: Sequential vs. Random IO Time to read 1000 blocks of size 8 KB?
Random access:trnd = 1000 * t = 1000 * (ts + tr + ttr) = 1000 * (10 + 4.17 + 0.16) = 1000 * 14.33 = 14330 ms
Sequential access:tseq = ts + tr + 1000 * ttr + N * ttrack-to-track seek time
= ts + tr + 1000 * 0.16 ms + (16 * 1000)/63 * 1 ms = 10 ms + 4.17 ms + 160 ms + 254 ms ≈ 428 ms
Need consider this gap in algorithms!
[Information Systems Class] 91
![Page 92: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/92.jpg)
Storage Manager Control all access to external storage (i.e., disks)
implements external storage hierarchy (SSD, tape, disks, …)
optimize heterogeneity of storageoutsmarts file system: operating system cachingwrite-ahead logging for redo and undo recoveryOracle, Google, etc. implement their own file system
Management of files and blockskeep track of files associated to the database (catalog)group set of blocks into pages (granularity of access)
Buffer managementsegmentation of buffer poolclever replacement policy; e.g., MRU for sequential
scanspin pages (no replacement while in use)
92
![Page 93: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/93.jpg)
Database = { files }
A file = variable-sized sequence of blocksBlock is the unit of transfer to disk. Typically, 512B
A page = fixed-sized sequence of blocks. A page contains records or index entries (special case blobs. One record spans multiple
pages) typical page size: 8KB for records; 16 KB for index
entriesPage is logical unit of transfer and unit of buffering
Blocks of same page are prefetched, stored on same track on disk
93
![Page 94: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/94.jpg)
Table Spaces, Files, and Tables
Table Space
Table
belongs
belongs
N
1
1 File
N
94
![Page 95: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/95.jpg)
Table Spaces Explicit management of files allocated by the
databaseEach file belongs to a table spaceEach table stored in a table spaceCluster multiple tables in the same file
DDL OperationsCREATE TABLESPACE wutz
DATAFILE a.dat SIZE 4MB, b.dat SIZE 3MB;ALTER TABLESPACE wutz ADD DATAFILE …; CREATE TABLE husten TABLESPACE wutz;
Warning: Full table space crash the DBMSClassic emergency situation in practice
95
![Page 96: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/96.jpg)
Buffer Management Keep pages in main memory as long as possible
Minimize disk I/Os in storage hierarchy
Critical questionsWhich pages to keep in memory (replace policy)?When to write updated pages back to disk (trans.
mgr.)?
General-purpose replacement policiesLRU, Clock, … (see operating systems class)LRU-k: replace page with k th least recent usage2Q: keep two queues: hot queue, cold queue
Access moves page to hot queueReplacement from cold queue 96
![Page 97: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/97.jpg)
Access Patterns of Databases Sequential: table scans
P1, P2, P3, P4, P5, …
Hiearchical: index navigationP1, P4, P11, P1, P4, P12, P1, P3, P8, P1, P2, P7, P1, P3, P9, …
Random: index lookupP13, P27, P3, P43, P15, …
Cyclic: nested-loops joinP1, P2, P3, P4, P5, P1, P2, P3, P4, P5, P1, P2, P3, P4, P5, …
97
![Page 98: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/98.jpg)
DBMin [Chou, DeWitt, VLDB 1985]
ObservationsThere are many concurrent queriesEach query is composed of a set of operatorsAllocate memory for each operator of each queryAdjust replacement policy according to access pattern
Examplesscan(T): 4 pages, MRU replacement indexScan(X)
200 pages for Index X, LRU replacement100 pages for Table T, Random replacement
hashJoin(R, S)200 pages for Table S
Allows to do load control and further optimizationsEconomic model for buffer allocation, priority-based BM,
… 98
![Page 99: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/99.jpg)
DBMin: Buffer Segmentation
scan(T)
DB
index(S.b)
SELECT * FROM T, SWHERE T.a=S.b
SELECT *FROM T
fetch(S)
scan(T)
DB Buffer Pool
99
![Page 100: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/100.jpg)
DBMS vs. OS: Double Page Fault DBMS needs Page X
Page X is not in the DB buffer pool DBMS evicts Page Y from DB buffer pool
make room for XBut, Page Y is not in the OS Cache
OS reads Page Y from disk (swap)
SummaryLatency: need to wait for (at least) two I/OsCost: If Y updated, up to three I/Os to write Y to diskUtilization: Same page held twice in main memory
If you are interested in DB/OS co-design, … 100
![Page 101: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/101.jpg)
Application
Query Processor
Indexes, Records
Manager, Pages
SQL / rel. algebra {tuples}
get/put block
Tra
nsac
tions
(Loc
king
, Lo
ggin
g)
Met
adat
a M
gmt
(Sch
ema,
Sta
ts)
Server (Network, App Buffers, Security)
Storage System (disks, SSDs, SAN, …)
Query Processor
Data Manager
Storage Manager
101
![Page 102: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/102.jpg)
Data Manager Maps records to pages
implement „record identifier“ (RID)
Implementation of IndexesB+ trees, R trees, etc. Index entry ~ Record (same mechanism)
Freespace Management Index-organized tables (IOTs)Various schemes
Implementation of BLOBs (large objects)variants of position trees
102
![Page 103: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/103.jpg)
Structure of a Record
Fixed length fieldse.g., number(10,2), date, char[100]direct access to these fields
Variable length fieldse.g., varchar[100]store (length, pointer) as part of a fixed-length fieldstore payload information in a variable-length fieldaccess in two steps: retrieve pointer + chase pointer
NULL ValuesBitmap: set 1 if value of a field is NULL
Bitmap fixed-length variable length fields
103
![Page 104: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/104.jpg)
Inside a Page
record identifier (rid):<pageno, slotno>indexes use rids to ref. records
record position (in page):slotno x bytes per slot
records can move in pageif records grow or shrinkif records are deletedno need to update indexes
104
![Page 105: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/105.jpg)
What happens when a page is full? Problem: A record grows because of an update
E.g., a varchar field is updated so that record grows
Idea: Keep a placeholder (TID)Move the record to a different pageKeep a „forward“ (TID) at „home“ page If record moves again, update TID at „home“ page
AssessmentAt most two I/Os to access a record; typically, only
oneFlexibility to move records within and across pagesNo need to update references to record (i.e., indexes)
105
![Page 106: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/106.jpg)
TID (Forwarding) Concept
911, 2
911 4711
4711, 3Wutz 23
106
![Page 107: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/107.jpg)
Freespace Management
Find a page for a new recordMany different heuristics conceivableAll based on a list of pages with free space
Append OnlyTry to insert into the last page of free space list. If no room in last page, create a new page.
Best FitScan through list and find min page that fits.
First Fit, Next FitScan through list and find first / next fit
Witnesses: Classify buckets IOT: organize all tuples in a B+ tree
Let the B+ tree take care of splitting and freespace mgmt. 107
![Page 108: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/108.jpg)
Application
Query Processor
Indexes, Records
Manager, Pages
SQL / rel. algebra {tuples}
get/put block
Tra
nsac
tions
(Loc
king
, Lo
ggin
g)
Met
adat
a M
gmt
(Sch
ema,
Sta
ts)
Server (Network, App Buffers, Security)
Storage System (disks, SSDs, SAN, …)
Query Processor
Data Manager
Storage Manager
108
![Page 109: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/109.jpg)
Meta-data Management: Catalog All meta-data stored in tables
Accessed internally using SQLEat your own dogfood
Kinds of meta-dataSchema: used to compile queries Table spaces: files used to store databaseHistograms: estimate result sizes; query optimizationParameters (cost of I/O, CPU speed, …): query
optimizationCompiled Queries: used for (JDBC) PreparedStatements Configuration: AppHeap Size, Isolation Level, …Users (login, password): used for securityWorkload Statistics: index advisors
109
![Page 110: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/110.jpg)
What does a Database System do? Input: SQL statement Output: {tuples}1. Translate SQL into a set of get/put req. to backend
storage2. Extract, process, transform tuples from blocks
Tons of optimizations Efficient algorithms for SQL operators (hashing, sorting) Layout of data on backend storage (clustering, free space) Ordering of operators (small intermediate results) Semantic rewritings of queries Buffer management and caching Parallel execution and concurrency Outsmart the OS Partitioning and Replication in distributed system Indexing and Materialization Load and admission control
+ Security + Durability + Concurrency Control + Tools
110
![Page 111: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/111.jpg)
Database Optimizations Query Processor (based on statistics)
Efficient algorithms for SQL operators (hashing, sorting) Ordering of operators (small intermediate results) Semantic rewritings of queries Parallel execution and concurrency
Storage Manager Load and admission control Layout of data on backend storage (clustering, free space) Buffer management and caching Outsmart the OS
Transaction Manager Load and admission control
Tools (based on statistics) Partitioning and Replication in distributed system Indexing and Materialization
111
![Page 112: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/112.jpg)
DBMS vs. OS Optimizations
Many DBMS tasks are also carried out by OSLoad controlBuffer managementAccess to external storageScheduling of processes…
What is the difference?DBMS has intimate knowledge of workloadDBMS can predict and shape access pattern of a queryDBMS knows the mix of queries (all pre-compiled)DBMS knows the contention between queriesOS does generic optimizations
Problem: OS overrides DBMS optimizations!112
![Page 113: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/113.jpg)
What to optimize? Feature Traditional Cloud
Cost [$] fixed optimize
Performance [tps, secs] optimize fixed
Scale-out [#cores] optimize fixed
Predictability [($)] - fixed
Consistency [%] fixed ???
Flexibility [#variants] - optimize
[Florescu & Kossmann, SIGMOD Record 2009]
Put $ on the y-axis of your graphs!!!
113
![Page 114: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/114.jpg)
Experiments [Loesing et al. 2010]
TPC-W Benchmark throuphput: WIPS latency: fixed depending on request typecost: cost / WIPS, total cost, predictability
PlayersAmazon RDS, SimpleDBS3/28msec [Brantner et al. 2008]Google AppEngineMicrosoft Azure
114
![Page 115: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/115.jpg)
Scale-up Experiments
115
![Page 116: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/116.jpg)
Cost / WIPS (m$)
Low Load Peak Load
Amazon RDS 1.212 0.005
S3 / 28msec - 0.007
Google AE/C 0.002 0.028
MS Azure 0.775 0.005
116
![Page 117: Project Part II Starting Point: Existing DB Application (Java + SQL) ideally, your system from Part I of the project otherwise, the student enrollment](https://reader035.vdocuments.site/reader035/viewer/2022062806/56649e6c5503460f94b6bed7/html5/thumbnails/117.jpg)
What do you need for Project Part 2 Storage Manager
Management of filesSimple buffer managementFree space management and new page allocation
Data ManagerSlotted pages
Query Processor Implementation of scan, join, group-by Iterator model (external for extra credit)
Catalog, Server, Transaction Manager -
Be pragmatic! Get it running! 117