Download - The MonetDB Architecture
![Page 1: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/1.jpg)
M.Kersten 2008 1
The MonetDB Architecture
Martin KerstenCWI
Amsterdam
![Page 2: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/2.jpg)
M.Kersten 2008 2
Execution Paradigm
DatabaseStructures
Queryoptimizer
Try to keep things simple
DBMSArchitecture
![Page 3: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/3.jpg)
M.Kersten Sep 2008
MonetDB quickstep
MonetDBkernel
MAPI protocol
JDBC
C-mapi lib
Perl
End-user application
ODBC PHP Python
SQL XQuery
RoR
![Page 4: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/4.jpg)
M.Kersten Sep 2008
The MonetDB Software Stack
XQuery
MonetDB 4 MonetDB 5
MonetDB kernel
SQL 03
Optimizers
SQL/XML
SOAP
Open-GIS
An advanced column-oriented DBMS
X100 Compile?
![Page 5: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/5.jpg)
M.Kersten 2008 5
SQL
MonetDB Server
MonetDB Kernel
MAL
MAL
The MonetDB Assembly Languageglues the layers together. - binary relational model operators - imperative constructs - functional abstractions
Original: P. A. Boncz, M. L. Kersten. MIL Primitives for Querying a Fragmented World. The VLDB Journal, 8(2):101-119, October 1999.
MonetDB quickstep
![Page 6: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/6.jpg)
MonetDB quickstep
SQL
MonetDB Server
Tactical Optimizers
MonetDB Kernel
MAL
MAL
function user.s3_1():void; X1:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",0); X6:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",1); X9:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",2); X13:bat[:oid,:oid] := sql.bind_dbat("sys","photoobjall",1); X8 := algebra.kunion(X1,X6); X11 := algebra.kdifference(X8,X9); X12 := algebra.kunion(X11,X9); X14 := bat.reverse(X13); X15 := algebra.kdifference(X12,X14); X18 := algebra.markT(X15,0@0); X19 := bat.reverse(X18); X20 := aggr.count(X19); sql.exportValue(1,"sys.","count_","int",32,0,6,X20,"");end s3_1;
select count(*) from photoobjall;
0 base table
1 insert
2 update
delete
![Page 7: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/7.jpg)
M.Kersten 2008 7
SQL
MonetDB Server
MAL optimizers
MonetDB Kernel
MAL
MAL
Strategic optimizer:– Exploit the lanuage the language– Rely on heuristics
Operational optimizer:– Exploit everything you know at runtime– Re-organize if necessary
Tactical MAL optimizer:–Modular optimizer framework–Focused on coarse grain resource optimization
MonetDB quickstep
![Page 8: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/8.jpg)
MonetDB quickstep
SQL
MonetDB Server
Tactical Optimizers
MonetDB Kernel
MAL
MAL
function user.s3_1():void; X1:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",0); X20 := aggr.count(X1); sql.exportValue(1,"sys.","count_","int",32,0,6,X20,"");end s3_1;
select count(*) from photoobjall;
Optimizer pipelines.
sql> select optimizer;inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,garbageCollector,dataflow,history,replication,multiplex
![Page 9: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/9.jpg)
MonetDB quickstep
SQL
MonetDB Server
Tactical Optimizers
MonetDB Kernel
MAL
MAL
function user.s3_1():void; X1:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",0); X20 := aggr.count(X1); sql.exportValue(1,"sys.","count_","int",32,0,6,X20,"");end s3_1;
select count(*) from photoobjall;
Kernel execution paradigms
Tuple-at-a-time pipelined
Operator-at-a-time
![Page 10: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/10.jpg)
M.Kersten 2008 10
MonetDB storage
DatabaseStructures
N-ary stores
PAX stores
Columnstores
Try to keep things simple
![Page 11: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/11.jpg)
M.Kersten 2008 11
John 32 HoustonOK
Early 80s: tuple storage structures for PCs were simple
Mary 31 HoustonOK
Easy to access at the cost of wasted space
Try to keep things simple
![Page 12: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/12.jpg)
M.Kersten 2008 12
Slotted pages Logical pages equated physical pages
32 John Houston
31 Mary Houston
Try to keep things simple
![Page 13: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/13.jpg)
M.Kersten 2008 13
Slotted pages Logical pages equated multiple physical pages
32 John Houston
31 Mary Houston
Try to keep things simple
![Page 14: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/14.jpg)
M.Kersten 2008 14
Not all attributes are equally important
Avoid things you don’t always need
![Page 15: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/15.jpg)
M.Kersten 2008 15
A column orientation is as simple and acts like an array
Attributes of a tuple are correlated by offset
Avoid moving too much around
![Page 16: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/16.jpg)
M.Kersten 2008 16
• MonetDB Binary Association TablesID Day Discount
10 4/4/98 0.19511 9/4/98 0.06512 1/2/98 0.17513 7/2/98 0
OID ID100 10101 11102 12103 13104 14
OID Day100 4/4/98101 9/4/98102 1/2/98103 7/2/98104 1/2/99
OID Discount100 0.195101 0.065102 0.175103 0104 0.065
Try to keep things simple
![Page 17: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/17.jpg)
M.Kersten 2008 17
Physical data organization• Binary Association Tables
head tail
100 10101 11102 12103 13104 14
Bat Unitfixed size
Densesequence
Memory mappedfiles
Try to avoid doing things twice
![Page 18: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/18.jpg)
M.Kersten 2008 18
• Binary Association Tables acceleratorsOID ID
100 10101 11102 12103 13104 14Hash-based
access
Try to avoid doing things twice
Column properties:key-nessnon-nulldenseordered
![Page 19: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/19.jpg)
M.Kersten 2008 19
• Binary Association Tables storage controlOID ID
100 Amsterdam101 Seattle102 New York103 London104 Paris
ID
AmsterdamSeattleNew YorkLondon
OID ID
100
101
102
103
104
A BAT can be usedas an encoding table
A VID datatypecan be used torepresent denseenumerations
Type remappingsare used to squeezespace
100
Try to avoid doing things twice
![Page 20: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/20.jpg)
M.Kersten 2008 20
• Column orientation benefits datawarehousing
• Brings a much tighter packaging and improves transport through the memory hierarchy
• Each column can be more easily optimized for storage using compression schemes
• Each column can be replicated for read-only access
Mantra: Try to keep things simple
![Page 21: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/21.jpg)
M.Kersten 2008 22
Execution Paradigm
Volcanomodel
Materialize All Model
Vectorizedmodel
Try to maximize performance
![Page 22: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/22.jpg)
M.Kersten 2008 23
Volcano Engines
Query
SELECT name, salary*.19 AS tax
FROMemployee
WHERE age > 25
![Page 23: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/23.jpg)
M.Kersten 2008 24
Operators
Iterator interface-open()-next(): tuple-close()
Volcano Engines
![Page 24: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/24.jpg)
M.Kersten 2008 25
Primitives
Provide computationalfunctionality
All arithmetic allowed in expressions, e.g. multiplication
mult(int,int) int
Volcano Engines
![Page 25: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/25.jpg)
M.Kersten 2008 26
• The Volcano model is based on a simple pull-based iterator model for programming relational operators.
• The Volcano model minimizes the amount of intermediate store
• The Volcano model is CPU intensive and can be inefficient
Try to maximize performance
Volcano paradigm
![Page 26: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/26.jpg)
M.Kersten 2008 27
MonetDB paradigm
• The MonetDB kernel is a programmable relational algebra machine
• Relational operators operate on ‘array’-like structures
Try to use simple a software pattern
![Page 27: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/27.jpg)
M.Kersten 2008 28
Operator implementation
• All algebraic operators materialize their result• GOOD: small code footprints• GOOD: potential for re-use• BAD : extra storage for intermediates• BAD: cpu cost for retaining it
• Local optimization decisions • Sortedness, uniqueness, hash index• Sampling to determine sizes• Parallelism options• Properties that affect the algorithms
Try to use simple a software pattern
![Page 28: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/28.jpg)
M.Kersten 2008 29
Operator implementation
• All algebraic operators materialize their result
• Local optimization decisions
• Heavy use of code expansion to reduce cost• 55 selection routines• 149 unary operations• 335 join/group operations• 134 multi-join operations• 72 aggregate operations
Try to use simple a software pattern
![Page 29: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/29.jpg)
M.Kersten 2008 30
Execution Paradigm
DatabaseStructures
Queryoptimizer
DBMSArchitecture
Try to avoid the search space trap
![Page 30: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/30.jpg)
Query optimization
• Alternative ways of evaluating a given query• Equivalent expressions• Different algorithms for each operation (Chapter 13)
• Cost difference between a good and a bad way of evaluating a query can be enormous• Example: performing a r X s followed by a selection
r.A = s.B is much slower than performing a join on the same condition
• Need to estimate the cost of operations• Depends critically on statistical information about
relations which the database must maintain• Need to estimate statistics for intermediate results to
compute cost of complex expressions
![Page 31: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/31.jpg)
Introduction (Cont.)
Relations generated by two equivalent expressions have the same set of attributes and contain the same set of tuples, although their attributes may be ordered differently.
![Page 32: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/32.jpg)
Introduction (Cont.)
• Generation of query-evaluation plans for an expression involves several steps:1. Generating logically equivalent expressions
• Use equivalence rules to transform an expression into an equivalent one.
2. Annotating resultant expressions to get alternative query plans
3. Choosing the cheapest plan based on estimated cost
• The overall process is called cost based optimization.
![Page 33: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/33.jpg)
Equivalence Rules
1. Conjunctive selection operations can be deconstructed into a sequence of individual selections.
2. 2. Selection operations are commutative.
3. Only the last in a sequence of projection operations is needed, the others can be omitted.
4. Selections can be combined with Cartesian products and theta joins. (E1 X E2) = E1 E2 1(E1 2 E2) = E1 1 2 E2
))(())((1221
EE
))(()(2121
EE
)())))((((121EE ttntt
![Page 34: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/34.jpg)
Equivalence Rules (Cont.)
5. Theta-join operations (and natural joins) are commutative.
E1 E2 = E2 E1
6. (a) Natural join operations are associative: (E1 E2) E3 = E1 (E2 E3)
(b) Theta joins are associative in the following manner:
(E1 1 E2) 2 3 E3 = E1 2 3 (E2 2 E3) where 2 involves attributes from only E2 and E3.
![Page 35: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/35.jpg)
Pictorial Depiction of Equivalence Rules
![Page 36: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/36.jpg)
Equivalence Rules (Cont.)
7. The selection operation distributes over the theta join operation under the following two conditions:(a) When all the attributes in 0 involve only the attributes of one of the expressions (E1) being joined.
0E1 E2) = (0(E1)) E2
(b) When 1 involves only the attributes of E1 and 2 involves only the attributes of E2.
1 E1 E2) = (1(E1)) ( (E2))
![Page 37: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/37.jpg)
Equivalence Rules (Cont.)
8. The projections operation distributes over the theta join operation as follows:(a) if it involves only attributes from L1 L2:
(b) Consider a join E1 E2. • Let L1 and L2 be sets of attributes from E1 and E2,
respectively. • Let L3 be attributes of E1 that are involved in join
condition , but are not in L1 L2, and• let L4 be attributes of E2 that are involved in join
condition , but are not in L1 L2.
))(())(()( 2......12.......1 2121EEEE LLLL
)))(())((().....( 2......121 42312121EEEE LLLLLLLL
![Page 38: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/38.jpg)
Equivalence Rules (Cont.)
9. The set operations union and intersection are commutative E1 E2 = E2 E1 E1 E2 = E2 E1
9. (set difference is not commutative).10.Set union and intersection are associative.
(E1 E2) E3 = E1 (E2 E3) (E1 E2) E3 = E1 (E2 E3)
9. The selection operation distributes over , and –. (E1 – E2) = (E1) – (E2) and similarly for and in place of –Also: (E1 – E2) = (E1) – E2
and similarly for in place of –, but not for
12.The projection operation distributes over union L(E1 E2) = (L(E1)) (L(E2))
![Page 39: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/39.jpg)
Multiple Transformations (Cont.)
![Page 40: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/40.jpg)
Optimizer strategies
• Heuristic• Apply the transformation rules in a specific order
such that the cost converges to a minimum
• Cost based• Simulated annealing• Randomized generation of candidate QEP• Problem, how to guarantee randomness
![Page 41: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/41.jpg)
Memoization Techniques
• How to generate alternative Query Evaluation Plans?• Early generation systems centred around a tree
representation of the plan • Hardwired tree rewriting rules are deployed to
enumerate part of the space of possible QEP• For each alternative the total cost is determined• The best (alternatives) are retained for execution
• Problems: very large space to explore, duplicate plans, local maxima, expensive query cost evaluation.
• SQL Server optimizer contains about 300 rules to be deployed.
![Page 42: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/42.jpg)
Memoization Techniques
• How to generate alternative Query Evaluation Plans?• Keep a memo of partial QEPs and their cost. • Use the heuristic rules to generate alternatives to
built more complex QEPs
• r1 r2 r3 r4
r1 r2 r2 r3 r3 r4 r1 r4
xLevel 1 plans
r3 r3Level 2 plans
Level n plans r4
r2 r1
![Page 43: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/43.jpg)
M.Kersten 2008 44
Ditching the optimizers
• Applications have different characteristics• Platforms have different characteristics• The actual state of computation is crucial
• A generic all-encompassing optimizer cost-
model does not work
![Page 44: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/44.jpg)
M.Kersten 2008 45
Code Inliner. Constant Expression Evaluator.
Accumulator Evaluations.Strength Reduction. Common Term Optimizer.
Join Path Optimizer. Ranges Propagation. Operator Cost Reduction. Foreign Key handling. Aggregate Groups.
Code Parallizer. Replication Manager. Result Recycler.
MAL Compiler. Dynamic Query Scheduler. Memo-based Execution. Vector Execution.
Alias Removal. Dead Code Removal. Garbage Collector.
Try to disambiguate decisions
![Page 45: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/45.jpg)
M.Kersten 2008 46
Execution Paradigm
DatabaseStructures
Queryoptimizer
DBMSArchitecture
No data from persistent store to the memory trash
![Page 46: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/46.jpg)
M.Kersten 2008 47
Execution paradigms
• The MonetDB kernel is set up to accommodate different execution engines
• The MonetDB assembler program is • Interpreted in the order presented• Interpreted in a dataflow driven manner• Compiled into a C program• Vectorised processing
• X100 project
No data from persistent store to the memory trash
![Page 47: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/47.jpg)
M.Kersten 2008 48
MonetDB/x100
Combine Volcano model withvector processing.
All vectors together should fit the CPU cache
Vectors are compressed
Optimizer should tune this,given the query characteristics.
ColumnBM (buffer manager)
X100 query engine
CPUcache
networkedColumnBM-s
RAM
![Page 48: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/48.jpg)
M.Kersten 2008 49
• Varying the vector size on TPC-H query 1
mysql, oracle,
db2
X100
MonetDB
low IPC, overhead
RAM bandwidth
bound
No data from persistent store to the memory trash
![Page 49: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/49.jpg)
Query evaluation strategy
• Pipe-line query evaluation strategy• Called Volcano query processing model• Standard in commercial systems and MySQL
• Basic algorithm:• Demand-driven evaluation of query tree.• Operators exchange data in units such as records• Each operator supports the following interfaces:– open,
next, close• open() at top of tree results in cascade of opens
down the tree.• An operator getting a next() call may recursively
make next() calls from within to produce its next answer.
• close() at top of tree results in cascade of close down the tree
![Page 50: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/50.jpg)
Query evaluation strategy
• Pipe-line query evaluation strategy• Evaluation:
• Oriented towards OLTP applications• Granule size of data interchange
• Items produced one at a time• No temporary files
• Choice of intermediate buffer size allocations
• Query executed as one process• Generic interface, sufficient to add the iterator
primitives for the new containers.• CPU intensive• Amenable to parallelization
![Page 51: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/51.jpg)
Query evaluation strategy
• Materialized evaluation strategy• Used in MonetDB• Basic algorithm:
• for each relational operator produce the complete intermediate result using materialized operands
• Evaluation:• Oriented towards decision support queries• Limited internal administration and dependencies• Basis for multi-query optimization strategy• Memory intensive• Amendable for distributed/parallel processing
![Page 52: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/52.jpg)
M.Kersten 2008 53
Outline
Execution Paradigm
DatabaseStructures
Indexing
![Page 53: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/53.jpg)
M.Kersten 2008 54
MaterializedViews
Cracking
Indexing TraditionalB-tree, Hash
Find a trusted fortune teller
![Page 54: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/54.jpg)
M.Kersten 2008 55
• Indices in database systems focus on:
• All tuples are equally important for fast retrieval
• There are ample resources to maintain indices
• MonetDB does not rely on B-trees
• MonetDB cracks the database into pieces
Find a trusted fortune teller
![Page 55: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/55.jpg)
M.Kersten 2008 56
Database Cracking• Cracking in a database
kernel?• Every database scan
is a physical reorganization
• This is crazy…
• Reorganization is utterly expensive…
• Better to have many (update) users pay less then one (query) user a lot
• It does not fit the Volcano-style query processor..
• It just doesn’t work that way…….
D1
Find a trusted fortune teller
![Page 56: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/56.jpg)
M.Kersten 2008 57
• Cracking in a database kernel?
• Every database scan is a physical reorganization
select * from D1 where ccd=‘j5’
D1
Find a trusted fortune teller
![Page 57: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/57.jpg)
M.Kersten 2008 58
• Cracking in a database kernel?
• Every database scan is a physical reorganization
D1
create view V as select * from D1 where ccd=‘j5’
D1
select * from D1 where not ccd=‘j5’
Find a trusted fortune teller
![Page 58: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/58.jpg)
M.Kersten 2008 60
• Cracking in a database kernel?
• Every database scan is a physical reorganization
D1
select * from D1 where ccd=‘j5’
select * from D1 where ccd=‘j7’
D1
select * from D1 where not ( ccd=‘j5’or ccd= ‘j7’)
Find a trusted fortune teller
![Page 59: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/59.jpg)
M.Kersten 2008 61
• Cracking in a database kernel?
• Every database scan is a physical reorganization
• Don’t pay the complete sort cost during update
• Incremental predicate index maintenance
• The first user pays !
D1
Try to avoid useless investments
![Page 60: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/60.jpg)
M.Kersten 2008
A simple range queryTry to avoid useless investments
![Page 61: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/61.jpg)
M.Kersten 2008
TPC-H query 6
Try to avoid useless investments
![Page 62: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/62.jpg)
M.Kersten 2008 64
• Cracking is easy in a column store and is part of the critical execution path
• Cracking works under high volume updates
Try to avoid useless investments
![Page 63: The MonetDB Architecture](https://reader035.vdocuments.site/reader035/viewer/2022062217/56813d42550346895da6ff5c/html5/thumbnails/63.jpg)
M.Kersten 2008 65
CstoreSQLserver
DB2
PostgreSQL
MySQL
Whoa MonetDB !Speed lines !