reducing the storage overhead of main-memory oltp ... · main-memory oltp databases with hybrid...
Post on 01-Aug-2020
5 Views
Preview:
TRANSCRIPT
Reducing the Storage Overhead of Main-Memory OLTP Databases with
Hybrid IndexesHuanchen Zhang
You are running out of memory
2
You are running out of memory
2
Buy more ?
You are running out of memory
2
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
utTPC-C on -Store
3
Mem
ory
(GB)
Disk tuplesIn-memory tuplesIndexes
4
8
0
Memory Limit = 5GB
4
The better way:Use memory more efficiently
5
Indexes are LARGEBenchmark % space for index
TPC-C
Voter
Articles
58%55%34%
Hybrid Index
34%41%18%
6
Our Contributions The hybrid index architecture
The Dual-Stage Transformation
Applied to 4 index structures- B+tree- Masstree
7
- Skip List- Adaptive Radix Tree (ART)
Performance Space30 – 70%
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
ut (t
xn/s
)Did we solve this problem?
TPC-
C on
-Sto
re
Stay tuned
8
How do hybrid indexes achievememory savings ?
9
Static
dynamic stage static stage
Hybrid Index: a dual-stage architecture
10
dynamic stage static stage
write merge
Inserts are batched in the dynamic stage
11
dynamic stage static stage
Reads search the stages in order
12
dynamic stage static stage
read
A Bloom filter improves read performance
13
dynamic stage static stage
read
write merge
Memory-efficient
Skew-aware
14
~~~~~~~~
dynamic stage static stage
merge
The Dual-Stage Transformation
15
Compaction
Reduction
Compression
The Dynamic-to-Static Rules
16
2 4
4
1 2
a b
6 8 10
3
c
4
d
5 5
e f
5
g
6
h
7
i
8
j
9
k
10
l
11
m
12
n
Compaction: minimize # of memory blocks
17
1 2 3
a b c
4 5 6
d h
7 8 9
i j k
10 11 12
l m n
e f g
3 6 9
17
Compaction: minimize # of memory blocks
1 2 3
a b c
4 5 6
d h
7 8 9
i j k
10 11 12
l m n
e f g
3 6 9
Reduction: minimize structural overhead
18
2 4
4
1 2
a b
6 8 10
3 4
c d
5 5
e f
5 6
g h
7 8
i j
9 10
k l
11 12
m n
1 2 3
a b c
4 5 6
d h
7 8 9
i j k
10 11 12
l m n
e f g
3 6 9
18
Reduction: minimize structural overhead
dynamic stage static stage
merge
The Dual-Stage Transformation
19
dynamic stage static stage
merge
The Dual-Stage Transformation
19
MergeQuestions:1. Partial?2. When?3. Blocking?
??
?
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
ut (t
xn/s
)Did we solve this problem?
TPC-
C on
-Sto
re
B+tree
20
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
ut (t
xn/s
)
60K
20K
Yes, we improved the DBMS’s capacity!TP
C-C
on-S
tore
B+tree
Hybrid
20
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
TakeAway:Largerworkingsetinmemory
Higherthroughput
Memorysavedbyindexes
This is just the BEGINNING
22
Conclusions
The hybrid index architecture
The Dual-Stage Transformation
Applied to 4 index structures
GENERAL
PRACTICAL
USEFUL
23
- B+tree- Masstree
- Skip List- Adaptive Radix Tree (ART)
1-844-88-CMUDB
Toll-Free Hotline:
top related