foundationdb technical overview v2 - qcon · foundationdb technical overview v2 created date:...
TRANSCRIPT
![Page 1: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/1.jpg)
Evan Tschannen
�1
![Page 2: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/2.jpg)
• Worked on FoundationDB for 8 years
• Touched every core component of the database
• Recently added multi-region asynchronous replication
Evan Tschannen
![Page 3: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/3.jpg)
• https://www.foundationdb.org
• https://github.com/apple/foundationdb
• Open Source, Apache License (v2)
![Page 4: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/4.jpg)
Why are there so many databases?
!4
![Page 5: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/5.jpg)
!5
• Tension between performance, scalability, and consistency
• A wide variety of data models
Why are there so many databases?
![Page 6: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/6.jpg)
FoundationDB Performance
• Organizes many instances of a single process database into a single database
• Read and write throughput 90% of the aggregate individual read and write throughput
• Single hop read latencies
• Four hop write latencies
![Page 7: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/7.jpg)
FoundationDB is a…
• Highly performant,
• Scalable,
• Ordered key value store,
• That supports ACID transactions.
![Page 8: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/8.jpg)
!8
A
B
C
Writer
Servers
Reader
Wait for 2 of 3 responses Wait for 2 of 3 responses
![Page 9: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/9.jpg)
!9
A
B
C
Writer
Servers
Reader
Wait for 2 of 3 responses Wait for 2 of 3 responses
![Page 10: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/10.jpg)
!10
A
B
C
Writer
Servers
Reader
Wait for 3 of 3 responses Wait for 1 of 3 responses
![Page 11: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/11.jpg)
!11
A
B
C
Writer
Servers
Reader
Wait for 3 of 3 responses Wait for 1 of 3 responses
![Page 12: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/12.jpg)
!12
A
B
C
Writer
Servers
Reader
Wait for 3 of 3 responses Wait for 1 of 3 responses
Versions(Stored in a different database)
D
0: A,B,C 101: B,C,D
![Page 13: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/13.jpg)
!13
Coordinators
Storage Servers
ZooKeeper (Failure Handling)
SQLite (Reads)
Transaction Logs
FoundationDB (Writes)
![Page 14: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/14.jpg)
!14
Coordinators
Storage Servers Transaction Logs
Cluster Controller
![Page 15: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/15.jpg)
!15
Coordinators
Storage Servers Transaction Logs
Cluster Controller
![Page 16: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/16.jpg)
!16
Get Read Version
ReadWrite
Commit
FoundationDB APIVersions
200
400
100 A : apple(200)
(A, 200)
(B, banana)
![Page 17: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/17.jpg)
!17
Get Read Version
ReadWrite
Commit
FoundationDB APIVersions
200
400
100 A : apple(200)
(A, 200)
(B, banana)300 A : orange
![Page 18: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/18.jpg)
!18
Coordinators
Storage Servers
Cluster Controller
Master
Proxies Resolvers Transaction Logs
![Page 19: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/19.jpg)
!19
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
![Page 20: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/20.jpg)
!20
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(B : banana)
Data Distribution
![Page 21: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/21.jpg)
!21
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(400)
(B : banana)
![Page 22: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/22.jpg)
!22
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(400)
(B : banana)
![Page 23: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/23.jpg)
!23
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(B : banana)
![Page 24: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/24.jpg)
!24
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(400)
(B : banana)
![Page 25: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/25.jpg)
!25
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
![Page 26: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/26.jpg)
!26
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(400)
(300)
(200)
![Page 27: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/27.jpg)
!27
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(400)
(400)
(300)
(200)
![Page 28: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/28.jpg)
!28
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
![Page 29: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/29.jpg)
!29
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
![Page 30: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/30.jpg)
!30
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
![Page 31: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/31.jpg)
!31
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
Data Distribution
![Page 32: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/32.jpg)
!32
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
![Page 33: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/33.jpg)
FoundationDB Performance
• Organizes many instances of a single process database into a single database
• Read and write throughput 90% of the aggregate individual read and write throughput
• Single hop read latencies
• Four hop write latencies
![Page 34: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/34.jpg)
Does it work?
• Run the entire distributed database in a single process
• Inject random failures
• Check correctness
• Reproduce errors
![Page 35: FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date: 11/7/2018 12:29:54 AM](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd865fafde6e3240c37faca/html5/thumbnails/35.jpg)
https://www.foundationdb.org
!35