data system - human fault-tolerance
DESCRIPTION
TRANSCRIPT
![Page 1: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/1.jpg)
Human fault-tolerance
Nathan MarzTwitter 1
![Page 2: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/2.jpg)
What is a data system?
A system that manages the storage and querying of data
![Page 3: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/3.jpg)
What is a data system?
A system that manages the storage and querying of data with a lifetime measured in years
![Page 4: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/4.jpg)
What is a data system?
A system that manages the storage and querying of data with a lifetime measured in years encompassing every version of the application to ever exist
![Page 5: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/5.jpg)
What is a data system?
A system that manages the storage and querying of data with a lifetime measured in years encompassing every version of the application to ever exist, every hardware failure
![Page 6: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/6.jpg)
What is a data system?
A system that manages the storage and querying of data with a lifetime measured in years encompassing every version of the application to ever exist, every hardware failure, and every human mistake ever made
![Page 7: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/7.jpg)
![Page 8: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/8.jpg)
![Page 9: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/9.jpg)
![Page 10: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/10.jpg)
![Page 11: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/11.jpg)
+
![Page 12: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/12.jpg)
![Page 13: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/13.jpg)
![Page 14: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/14.jpg)
![Page 15: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/15.jpg)
![Page 16: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/16.jpg)
We don’t know how to make perfect software
![Page 17: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/17.jpg)
PaxosreplicationCRC’sBASEquorum
tunable consistencyactive anti-entropymerkle treesread repairconsistent hashing
Machine fault-tolerance
![Page 18: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/18.jpg)
![Page 19: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/19.jpg)
The worst consequence is data loss or data corruption
![Page 20: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/20.jpg)
As long as an error doesn’t lose or corrupt good data,
you can fix what went wrong
![Page 21: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/21.jpg)
This brings us to mutability
![Page 22: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/22.jpg)
Mutability
![Page 23: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/23.jpg)
The U and D in CRUD
![Page 24: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/24.jpg)
Mutable systems inherently lack human fault-tolerance
![Page 25: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/25.jpg)
Very easy for a mistake to corrupt or lose data
![Page 26: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/26.jpg)
Immutability
![Page 27: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/27.jpg)
Immutability• An immutable system captures a historical record of events• Each event happens at a particular time and is always true
![Page 28: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/28.jpg)
Capturing change with mutable data model
Person Location
Sally Philadelphia
Bob Chicago
Person Location
Sally New York
Bob Chicago
Sally moves to New York
![Page 29: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/29.jpg)
Capturing change with immutable data model
Person Location Time
Sally Philadelphia 1318358351
Bob Chicago 1327928370
Person Location Time
Sally Philadelphia 1318358351
Bob Chicago 1327928370
Sally New York 1338469380
Sally moves to New York
![Page 30: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/30.jpg)
Immutability greatly restricts the range of errors that can cause
data loss or data corruption
![Page 31: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/31.jpg)
Vastly more human fault-tolerant
![Page 32: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/32.jpg)
Plus much easier to reason about systems based on immutability
![Page 33: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/33.jpg)
What can we conclude?
![Page 34: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/34.jpg)
Your source of truth should be immutable
![Page 35: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/35.jpg)
Rather than build systems like this...
ApplicationMutable database
SOURCE OF TRUTH
![Page 36: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/36.jpg)
Rather than build systems like this...
Application
MySQLCassandra
HBaseVoldemort
Riak...
SOURCE OF TRUTH
![Page 37: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/37.jpg)
...build them like this
ApplicationViewImmutable
data
SOURCE OF TRUTH
![Page 38: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/38.jpg)
...build them like this
Application
CassandraHBase
VoldemortRiak
ElephantDB...
HDFS
SOURCE OF TRUTH
![Page 39: Data System - Human fault-tolerance](https://reader033.vdocuments.site/reader033/viewer/2022051210/54c672494a795913618b46a9/html5/thumbnails/39.jpg)
Long live immutability(forgive the pun)