nosql databases - id2221kth.github.io · abortedwithout a ecting the database. i consistency a...
TRANSCRIPT
![Page 3: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/3.jpg)
Where Are We?
2 / 89
![Page 4: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/4.jpg)
Database and Database Management System
I Database: an organized collection of data.
I Database Management System (DBMS): a software to capture and analyze data.
3 / 89
![Page 5: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/5.jpg)
Three Database Revolutions
[Guy Harrison, Next Generation Databases: NoSQLand Big Data, 2015]
4 / 89
![Page 6: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/6.jpg)
Early Database Systems
I There were databases but no Database Management Systems (DBMS).
[Guy Harrison, Next Generation Databases: NoSQLand Big Data, 2015]
5 / 89
![Page 7: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/7.jpg)
The First Database Revolution
I Navigational data model: hierarchical model (IMS) and network model (CODASYL).
I Disk-aware
[Guy Harrison, Next Generation Databases: NoSQLand Big Data, 2015]
6 / 89
![Page 8: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/8.jpg)
The Second Database Revolution
I Relational data model: Edgar F. Codd paper• Logical data is disconnected from physical information storage
I ACID transactions• Atomic, Consistent, Isolated, Durable
I SQL language
I Object databases• Information is represented in the form of objects
7 / 89
![Page 9: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/9.jpg)
ACID Properties
I Atomicity• All included statements in a transaction are either executed or the whole transaction is
aborted without affecting the database.
I Consistency• A database is in a consistent state before and after a transaction.
I Isolation• Transactions can not see uncommitted changes in the database.
I Durability• Changes are written to a disk before a database commits a transaction so that committed
data cannot be lost through a power failure.
8 / 89
![Page 10: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/10.jpg)
The Third Database Revolution
I NoSQL databases: BASE instead of ACID.
I NewSQL databases: scalable performance of NoSQL + ACID.
[http://ithare.com/nosql-vs-sql-for-mogs]
9 / 89
![Page 11: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/11.jpg)
Three Waves of Database Technology
[Guy Harrison, Next Generation Databases: NoSQLand Big Data, 2015]
10 / 89
![Page 12: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/12.jpg)
SQL vs. NoSQL Databases
11 / 89
![Page 13: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/13.jpg)
Relational SQL Databases
I The dominant technology for storing structured data in web and business applications.
I SQL is good• Rich language and toolset• Easy to use and integrate• Many vendors
I They promise: ACID
12 / 89
![Page 14: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/14.jpg)
SQL Databases Challenges
I Web-based applications caused spikes.• Internet-scale data size• High read-write rates• Frequent schema changes
I RDBMS were not designed to be distributed.
13 / 89
![Page 15: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/15.jpg)
Scaling SQL Databases is Expensive and Inefficient
[http://www.couchbase.com/sites/default/files/uploads/all/whitepapers/NoSQLWhitepaper.pdf]
14 / 89
![Page 16: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/16.jpg)
NoSQL
I Avoids:• Overhead of ACID properties• Complexity of SQL query
I Provides:• Scalablity• Easy and frequent changes to DB• Large data volumes
15 / 89
![Page 17: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/17.jpg)
NoSQL Cost and Performance
[http://www.couchbase.com/sites/default/files/uploads/all/whitepapers/NoSQLWhitepaper.pdf]
16 / 89
![Page 18: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/18.jpg)
SQL vs. NoSQL
[http://www.couchbase.com/sites/default/files/uploads/all/whitepapers/NoSQLWhitepaper.pdf]
17 / 89
![Page 19: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/19.jpg)
ACID vs. BASE
18 / 89
![Page 20: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/20.jpg)
Availability
I Replicating data to improve the availability of data.
I Data replication• Storing data in more than one site or node
19 / 89
![Page 21: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/21.jpg)
Consistency
I Strong consistency• After an update completes, any subsequent access will return the updated value.
I Eventual consistency• Does not guarantee that subsequent accesses will return the updated value.• Inconsistency window.• If no new updates are made to the object, eventually all accesses will return the last
updated value.
20 / 89
![Page 22: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/22.jpg)
CAP Theorem
I Consistency• Consistent state of data after the execution of an operation.
I Availability• Clients can always read and write data.
I Partition Tolerance• Continue the operation in the presence of network partitions.
I You can choose only two!
21 / 89
![Page 23: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/23.jpg)
Consistency vs. Availability
I The large-scale applications have to be reliable: availability, consistency, partitiontolerance
I Not possible to achieve with ACID properties.
I The BASE approach forfeits the ACID properties of consistency and isolation in favorof availability and performance.
22 / 89
![Page 24: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/24.jpg)
BASE Properties
I Basic Availability• Possibilities of faults but not a fault of the whole system.
I Soft-state• Copies of a data item may be inconsistent
I Eventually consistent• Copies becomes consistent at some later time if there are no more updates to that data
item
23 / 89
![Page 25: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/25.jpg)
ACID vs. BASE
[https://www.guru99.com/sql-vs-nosql.html]
24 / 89
![Page 26: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/26.jpg)
NoSQL Data Models
25 / 89
![Page 27: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/27.jpg)
NoSQL Data Models
[http://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques]
26 / 89
![Page 28: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/28.jpg)
Key-Value Data Model
I Collection of key/value pairs.
I Ordered Key-Value: processing over key ranges.
I Dynamo, Scalaris, Voldemort, Riak, ...
27 / 89
![Page 29: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/29.jpg)
Column-Oriented Data Model
I Similar to a key/value store, but the value can have multiple attributes (Columns).
I Column: a set of data values of a particular type.
I Store and process data by column instead of row.
I BigTable, Hbase, Cassandra, ...
28 / 89
![Page 30: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/30.jpg)
Document Data Model
I Similar to a column-oriented store, but values can have complex documents.
I Flexible schema (XML, YAML, JSON, and BSON).
I CouchDB, MongoDB, ...
{
FirstName: "Bob",
Address: "5 Oak St.",
Hobby: "sailing"
}
{
FirstName: "Jonathan",
Address: "15 Wanamassa Point Road",
Children: [
{Name: "Michael", Age: 10},
{Name: "Jennifer", Age: 8},
]
}
29 / 89
![Page 31: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/31.jpg)
Graph Data Model
I Uses graph structures with nodes, edges, and properties to represent and store data.
I Neo4J, InfoGrid, ...
[http://en.wikipedia.org/wiki/Graph database]
30 / 89
![Page 32: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/32.jpg)
BigTable
31 / 89
![Page 33: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/33.jpg)
BigTable
I Lots of (semi-)structured data at Google.• URLs, per-user data, geographical locations, ...
I Distributed multi-level map
I CAP: strong consistency and partition tolerance
32 / 89
![Page 34: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/34.jpg)
Data Model
33 / 89
![Page 35: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/35.jpg)
Data Model (1/7)
I Column-Oriented data model
I Similar to a key/value store, but the value can have multiple attributes (Columns).
I Column: a set of data values of a particular type.
I Store and process data by column instead of row.
34 / 89
![Page 36: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/36.jpg)
Data Model (2/7)
I In many analytical databases queries, few attributes are needed.
I Column values are stored contiguously on disk: reduces I/O.
[Lars George, Hbase: The Definitive Guide, O’Reilly, 2011]
35 / 89
![Page 37: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/37.jpg)
Data Model (3/7)
I Table
I Distributed multi-dimensional sparse map
36 / 89
![Page 38: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/38.jpg)
Data Model (4/7)
I Rows
I Every read or write in a row is atomic.
I Rows sorted in lexicographical order.
37 / 89
![Page 39: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/39.jpg)
Data Model (5/7)
I Column
I The basic unit of data access.
I Column families: group of (the same type) column keys.
I Column key naming: family:qualifier
38 / 89
![Page 40: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/40.jpg)
Data Model (6/7)
I Timestamp
I Each column value may contain multiple versions.
39 / 89
![Page 41: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/41.jpg)
Data Model (7/7)
I Tablet: contiguous ranges of rows stored together.
I Tablets are split by the system when they become too large.
I Each tablet is served by exactly one tablet server.
40 / 89
![Page 42: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/42.jpg)
System Architecture
41 / 89
![Page 43: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/43.jpg)
BigTable System Structure
[https://www.slideshare.net/GrishaWeintraub/cap-28353551]
42 / 89
![Page 44: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/44.jpg)
Main Components
I Master
I Tablet server
I Client library
43 / 89
![Page 45: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/45.jpg)
Master
I Assigns tablets to tablet server.
I Balances tablet server load.
I Garbage collection of unneeded files in GFS.
I Handles schema changes, e.g., table and column family creations
44 / 89
![Page 46: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/46.jpg)
Tablet Server
I Can be added or removed dynamically.
I Each manages a set of tablets (typically 10-1000 tablets/server).
I Handles read/write requests to tablets.
I Splits tablets when too large.
45 / 89
![Page 47: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/47.jpg)
Client Library
I Library that is linked into every client.
I Client data does not move though the master.
I Clients communicate directly with tablet servers for reads/writes.
46 / 89
![Page 48: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/48.jpg)
Building Blocks
I The building blocks for the BigTable are:• Google File System (GFS)• Chubby• SSTable
47 / 89
![Page 49: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/49.jpg)
Google File System (GFS)
I Large-scale distributed file system.
I Store log and data files.
48 / 89
![Page 50: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/50.jpg)
Chubby Lock Service
I Ensure there is only one active master.
I Store bootstrap location of BigTable data.
I Discover tablet servers.
I Store BigTable schema information and access control lists.
49 / 89
![Page 51: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/51.jpg)
SSTable
I SSTable file format used internally to store BigTable data.
I Chunks of data plus a block index.
I Immutable, sorted file of key-value pairs.
I Each SSTable is stored in a GFS file.
50 / 89
![Page 52: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/52.jpg)
Tablet Serving
51 / 89
![Page 53: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/53.jpg)
Master Startup
I The master executes the following steps at startup:
• Grabs a unique master lock in Chubby, which prevents concurrent master instantiations.
• Scans the servers directory in Chubby to find the live servers.
• Communicates with every live tablet server to discover what tablets are already assignedto each server.
• Scans the METADATA table to learn the set of tablets.
52 / 89
![Page 54: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/54.jpg)
Tablet Assignment
I 1 tablet → 1 tablet server.
I Master uses Chubby to keep tracks of live tablet serves and unassigned tablets.• When a tablet server starts, it creates and acquires an exclusive lock in Chubby.
I Master detects the status of the lock of each tablet server by checking periodically.
I Master is responsible for finding when tablet server is no longer serving its tabletsand reassigning those tablets as soon as possible.
53 / 89
![Page 55: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/55.jpg)
Finding a Tablet
I Three-level hierarchy.
I The first level is a file stored in Chubby that contains the location of the root tablet.
I Root tablet contains location of all tablets in a special METADATA table.
I METADATA table contains location of each tablet under a row.
I The client library caches tablet locations.
54 / 89
![Page 56: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/56.jpg)
Tablet Serving (1/2)
I Updates committed to a commit log.
I Recently committed updates are stored in memory - memtable
I Older updates are stored in a sequence of SSTables.
55 / 89
![Page 57: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/57.jpg)
Tablet Serving (2/2)
I Strong consistency• Only one tablet server is responsible for a given piece of data.• Replication is handled on the GFS layer.
I Trade-off with availability• If a tablet server fails, its portion of data is temporarily unavailable until a new server
is assigned.
56 / 89
![Page 58: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/58.jpg)
Loading Tablets
I To load a tablet, a tablet server does the following:
I Finds locaton of tablet through its METADATA.• Metadata for a tablet includes list of SSTables and set of redo points.
I Read SSTables index blocks into memory.
I Read the commit log since the redo point and reconstructs the memtable.
57 / 89
![Page 59: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/59.jpg)
BigTable vs. HBase
BigTable HBase
GFS HDFS
Tablet Server Region Server
SSTable StoreFile
Memtable MemStore
Chubby ZooKeeper
58 / 89
![Page 60: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/60.jpg)
HBase Example
# Create the table "test", with the column family "cf"
create ’test’, ’cf’
# Use describe to get the description of the "test" table
describe ’test’
# Put data in the "test" table
put ’test’, ’row1’, ’cf:a’, ’value1’
put ’test’, ’row2’, ’cf:b’, ’value2’
put ’test’, ’row3’, ’cf:c’, ’value3’
# Scan the table for all data at once
scan ’test’
# To get a single row of data at a time, use the get command
get ’test’, ’row1’
59 / 89
![Page 61: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/61.jpg)
Cassandra
60 / 89
![Page 62: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/62.jpg)
Cassandra
I A column-oriented database
I It was created for Facebook and was later open sourced
I CAP: availability and partition tolerance
61 / 89
![Page 63: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/63.jpg)
Borrowed From BigTable
I Data model: column oriented• Keyspaces (similar to the schema in a relational database), tables, and columns.
I SSTable disk storage• Append-only commit log• Memtable (buffering and sorting)• Immutable sstable files
62 / 89
![Page 64: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/64.jpg)
Data Partitioning (1/2)
I Key/value, where values are stored as objects.
I If size of data exceeds the capacity of a single machine: partitioning
I Consistent hashing for partitioning.
63 / 89
![Page 65: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/65.jpg)
Data Partitioning (2/2)
I Consistent hashing.
I Hash both data and node ids using the same hash function in a same id space.
I partition = hash(d) mod n, d: data, n: the size of the id space
id space = [0, 15], n = 16
hash("Fatemeh") = 12
hash("Ahmad") = 2
hash("Seif") = 9
hash("Jim") = 14
hash("Sverker") = 4
64 / 89
![Page 66: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/66.jpg)
Replication
I To achieve high availability and durability, data should be replicated on multiplenodes.
65 / 89
![Page 67: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/67.jpg)
Adding and Removing Nodes
I Gossip-based mechanism: periodically, each node contacts another randomly selectednode.
66 / 89
![Page 68: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/68.jpg)
Cassandra Example
# Create a keyspace called "test"
create keyspace test
with replication = {’class’: ’SimpleStrategy’, ’replication_factor’: 1};
# Print the list of keyspaces
describe keyspaces;
# Navigate to the "test" keyspace
use test
# Create the "words" table in the "test" keyspace
create table words (word text, count int, primary key (word));
# Insert a row
insert into words(word, count) values(’hello’, 5);
# Look at the table
select * from words;
67 / 89
![Page 69: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/69.jpg)
Neo4j
68 / 89
![Page 70: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/70.jpg)
Neo4j
I A graph database
I The relationships between data is equally important as the data itself
I Cypher: a declarative query language similar to SQL, but optimized for graphs
I CAP: strong consistency and availability
69 / 89
![Page 71: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/71.jpg)
Data Model (1/4)
I Node (Vertex)• The main data element from which graphs are constructed.• A waypoint along a traversal route
70 / 89
![Page 72: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/72.jpg)
Data Model (2/4)
I Relationship (Edge)
I May contain• Direction• Metadata, e.g., weight or relationship type
71 / 89
![Page 73: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/73.jpg)
Data Model (3/4)
I Label• Define node category (optional)• Can have more than one
72 / 89
![Page 74: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/74.jpg)
Data Model (4/4)
I Properties• Enrich a node or relationship
73 / 89
![Page 75: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/75.jpg)
Example
[Ian Robinson et al., Graph Databases, 2015]
74 / 89
![Page 76: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/76.jpg)
How a Graph is Physically Stored in Neo4j? (1/2)
I Neo4j stores graph data in a number of different store files.
I Each store file contains the data for a specific part of the graph.• Separate stores for nodes, relationships, labels, and properties.
I The division of storage responsibilities facilitates performant graph traversals.
[Ian Robinson et al., Graph Databases, 2015]
75 / 89
![Page 77: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/77.jpg)
How a Graph is Physically Stored in Neo4j? (2/2)
[Ian Robinson et al., Graph Databases, 2015]
76 / 89
![Page 78: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/78.jpg)
77 / 89
![Page 79: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/79.jpg)
What is Cypher?
I Declarative query language
I (): Nodes
I []: Relationships
I {}: Properties
78 / 89
![Page 80: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/80.jpg)
Cypher Example (1/4)
// Match all nodes
MATCH (n)
RETURN n;
// Match all nodes with a Person label
MATCH (n:Person)
RETURN n;
// Match all nodes with a Person label and property name is ’Tom Hanks’
MATCH (n:Person {name: ’Tom Hanks’})
RETURN n;
79 / 89
![Page 81: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/81.jpg)
Cypher Example (2/4)
// Return nodes with label Person and name property equals ’Tom Hanks’
MATCH (p:Person)
WHERE p.name = ’Tom Hanks’
RETURN p;
// Return nodes with label Movie, released property is between 1991 and 1999
MATCH (m:Movie)
WHERE m.released > 1990 AND m.released < 2000
RETURN m;
// Find all the movies Tom Hanks acted in
MATCH (:Person {name:’Tom Hanks’})-[:ACTED_IN]->(m:Movie)
RETURN m.title;
80 / 89
![Page 82: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/82.jpg)
Cypher Example (3/4)
// Find all the movies Tom Hanks directed and order by latest movie
MATCH (:Person {name:’Tom Hanks’})-[:DIRECTED]->(m:Movie)
RETURN m.title, m.release ORDER BY m.release DESC;
// Find all of the co-actors Tom Hanks has ever worked with
MATCH (:Person {name:’Tom Hanks’})-->(:Movie)<-[:ACTED_IN]-(coActor:Person)
RETURN coActor.name;
81 / 89
![Page 83: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/83.jpg)
Cypher Example (4/4)
// Find nodes with an ACTED_IN relationship
MATCH (p)-[:ACTED_IN]->()
RETURN p
// Find Person nodes with an ACTED_IN or DIRECTED_IN relationship
MATCH (p:Person)-[:ACTED_IN|DIRECTED]->()
RETURN p
// Find Person nodes who do not have an ACTED_IN relationship
MATCH (p:Person)
WHERE NOT (p)-[:ACTED_IN]->()
RETURN p
82 / 89
![Page 84: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/84.jpg)
Summary
83 / 89
![Page 85: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/85.jpg)
Summary
I NoSQL data models: key-value, column-oriented, document-oriented, graph-based
I Sharding and consistent hashing
I ACID vs. BASE
I CAP (Consistency vs. Availability)
84 / 89
![Page 86: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/86.jpg)
Summary
I BigTable
I Column-oriented
I Main components: master, tablet server, client library
I Basic components: GFS, SSTable, Chubby
I CP
85 / 89
![Page 87: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/87.jpg)
Summary
I Cassandra
I Column-oriented (similar to BigTable)
I Consistency hashing
I Gossip-based membership
I AP
86 / 89
![Page 88: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/88.jpg)
Summary
I Neo4j
I Graph-based
I Cypher
I CA
87 / 89
![Page 89: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/89.jpg)
References
I F. Chang et al., Bigtable: A distributed storage system for structured data, ACMTransactions on Computer Systems (TOCS) 26.2, 2008.
I A. Lakshman et al., Cassandra: a decentralized structured storage system, ACMSIGOPS Operating Systems Review 44.2, 2010.
I I. Robinson et al., Graph Databases (2nd ed.), O’Reilly Media, 2015.
88 / 89
![Page 90: NoSQL Databases - id2221kth.github.io · abortedwithout a ecting the database. I Consistency A database is in aconsistentstate before and after a transaction. I Isolation Transactions](https://reader034.vdocuments.site/reader034/viewer/2022042307/5ed37b13847f87317f77bcff/html5/thumbnails/90.jpg)
Questions?
AcknowledgementsSome content of the Neo4j slides were derived from Ljubica Lazarevic’s slides.
89 / 89