NoSQL and MongoDB

1

2

Introduction to NoSQL

3

Based on a presentation by Traversy Media

What is NoSQL?Not only SQLSQL means

Relational modelStrong typingACID complianceNormalization…

NoSQL means more freedom or flexibility

4

Relevance to Big DataData gets biggerTraditional RDBMS cannot scale wellRDBMS is tied to its data and query processing modelsNoSQL relaxes some of the restrictions of RDBMS to provide a better performance

5

Advantages of NoSQLHandles Big DataData Models – No predefined schemaData Structure – NoSQL handles semi-structured dataCheaper to manageScaling – Scale out / horizonal scaling

6

Advantages of RDBMSBetter for relational dataData normalizationWell-established query language (SQL)Data IntegrityACID Compliance

7

Types of NoSQL DatabasesDocument Databases [MongoDB, CouchDB]Column Databases [Apache Cassandra]Key-Value Stores [Redis, Couchbase Server]Cache Systems [Redis, Memcached]Graph Databases [Neo4J]Streaming Systems [FlinkDB, Storm]

8

Structured/Semi-structured

9

ID Name Email …

1 Jack jack@example.com

2 Jill jill@example.net

3 Alex alex@example.org

Document 1

{ “id”: 1, “name”:”Jack”, “email”: “jack@example.com”, “address”: {“street”: “900 university ave”, “city”: “Riverside”, state: “CA”}, “friend_ids”: [3, 55, 123]}

Document 2

{ “id”: 2, “name”: “Jill”, “email”: “jill@example.net”, “hobbies”: [“hiking”, “cooking”]}

Columnar Data Store

10

ID

1

2

3

Name

Jack

Jill

Alex

Email

…

…

…

ID Name Email …

1 Jack jack@example.com

2 Jill jill@example.net

3 Alex alex@example.org

Key-value Stores

11

1 à Jack jack@example.com …

2 à Jill jill@example.net …

3 à Alex alex@example.org …

ID Name Email …

1 Jack jack@example.com

2 Jill jill@example.net

3 Alex alex@example.org

Document DatabaseMongoDB

12

Document Data ModelRelational model (RDBMS)

DatabaseRelation (Table) : Schema

Record (Tuple) : Data

Document ModelDatabase

Collection : No predefined schemaDocument : Schema+data

No need to define/update schemaNo need to create collections

13

Document 1

{ “id”: 1, “name”:”Jack”, “email”: “jack@example.com”, “address”: {“street”: “900 university ave”, “city”: “Riverside”, state: “CA”}, “friend_ids”: [3, 55, 123]}

Document FormatMongoDB natively works with JSON documentsFor efficiency, documents are stored in a binary format called BSON (i.e., binary JSON)Like JSON, both schema and data are stored in each document

14

How to Use MongoDB

15

db.users.insert({name: “Jack”, email: “jack@example.com”});

Install: Check the MongoDB websitehttps://docs.mongodb.com/manual/installation/

db.users.find();db.users.find({name: “Jack”});

db.users.update({name: "Jack"}, {$set: {hobby: "cooking"}});updateOne, updateMany, replaceOne

db.users.remove({name: "Alex"});deleteOne, deleteMany

Create collection and insert a document

Retrieve all/some documents

Update

Delete

https://docs.mongodb.com/manual/crud/

Schema ValidationYou can still explicitly create collections and enforce schema validation

16

db.createCollection("students", {validator: { $jsonSchema: {bsonType: "object",required: [ "name", "year", "major", "address" ],properties: {name: {bsonType: "string",description: "must be a string and is required" },

…}

}}}

https://docs.mongodb.com/manual/core/schema-validation/

Storage LayerPrior to MongoDB 3.2, only B-tree was available in the storage layerTo increase its scalability, MongoDB added LSM Tree in later versions after it acquired WiredTiger

17

mongod --wiredTigerIndexConfigString "type=lsm,block_compressor=zlib"

Override default configuration

LSM Vs B-tree

18https://github.com/wiredtiger/wiredtiger/wiki/Btree-vs-LSM

IndexingLike RDBMS, document databases use indexes to speed up some queries

MongoDB uses B-tree as an index structure

19https://docs.mongodb.com/manual/indexes/

Index TypesDefault unique _id indexSingle field index

db.collection.createIndex({name: -1});Compound index (multiple fields)

db.collection.createIndex( { name: 1, score: -1});Multikey indexes (for array fields)

Creates an index entry for each value

20https://docs.mongodb.com/manual/indexes/

Index TypesGeospatial index (for geospatial points)

Uses geohash to convert two dimensions to one dimension2d indexes: For Euclidean spaces2d sphere: spherical (earth) geometryWorks with multikey indexes for multiple locations (e.g., pickup and dropoff locations for taxis)

Text Indexes (for string fields)Automatically removes stop wordsStems the works to store the root only

Hashed Indexes (for point lookups)21

Geohashes

22

Additional Index FeaturesUnique indexes: Rejects duplicate keysSparse Indexes: Skips documents without the index field

In contrast, non-sparse indexes assume a null value if the index field does not exist

Partial indexes: Indexes only a subset of records based on a filter.

23

db.restaurants.createIndex({ cuisine: 1, name: 1 },{ partialFilterExpression: { rating: { $gt: 5 } } }

)

Comparison of data typesMin key (internal type)NullNumbers (32-bit integer, 64-bit integer, double)Symbol, StringObjectArrayBinary dataObject IDBooleanDate, timestampRegular expressionMax key (internal type)

24https://docs.mongodb.com/v3.6/reference/bson-type-comparison-order/

Comparison of data typesNumbers: All converted to a common typeStrings

Alphabetically (default)Collation (i.e., locale and language)

Arrays<: Smallest value of the array>: Largest value of the arrayEmpty arrays are treated as null

ObjectCompare fields in the order of appearanceCompare <name,value> for each field

25

Distributed ProcessingTwo methods for distributed processing

Replication (Similar to MySQL)Sharding (True horizontal scaling)

26

Replicationhttps://docs.mongodb.com/manual/replication/

Shardinghttps://docs.mongodb.com/manual/sharding/

Distributed Index StructureLog-structured Merge Tree (LSM)

27

Big Data IndexingHadoop and Spark are good in scanning large filesWe would like to speed up point and range queries on big data for some queriesHDFS limitation: Random updates are not allowedLog-structured Merge Tree (LSM-Tree) is adopted to address this problem.

28

RDBMS Indexing

29

New record

Index

Log

Index Update

30

New record

Randomly updated disk page(s)

Append a disk page

LSM TreeKey idea: Use the log as the indexRegularly: Merge the logs to consolidate the index (i.e., remove redundant entries)

31

New records LogLogLogLogLog

Flush Merge

Bigger log

O’Neil, Patrick, Edward Cheng, Dieter Gawlick, and Elizabeth O’Neil. "The log-structured merge-tree (LSM-tree)." Acta Informatica 33, no. 4 (1996): 351-385.

LSM in Big DataFirst major application: BigTable (Google)

32

0

20

40

60

80

100

120

1997199819992000200120022003200420052006200720082009201020112012201320142015201620172018

Citations

Citations

First report from Google mentioning LSM

BigTable paper

LSM in Big DataBuffer data in memory (memory component)Flush records to disk into an LSM as a disk component (sequential write)Disk components are sorted by keyCompact (merge) disk components in the background (sequential read/write)

33

ConclusionMongoDB is a document database that is geared towards high update rates and transactional queriesIt adopts JSON as a data modelIt provides the flexibility to insert any kind of data without schema definitionLSM Tree is used for indexingWeak types are handled using a special comparison method for all types

34