high performance nosql with mongodb - sdd 2016sddconf.com/brands/sdd/library/high-performance... ·...

High Performance NoSQL with MongoDB

Michael Kennedy | @mkennedy | michaelckennedy.net

History of NoSQLJune 11th, 2009, San Francisco, USA

Johan Oskarsson (from http://last.fm/) organized a meetup to discuss advances in data storage which were all using distributed databases leveraging clusters. He asked the group for a short term they could use as a hashtag. [1]

Eric Evans (not of DDD fame) proposed #NoSQL and it stuck.

Michael's NoSQL Definition

Database systems which are cluster-friendly and which trade inter-entity relationships for both simplicity and performance.

Four types of "NoSQL" DBs• Key Value Stores

– Amazon DynamoDB– Redis

• Column-Oriented databases– Hbase– Cassandra– Google BigQuery

• Graph Databases– Neo4J– OrientDB

• Document Databases– MongoDB– CouchDB– DocumentDB (on Azure)

Key-value data storage

Column Oriented DBs

Graph DBs

Document DBs

Not so different

How much do you need perf?

Image credit: nerovivo

Relational 3NF models are complex

Document DBs for simplicity

Document db style

Document DBs for simplicity

Document db style

Single server performance

Single biggest performance problem (and fix)?

Incorrect indexes(too few or too many)

• Be data-driven: profile and then add indexesAdding indexes

• Indexes are more important than for RDBMSes

Adding indexes

Demo time

Step 1: Enable profiling

Step 2: Run common queries

Step 3: Analyze system.profile

Step 4: Add indexes for slow

Step 5: GOTO 1

Scaling out

Image credit: johnantoniImage credit: Torkild Retvedt

Scaling out• Scale-out is the great promise of NoSQL• MongoDB has two modes of scale out

– Sharding – Replication

Real-word statistics from one company

120,000 DB operations / second2GB of app-to-db I/O / second

Replication vs. scalability• Sharding is the primary way to improve single query speed• Replication is not the primary way to scale

– even though you may get better read performance, not much better write performance unless very read heavy

Server 1A-B-C-D-E

Server 4A-B-C-D-E

Server 2A-B-C-D-E

Server 3A-B-C-D-E

Server 5A-B-C-D-E

Server 1A

Server 4D

Server 2B

Server 3C

Server 5E

Replication Sharding

Sharding

Weather data from the entire 20th century in MongoDBCase study by MongoDB Inc:

http://www.mongodb.com/presentations/weather-century-part-2-high-performance

Scaling via Sharding – an example

• 2.5billiondatapoints• 4Terabyte(1.6kperdocument)

Data size and quantity

{ "st" : "u725053","ts" : ISODate("2013-06-03T22:51:00Z"),"airTemperature" : {

"value" : 21.1,"quality" : "5"

},"atmosphericPressure" : {

"value" : 1009.7,"quality" : "5"

Sample record (JSON)

class WeatherRecord{

public string st {get; set;}public DateTime ts {get; set;}public Temp airTemperature {get; set;}public Pressure atmosphericPressure {get; set;}

class Temp{

public int value {get; set;}public string quality {get; set;}

}class Pressure{

public int value {get; set;}public string quality {get; set;}

Sample record in C#

Asingleserverwithareallybigdisk

Application mongod

i2.8xlarge251 GB RAM

6 TB SSD

c3.8xlarge

Scale Up

AreallybigclusterwhereeverythingisinRAM

Application / mongos

...100 x r3.2xlarge

61 GB RAM@

100 GB disk

mongod

c3.8xlarge

Scale out configuration

AreallybigclusterwhereeverythingisinRAM

Application / mongos

...100 x r3.2xlarge

61 GB RAM@

100 GB disk

mongod

Can scale even more

$60,000 / yr

$700,000 / yr

Cost per year in AWS?

single server cluster

ms avg

max. throughput: 40,000/s 610,000/s

(10 mongos)

db.data.find({"st" : "u747940","ts" : ISODate("1969-07-16T12:00:00Z")})

Performance: single time and place

single server cluster

ms avg

max.throughput: 20/s 430/s

(10 mongos)

targeted query

db.data.find({"st" : "u747940","ts" : {"$gte": ISODate("1989-01-01"),

"$lt" : ISODate("1990-01-01")}})

Performance: 1 year's weather

61.8 °C = 143 °F

2 minCluster

4 h 45 minSingle Server

142x faster

db.data.aggregate([{ "$match" : { "airTemperature.quality" :

{ "$in" : [ "1", "5" ] } } },

{ "$group" : { "_id" : null,"maxTemp" : { "$max" :

"$airTemperature.value" } } }])

Analytics

Get the code and data

https://github.com/mikeckennedy/sdd2016

talkpython.fm

Want to go deeper?

training.talkpython.fm

michaelckennedy.netmikeckennedy@gmail.com

@mkennedy

high performance nosql with mongodb - sdd 2016sddconf.com/brands/sdd/library/high-performance... ·...

Documents

ultraismo sdd

dear sdd --> siddharta

hematology- 2013 sdd

efficiency metric sdd

programa sdd 2010

self-described data - sdd status and plans · sdd...

{sdd}€¦ · developmentor michael kennedy | @mkennedy |...

sdd offline status

ii. chaînage, sdd séquentielles

sdd example

sdd hsc summary

high performance nosql with mongodb - sdd 2016 · pdf...

job desc sdd

sdd ecodigoetica

bab i sp sdd

sdd braila 1 15.03.10

ipad presentation sdd

sdd - qubicom.co.kr

{sdd}sddconf.com/brands/sdd/library/an_introduction_to_python... ·...

hsc syllabus for sdd