Choosing the Right NOSQL Database

Tobias IvarssonHacker @ Neo Technology

twitter: @thobe / @neo4j / #neo4jemail: tobias@neotechnology.comweb: http://neo4j.org/web: http://thobe.org/

2Image credit: http://browsertoolkit.com/fault-tolerance.png

3Image credit: http://browsertoolkit.com/fault-tolerance.png

4Image credit: http://browsertoolkit.com/fault-tolerance.png

This is still the view a lot of people have of NOSQL.

The Technologies

๏Graph Databases- Neo4j

๏Document Databases- MongoDB

๏Column Family Database- Cassandra

5

Neo4j is a Graph Database

6

Graph databases FOCUS on the interconnection between entities.

6

IS_A

Neo4j Graph Database

Graph databases FOCUS on the interconnection between entities.

Other Graph Databases๏Neo4j

๏Sones GraphDB

๏ Infinite Graph (by Objectivity)

๏AllegroGraph (by Franz inc.)

๏HypergraphDB

๏ InfoGrid

๏DEX

๏VertexDB

๏FlockDB7

Document Databases

8

Document Databases๏MongoDB

๏Riak

๏CouchDB

๏SimpleDB (internal at Amazon)

9

ColumnFamily DBs

10

ColumnFamily Databases๏Cassandra

๏BigTable (internal at Google)

๏HBase (part of Hadoop)

๏Hypertable

11

Application 1:Blog system

12

Requirements for a Blog System

13

๏Get blog posts for a specific blog ordered by date

• possibly filtered by tag

๏Blogs can have an arbitrary number of blog posts

๏Blog posts can have an arbitrary number of comments

the choice:Document DB

14

“Schema” design

15

๏Represent each Blog as a Collection of Post documents

๏Represent Comments as nested documents in the Post documents

Creating a blog post

16

import com.mongodb.Mongo;import com.mongodb.DB;import com.mongodb.DBCollection;import com.mongodb.BasicDBObject;import com.mongodb.DBObject;// ...Mongo mongo = new Mongo( "localhost" ); // Connect to MongoDB// ...DB blogs = mongo.getDB( "blogs" ); // Access the blogs databaseDBCollection myBlog = blogs.getCollection( "myBlog" );

DBObject blogPost = new BasicDBObject();blogPost.put( "title", "JavaOne 2010" );blogPost.put( "pub_date", new Date() );blogPost.put( "body", "Publishing a post about JavaOne in my

MongoDB blog!" );blogPost.put( "tags", Arrays.asList( "conference", "java" ) );blogPost.put( "comments", new ArrayList() );

myBlog.insert( blogPost );

Retrieving posts// ...import com.mongodb.DBCursor;// ...

public Object getAllPosts( String blogName ) {DBCollection blog = db.getCollection( blogName );return renderPosts( blog.find() );

}

public Object getPostsByTag( String blogName, String tag ) {DBCollection blog = db.getCollection( blogName );return renderPosts( blog.find(

new BasicDBObject( "tags", tag ) ) );}

private Object renderPosts( DBCursor cursor ) {// order by publication date (descending)cursor = cursor.sort( new BasicDBObject( "pub_date", -1 ) );// ...

} 17

Adding a commentDBCollection myBlog = blogs.getCollection( "myBlog" );// ...

void addComment( String blogPostId, String message ) {DBCursor posts = myBlog.find(

new BasicDBObject( "_id", blogPostId );if ( !posts.hasNext() ) throw new NoSuchElementException();

DBObject blogPost = posts.next();

List comments = (List)blogPost.get( "comments" );comments.add( new BasicDBObject( "message", message )

.append( "date", new Date() ) );

myBlog.save( blogPost );}

18

Application 2:Twitter Clone

19

Requirements for a Twitter Clone

20

๏Handle high load - especially high write load

•Twitter generates 300GB of tweets / hour (April 2010)

๏Retrieve all posts by a specific user, ordered by date

๏Retrieve all posts by people a specific user follows, ordered by date

the choice:ColumnFamily DB

21

Schema design

22

๏Main keyspace: “Twissandra”, with these ColumnFamilies:

•User - user data, keyed by user id (UUID)

•Username - inverted index from username to user id

• Friends - who is user X following?

• Followers - who is following user X?

•Tweet - the actual messages

•Userline - timeline of tweets posted by a specific user

•Timeline - timeline of tweets posted by usersthat a specific user follows

... that’s a lot of denormalization ...๏ColumnFamilies are similar to tables in an RDBMS

๏Each ColumnFamily can only have one Key

๏This makes the data highly shardable

๏Which in turn enables very high write throughput

๏Note however that each ColumnFamily will require its own writes

•There are no ACID transactions

•YOU as a developer is responsible for Consistency!

• (again, this gives you really high write throughput)

23

Create user

24

new_useruuid = str(uuid())

USER.insert(useruuid, {'id': new_useruuid,'username': username,'password': password})

USERNAME.insert(username, {'id': new_useruuid})

FRIENDS.insert(useruuid, {frienduuid: time.time()})FOLLOWERS.insert(frienduuid, {useruuid: time.time()})

Follow user

Create messagetweetuuid = str(uuid())timestamp = long(time.time() * 1e6)

TWEET.insert(tweetuuid, {'id': tweetuuid,'user_id': useruuid,'body': body,'_ts': timestamp})

message_ref = {struct.pack('>d'),timestamp: tweetuuid}

USERLINE.insert(useruuid, message_ref)

TIMELINE.insert(useruuid, message_ref)for otheruuid in FOLLOWERS.get(useruuid, 5000): TIMELINE.insert(otheruuid, message_ref)

25

Get messages

timeline = TIMELINE.get(useruuid,column_start=start,column_count=NUM_PER_PAGE,column_reversed=True)

tweets = TWEET.multiget( timeline.values() )

timeline = USERLINE.get(useruuid,column_start=start,column_count=NUM_PER_PAGE,column_reversed=True)

tweets = TWEET.multiget( timeline.values() )

26

For all users this user follows

By a specific user

Application 3:Social Network

27

Requirements for a Social Network

28

๏ Interact with friends

๏Get recommendations for new friends

๏View the social context of a personi.e. How do I know this person?

the choice:Graph DB

29

“Schema” design

30

๏Persons represented by Nodes

๏Friendship represented by Relationships between Person Nodes

๏Groups represented by Nodes

๏Group membership represented by Relationshipfrom Person Node to Group Node

๏ Index for Person Nodes for lookup by name

๏ Index for Group Nodes for lookup by name

A small social graph example

31

Nebuchadnezzar crew

Agent taskforce

Morpheus

Agent Smith

Agent Brown

Cypher

Trinity

Thomas Anderson

Tank

Dozer

Qualifier: F

amily

Qualifier: Lovers

FRIENDSHIP

MEMBERSHIP

Creating the social graph

32

GraphDatabaseService graphDb = new EmbeddedGraphDatabase(GRAPH_STORAGE_LOCATION );

IndexService indexes = new LuceneIndexService( graphDb );Transaction tx = graphDb.beginTx();try {

Node mrAnderson = graphDb.createNode();mrAnderson.setProperty( "name", "Thomas Anderson" );mrAnderson.setProperty( "age", 29 );indexes.index( mrAnderson, "person", "Thomas Anderson" );Node morpheus = graphDb.createNode();morpheus.setProperty( "name", "Morpheus" );morpheus.setProperty( "rank", "Captain" );indexes.index( mrAnderson, "person", "Morpheus" );Relationship friendship = mrAnderson.createRelationshipTo(

morpheus, SocialGraphTypes.FRIENDSHIP );

tx.success();} finally {

tx.finish();}

Making new friendsNode person1 = indexes.getSingle( "persons", person1Name );Node person2 = indexes.getSingle( "persons", person2Name );

person1.createRelationshipTo(person2, SocialGraphTypes.FRIENDSHIP );

Node person = indexes.getSingle( "persons", personName );Node group = indexes.getSingle( "groups", groupName );

person.createRelationshipTo(group, SocialGraphTypes.MEMBERSHIP );

33

Joining a group

How do I know this person?Node me = ...Node you = ...

PathFinder shortestPathFinder = GraphAlgoFactory.shortestPath(Traversals.expanderForTypes(

SocialGraphTypes.FRIENDSHIP, Direction.BOTH ),/* maximum depth: */ 4 );

Path shortestPath = shortestPathFinder.findSinglePath(me, you);

for ( Node friend : shortestPath.nodes() ) {System.out.println( friend.getProperty( "name" ) );

}

34

Recommend new friendsNode person = ...

TraversalDescription friendsOfFriends = Traversal.description().expand( Traversals.expanderForTypes(

SocialGraphTypes.FRIENDSHIP, Direction.BOTH ) ).prune( Traversal.pruneAfterDepth( 2 ) ).breadthFirst() // Visit my friends before their friends.//Visit a node at most once (don’t recommend direct friends).uniqueness( Uniqueness.NODE_GLOBAL ).filter( new Predicate<Path>() {

// Only return friends of friendspublic boolean accept( Path traversalPos ) {

return traversalPos.length() == 2;}

} );

for ( Node recommendation : friendsOfFriends.traverse( person ).nodes() ) {System.out.println( recommendedFriend.getProperty("name") );

} 35

When to use Document DB (e.g. MongoDB)๏When data is collections of similar entities

•But semi structured (sparse) rather than tabular

•When fields in entries have multiple values

36

When to use ColumnFamily DB (e.g. Cassandra)๏When scalability is the main issue

•Both scaling size and scaling load

‣In particular scaling write load

๏Linear scalability (as you add servers) both in read and write

๏Low level - will require you to duplicate data to support queries

37

When to use Graph DB (e.g. Neo4j)๏When deep traversals are important

๏For complex and domains

๏When how entities relate is an important aspect of the domain

38

When not to use a NOSQL Database๏RDBMSes have been the de-facto standard for years, and still have

better tools for some tasks

• Especially for reporting

๏When maintaining a system that works already

๏Sometimes when data is uniform / structured

๏When aggregations over (subsets) of the entire dataset is key

๏But please don’t use a Relational database for persisting objects

39

Complex problem? - right tool for each job!

40Image credits: Unknown :’(

Polyglot persistence

41

๏Use multiple databases in the same system- use the right tool for each part of the system

๏Examples:

•Use an RDBMS for structured data and a Graph Database for modeling the relationships between entities

•Use a Graph Database for the domain model and a Document Database for storing large data objects

http://neotechnology.com

- the Graph Database company