persistence smoothie

Persistence SmoothieBlending SQL and NoSQL

Michael BleighIntridea, Inc.

photo by Nikki L. via Flickr

Thursday, March 11, 2010

present.ly


tweetstream hashieacts-as-taggable-on

subdomain-fu seed-fumustache_json

github.com/intridea


@mbleigh


You’ve (probably)heard a lot about

NoSQL


NoSQL is a new way to think about

persistence


AtomicityConsistency

IsolationDurability


DenormalizationEventual Consistency

Schema-FreeHorizontal Scale


NoSQL tries to scale (more) simply


NoSQL is going mainstream


New York TimesBusiness InsiderBBC ShopWikiGitHub Meebo

Disqus SourceForgeSony Digg


...but not THAT mainstream.


A word of caution...


NoSQL can divide by zero


NoSQL can divide by zero

NoSQL doesn’t

sleep, it waits

NoSQL counted to infinity, twiceThursday, March 11, 2010

NoSQL is a (growing) collection of tools, not

a new way of life


Key-Value StoresDocument Databases

Column StoresGraph Databases


Key-Value Stores


Redis

• Key-value store + datatypes

• Lists, (Scored) Sets, Hashes

• Cache-like functions (expiration)

• (Mostly) In-Memory


Riak

• Combo key-value store and document database

• HTTP REST interface

• “Link walking”

• Map-Reduce


Map/Reduce

• Massively parallel way to process large datasets

• First you scour data and “map” a new set of data

• Then you “reduce” the data down to a salient result


map = function() { this.tags.forEach(function(tag) { emit(tag, {count: 1}); });}

reduce = function(key, values) { var total = 0; for (var i = 0; i < values.length; i++) { total += values[i].count; return {count: total};}


Tokyo CabinetDynomite

MemcachedDBVoldemort


Document Databases


MongoDB

• Document store that speaks BSON (Binary JSON)

• Indexing, simple query syntax

• GridFS

• Deliberate MapReduce


CouchDB

• JSON Document Store

• HTTP REST Interface

• Incremental MapReduce

• Intelligent Replication


Column-Oriented Datastores


Cassandra

• Built by Facebook, used by Twitter

• Pure horizontal scalability

• Schemaless


Graph Databases


Neo4J


When should I use this stuff?


Complex, slow joins for “activity stream”


Complex, slow joins for “activity stream”

Denormalize, use Key-Value Store


Variable schema,vertical interaction


Variable schema,vertical interaction

Document Databaseor Column Store


Modeling multi-step relationships


Modeling multi-step relationships

Graph Database


NoSQL solves real scalability and data

design issues


Ben Scofieldbit.ly/state-of-nosql


Ready to go?


Just one problem...


Your data is already in a SQL database


We CAN all just get along.


Three Ways


The Hard(ish) Way


The Easy Way


A Better Way?


The Hard Way:Do it by hand.


class Post include MongoMapper::Document key :title, String key :body, String key :tags, Array key :user_id, Integer def user User.find_by_id(self.user_id) end def user=(some_user) self.user_id = some_user.id endend

class User < ActiveRecord::Base def posts(options = {}) Post.all({:conditions => {:user_id => self.id}}.merge(options)) endend


Pros & Cons

• Simple, maps to your domain

• Works for small, simple ORM intersections

• MUCH simpler in Rails 3

• Complex relationships are a mess

• Makes your models fat

• As DRY as the ocean


The Easy Way:DataMapper


DataMapper

• Generic, relational ORM

• Speaks pretty much everything you’ve ever heard of

• Implements Identity Map

• Module-based inclusion


DataMapper.setup(:default, "mysql://localhost")DataMapper.setup(:mongodb, "mongo://localhost/posts")

class Post include DataMapper::Resource def self.default_repository_name; :mongodb; end property :title, String property :body, String property :tags, Array belongs_to :userend

class User include DataMapper::Resource property :email, String property :name, String has n, :postsend


Pros & Cons

• The ultimate Polyglot ORM

• Simple relationships between persistence engines are easy

• Jack of all trades, master of none

• Perpetuates (sometimes) false assumptions

• Legacy stuff is in ActiveRecord anyway


Is there a better way?


Maybe.


Gloo: Cross-ORM Relationship Mapper

github.com/intridea/gloo


0.0.0.prealpha.1


Can’t we just sit down and talk to

each other?


class Post include MongoMapper::Resource key :title, String key :body, String key :tags, Array gloo :active_record do belongs_to :user endend

class User < ActiveRecord::Base gloo :mongo_mapper do many :posts endend


Goals/Status

• Be able to define relationships on the terms of any ORM from any class, ORM or not

• Right Now: Partially working ActiveRecord relationships

• Doing it wrong? Maybe


Code Time: Schema4Less


Social Storefront

• Dummy application of a store that lets others “follow” your purchases (a less creepy Blippy?)

• Four requirements:

• users

• purchasing

• listings

• social graph


Users

• I already have an authentication system

• I’m happy with it

• It’s Devise and ActiveRecord

• Stick with SQL


Purchasing

• Users need to be able to purchase items from my storefront

• I can’t lose their transactions

• I need full ACID

• I’ll use MySQL


Social Graph

• I want activity streams and one and two way relationships

• I need speed

• I don’t need consistency

• I’ll use Redis


Product Listings

• I am selling both movies and books

• They have very different properties

• Products are relatively non-relational

• I’ll use MongoDB


Demo and Walkthrough


Wrapping Up


These systems can (and should) live and

work together


Most important step is to actually think about data design


When you have a whole bag of tools, things stop looking

like nails


Questions?


persistence smoothie

Technology