zero to mongo in 60 hours

Zero to Mongo in 60 Hours

Ryan AngillyMyPunchbowl.com

@angilly

Wednesday, April 28, 2010

I’m a pretty awesome dude

• Electrical Engineer by education

• Ex-EMC’er

• Founded MessageSling.com

• Ex-EMC’er

• Entered the deadpool in January

• Ex-EMC’er

• Senior developer @ MyPunchbowl.com

• Ex-EMC’er

• Built several web apps

• Ex-EMC’er

• All Ruby. Mostly Rails

• Ex-EMC’er

• All SQL

• Ex-EMC’er

• All SQL

• No experience w/ NoSQL/Mongo as of September ’09

• Ex-EMC’er

• All SQL

• No experience w/ NoSQL/Mongo as of September ’09

• Want to provide a beginner’s perspective

MyPunchbowl is a pretty awesome company

• Leader in start to finish party planning

• Connecting party planners to party vendors -- subject of this talk

• Currently employs 11 people

• Funded over 2 rounds by Intel & Contour

• Strategic partnership w/ OTC among others

• Amount of traffic, users, events, invites, vendors, etc... make our data sets large enough to be interesting :)

• We’re talking many millions of ‘things’

Our engineers play with everything

Search by category and location

Search by business name and location

Track searches

Track vendor impressions

MongoDB + MyPunchbowl

A tale in conversations

Tracking requirement a good excuse to finally use MongoDB

Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff.

Blake: Cool.

Matt (CEO): What’s that?

Blake: Cool.

Blake & Me: Something cool.

Blake: Cool.

Blake & Me: Something cool.

Matt (CEO): K

6 reasons to use MongoDB

• Easy to get running (~5 minutes on OSX)

• Open Source

• Support in multiple (computer) languages

• Prototype in Ruby, move to Java if needed

• Very active development

• Full featured

• Great ecosystem

MongoDB feels right

Me: MongoDB gives me the warm fuzzies that Rails did.

MongoDB feels right

Darren: Just like that Nunemaker post.

MongoDB feels right

Me: ? *runs to google*

MongoDB feels right

Me: ? *runs to google*

Support is INSANE

11:35pm. Wednesday. Founder.60 second response time.

2.5 days? Really?

0 Days 1 Day 2 Days 200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Play around with available OSS

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Figure out document ‘schema’

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Write some stuff on top of mongo-ruby-driver

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Build test rig

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Setup Configuration Management

Build test rig

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Get production ready

Build test rig

200+ Daysand on...

2.5 days to MongoDB

0 Days 1 Day 2 Days

Decision to use

MongoDB

Get production ready

Build test rig

Deploy

200+ Daysand on...

2.5 days to MongoDB

A stroll through Mongo’s OSS ecosystem

• mongodb (C++)

• mongo shell (JS via SpiderMonkey)

• mongodb (C++)

• mongo-ruby-driver

• mongodb (C++)

• mongo-java-driver

• mongodb (C++)

• mongo_record (Ruby)

• mongodb (C++)

• mongo_mapper (Ruby)

• mongodb (C++)

• mongo_mapper (Ruby)

• Integration w/ Rails

Document ‘schema’

• What do documents look like?

• Example Document

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• How do you update the documents?

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• Mongo shell command

coll.update({‘city’: ‘Boston’,

‘state’: ‘MA’,

‘date’: 1272153600,

‘category_id’: 1},

{‘$inc’: {‘occurrences’: 1}},

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

‘date’: 1272153600,

Document to match

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

‘date’: 1272153600,

Document to match

Operation to perform

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

‘date’: 1272153600,

Document to match

Operation to perform

‘upsert’: update if it’s there

insert if it’s not

Cuhhraazy Indexes

• Original Document

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• 40k cities, 200 days, 60 categories

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• 480M potential documents

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• Composite indexes

• state_1_city_1

• state_1_date_1_city_1

• category_id_1_date_1

• date_1

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• state_1_city_1

• date_1

• Flexibility & performance inquerying and aggregating

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• state_1_city_1

• date_1

• More Complex Document w/ Embedded Document

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 9581555,

category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

zip_codes: [“01518”]

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• state_1_city_1

• date_1

city: “Boston”,

state: “MA”,

date: 1272153600,

category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

• Additional, complex data down the road?

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• state_1_city_1

• date_1

city: “Boston”,

state: “MA”,

date: 1272153600,

category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

• Add deep embedded indexes!

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• state_1_city_1

• date_1

city: “Boston”,

state: “MA”,

date: 1272153600,

category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

• ‘mobile_sources.zip_codes’

Cuhhraazy Indexes

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

• state_1_city_1

• date_1

city: “Boston”,

state: “MA”,

date: 1272153600,

category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

• ‘mobile_sources.zip_codes’

• ‘mobile_sources.browsers.iphone_4g’

Let’s write some Ruby

Let’s write some Rubyclass SearchStats

cattr_accessor :db, :collection

self.collection = $mongo_db.collection ‘searches‘

def self.record_search(city, state, category_id, vendor_id)

collection.update({:vendor_id => vendor_id,

:city => city,

:state => state,

:category_id => category_id,

:date => Time.now.utc.beginning_of_day.to_i},

{'$inc' => {:occurrence => 1}},

{:upsert => true})

:city => city,

:state => state,

{:upsert => true})

irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42)

:city => city,

:state => state,

{:upsert => true})

irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42)

mongo> db.searches.findOne()

{ "city" : "Boston",

"date" : 1272326400,

"impression" : 1,

"category_id" : 9,

"state" : "MA",

"vendor_id" : 42 }

TESTING

TESTING• Pretty much the same as anything else

• test/unit/search_stats_test.rb

• config/environments/test.rb

• $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’

• SearchStatsTest#setup & #teardown

• No transactions

• ActiveSupport spoils us

• No transactions

• ActiveSupport spoils us

• Gotta clean up after yourself

Other developers gotta use it too

• Rails’ database.yml takes care of MySQL

• Gotta switch up MongoDBs on your own

• Simple config system file inspired by another Nunemaker post.

• No database migrations!

• Must write rake tasks to create indexes unless your ORM takes care of it for you

Making production ready

• Replication was iffy in 1.0.0 (solid now)

• Nightly dumps w/ mongoexport

• Upload to S3

• Monitor process w/ monit

Deploy

• Original deployment uneventful

Deploy

• Just worked

Deploy

• Just worked

• Future deployments that required a database upgrade required downtime (bummer)

Deploy

• Just worked

• Future deployments that required a database upgrade required downtime (bummer)

• We wanted to play with the geospatial search

Day 200

• One production process

Day 200

• One database

Day 200

• One database

• Daily dumps using mongoexport + upload to S3

Day 200

• One database

• Replication not running

Day 200

• One database

• Several collections

Day 200

• One database

• 100k to >10M documents

Day 200

• One database

• 100k to >10M documents

• Scores of deep, composite indexes

What tripped me up?

Simple Test

should "get impressions_by_date" do

record1 = {:date => 100, :impression => 10}

record2 = {:date => 90, :impression => 8}

Mongo::Vendor.collection.insert record1

Mongo::Vendor.collection.insert record2

assert_same_elements [record1, record2],

Mongo::Vendor.impressions_by_date

Crazy Failure

1) Failure:

test: Mongo::Vendor should get

impressions_by_date. (Mongo::VendorTest)

-<{{"date"=>100.0, "csum"=>10.0}=>1,

{"date"=>90.0, "csum"=>8.0}=>1}>

+<{{"date"=>90.0, "csum"=>8.0}=>1,

{"date"=>100.0, "csum"=>10.0}=>1}>

Mongo’s BSON OrderedHash doesn’t behave like ActiveSupport::OrderedHash

irb(main):001:0> require 'active_support'irb(main):002:0> include ActiveSupportirb(main):003:0> oh = OrderedHash.newirb(main):004:0> oh[:b] = 2irb(main):005:0> oh[:c] = 3irb(main):006:0> oh[:a] = 1irb(main):007:0> a = {:a => 1, :b => 2, :c => 3}irb(main):008:0> a == oh=> trueirb(main):009:0> oh == a=> true

irb(main):001:0> require 'mongo'irb(main):002:0> oh = OrderedHash.newirb(main):003:0> oh[:b] = 2irb(main):004:0> oh[:c] = 3irb(main):005:0> oh[:a] = 1irb(main):006:0> a = {:a => 1, :b => 2, :c => 3}irb(main):007:0> a == oh=> true

irb(main):008:0> oh == a=> false

And it turns out...

MongoDB will not add indexes to your collections.

And it turns out...

MongoDB will not add indexes to your collections.

Unless you tell it to.

add indexes

But it’ll help pick up the pieces

Before index> db.vendors.findOne({'date': {'$gt': 1271895200}}){ ... }Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms

During index> db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendorsSun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" }! ! ...Sun Apr 25 01:21:35 ! done building bottom layer, going to commitSun Apr 25 01:21:35 done for ~10M records 78.56secsSun Apr 25 01:21:35 insert v_production.system.indexes 78560ms

After Index> db.vendors.find({'date': 1272153600}).explain(){! ...! "nscanned" : 11220,! "millis" : 12,! ...}

Full collection scan ~3.5s

Index added ~79sNo locks.

During index creation, system completely CRUD responsive. No impact to

MyPunchbowl.com

Up next: Top Secret Project

• Project involving

• SCREAMS for a document-based solution

• mongosphinx

• MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building

• Geo-spatial search not yet ellipsoidal (or even spherical)

• mongosphinx

• paperclip+mongomapper

• mongosphinx

• MyPunchbowl user/event/invite data mining/visualization

• mongosphinx

• Charding + MapReduce = Nerdstorm

• mongosphinx

• Charding + MapReduce = Nerdstorm

• Building tools to speed up ETL out of MySQL and into MongoDB

The end

Questions?

Contact Me

ryan@angilly.com

http://www.mypunchbowl.com

http://ryanangilly.com

@angilly

/4 logos, wget’d from all over the Internets.

Dunce Cap http://steynian.files.wordpress.com/2009/09/dunce-cap.jpg

http://railstips.org/blog/archives/2009/12/18/why-i-think-mongo-is-to-databases-what-rails-was-to-frameworks/

http://railstips.org/blog/archives/2009/11/10/config-so-simple-your-mama-could-use-it/

Vendors configuration http://gist.github.com/380213

zero to mongo in 60 hours

pretty awesome companyleader

party vendors

party planners

nish party planning

education exemcer

web apps exemcer

senior developer

experience wnosqlmongo

Technology

aspnetwebapi mongo

from zero to mongo, art.sy experience w/ mongodb

mongo gridfs

mongo db japan

full metal mongo

mongo replicaset

webinar | zero to account plan in 6 contact hours

mongo db introduction

mongo bbmw

mongo scaling

mongo + php

mongo santamaria

hello mongo

model zero hours contract of employment

mongo db y mongo db atlas - pandorafms.com

zero-hours contracts - insecurity of flexibility?

mongo db basics

mongo an alternative database system. installing mongo we...

mongo bongo

zero hours contracts (house of commons)