zero to mongo in 60 hours

Zero to Mongo in 60 Hours

Ryan AngillyMyPunchbowl.com

@angilly

Wednesday, April 28, 2010

I’m a pretty awesome dude



• Electrical Engineer by education




• Ex-EMC’er




• Ex-EMC’er

• Founded MessageSling.com




• Ex-EMC’er


• Entered the deadpool in January




• Ex-EMC’er



• Senior developer @ MyPunchbowl.com




• Ex-EMC’er




• Built several web apps




• Ex-EMC’er





• All Ruby. Mostly Rails




• Ex-EMC’er






• All SQL




• Ex-EMC’er






• All SQL

• No experience w/ NoSQL/Mongo as of September ’09




• Ex-EMC’er






• All SQL

• No experience w/ NoSQL/Mongo as of September ’09

• Want to provide a beginner’s perspective


MyPunchbowl is a pretty awesome company



• Leader in start to finish party planning




• Connecting party planners to party vendors -- subject of this talk





• Currently employs 11 people






• Funded over 2 rounds by Intel & Contour







• Strategic partnership w/ OTC among others








• Amount of traffic, users, events, invites, vendors, etc... make our data sets large enough to be interesting :)








• Amount of traffic, users, events, invites, vendors, etc... make our data sets large enough to be interesting :)

• We’re talking many millions of ‘things’


Our engineers play with everything


Search by category and location


Search by category and location

Search by business name and location


Track searches


Track searches

Track vendor impressions


MongoDB + MyPunchbowl

A tale in conversations


Tracking requirement a good excuse to finally use MongoDB



Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff.




Blake: Cool.




Blake: Cool.

Matt (CEO): What’s that?




Blake: Cool.


Blake & Me: Something cool.




Blake: Cool.


Blake & Me: Something cool.

Matt (CEO): K


6 reasons to use MongoDB

• Easy to get running (~5 minutes on OSX)

• Open Source

• Support in multiple (computer) languages

• Prototype in Ruby, move to Java if needed

• Very active development

• Full featured

• Great ecosystem


MongoDB feels right


MongoDB feels right

Me: MongoDB gives me the warm fuzzies that Rails did.


MongoDB feels right


Darren: Just like that Nunemaker post.


MongoDB feels right


Darren: Just like that Nunemaker post.

Me: ? *runs to google*


Support is INSANE

11:35pm. Wednesday. Founder.60 second response time.


2.5 days? Really?


2.5 days? Really?

Yes


2.5 days? Really?

Yes*


0 Days 1 Day 2 Days 200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB

200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB

Play around with available OSS

200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB


Figure out document ‘schema’

200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB



Write some stuff on top of mongo-ruby-driver

200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB



Build test rig


200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB



Setup Configuration Management

Build test rig


200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB



Get production ready


Build test rig


200+ Daysand on...

2.5 days to MongoDB


0 Days 1 Day 2 Days

Decision to use

MongoDB



Get production ready


Build test rig


Deploy

200+ Daysand on...

2.5 days to MongoDB


A stroll through Mongo’s OSS ecosystem



• mongodb (C++)



• mongodb (C++)

• mongo shell (JS via SpiderMonkey)



• mongodb (C++)


• mongo-ruby-driver



• mongodb (C++)



• mongo-java-driver



• mongodb (C++)




• mongo_record (Ruby)



• mongodb (C++)





• mongo_mapper (Ruby)



• mongodb (C++)





• mongo_mapper (Ruby)

• Integration w/ Rails


Document ‘schema’



• What do documents look like?




• Example Document

{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}





{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}

• How do you update the documents?





{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}


• Mongo shell command

coll.update({‘city’: ‘Boston’,

‘state’: ‘MA’,

‘date’: 1272153600,

‘category_id’: 1},

{‘$inc’: {‘occurrences’: 1}},

true)





{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}





‘date’: 1272153600,



true)

Document to match





{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}





‘date’: 1272153600,



true)

Document to match

Operation to perform





{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}





‘date’: 1272153600,



true)

Document to match

Operation to perform

‘upsert’: update if it’s there

insert if it’s not


Cuhhraazy Indexes


Cuhhraazy Indexes

• Original Document

{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}

• 40k cities, 200 days, 60 categories


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}


• 480M potential documents


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}



• Composite indexes

• state_1_city_1

• state_1_date_1_city_1

• category_id_1_date_1

• date_1


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}




• state_1_city_1



• date_1

• Flexibility & performance inquerying and aggregating


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}




• state_1_city_1



• date_1


• More Complex Document w/ Embedded Document

{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 9581555,

category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

},

zip_codes: [“01518”]

}

}


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}




• state_1_city_1



• date_1



{

city: “Boston”,

state: “MA”,

date: 1272153600,


category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

},


}

}

• Additional, complex data down the road?


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}




• state_1_city_1



• date_1



{

city: “Boston”,

state: “MA”,

date: 1272153600,


category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

},


}

}


• Add deep embedded indexes!


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}




• state_1_city_1



• date_1



{

city: “Boston”,

state: “MA”,

date: 1272153600,


category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

},


}

}



• ‘mobile_sources.zip_codes’


Cuhhraazy Indexes


{

city: “Boston”,

state: “MA”,

date: 1272153600,

occurrences: 1,

category_id: 1

}




• state_1_city_1



• date_1



{

city: “Boston”,

state: “MA”,

date: 1272153600,


category_id: 1,

mobile_sources: {

browsers: {

windows_mobile: 1,

palm_os: 0,

iphone_4g: 9481554

},


}

}



• ‘mobile_sources.zip_codes’

• ‘mobile_sources.browsers.iphone_4g’


Let’s write some Ruby


Let’s write some Rubyclass SearchStats

cattr_accessor :db, :collection

self.collection = $mongo_db.collection ‘searches‘

def self.record_search(city, state, category_id, vendor_id)

collection.update({:vendor_id => vendor_id,

:city => city,

:state => state,

:category_id => category_id,

:date => Time.now.utc.beginning_of_day.to_i},

{'$inc' => {:occurrence => 1}},

{:upsert => true})

end

end







:city => city,

:state => state,




{:upsert => true})

end

end

irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42)







:city => city,

:state => state,




{:upsert => true})

end

end

irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42)

mongo> db.searches.findOne()

{ "city" : "Boston",

"date" : 1272326400,

"impression" : 1,

"category_id" : 9,

"state" : "MA",

"vendor_id" : 42 }


TESTING


TESTING• Pretty much the same as anything else



• test/unit/search_stats_test.rb




• config/environments/test.rb





• $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’






• SearchStatsTest#setup & #teardown







• No transactions







• No transactions

• ActiveSupport spoils us







• No transactions

• ActiveSupport spoils us

• Gotta clean up after yourself


Other developers gotta use it too



• Rails’ database.yml takes care of MySQL




• Gotta switch up MongoDBs on your own





• Simple config system file inspired by another Nunemaker post.






• No database migrations!






• No database migrations!

• Must write rake tasks to create indexes unless your ORM takes care of it for you


Making production ready



• Replication was iffy in 1.0.0 (solid now)




• Nightly dumps w/ mongoexport





• Upload to S3





• Upload to S3

• Monitor process w/ monit


Deploy


Deploy

• Original deployment uneventful


Deploy


• Just worked


Deploy


• Just worked

• Future deployments that required a database upgrade required downtime (bummer)


Deploy


• Just worked

• Future deployments that required a database upgrade required downtime (bummer)

• We wanted to play with the geospatial search


Day 200


Day 200

• One production process


Day 200


• One database


Day 200


• One database

• Daily dumps using mongoexport + upload to S3


Day 200


• One database


• Replication not running


Day 200


• One database



• Several collections


Day 200


• One database




• 100k to >10M documents


Day 200


• One database




• 100k to >10M documents

• Scores of deep, composite indexes


What tripped me up?


Simple Test

should "get impressions_by_date" do

record1 = {:date => 100, :impression => 10}

record2 = {:date => 90, :impression => 8}

Mongo::Vendor.collection.insert record1

Mongo::Vendor.collection.insert record2

assert_same_elements [record1, record2],

Mongo::Vendor.impressions_by_date

end


Crazy Failure

1) Failure:

test: Mongo::Vendor should get

impressions_by_date. (Mongo::VendorTest)

...

-<{{"date"=>100.0, "csum"=>10.0}=>1,

{"date"=>90.0, "csum"=>8.0}=>1}>

+<{{"date"=>90.0, "csum"=>8.0}=>1,

{"date"=>100.0, "csum"=>10.0}=>1}>


Mongo’s BSON OrderedHash doesn’t behave like ActiveSupport::OrderedHash


irb(main):001:0> require 'active_support'irb(main):002:0> include ActiveSupportirb(main):003:0> oh = OrderedHash.newirb(main):004:0> oh[:b] = 2irb(main):005:0> oh[:c] = 3irb(main):006:0> oh[:a] = 1irb(main):007:0> a = {:a => 1, :b => 2, :c => 3}irb(main):008:0> a == oh=> trueirb(main):009:0> oh == a=> true




irb(main):001:0> require 'mongo'irb(main):002:0> oh = OrderedHash.newirb(main):003:0> oh[:b] = 2irb(main):004:0> oh[:c] = 3irb(main):005:0> oh[:a] = 1irb(main):006:0> a = {:a => 1, :b => 2, :c => 3}irb(main):007:0> a == oh=> true




irb(main):001:0> require 'mongo'irb(main):002:0> oh = OrderedHash.newirb(main):003:0> oh[:b] = 2irb(main):004:0> oh[:c] = 3irb(main):005:0> oh[:a] = 1irb(main):006:0> a = {:a => 1, :b => 2, :c => 3}irb(main):007:0> a == oh=> true


irb(main):008:0> oh == a=> false


And it turns out...


And it turns out...

MongoDB will not add indexes to your collections.


And it turns out...

MongoDB will not add indexes to your collections.

Unless you tell it to.


add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes


But it’ll help pick up the pieces

Before index> db.vendors.findOne({'date': {'$gt': 1271895200}}){ ... }Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms

During index> db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendorsSun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" }! ! ...Sun Apr 25 01:21:35 ! done building bottom layer, going to commitSun Apr 25 01:21:35 done for ~10M records 78.56secsSun Apr 25 01:21:35 insert v_production.system.indexes 78560ms

After Index> db.vendors.find({'date': 1272153600}).explain(){! ...! "nscanned" : 11220,! "millis" : 12,! ...}






Full collection scan ~3.5s







Index added ~79sNo locks.








12ms!








12ms!

During index creation, system completely CRUD responsive. No impact to

MyPunchbowl.com


Up next: Top Secret Project



• Project involving




• SCREAMS for a document-based solution





• mongosphinx

• MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building

• Geo-spatial search not yet ellipsoidal (or even spherical)





• mongosphinx



• paperclip+mongomapper





• mongosphinx




• MyPunchbowl user/event/invite data mining/visualization





• mongosphinx





• Charding + MapReduce = Nerdstorm





• mongosphinx





• Charding + MapReduce = Nerdstorm

• Building tools to speed up ETL out of MySQL and into MongoDB


The end

Questions?


Contact Me

[email protected]

http://www.mypunchbowl.com

http://ryanangilly.com

@angilly


mailto:[email protected]

mailto:[email protected]





Links

Slide 3/4 logos, wget’d from all over the Internets.

Dunce Cap http://steynian.files.wordpress.com/2009/09/dunce-cap.jpg

http://railstips.org/blog/archives/2009/12/18/why-i-think-mongo-is-to-databases-what-rails-was-to-frameworks/

http://railstips.org/blog/archives/2009/11/10/config-so-simple-your-mama-could-use-it/

Vendors configuration http://gist.github.com/380213


http://steynian.files.wordpress.com/2009/09/dunce-cap.jpg

http://steynian.files.wordpress.com/2009/09/dunce-cap.jpg





http://gist.github.com/380213

http://gist.github.com/380213

zero to mongo in 60 hours

Technology

pretty awesome companyleader

party vendors

party planners

nish party planning

education exemcer

web apps exemcer

senior developer

experience wnosqlmongo