mongo db athens user group replication and high availability
TRANSCRIPT
![Page 1: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/1.jpg)
MongoDB Athens User GroupAthens, Greece 16/1/2013
Alex [email protected]
Replication and High Availability
![Page 2: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/2.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
History
● Oracle shop
● Non existent OLAP
● Queries in live DB
![Page 3: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/3.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Initial investigation
● CouchDB
● Riak
● Hbase
● Cassandra
● MongoDB
● Voldemort
![Page 4: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/4.jpg)
History
● Missed all the fun while being in the States
● Hadoop / HBase / llama / Pig
● Sharded MySQL
● Voldemort
● Huge RAC deployments
● Following MongoDB since 1.4 (Replica sets? Nah.. Sharding?..alpha)
![Page 5: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/5.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting and Analytics
● Settled on MongoDB
● Document oriented
● No clue about schema at the time
● No clue about what are we going to do with client data
![Page 6: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/6.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Prototyping, version 0.5
● MT, MO, User collections
● Sync Map Reduce for reporting
● One DB to rule them all
![Page 7: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/7.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Show stoppers
● Single server deployment
● Global write lock
● MR in real time
![Page 8: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/8.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Results
● Demo Christmas Eve 2010
![Page 9: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/9.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Results
● Demo Christmas Eve 2010
● Slow....
![Page 10: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/10.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting, version 1.0
● Spring Batch for async computations
● Quartz scheduler firing every 3 minutes
● Separate nodes for OLTP and OLAP Dbs
● Custom cloneCollection()
![Page 11: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/11.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Real world kicks in
● Everything designed for online integration
● Huge client coming in offering offline integration
● Ride the cloud wagon!
![Page 12: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/12.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 2 (the real world)
● Files coming in via FTP containing all sorts of time inconsistencies
● No longer a linear timeline of events, more like a soup of results
![Page 13: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/13.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
MongoDB on EC2
● 2 replica sets of 2 nodes+arb
● Arbiters crossed wrt replica sets
● Third node could be different availability zone
![Page 14: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/14.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
![Page 15: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/15.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
“server seen down”
![Page 16: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/16.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
“server seen down”
![Page 17: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/17.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Can afford 1 failure with fully functional cluster
● Can afford 2 failures with partially functional cluster**
** Terms and conditions may apply
![Page 18: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/18.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Rolling upgrades without DB downtime
● Schemaless, document oriented offers great flexibility in application terms
![Page 19: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/19.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Unix level tweaks:
– raise ulimit
– raise tcp timeout
– Noatime nodiratime
– XFS, ext4
– LVM for snapshotting
● Mongo level tips:
– Use journaling. USE JOURNALING
![Page 20: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/20.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
EC2 specific tips:
– Can and will steal back time, plan for it
– Can get flaky at times..
– Design around EBS
![Page 21: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/21.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● EC2 storage:
– Local storage. Ephemeral
– EBS storage. Lasts but not strong durability guarantees.
– S3 storage. Lasts more, slower
![Page 22: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/22.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replica Sets Configuration
● Settled for EBS storage.
● Nightly backups, 30 day window
![Page 23: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/23.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 3
● Aggregation Framework effort led by Chris Westin
– Simpler way to perform Map Reduce jobs without all the pain of JS
– Integrates cleanly with our business logic
● Initial design on sharding
– More on that next..
![Page 24: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/24.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 3
● Aggregation framework for both storing and retrieving aggregate data
– New collection for double checking results with MR.
● Faster, simpler, most of the times fits in our problem domain.
● Worked better in dev than production versions ;)
![Page 25: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/25.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Reporting Version 3
● More fine grained write semantics.
– WriteConcern.SAFE for most write queries
– .REPLICAS_SAFE for non idempotent queries that are costly to recompute
● Do you feel lucky punk?
– Reactive Mongo
● Asynchronous & Non-Blocking Scala Driver for MongoDB
– Brings the best of WriteConcern.SAFE and WriteConcern.NORMAL
![Page 26: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/26.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replication and High Availability Take aways
● Use delayed members
● Size your oplog
● Use writeconcern and readpreference to balance between providing fresh data and overloading servers
● Failover happens automagically but not instantaneously
● Think your security model
![Page 27: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/27.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replication and High Availability Take aways
● More important: Think who has access to your systems.
– No commit, no rollback
● Prepare people for change
– Educate non engineers
– Use morphia
![Page 28: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/28.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Replication and High Availability Take aways
● Audit – audit – audit
– Monitor closely your MongoDB servers for potential bottlenecks
● mms.10gen.com great tool to do so
– Github is your friend:
● https://github.com/mongolab/dex
![Page 29: Mongo DB Athens user group replication and high availability](https://reader030.vdocuments.site/reader030/viewer/2022032422/55a8e2fa1a28ab4a6a8b4733/html5/thumbnails/29.jpg)
(c) Alex Giamas, Persado Inc, All rights reserved
Q&A
Ask me anything...
or drop me a line: