MongoDB

Debugging Performance Problems

@ConradIrwin

50M crashes/day

4TB data / 100GB index

15 nodes / 3* availability zones

I don't care if MongoDB is slow

I care if my app is slow

How to make my app fast again if it's slow because of the way I'm using MongoDB

What does slow mean?

Performance over time

Performance over time

It's slow :(

Solution 1

Denormalize

It's fast :)

It's slow :(

db.errors. find({project_id: x}). sort({ _id: -1}). limit(30)

db.errors. find({project_id: x}). sort({ _id: -1}). limit(30). explain()

{ "cursor" : "BtreeCursor _id_ reverse", "isMultiKey" : false, "n" : 0, "nscannedObjects" : 227756, "nscanned" : 227756, "nscannedObjectsAllPlans" : 227756, "nscannedAllPlans" : 227756, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 1779, "nChunkSkips" : 0, "millis" : 461, "indexBounds" : { "_id" : [ [ { "$maxElement" : 1 }, { "$minElement" : 1 } ] ] }, "server" : "Jaroussky.local:27017", "filterSet" : false}

{"cursor" : "BtreeCursor _id_ reverse",

"nscanned" : 227756,

"indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] }}

Solution 2

Index

db.errors.ensureIndex( {project_id: 1, _id: -1})

{"cursor" : "BtreeCursor project_id_1__id_1","nscanned" : 6,"indexBounds" : { "project_id" : [[ ObjectId(x),ObjectId(x) ]], "_id" : [[ {"$minElement" : 1}, {"$maxElement" : 1} ]]}}

It's fast :)

It's slow :(

Only 1 query...

Indexed properly...

mongostat

insert query update delete getmore command flushes mapped 5 97 57 *0 95 127|0 0 320g 5 98 61 *0 113 146|0 0 320g 8 94 61 *0 95 137|0 0 320g

vsize res faults locked db idx miss % qr|qw ar|aw 641g 8.65g 7 bugsnag:47.7% 0 0|0 0|0 641g 8.66g 2 bugsnag:21.0% 0 0|0 0|0 641g 8.64g 3 bugsnag:23.4% 0 0|0 0|0

netIn netOut conn set repl time 47k 73k 145 bugsnag1 PRI 06:34:05 63k 99k 146 bugsnag1 PRI 06:34:06 98k 124k 146 bugsnag1 PRI 06:34:07

command locked db time 127|0 bugsnag:47.7% 06:34:05 146|0 bugsnag:21.0% 06:34:06 137|0 bugsnag:23.4% 06:34:07

Solution 3

Shard

sh.shardCollection("errors"{ project_id: 1, _id: -1})

command locked db time 83|0 bugsnag:3.1% 06:46:12 69|0 bugsnag:4.1% 06:46:13 73|0 bugsnag:2.4% 06:46:14

It's fast :)

It's slow :(

Only 1 query...

Indexed properly...

Lock % ok...

iostat

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-szxvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41

avgqu-sz await r_await w_await svctm %util 11.02 14.54 5.33 14.73 0.60 45.60 8.04 12.12 0.00 12.12 0.51 33.60 13.01 17.92 0.00 17.92 0.50 36.40

rkB/s wkB/s await %util 7660 138 14.73 83.60 2248 2043 12.12 65.20 7905 436 17.92 45.60

db.stats()['indexSize']= 3852071824

= 3932614656free -b (m3.medium)

Solution 4

Scale

db.stats()['indexSize']= 3852071824

= 15775363072free -b (r3.large)

It's fast :)

It's slow :(

Main query seems fine...

"Quick" queries sometimes slow...

mongotop

ns total read write 2014-06-22T19:11:35 bugsnag.events 95ms 0ms 95ms bugsnag.errors 80ms 6ms 74msbugsnag.system.namespaces 26ms 26ms 0ms bugsnag.projects 12ms 5ms 7ms bugsnag.users 15ms 2ms 13ms bugsnag.error_aggregates 4ms 0ms 4ms bugsnag.deploys 4ms 3ms 1ms bugsnag.event_tallies 3ms 0ms 3ms

ns tot r w bugsnag.events 95 0 95 bugsnag.errors 80 6 74bugsnag.projects 12 5 7 bugsnag.users 15 7 8

Solution 5

Tag shards

It's fast :)

Solution 1

Denormalize

Solution 2

Index

Solution 3

Shard

Solution 4

Scale

Solution 5

Tag shards

Solution 6

...

NewRelic / Skylight

explain()

mongostat

iostat

mongotop

...

It will be slow

You can speed it up

Thanks!@ConradIrwin