mongodb europe 2016 - debugging mongodb performance

Asya Kamsky, Lead Product Manager MongoDB

Diagnostics and Debugging 3.4

“ I thought of my old teacher Joe Bell, ... of his eerie trick of spotting details. If he were a detective he would surely reduce this ... business to something nearer an exact science. —Arthur Conan Doyle

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

Understanding The Patient

#MDBW16

Understanding The Patient

Understand the system

Knowledge

Monitor trends over time

Trends

Record all metrics "at rest"

Baseline

Gathering Data “Data! Data! Data! I can't make bricks without clay.”

-- Sherlock Holmes, The Adventure of the Copper Beeches

“There is nothing like first-hand evidence.” � -- Sherlock Holmes, A Study in Scarlet

Available Tools

#MDBW16

Available Tools: Say "AH"

#MDBW16


> db.isMaster( ) > rs.conf( ) > rs.status( ) > sh.status( )

> db.version( ) > db.serverCmdLineOpts( )

#MDBW16


#MDBW16

> show dbs > show collections > db.getCollectionNames().forEach(function(c) { printjson(db.getCollection(c).getIndexes()); });


#MDBW16


#MDBW16

Available Tools: Compass

#MDBW16

Available Tools: OS

% free % blockdev --report % dmesg % ulimit -a % ifconfig, ip <...>, iptables % iostat % netstat % top; htop % perf % iperf3

#MDBW16

Available Tools: OS for MongoDB % mongostat % mongotop

mongod logs

2014-09-01T15:50:03.184-0004 [conn208] query

data.activity query: { $query: { _updated_at: { $gt: new

Date(1396459946346) }, _to: "_UserA" } }, $orderby:

{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,

_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692

nscannedObjects:19692 numYields:170 locks(micros) r:

283957 nreturned:65 reslen:24939 342ms

2014-09-01T15:50:03.184-0004 [conn208] query

data.activity query: { $query: { _updated_at: { $gt: new

Date(1396459946346) }, _to: "_UserA" } }, $orderby:

{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,

_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692

nscannedObjects:19692 numYields:170 locks(micros) r:

283957 nreturned:65 reslen:24939 342ms

mongod logs

2015-05-26T23:33:32.774-0500 I

2015-05-28T12:37:11.440-0500 I

2015-05-28T12:38:35.839-0500 I

D, I, W, E, F

3.0 mongod logs

2015-05-26T23:33:32.774-0500 I COMMAND

2015-05-28T12:37:11.440-0500 I NETWORK

2015-05-28T12:38:35.839-0500 I QUERY

3.0

COMMAND, NETWORK, QUERY, REPL, ACCESS, INDEX, JOURNAL, SHARDING, WRITE, -

mongod logs

2015-05-26T23:33:32.774-0500 I COMMAND [conn45] command admin.$cmd command: listDatabases { listDatabases: 1.0 } ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:393 locks:{ Global: { acquireCount: { r: 12 } }, Database: { acquireCount: { r: 6 } } } 321ms

2015-05-28T12:37:11.440-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:48625 #183 (21 connections now open)

2015-05-28T12:38:35.839-0500 I QUERY [conn183] getmore tableau.flights201406 query: { origin_city_market_id: 31703.0 } cursorid:61957110347 ntoreturn:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:176 nreturned:22579 reslen:1332181 locks:{ Global: { acquireCount: { r: 354 } }, Database: { acquireCount: { r: 177 } }, Collection: { acquireCount: { r: 177 } } } 114ms

3.0 mongod logs

3.2 mongod logs

2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c

command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt: 4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }

keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1

keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187

locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms

3.2 mongod logs

3.4 mongod logs

3.4 mongod logs

2016-06-01T15:30:04.373-0700 I COMMAND [conn99] command socialite.following command: aggregate { aggregate: "following", pipeline: [ { $match: { _f: "45705" } }, { $group: { _id: null, followees: { $addToSet: "$_t" } } }, { $lookup: { from: "following", localField: "followees", foreignField: "_f", as: "fofollowees" } }, { $project: { fofs: { $setUnion: [ "$followees", "$fofollowees._t" ] } } } ] } planSummary: IXSCAN { _f: 1, _t: 1 } keysExamined:1 docsExamined:0 numYields:11 reslen:316214 locks:{ Global: { acquireCount: { r: 3800 } }, Database: { acquireCount: { r: 1900 } }, Collection: { acquireCount: { r: 1900 } } } protocol:op_query 218ms

3.4 mongod logs

2016-06-25T23:38:27.346-0500 I WRITE [conn128] update ycsb.usertable query: { _id: "user7074965863272626663" } planSummary: IDHACK update: { $set: { field1: BinData(0, 23...) } } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 numYields:1 locks:{ Global: { acquireCount: { r: 3, w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 11ms

> db.getLogComponents()

> db.getLogComponents(){ "verbosity" : 1,

"accessControl" : { "verbosity" : -1},

"command" : { "verbosity" : -1},"control" : { "verbosity" : -1},

"geo" : { "verbosity" : -1},

"index" : { "verbosity" : -1},"network" : { "verbosity" : -1},

"query" : { "verbosity" : -1},

"replication" : { "verbosity" : -1},"sharding" : { "verbosity" : -1},

"storage" : { "verbosity" : -1,

"journal" : { "verbosity" : -1}},"write" : { "verbosity" : -1}

}

> db.getLogComponents(){ "verbosity" : 1,

"accessControl" : { "verbosity" : -1},

"command" : { "verbosity" : -1},"control" : { "verbosity" : -1},

"executor" : { "verbosity" : -1},

"geo" : { "verbosity" : -1},"index" : { "verbosity" : -1},

"network" : { "verbosity" : -1},

"query" : { "verbosity" : -1},"replication" : { "verbosity" : -1},

"sharding" : { "verbosity" : -1},

"storage" : { "verbosity" : -1,"journal" : { "verbosity" : -1}},

"write" : { "verbosity" : -1},

"ftdc" : { "verbosity" : -1}}

> db.setLogLevel( logLevel, component )

> db.setLogLevel( 1, "sharding" )

> db.setLogLevel( 2, "query" )

"It is of the highest importance ... to be able to recognize, �out of a number of facts, which are incidental and which vital. " �� Sherlock Holmes, The Reigate Puzzle

https://github.com/rueckstiess/mtools

by Thomas Rückstieß

#MDBW16

mplotqueries $ mplotqueries --help usage: mplotqueries [OPTIONS] [logfile [logfile ...]] A script to plot various information from logfiles. ...

#MDBW16


optional arguments:

--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event} type of plot (default=scatter with --yaxis duration)

#MDBW16


optional arguments:

--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event} type of plot (default=scatter with --yaxis duration) --group GROUP specify value to group on. All basic plot types can group on 'namespace', 'operation', 'thread', 'pattern' ...

"... what is out of the common is usually a guide� rather than a hindrance." � — Sherlock Holmes, A Study in Scarlet

mplotqueries

% mplotqueries firstmongo.log --type nscanned/n

% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms

mplotqueries

% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms 2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172 nreturned:56 reslen:18745 13086ms

mplotqueries

% mplotqueries updates?.log

"Eliminate all other factors, and the one which remains must be the truth." Sherlock Holmes -The Sign of Four

#MDBW16

Available Tools

% mongostat

#MDBW16

mongostat

#MDBW16

--discover

mongostat

#MDBW16

mongostat insert query update delete getmore command % dirty % used flushes vsize res faults qr|qw ar|aw netIn netOut conn ReplSetName role ts

#MDBW16

Trends

#MDBW16

Available Tools

#MDBW16

Available Tools: in the Cloud

% mplotqueries –type connchurn

#MDBW16

db.currentOp()

#MDBW16

db.currentOp()

> db.currentOp({secs_running:{$gt:10}})

{ "desc" : "conn3482",

"threadId" : "140512575297280",

"connectionId" : 3482,

"client_s" : "10.0.149.179:55295",

"active" : true,

"opid" : "SocialiteAtlas-shard-2:55547103",

"secs_running" : 13,

"microsecs_running" : NumberLong(13483812),

"op" : "query",

"ns" : "db1.coll1",

"numYields" : 48815,

"query" : {

"field1" : 5,

#MDBW16

db.currentOp()

> db.currentOp({desc:/^conn/,secs_running:{$gt:0}}).inprog.forEach(function(op) { print(op.opid, op.op, op.ns, op.secs_running); });

SocialiteAtlas-shard-0:68479014 query socialite.content 3








>

db.killOp ( opid )

#MDBW16

Asya Kamsky, Lead Product Manager MongoDB

Diagnostics and Debugging 3.4

Title Slide Option 1 Click here to add speaker name and title

Title Slide Option 2 Click here to add speaker name and title

This is divider slide option 2

#MDBW16

Click to add title: keep it to 56 characters w/spaces

This is a typical content slide with full width body.

•  First level bullet list •  Second level bullet list

•  Third level bullet list

#MDBW16

Two content

Click to add text.



Click to add text.



#MDBW16

Left content

Click to add text.


#MDBW16

Pie Chart

64%

25%

11%

1st Qtr 2nd Qtr 3rd Qtr

1st Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .

2nd Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .

3rd Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .

#MDBW16

Bar Graph

0

1

2

3

4

5

6

Category 1 Category 2 Category 3 Category 4

Chart Title

Series 1 Series 2 Series 3

#MDBW16

Column Header 1 Column Header 2 Column Header 3 Column Header 4 Column Header 5

Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet



Table

#MDBW16

Title only

#MDBW16

Coding Example – Light Background

// Retrieve var MongoClient = require('mongodb').MongoClient;

// Connect to the db

MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {

if(err) { return console.dir(err); }

db.collection('test', function(err, collection) {});

db.collection('test', {w:1}, function(err, collection) {});

db.createCollection('test', function(err, collection) {});

db.createCollection('test', {w:1}, function(err, collection) {});

});

#MDBW16

Coding Example – Dark Background

// Retrieve var MongoClient = require('mongodb').MongoClient;

// Connect to the db

MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {

if(err) { return console.dir(err); }

db.collection('test', function(err, collection) {});

db.collection('test', {w:1}, function(err, collection) {});

db.createCollection('test', function(err, collection) {});

db.createCollection('test', {w:1}, function(err, collection) {});

});

#MDBW16

List slide – can also be used for agenda

Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.

Lorem Ipsum 01 Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.


Lorem Ipsum 02




Lorem Ipsum 05

#MDBW16

Columns and icons with copy (option 1)

Lorem ipsum dolor sit amet, onsectetur

adipiscing elit. Praesent sodales odio sit amet odio tristique.

Linked Lorem ipsum dolor sit

amet, onsectetur adipiscing elit.

Praesent sodales odio sit amet odio tristique.

Planning Lorem ipsum dolor sit



Writing Lorem ipsum dolor sit



Research

#MDBW16

Columns and icons with copy (option 2)

Lorem ipsum dolor sit amet, onsectetur

adipiscing elit. Praesent sodales odio

sit amet odio tristique .

Linked Lorem ipsum dolor sit


Praesent sodales odio sit amet odio tristique .

Planning Lorem ipsum dolor sit



Writing Lorem ipsum dolor sit



Research

#MDBW16

Timeline or progress

2013



2014

2015



2016

“ Quote sample. Lorem ipsum dolor sit amet, onsectetur adipiscing elit amet sodales. Praesent sodales odio sit amet odio tristique. Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique. Lorem ipsum dolor sit amet, onsectetur adipiscing elit.”

mongodb europe 2016 - debugging mongodb performance

Data & Analytics