MongoDB in Production

Daniel Coupal Senior Technical Services Engineer MongoDB

Goal of the Presentation

1.  Walk you over common deployment problems seen by the support team

2.  Have an interactive session

Who am I?

• Daniel Coupal • Previously a developer, manager and consultant • 3 years with MongoDB • Roles

•  Technical Support Engineer •  Consulting engagements with customers •  Trainer on our Advanced Internals class for new hires

MongoDB Support Team Main support offices in …

New York, Austin, Palo Alto, Sydney, Delhi, Tel Aviv and Dublin

Who are you?

• How long have you been using MongoDB? •  In Production or Development stage?

• What presentations have you seen about MongoDB? •  Been to MongoDB Conferences? •  MongoDB Days? •  Meetups or other MongoDB events?

• What have you learned? •  On-line training classes from University.mongodb.com? •  Books on MongoDB?

What do you want to learn today?

•  The Story of NoSQL • MongoDB, the Company • Deploying guidelines • Common issues seen in Production •  •  • 

The Story of NoSQL

Once upon a time

The Sun was shinning on the land of the Oracle …

Then came the Web

We’re gonna need a bigger database

NoSQL databases

• Make use of newer technology • Scale horizontally (add more servers, instead of a bigger server) • Store richer data shapes vs “square data” in RDBMS •  Faster for a given application • High Availability and Replication built in • Distributed Systems

Data is like Oil

• Precious resource •  Even more when refined

• What was easy to extract/store has been done • Most of the data not in databases is likely not square and/or large datasets • New technology help you address what has not been processed

• MongoDB is in a good position to be the database of choice for the remaining data to be discovered

MongoDB – The Company

MongoDB - Company Overview

600 employees 1,000+ customers

Over $300 million in funding 30+ offices around the world

MongoDB - Timeline

•  Originaly, 10gen •  Founded in 2007 •  Released MongoDB 1.0 in 2009

•  MongoDB Inc •  Since 2013 •  Acquired WiredTiger in 2014

•  MongoDB •  Open source software •  Contributions on tools, drivers, … •  Most popular NoSQL database

Forrester Analysis – Q3 2016

• MongoDB Leader in •  Strategy •  Current Offering

MongoDB - We hire!

Positions open in Palo Alto, Austin, NYC and for remote workers •  http://www.mongodb.com/careers/positions Technical service engineers in Palo Alto • MongoDB • MongoDB Tools • Proactive support

Deploying Guidelines

Security

• SSL/X.509 • Kerberos/LDAP • Data encryption at rest • Users •  Firewall/security groups

Where are you deploying?

• Own servers •  AWS •  Azure • Google

Deployment platform

• MongoDB Ops Manager • MongoDB Cloud Manager • Chef • Puppet • Docker

Topology

• How many nodes? •  Primary / Secondary / Secondary (PSS) •  Primary / Secondary / Arbiter (PSA) •  More data nodes?

• How many data centers?

Backups

• Ops/Cloud Manager •  File System Snapshots • mongodump/mongorestore

Most Common Problems in the Field (not in any order)

Issue #1 – The NoSQL World is different

#1 – The NoSQL World is different

#1 – The NoSQL World is different

Relational World NoSQL World Step 1 - Model the data Step 1 - Define the access patterns to the data Step 2 - Write the queries to access the data Step 2 - Model the data Step 3 - Denormalize the data

Example: Schema in MongoDB physicians

hospitals

patients

records procedures

Issue #2 – Have a Test Environment

#2 – Have a Test Environment

•  Excellent (A) •  identical cluster •  restore production data in test/staging/QA environment

•  Acceptable (B)

•  less shards with identical hardware and same amount of data per shard

•  Less desirable (C) •  smaller hardware •  much less data •  non representative data

•  Not acceptable (F)

•  No test environment

Issue #3 – Look for Clues of Bad Queries

#3 – Look for Clues of Bad Queries (slow queries are listed in mongod log)

3.2 format 2016-05-05T12:52:05.071-0700 I COMMAND [conn16] command test.twitter command: find { find: "twitter", filter: { user.profile_background_image_url: /biatwitter/ } } planSummary: COLLSCAN keysExamined:0 docsExamined:102856 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:3077 locks:{ Global: { acquireCount: { r: 1608 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { r: 804 } } } protocol:op_command 151ms Amount of work done

How many results? Query should be fast!

#3 – Look for Clues of Bad Queries (slow queries in mongod log)

2.6 format 2016-05-05T12:54:04.432-0700 [conn30] query test.twitter query: { user.profile_background_image_url: /biatwitter/ } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:102856 nscannedObjects:102856 keyUpdates:0 numYields:0 locks(micros) r:170768 nreturned:2 reslen:2988 170ms

3.0 format

2016-05-05T12:53:42.291-0700 I COMMAND [conn9039] query test.twitter query: { user.profile_background_image_url: /biatwitter/ } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:102856 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:2988 locks:{ Global: { acquireCount: { r: 1608 } }, MMAPV1Journal: { acquireCount: { r: 804 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { R: 804 } } } 159ms

3.2 format

2016-05-05T12:52:05.071-0700 I COMMAND [conn16] command test.twitter command: find { find: "twitter", filter: { user.profile_background_image_url: /biatwitter/ } } planSummary: COLLSCAN keysExamined:0 docsExamined:102856 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:3077 locks:{ Global: { acquireCount: { r: 1608 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { r: 804 } } } protocol:op_command 151ms

#3 – Look for Clues of Bad Queries

• mtools for plots and stats on queries

#3 – Look for Clues of Bad Queries

• Cloud Manager (Premium groups) or Ops Manager •  Suggested indexes !!!

• Cloud Manager and Ops Manager •  charts

•  Logs • Profiler •  explain()

Issue #4 – Incorrect Production Settings

#4 – Incorrect Production Settings

•  Hardware •  Don’t use:

•  NAS •  other FS than xfs on Linux for WiredTiger •  other FS than ext4 or xfs on Linux for MMAPv1

•  VM Settings •  Don’t use ballooning/overcommit resources

•  OS Settings •  ulimits high enough •  Transparent Huge Pages off •  Turn NUMA off, or use numactl to start the processes •  low readahead for disks (32 instead of default of 256)

#4 – Incorrect Production Settings

• Production notes https://docs.mongodb.org/manual/administration/production-notes/

• Warnings •  At startup and in mongod.log •  In Cloud Manager/Ops Manager

• Support ticket mdiag (in GitHub) will gather info to check

Issue #5 - Cloud/Ops Manager Agents always initiate communications

#5 - Cloud/Ops Manager Agents always initiate communications

#5 - Cloud/Ops Manager Agents always initiate communications

Monitoring Agent

Secondary Secondary Primary

Issue #6 - Knowledge is King

#6 - Knowledge is King

• Use the right resources •  MongoDB.com/presentations •  Conferences: MongoDB Days/MongoDB World •  Consulting packages

•  Production readiness, Rapid Start, Performance Tuning, Database Modernization, … •  Commercial Support •  Free Support (Google groups and Stack Overflow) •  Training:

•  On-site classes •  Public trainings classes @ MongoDB offices •  Free education classes (M101 per language, M102, M202)

Conclusion

Take Away

1.  Remember 3 problems out of the 6 from this presentation

2.  Make 1 change in your application or deployment to prevent one issue

3.  If you do, this will be a good return on your hour spent at this presentation

Thank you!

Extras