What’s New in MongoDB 3.0

Osmar Olivo

Product Manager

Agenda

• Pluggable Storage Engines

• WiredTiger Integration

• Document- Level Locking

• Compression

• WT Internals

• Upgrade/Installation

• Options

Pluggable Storage Engines

Storage Engine Layer

● Vision: Many storage engines optimized for

many different use cases

● One data model, one API, one set of

operational concerns – but under the hood,

many options for every use case under the sun

Storage Engine Layer

Content

Repo

IoT Sensor

BackendAd Service

Customer

AnalyticsArchive

MongoDB Query Language (MQL) + Native Drivers

MongoDB Document Data Model

MMAP V1 WT In-Memory ? ?

Supported in MongoDB 3.0 Future Possible Storage Engines

Ma

na

ge

me

nt

Se

curity

Example Future State

Experimental

WiredTiger Integration

Why WT?

• WT addresses weaknesses of MMAPv1

– Compression

– Online Compaction

– Highly concurrent and vertically scalable

• Document level locking

• Allows full hardware utilization

• More tunable

– Higher ceiling for potential improvement

Why WT

• Strong Foundation

– Authors Former Members of Berkeley DB

team

• WT product and team acquired by

MongoDB

• Standalone Engine already in use in large

deployments including Amazon

Document Level Concurrency

Document Level Concurrency

• Improved Concurrency

– Uses algorithms to minimize contention

between threads• One thread yields on contention to same document

• Atomic update replaces latching/locking

– Writes no longer block all other writes

– CPU utilization directly correlates with

performance

Compression

Compression

• Compression is on in WT by default

• 2.8 supports two compression algorithms

– snappy ( default)

• Good compression benefits with little

CPU/performance impact

– zlib

• Extremely good compression at a cost of

additional CPU/degraded performance

WT Internals

WT Internals

• File format

– Data Stored as conventional B+ tree on disk

– Inserts into one big file per collection

– In memory, on disk format is decoupled

• Variable-length pages on disk

• Inserts into in-memory skiplists

WT Internals

• File Layout

– Each collection & index stored in own file

• Can only use one

– Will fail to start if MMAPv1 files found in

dbpath

– No in-place updates

• Rewrites document to end of file every

time

• No padding factor

WT Internals

• File Layout

– Does not extend concept of DB to disk layout

• Concept of DB purely logical

– Journal has own folder under dbpath

– Allows for storing indexes on separate

volumes easily

WT Internals

• Deprecate MMAPv1-specific catalog metadata

– system.indexes & system.namespaces

– System metadata should be accessed via

explicit commands going forward

WT Internals

• Cache

– WT uses two caches

• The WiredTiger engine cache

• The file system cache

• WT cache uses higher value of 50% of

system memory or 1GB (by default)

WT Internals

• When is data written to disk?

– Checkpoints

• By default, performed every 60 seconds or

after 2GB are written

• Analogous to MMAPv1 data file flushes

– When the WT cache is full pages will be

written to disk

WT Internals

• Durability

– WT uses a write ahead transaction log

• Cannot set journalCommitInterval

• Can use j:true write concern

• Can use --journal or –nojournal

• Old journal files truncated after each checkpoint

– Data files are always consistent, no longer have

need for journal recovery after unclean shutdown

• Will only lose data since last checkpoint (< 60 s.)

WT Internals

• Supported Platforms

– Linux

– Windows

– Mac OSX

• Not Supported Platforms

– NO Solaris (yet)

– NO 32Bit (ever)

How Do I Install/Upgrade It?

• Installation

– Starting from scratch add 1 additional flag

when launching mongod

• --storageEngine=wiredTiger

How Do I Install/Upgrade It?

• Upgrade Procedure

– 2 ways

– Mongodump/Mongorestore

– Initial sync a new replica member running

WT

• We support running replicas with mixed

storage engines

– CANNOT copy raw data files

• WT will fail to start if wrong data format in

dbpath

What Options Are Relevant?

• Options that still apply

– journal

– nojournal

– repair

– repairPath

– upgrade

What Options Are Relevant?

• MMAPV1 only options

– quota

– quotaFiles

– noprealloc

– nssize

– smallfiles

– syncdelay

– journalOptions

– journalCommitInterval

What Options Are Relevant?

• File Format specific options removed from

tools.

– --dbpath

Questions?