wiredtiger & what's new in 3.0
Post on 06-Jan-2017
870 Views
Preview:
TRANSCRIPT
WiredTiger & What's New in 3.0
3
I'm Excited
4
AgendaMongoDB 3.0Storage Engine APIWiredTiger Architecture
6
Benefits
7
Performance!
8
Compression in Action
(Flights database, ht Asya)
But 3.0 is more than WiredTiger
http://www.livingincebuforums.com/ipb/uploads/monthly_10_2011/post-198-0-67871200-1318223706.jpg
Pluggable Storage Engine API
11
MongoDB Architecture
Content Repo
IoT Sensor Backend Ad Service Customer
Analytics Archive
MongoDB Query Language (MQL) + Native Drivers
MongoDB Document Data Model
MMAP V1 WT In-Memory ? ?
Supported in MongoDB 3.0 Future Possible Storage Engines
Man
agem
ent
Sec
urity
Example Future State
Experimental
12
Storage Engine API• Allows to "plug-in" different storage engines
– Different work sets require different performance characteristics – mmapv1 is not ideal for all workloads– More flexibility
• Can mix storage engines on same replica set/sharded cluster• Opportunity to integrate further ( HDFS, native encrypted, hardware
optimized …)
Storage Engines
2 Storage Engines Available
… and more in the making!MMAPv1 WiredTiger
15
MongoDB Architecture
Content Repo
IoT Sensor Backend Ad Service Customer
Analytics Archive
MongoDB Query Language (MQL) + Native Drivers
MongoDB Document Data Model
MMAP V1 WT In-Memory ? ?
Supported in MongoDB 3.0 Future Possible Storage Engines
Man
agem
ent
Sec
urity
Example Future State
Experimental
mongod --storageEngine wiredTiger
MMAPv1
https://angrytechnician.files.wordpress.com/2009/05/memory.jpg
18
MMAPv1
19
MMAPv1
• Improved concurrency control
• Great performance on read-heavy workloads
• Data & Indexes memory mapped into virtual address space
• Data access is paged into RAM
• OS evicts using LRU
• More frequently used pages stay in RAM
3.0 Default
http://cdn.theatlantic.com/static/infocus/ngt051713/n10_00203194.jpg
WiredTiger
21
What is WiredTiger?• Storage engine company founded by BerkeleyDB alums• Recently acquired by MongoDB• Available as a storage engine option in MongoDB 3.0
22
Motivation for WiredTiger• Take advantage of modern hardware:
– many CPU cores– lots of RAM
• Minimize contention between threads– lock-free algorithms, e.g., hazard pointers– eliminate blocking due to concurrency control
• Hotter cache and more work per I/O– compact file formats– compression
WiredTiger Architecture
24
WiredTiger Architecture
All operations will be handled atomically through the transaction mechanism
For data access methods MongoDB is
using the Row storage, although we are working on more options for future
versions
In-memory Performance
26
Traditional B+tree (ht wikipedia)
27
Trees in cache
28
Hazard Pointers
1 memory flush
29
Pages in cache
30
In-memory performance• Trees in cache are optimized for in-memory access• Follow pointers to traverse a tree
– no locking to access pages in cache• Keep updates separate from clean data• Do structural changes (eviction, splits) in background threads
Document-level Concurrency
32
What is Concurrency Control?
• Computers have– multiple CPU cores– multiple I/O paths
• To make the most of the hardware, software has to execute multiple operations in parallel
• Concurrency control has to keep data consistent• Common approaches:
– locking– keeping multiple versions of data (MVCC)
33
WiredTiger Concurrency Control
34
Pages in cacheskiplist
35
Multiversion Concurrency Control (MVCC)
• Multiple versions of records kept in cache• Readers see the committed version before the transaction started
– MongoDB “yields” turn large operations into small transactions• Writers can create new versions concurrent with readers• Concurrent updates to a single record cause write conflicts
– MongoDB retries with back-off
36
MVCC In Action
updateupdate
Compression
38
WiredTiger Page I/O
39
Trees in cache
read required
40
Pages in cache
reconciled during write
41
Checksums• A checksum is stored with every page• Checksums are validated during page read
– detects filesystem corruption, random bitflips• WiredTiger stores the checksum with the page address (typically
in a parent page)– extra safety against reading a stale page image
42
Compression• WiredTiger uses snappy compression by default in MongoDB• Supported compression algorithms:
– snappy [default]: good compression, low overhead– zlib: better compression, more CPU– none
• Indexes also use prefix compression– stays compressed in memory
43
Compression in Action
(Flights database, ht Asya)
What's Next
45
46
WiredTiger LSM support
• Out-of-order insert workloads• Data set much larger than cache• Query performance is not (as) important• Resource overhead of background maintenance
acceptable
47
48
What’s next for WiredTiger?
• Tune for (many) more workloads– avoid stalls during checkpoints with 100GB+ caches– make capped collections (including oplog) more efficient– Mark Callaghan seeing ~2x speedup in 3.1 snapshots:
http://smalldatum.blogspot.com/ • Make Log Structured Merge (LSM) trees work well with MongoDB
– out-of-cache, write-heavy workloads• Adding encryption• More advanced transactional semantics in the storage engine API
49
Updates and Upgrades
• Can not– Can't copy database files– Can't just restart w/ same dbpath
• Yes we can!– Initial sync from replica set works perfectly!– mongodump/restore
• Rolling upgrade of replica set to WT:– Shutdown secondary – Relaunch w/ --storageEngine=wiredTiger– Rollover
50
Gotchas
• No 32-bit Support– WT is 64bit only
• system.indexes & system.namespaces deprecated– Explicit commands: db.getIndexes() db.getCollectionNames()
https://tingbudongchine.files.wordpress.com/2012/08/lemonde1.jpeg
Take Aways
52
MongoDB 3.0
• Pluggable Storage Engine API• Storage Engines• Large Replica Sets• Big Polygon• Security Enhancements – SCRAM • Audit Trail• Simplified Operations – Ops Manager• Tools Rewrite
53
Benefits
54
Performance!
http://www.mongodb.com/lp/white-paper/benchmark-report
http://cl.jroo.me/z3/v/D/C/e/a.baa-Too-many-bicycles-on-the-van.jpg
Norberto LeiteTechnical Evangelistnorberto@mongodb.com@nleite
Obrigado!
top related