best practices for running mongodb on aws - aws may 2016 webinar series
TRANSCRIPT
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rahul Bhartia, Partner Solution Architect
24th May 2016
MongoDB on AWSAn introduction and best-practices for AWS
MongoDB : Basics
Document Model
RDBMS
{ first_name: ‘Paul’, surname: ‘Miller’, city: ‘London’, location: [45.123,47.232], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ]}
MongoDB
Features
Rich Queries Find Paul’s carsFind everybody in London with a car between 1970 and 1980
Geospatial Find all of the car owners within 5km of Trafalgar Sq.
Text Search Find all the cars described as having leather seats
Aggregation Calculate the average value of Paul’s car collection
Map Reduce What is the ownership pattern of colors by geography over time (is purple trending in China?)
MongoDB: New features in 3.2
New Storage engines
In-memory
Run your most demanding, real-time apps with
the in-memory storage engine (beta)
Encrypted
Secure your data-at-rest with the encrypted
storage engine
Data Governance: Document Validation
Implement data governance without sacrificing
the agility that comes from a dynamic schema
Dynamic Lookup
Combine data from multiple collections with left
outer joins for richer analytics & more flexibility
in data modeling
Richer In-Database Analytics & Search
New Aggregation operators extend options for
performing analytics with lower developer
complexity
Array Operators
Math Operators Text
$slice,
$arrayElemAt,
$concatArrays,
$filter, $min,
$max, $avg,
$sum, and
more.
$stdDevSamp,
$stdDevPop,
$sqrt,$abs,
$trunc, $ceil,
$floor, $log,
$pow
$exp, and
more.
Case sensitive text search and support for languages such as Arabic, Farsi, Chinese, and more.
MongoDB Connector for BI
Visualize and explore multi-structured data using
SQL-based BI platforms.
Connector for BIProvides Schema
Translates QueriesTranslates Response
Compass: The GUI for MongoDB
Visually explore your data and schema.
Run ad hoc queries in seconds
Make smarter decisions about indexing,
document validation, and more.
No command line needed
MongoDB: High Availability & Scalability
Replica Sets
Replica set – 2 to 50 copies
Replica sets make up a self-healing ‘shard’
Replica sets address - High availability, Maintenance (e.g., HW swaps) and Disaster Recovery
Application
Driver
Primary
Secondary
Secondary
Replication
Replica Sets – Workload Isolation
Replica sets enable workload isolation
Example: Operational workloads on the primary
node, analytical workloads on the secondary
nodes
Application
PrimaryIn-memory
SecondaryWiredTiger
User DataSessions, Cart,Recommendations
SecondaryWiredTiger
PersistedUser Data
Elastic Scalability: Automatic Sharding
Increase or decrease capacity as you go
Automatic load balancing
Three types of sharding - Hash-based, Range-based and Tag-aware
Shard 1
Shard 2
Shard 3
Shard N
Horizontally Scalable
Query Routing
Multiple query optimization models
Each of the sharding options are appropriate for different apps / use cases
MongoDB: AWS Deployment Architectures
AWS EC2 Instance Types
General Purpose - M3, M4• Start with General Purpose instances and EBS GP2
Compute-optimized - C3, C4• WiredTiger - Write performance due to document-level concurrency control
Memory-optimized - R3• In memory storage engine: NEW!• Larger working set - MMAPv1 with read-intensive applications
Storage-optimized - I2, D2• Local instance store but lost when instance is stopped or terminated• Always use with higher replication
https://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#deployment-notes
Amazon EBS Volumes
EBS GP2 for general workloads and EBS PIOS for consistent performancehttps://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#storage-considerations
EBS-optimized Instances provides additional/dedicated bandwidthhttp://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html
EBS snapshots for recovery and backupshttps://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#backup-restore-verify
AWS Global Footprint
High Availability: Intra-Region
High Availability: Intra-Region
High Availability: Inter-Region
Disaster Recovery: Inter-Region
Disaster Recovery: Inter-Region
Go global
Deploy Global, Access Local
Replicate Data Near Users
MongoDB - Management
The Best Way to Run MongoDB
Ops Manager and Cloud Manager allows you
leverage and automate the best practices.
10x-20x more efficient operations
Complete performance visibility
Protection from data loss
Performance optimization
How they work
Ops or Cloud Manager
mongod mongodmongod
Agent Agent Agent
Amazon QuickStart
https://aws.amazon.com/quickstart/
VPC with private and public subnets
Instance role with fine-grained permissions.
Security groups
A fully customized MongoDB cluster - Replica sets, shards, and config servers and customized EBS storage
Monitoring & Alerting
Over 100+ database-related metrics
Dozens of optimized charts
Custom alerts so incidents do not become emergencies
Do-It-Yourself
Set of utilities distributed with MongoDB – mongostat, mongotop
Database commands – serverStatus, dbStats, collStats etc
Self-hosted – Ganglia, mtop, munin, nagios
Hosted (SaaS) – New Relic, Datadog, Server Density
https://docs.mongodb.com/manual/administration/monitoring/
Backup with Point-in-time Recovery
Cluster-wide snapshots of sharded clusters
Restore to precisely the moment you need, quickly and safely with point-in-time restores
Manual Backup Considerations
Consider a hidden member in a Replica set
Consider EBS Snapshots
Consider Journaling (Write Ahead Log), and allow for DB durability in case of a fault
Ensure consistency by using db.fsyncLock()
https://docs.mongodb.com/ecosystem/tutorial/backup-and-restore-mongodb-on-amazon-ec2/
Visual Query Profiler
Identify the slow-running queries across your cluster with just the click of a button
Index suggestions to improve your query performance
Automate rolling index builds to reduce operational overhead and the risk of failovers
Resources
MongoDB on AWS best practices:http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/
MongoDB production Noteshttp://docs.mongodb.org/manual/administration/production-notes/
MongoDB Documentationhttp://docs.mongodb.org
Thank you!