amazon documentdb (with mongodb compatibility) deep dive 1 - am… · aurora communitycommercial...

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Amazon DocumentDB (with MongoDB compatibility) Deep Dive

Jeff DuffyDocumentDB Specialist SA

© 2020, Amazon Web Services, Inc. or its Affiliates.

AWS: Purpose-built databases

Relational Key-value Document In-memory Graph

Amazon DynamoDB

Amazon Neptune

Amazon Relational Database Service

(RDS)

Aurora CommercialCommunity

Amazon ElastiCache

Amazon Keyspaces

Amazon DocumentDB

Time-series Ledger

Amazon Timestream

Amazon Quantum

Ledger DatabaseRedis Memcached

Wide Column

© 2020, Amazon Web Services, Inc. or its Affiliates.

Amazon DocumentDBFast, scalable, and fully managed MongoDB-compatible database service

Fast Scalable Fully managed MongoDB compatible

Millions of requests per second with millisecond latency

Separation of compute and storage scales both

independently; scale out to 15 read replicas in minutes

Managed by AWS: no hardware provisioning; auto patching, quick setup,

secure, and automatic backups

Compatible with MongoDB 3.6; use the same SDKs, tools, and

applications with Amazon DocumentDB

Purpose-built document database engineered for the cloud

© 2020, Amazon Web Services, Inc. or its Affiliates.

Customers

Learn more: https://aws.amazon.com/documentdb/customers/

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Document Databases

© 2020, Amazon Web Services, Inc. or its Affiliates.

(Client)

(App) (Database)

JSON

!=Evolution of document databases

Relational

JSON became the de facto

data interchange format

Friction when converting JSON to the relational

model

Object-relational mappers (ORMs) were created to

minimize friction

Document databases solved

the problem

JSON

© 2020, Amazon Web Services, Inc. or its Affiliates.

Documents (objects/JSON) are common application data models

Documents map naturally to how humans model data

Document databases store JSON-like documents

Document databases provide flexible schema and indexing

Ad hoc querying and aggregations

Denormalized data

modelNormalized data model

Why document databases?

© 2020, Amazon Web Services, Inc. or its Affiliates.

Use cases for document data

Mobile

Retail and marketing

User profilesCatalog

Content management

Personalization

© 2020, Amazon Web Services, Inc. or its Affiliates.

{userid: 181276,username: "sue1942",name: {first: "Susan",

last: "Benoit"}}

{userid: 181276,username: "sue1942",name: {first: "Susan",

last: "Benoit"},ExplodingSnails: {

hi_score: 3185400,global_rank: 5139,bonus_levels: true

},promotions: ["new user","5%","snail lover"]

}

{userid: 181276,username: "sue1942",name: {first: "Susan",

last: "Benoit"},ExplodingSnails: {

hi_score: 3185400,global_rank: 5139,bonus_levels: true

}}

Document Use Case: Gaming User Profiles

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Challenges scaling document databases

© 2020, Amazon Web Services, Inc. or its Affiliates.

API

Query processor

Caching

Logging

Storage

Application

Not designed for the cloud

Single monolithic architectures

Scale monolithicallyFail monolithically

Traditional Database Architecture Challenges

© 2020, Amazon Web Services, Inc. or its Affiliates.

Replication

Challenge: Add read capacity on-demand

Node 1 Node 2 Node 3 Node 4Disk Disk Disk Disk

© 2020, Amazon Web Services, Inc. or its Affiliates.

Replication

Challenge: Recover quickly from node failure

Node 1 Node 2 Node 3 Node 3’Disk Disk Disk Disk

© 2020, Amazon Web Services, Inc. or its Affiliates.

Challenge: Scale storage as data grows

Node

Storage Volume

Node

Storage Volume

Storage Volume

Node

© 2020, Amazon Web Services, Inc. or its Affiliates.

Operational Node

Backup Node

Challenge: Backup data without affecting performance

Snapshot Snapshot

© 2020, Amazon Web Services, Inc. or its Affiliates.

Challenge: Data Durability

Node

Storage Volume

Node

Storage Volume Storage Volume

NodeReplication Replication

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Amazon DocumentDBPurpose-built and engineered

for the cloud

© 2020, Amazon Web Services, Inc. or its Affiliates.

API

Query processor

Caching

Logging

Storage

Scale compute

Scale storage

Decouple compute and storage

Compute layer

Storage layer

© 2020, Amazon Web Services, Inc. or its Affiliates.

Instance (Replica)

Distributed storage volume

AZ1 AZ2 AZ3

Read

s

Instance (Primary)

Instance (Replica)

Read

s Writes Re

ads

Compute

Storage

Amazon DocumentDB: Cloud Native Architecture

© 2020, Amazon Web Services, Inc. or its Affiliates.

Instance (Replica)

Distributed storage volume

AZ1 AZ2 AZ3

Read

s

Instance (Primary)

Instance (Replica)

Read

s Writes Re

ads

Instance (Replica)

Read

s

Compute

Storage

Challenge: Add read capacity on-demand

~8-10 mins

© 2020, Amazon Web Services, Inc. or its Affiliates.

Distributed storage volume

Challenge: Quickly recover from node failure

~8-10 mins

~30 secs

Read

s

Writes

Read

s

Read

s WritesRe

ads

Read

s

Instance (Replica)

Instance (Primary)

Instance (Replica)Instance (Primary)

Instance (Replica)

© 2020, Amazon Web Services, Inc. or its Affiliates.

Distributed storage volume

Challenge: Scale storage as data grows

Grows Automatically from

10GB-64TB

Compute

Storage

Distributed storage volume

Storage

Read

s

WritesRe

ads

Read

s

Instance (Replica)

Instance (Primary)

Instance (Replica)

© 2020, Amazon Web Services, Inc. or its Affiliates.

Distributed storage volume

Instance (Replica)

Read

s

Instance (Primary)

Instance (Replica)

Read

s

Writes

Amazon S3

Continuous Backups(PITR)

Snapshots (Automated and Manual)

Read

s

Compute

Storage

Challenge: Backup data without affecting performance

© 2020, Amazon Web Services, Inc. or its Affiliates.

Distributed storage volume

AZ1 AZ2 AZ3

Instance (Replica)

Read

s

Instance (Primary)

Instance (Replica)

Read

s

Writes

Read

s

Instances:

Environment:

Durability: highly durable

1

dev/testproduction

2

99%Availability goal: 99.9%

3

99.99%

Challenge: Data Durability

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Service overview

© 2020, Amazon Web Services, Inc. or its Affiliates.

Automated maintenance

Automated upgrades and patching

Failed instances are automatically replaced

Automatic failover and recovery

Fully managed

AWS Integration

CloudWatch, CloudTrail, CloudFormation, VPC,

Secrets Manager

Pay-as-you-go pricing

Per-second instance billing, no long-term

commitment

“Our engineering teams now spend less time on operations like backup scripts, scale testing, and managing high availability and instead are able to focus on developing new capabilities for our customers.”

© 2020, Amazon Web Services, Inc. or its Affiliates.

MongoDB compatible

MongoDB 3.6 Same drivers, tools

Replica sets

Compatible with MongoDB Community Edition 3.6

Use the same MongoDB drivers and tools with Amazon DocumentDB

Read scaling is easy with replica set emulation

“Getting started with DocumentDB was also simple and we migrated our application in a couple of days without needing to make any meaningful code changes. Everything just worked.”

© 2020, Amazon Web Services, Inc. or its Affiliates.

Why choose the MongoDB API?

Source: http://db-engines.com

• Over 60 million downloads

• Easy for developers to get started

• MongoDB API is powerful

• Managing MongoDB is hard

© 2020, Amazon Web Services, Inc. or its Affiliates.

Scale out in minutes

Scale up in minutes

Autoscalestorage

Load balancing

Scale to 15 read replicas

Scale from 16 to 768 GiB

of RAM

Storage automatically grows from

10 GB to 64 TB

Scale reads across replicas

Scalable

© 2020, Amazon Web Services, Inc. or its Affiliates.

Launch

JAN

2019

MAR

201

9

Frankfurt

JUN

201

9

MAY

201

9

Secrets Manager

JUL

2019

Tokyo

Per-second Billing

AUG

2019

Sydney

FEB

2019

DDLAuditing

APR

2019

… with more to come in 2020!

OCT

201

9

Start/Stop Cluster

DEC

2019

FEB

2020

London

Canada

APR

2020

Slow query logger

Glue ETL

MAR

202

0

RBACAggregation

Operators

Change Streams

Mumbai

ParisSingapore

Aggregation Operators

Deletion ProtectionSeoul

Aggregation Operators

Working Backwards

© 2020, Amazon Web Services, Inc. or its Affiliates.

Scale Reads

Sharding motivator

Scale Writes

Scale Storage

Add up to 15 read replicas

Scale vertically with very low impact

DocumentDB approach

Storage scales automatically to 64TB

Why not shard?

© 2020, Amazon Web Services, Inc. or its Affiliates.

Security and compliance

VPC Only Encrypted by default

Safe defaults Compliance

Strict network isolation with Amazon Virtual Private Cloud

(VPC)

Encryption at rest with AWS KMS

Encryption in transit with TLS

Best practices are the defaults

PCI DSS,ISO 9001, 27001, 27017,

and 27018,SOC 1, 2 & 3, HIPAA

© 2020, Amazon Web Services, Inc. or its Affiliates.

Backup and recovery

Automatic backups 35 days of PITR

Archive snapshots

Automatic, incremental, and continuous backups

Backups do not affect database performance

Point-in-time recovery (PITR) for up to 35 days

Keep snapshots for as long as you need

No performance impact

"Adopting Amazon DocumentDB is a game-changer because we offload management, security, and backup of our MongoDB databases to AWS. With Amazon DocumentDB, we can add or scale instances in minutes, regardless of data size. Further, we get automatic backups and point-in-time restore capabilities, which far exceed other managed DB services at less cost.”

© 2020, Amazon Web Services, Inc. or its Affiliates.

Distributed storage volume

Read

s

Read

s Writes

Amazon S3

Read

s

Compute

Storage

Instances: Size/hr * count (db.r5.large $0.277/hr)

I/O: Count ($0.20/million)

Storage: GB/mo

Backup: GB/mo(100% Free! then $0.021/GB)

Pricing (us-east-1)

($0.10/GB)

© 2020, Amazon Web Services, Inc. or its Affiliates.

Offline migration using MongoDB utilities:Get started quickly, great for POCs

Online migration using AWS DMS Near-zero downtime migration

Hybrid migration leverages both solutionsBest of both worlds

Migration GuideLearn how to migrate from MongoDB to Amazon DocumentDB

All options support MongoDB on-premises and EC2, for both replica sets and sharded clusters

https://docs.aws.amazon.com/documentdb/latest/developerguide/docdb-migration.html

© 2020, Amazon Web Services, Inc. or its Affiliates.

Purpose builtThe right tool for the right

job

Thank you