amazon dynamodb: a serverless nosql database for scale and

Amazon DynamoDB: a

serverless NoSQL Database

for scale and performance

Venkat Govindarajan

Sr. Database Specialist SA

Robert McCauley

Sr. DynamoDB Specialist SA

Agenda

• Relational Databases

• Why NoSQL and DynamoDB?

• Adaptive Capacity

• Managing Throughput

• New Responsibilities

• Microservice Architecture with DynamoDB

• Global Tables & DEMO

• Data Access Functions

• NoSQL Workbench DEMO

• Migrating to DynamoDB

The NoSQL Way

Database evolution – A paradigm shiftLE

FOCUS ON BUSINESS LOGIC

Managed Databases

Easier to setup and launch

Automated patching and backups

Scale with read replicas

Database evolution – A paradigm shiftLE

FOCUS ON BUSINESS LOGIC

Predictable fast performance

Fault tolerance built-in

Pay for value

Easy access via API

SERVERLESS NoSQL

Purpose-builtThe right tool for the right

Purpose-built databases

FINRA uses different AWS databases for different jobs

Market data search:

• Massive data volume

• Need quick lookups for searches

Amazon Aurora Amazon RDS

AmazonDynamoDB

Amazon DocumentDB

Amazon Neptune

Amazon ElastiCache

Session state:

• In-memory store for sub-millisecond site rendering

Data collection

Fraud detection

Relational data:

• Referential integrity

• Primary transactional database

Wait, what’s wrong with my existing relational DB?

• Monolith

• Hard to scale

Monolith Issue #1 – Shared Everything

Monolith Issue #2 – buffer cache

Buffer Cache – 64 GB

Data – 50 GBData – 500 GB

SELECT * FROM TableWHERE Column15 = "xyz"

Year 1: here you go!

Year 2: please wait..…

Monolith Issue #3 – nested queries

SELECT * FROM (

SELECT * …

Temp Space

Monolith Issue #4 –stored procedures

Stored Procedures

IF Column7 > :xTHEN

UPDATE..ELSE

INSERT INTO

Data Statistics

Compiled Procedures

Hard to Scale – JOINs are taxing

customers products reviewscarts

Cart.JSON

Hard to Scale – Sharding your data

Cust 234

• Different sizes

• Different

velocities

• Expanding and

migrating data

is difficult

Hard to Scale – Heavy proprietary connections• Connection count

maximum

• Install custom

drivers (~50mb)

• Custom ports

• Custom DB user

directory

• Best for fixed app

servers or desktops

Connection

Manager

Function.

monolithic application

+ DBA army

Lesson learned: decompose for agility

microservices

+ 2 pizza teams

Dev el o p m en t t r a n s fo r m a t i o n a t A m a z o n

Purpose built data stores, improved resiliency, API

communication

Why NoSQL?

Scaling Databases

Traditional SQL NoSQL

Scale up

Scale out to many shards

There is a way to design data that’s horizontally scalable.

Super Bowl Peak Consumed Capacity

Provisioned Capacity

Global-Scale Events: Elastic is the New Normal

Developer-friendly – Document PUT & GET to an API

Cart.JSON API

Why DynamoDB?

Amazon DynamoDB

Document or key-value Scales to any workloadFully managed NoSQL

Access control Event-driven programmingFast and consistent

Amazon DynamoDB - Features

NoSQL Workbench Point-in-time recovery

Time To Live (TTL)

DynamoDB local

CloudWatch

Contributor InsightsEncryption at rest DynamoDB Accelerator

On-demand backup

Amazon DynamoDB - Features

99.999% SLADynamoDB Streams Global Tables

Adaptive CapacityAuto Scaling

Transactions

Server 1

Table 1 Table 2 Table 3

Server N

You work with tables…

1K WCU/ 3K RCU

up to 10 GB

DynamoDB does the rest under the hood…

Storage – Serverless like S3

MyTableMyTable

Create a Table

Put "Items"

Pay for size stored

No storage provisioning needed

Horizontal scaling with DynamoDB

Workload:

data volume, reads, writes

DynamoDB resources:

storage, read, and write capacity

Adaptive Capacity

The uneven access problem common to NoSQL

Across data and time

Scaling NoSQL databases

Most NoSQL databases DynamoDB

Servers and clusters DynamoDB: partitions

P1DBP25

Basic premise: There is a way to shard data that’s horizontally scalable.

Adaptive Capacity - Core Functions

• Dynamic partitioning

• High-traffic item isolation

• Throughput boosting

Server 1

Dynamic Partitioning – Storage Growth

Server 1

Server 2

Server 1

Server 2

Server 1

Server 2 Server 3

Server 1

Server 2 Server 3

Server 1

Server 4 Server 2 Server 3

A D B C

High-traffic item isolation

Server 1

Server 4 Server 2 Server 3

A D B C

Adaptive capacity throughput boosting

Read/Write Capacity Modes

Provisioned and On-

demand Capacity Modes

Read Capacity Unit

Item size Strongly consistent reads

4 KB 1 1 RCU

Item size Eventually consistent reads

Item size Transactional reads

4 KB 2 1 RCU

4 KB 1 2 RCU

Read Capacity Unit - Examples

Item size Strongly consistent reads

8 KB * (1 RCU / 4 KB) * (1) = 2 RCU

Conversion ratio

Item size Eventually consistent reads

8 KB * (1 RCU / 4 KB) * (1 / 2) = 1 RCU

Conversion ratio

Item size Transactional reads

8 KB * (1 RCU / 4 KB) * (2) = 4 RCU

Conversion ratio

Write Capacity Unit

Item size Standard writes

1 KB 1 1 WCU

Item size Transactional writes

1 KB 1 2 WCU

Write Capacity Unit - Examples

Item size Standard writes

3 KB * (1 WCU / 1 KB) * (1) = 3 WCU

Conversion ratio

Item size Transactional writes

3 KB * (1 WCU / 1 KB) * (2) = 6 WCU

Conversion ratio

• Add any number of items to a table

• Item max size = 400 KB

• Scaling is achieved through partitioning

• Partition max capacity = (3000 RCU) + (1000 WCU)

• Global Secondary Index (GSI)

• GSI has its own capacity settings beyond the base table

Throughput Considerations

Provisioned Mode -

Auto Scaling

DynamoDB on-demand capacity mode

Start at zero

No limit

Features

• No capacity planning, provisioning, or

reservations—simply make API calls

• Pay only for the reads and writes you

perform

Key benefits

• Eliminates tradeoffs of

overprovisioning or underprovisioning

• Instantly accommodates your workload

as traffic ramps up or down

On-demand scaling properties

Base throughput

• Up to 4,000 write request units: 4,000 writes per second

• Up to 12,000 read request units: 24,000 eventually consistent reads

per second

• Any linear combination of the two

Maximum throughput

Unlimited!

Pay per request: Use nothing, pay nothing

Maximum Scaling Capabilities:Up to twice your previous peak

DynamoDB request rate

Maximum throughput

On-demand tables do not “scale down”Maximum throughput

Maximum throughput

Use provisioned mode

• Steady workloads

• Gradual ramps

• Events with known traffic

Use on-demand mode

• Unpredictable workloads

• Frequently idle workloads

• Events with unknown traffic

• “Set it and forget it”

Consider your tolerance for operational overheadand overprovisioning

Post-launch Switchover: Conversion after load stabilization

Change

Capacit

y Mode

Amazon CloudWatch Contributor Insights for DynamoDB

New Roles and

Responsibilities

New Roles & Responsibilities

DBA → Cloud DBA

Developer += Data Architect

New Roles – Cloud DBA

Monitor CloudWatch:

• Read Throttles

• Write Throtttles

• Returned Bytes

• Request Latency (AVG)

• User Errors

• System Errors

Move to or from On Demand

Perform any Backups &

Restores

Enable Point In Time Recovery

Monitor Global Table

Replication

Manage user access via IAM

or Cognito

Define IAM Policies for

types of users

Define Encryption Key via

KMS service (optional)

New Roles – Data Architect

Model:

Table Keys

NoSQL Workbench for

DynamoDB

CustomerA Order1#Line1 Yoga Mat Qty: 3

CustomerA Order1#Line2 Kettlebell Qty: 2

CustomerA Order2#Line1 Jump Rope Qty: 4

Model :

Global Secondary Indexes

Re-groups Items on an

alternate key

Kettlebell Order1#Line2 CustomerA Qty: 2

Kettlebell Order2#Line3 CustomerB Qty: 7

Yoga Mat Order1#Line1 CustomerC Qty: 3

Yoga Mat Order2#Line2 CustomerA Qty: 2

Jump Rope Order2#Line1 CustomerA Qty: 4

DENORMALIZE!

Duplicate data to avoid

looups and JOINs

Peach 2020-06-10 Qty: 2

Customer:

{name: “Patty”,

street: “123 Main”,

City:”Portland”}

Peach 2020-06-12 Qty: 5

Customer:

{name: “Timmy”,

street: “456 Nook”,

City:”San Diego”}

Peach 2020-07-25 Qty: 4

Customer:

{name: “Sam”,

street: “789 Elm”,

City:”Providence”}

Define Conventions:

Overload different Item

types to store complete

Item Collections

Turnip 2020-06-10 Qty: 2

Customer:

{name: “Patty”,

street: “123 Main”,

City:”Portland”}

Turnip 2020-06-12 Qty: 5

Customer:

{name: “Timmy”,

street: “456 Nook”,

City:”San Diego”}

Turnip 2020-07-25 Qty: 4

Customer:

{name: “Sam”,

street: “789 Elm”,

City:”Providence”}

Turnip DESCRIPTION

Turnip Description: This turnip is a

fast growing, crisp, fine-grained

turnip that keeps its mild sweet

flavor yet grows larger than…

Define Data Types and Item Lifecycle

Format DynamoDB Type Bytes Stored

Unix Epoch Date

32 bit Int

NUMBER:

15899886456

STRING:

158998864510

ISO 8601 StringSTRING:

2020-05-20T14:26:59Z20

Implementing Microservice

Architectures

with Amazon DynamoDB and AWS

Lambda

Example

Lambda

Amazon

DynamoDBAmazon API

Gateway

Amazon

DynamoDB

Streams

Lambda

Amazon

Athena

DynamoDB

Query sideOnline Transaction

Processing (OLTP) side

A serverless app

Building Blocks

Lambda

Amazon

DynamoDB

Amazon API

Gateway

1. API Gateway + Lambda + DynamoDB: core building block

2. DynamoDB Streams + AWS Lambda: building block for reliable event delivery

Amazon

DynamoDB

Streams

LambdaDynamoDB

DynamoDB Streams and Lambda

Partition A

Partition B

Partition C

Updates GetRecords

DynamoDB

DynamoDB Streams

Lambda

function

DynamoDB Streams + AWS Lambda = reliable “exactly once” event delivery

Example: Transactions and Querying

• An ecommerce app

• Relational vs. nonrelational data models

• Transactions in a distributed microservices architecture

• Enabling querying in a distributed microservices architecture

Example: Ecommerce App Using Microservices

Shopping cart

service

Product

catalog

service

Account

service

Lambda

Gateway

DynamoDBDB1 DB2 DB3

Shopping Cart Data Model

CartId: 1,Version: 3

CART_ITEM

ItemId:2, Qty: 1,

CartId: 1

Relational database

CartID:2,

Version: 3,

Items: [

{ID: 2, Qty: 1},

{ID: 5, Qty: 2}]

Nonrelational

database

Normalized schema:

multiple tables

Denormalized schema:

aggregates in a single table

Data Consistency on Cart Updates

Example: add/remove shopping cart items

1.get cart => vread = Version2.update cart:

IF Version = vreadadd/remove cart items++Version

ELSE go back to Step 1.

Use ConditionExpression

in DynamoDB

Optimistic concurrency control (OCC)

Updating Cart: Data Consistency Using OCC

CartID:2,

Version: 3,

CartItems: [

{ID: 2, Qty: 1},{ID: 5, Qty: 2}]

✓ Use conditions to implement OCC, ensuring data

consistency

✓ GetItem call can eventually be consistent

1. Get the cart: GetItem

{ "TableName": “Cart”,

"Key": {“CartId”: {“N”: “2”}}

2. Update the cart: conditional PutItem (or UpdateItem)

{ "TableName": “Cart”,

"Item": {

“CartID”: {“N”: “2”},

“Version”: {“N”:”4”},

“CartItems”: {…}

"ConditionExpression": “Version = :v1”,

"ExpressionAttributeValues": {":v1": {“N”: “3”}}

Global Tables

Enhancements to Global Tables

Features

• Add regions on the fly

• Convert single-region tables to global

• Reduced write capacity consumption

Key benefits

• Flexible regional footprint

• Save on replicated write costs

Replica (N. America)

Replica (Europe)

Replica (Asia)

Global table

High Level Architecture

Region

Region 1

Region

Region 2

StreamReader

Multi-Primary Replication

Region

Region 1

Region

Region 2

StreamReader

Multi-Region Multi-Primary Replication

Region

Region 1

StreamReader

Region

Region 3

StreamReader

Region

Region 2

StreamReader

Amazon DynamoDBI M P O R T A N T 2 0 2 0 L A U N C H E S A T A G L A N C E

Export to Amazon S3

A native feature of DynamoDB

that allows you to export

DynamoDB table data to Amazon

Simple Storage Service

(Amazon S3) – no code required.

Build materialized views that

combine and replicate data across

multiple data stores without you

having to write custom code.

AWS Glue Elastic Views

PartiQL

Stream your DynamoDB data to

Kinesis Data Streams to take

advantage of longer retention

time and other Kinesis services.

Change data capture for Amazon Kinesis Data Streams

PartiQL

Backup Export to Amazon S3

Amazon

Recovery

Compressed objects

• JSON format

• ION format

Analyze Exported Data with Athena, QuickSight, Sagemaker,

Amazon

Athena

SQL query

How Disney+ scales globally on DynamoDB

Disney+ on DynamoDB

Bookmark

global table

Amazon Kinesis

Data Streams

Bookmark

writer

Content API

Telemetry

service

Data Access Functions

Data Access Functions – Show cost per read

Request Parameter "Return Consumed Capacity"

Evaluate efficiency of Filters

Querying Data – next & previous

Peach 2020-06-20 Qty: 3

Peach 2020-06-21 Qty: 1

Cherry 2020-06-10 Qty: 2

Cherry 2020-06-12 Qty: 5

Cherry 2020-07-25 Qty: 4

Coconut 2020-05-28 Qty: 2

Pear 2020-06-13 Qty: 1

Pear 2020-06-15 Qty: 4

GET-ITEM to return a single item

QUERY can return a single next item

Key Condition

Expression

PKey = “Cherry”

SKey > “2020-06-12”

Limit 1

Scan Forward TRUE

Data Access Functions – Writes

Carefully structuring Items unlocks powerful Query patterns

CustomerA Order1#Line1#RETURN RETURN Yoga Mat Qty: 3

CustomerA Order2#Line1 Jump Rope Qty: 4

CustomerA STATUS#HOLD HOLD

StatusDate

DONE_2014-10-02

IN_PROGRESS_2014-10-08

IN_PROGRESS_2014-10-03

PENDING_2014-09-30

PENDING_2014-10-03

Status

IN_PROGRESS

PENDING

2018-10-02

2018-10-08

2018-10-03

2018-09-30

Data Access Functions – Writes

Combine values together for composite sort key

Query with Sort Key Expression: BEGINS_WITH('PENDING')

A client-side application that helps you build scalable, high-performance data models

Simplifies query development and testing

A rich GUI-based tool that helps you visualize data models and perform DynamoDB operations

Available for Windows, macOS, and Linux

NoSQL WorkbenchEnterprise ready

Data modeler

Visualizer

Operation builder

Demo: Generating Data Access Functions with

the NoSQL Workbench

Migrating to DynamoDB

Existing Tables to Single Table Design

customers

products reviews

carts DynamoDB

Single Table

Put-Item

BatchWriteItem

CREATE VIEW

SQL UNION ALL

Single Table Design- Generic Keys, Overload Items, GSIs

Cart1 Peach Peach CustomerA … …

Cart1 Coconut Coconut CustomerA … …

CustomerA CustomerA CustomerA … …

CustomerB CustomerB CustomerB … …

Peach Order2#Line3 Peach … …

Review3 Review3 Peach CustomerC … …

PKey SKey

Existing Table Data prepared by View SQL

customers

products reviews

carts Single Table

Dual Writes to Both Systems

customers

products reviews

cartsSingle Table

Application

verification

One table or many?

Denormalize and overload the table to

enable item collections and GSI queries

Avoid third normal form, foreign key

constraints, and JOINS

Consider one table per micro service

Use multiple tables as needed!

DynamoDBFast and flexible NoSQL database service for any scale

Performance at scale

• Handles millions of requests per second

• Delivers single-digit-millisecond latency

• Automated global replication

• New advanced streaming with Amazon Kinesis Data Streams for DynamoDB

Enterprise ready

• ACID transactions

• Encryption at rest

• Continuous backups (PITR), and on-demand backup and restore

• NoSQL Workbench

• Export table data to S3

• PartiQL (a SQL-compatible query language) support

No servers to manage

• Maintenance free

• Auto scaling

• On-demand capacity mode

• Change data capture for integration with AWS Lambda, Amazon Redshift, Amazon Elasticsearch Service

Amazon DynamoDB: a

serverless NoSQL Database

for scale and performance

@dynamodb

bit.ly/ddbefficiency

amazon-dynamodb-labs.com

Thank you for attending!

amazon dynamodb: a serverless nosql database for scale and

Documents

nosql database strategy · cloudfront advertisement...

amazon dynamodb

aws re:invent 2016| dat318 | migrating from rdbms to nosql:...

tutorial on scalable cloud-databases in research and...

utilizando nosql para big data com dynamodb

aws の nosql 入門〜amazon elasticache, amazon...

serverless architectureにおけるnosql services...

how to expedite design and development of serverless ·...

nosql&databases - wordpress.com ·...

building serverless applications...@model top-level entity;...

amazon dynamodb テーブル設計と実践 tips · pdf...

dynamodb - tutorialspoint.com · dynamodb is a...

nosql and aws dynamodb

dynamodb dg

dynamodb media server (2012-06-12 dynamodb+emr...

adem hadrovic · dynamodb, serverless, nodejs, react with...

asa: amazon shopping assistant€¦ · aws lambda & aws api...

comparing the use of amazon dynamodb and apache hbase for...

dynamodb presentation

comparaison de l'utilisation d'amazon dynamodb et d'apache...