aws summit 2013 | auckland - understanding aws storage options

Jeff Putt

Understanding AWS Storage Options

Business Development Manager, APAC, AWS

AWS Storage Options

Object Storage Block Storage

Connect AWS Storage with On-Premises Environments

For NoSQL and Relational Database storage, catch the dedicated sessions

+ +

We are sincerely eager to hear your feedback on this

presentation and on re:Invent.

Please fill out an evaluation form when you have a chance. We are constantly producing more data

We are sincerely eager to hear your feedback on this

presentation and on re:Invent.

Please fill out an evaluation form when you have a chance.

From all types of industries

#1

Object Storage

●○○

Amazon S3 Simple Storage Service

99.999999999%

Durability

Over 1 Trillion

Unique Customer Objects

750,000+ Peak Transactions Per Second

1. High Durability 2. Unlimited Capacity 3. Low latency 4. High Volume Static Websites 5. Security & Compliance 6. Delivery via CDN

S3 gives you So you can build Mobile Apps Music Apps Video networks Video Apps Backup Apps Photo Apps Static Websites Ad networks Anything else you’d like!

Amazon Glacier Low-Cost Archiving Service

1¢

Per GB / Month

$120

Per TB / Year

99.999999999%

Durability

3−5 Hours

Data Retrieval

Storage Costs

vs.

Retrieval Costs

1. High Durability 2. Low cost 3. 3-5 hour latency

Glacier gives you So you can • Archive files • Build Hierarchal Storage

Management in the cloud

Amazon S3 and Glacier can help you build…

Gaming Commerce

Digital Media Distribution

Websites Social Media

Photo/Video Sharing and

Backup

Big Data

Store unlimited objects in Amazon S3 and Amazon Glacier

S3 Glacier

Access both via SDKs and via third-party software

S3 Glacier

Internet Web

Services API

HTTP(S)

Via SDK Via 3rd parties

Or send S3 files directly to Glacier

S3 Glacier

S3 storage costs as low as 4.7 cents per GB

Serve static content at high volumes

Deliver content with lower latency via Amazon CloudFront

Smugmug is home to millions of customers sharing

Billions of photos

Adds over 20,000 tracks of music a day

Nasdaq FinQloud is a WORM compliant archive for

Broker Dealer financial data

Shazam hit 500,000 writes per second

during the Super Bowl

nearmap Aerial Imagery

nearmap routinely captures hi-res aerial images

covering 75% of Australia’s population

nearmap provides browser-based access to up-to-

date and historical aerial imagery to thousands of

companies, large and small

30

Clear

31

Current

Images are delivered within

7-14 days, rather than 3-6

months after flying

6mo

Traditional

nearmap

Processing Time

3mo 9mo 1yr

Brisbane floods – nearmap was the only company to capture the

flooded areas and publish within days

32

Bundaberg Flood

33

34

Bundaberg Flood

35

Bundaberg Flood : Before

36

Bundaberg Flood : After

37

Bundaberg Flood : Before

38

Bundaberg Flood : After

39

Bundaberg Flood : Before

40

Bundaberg Flood : After

nearmap Tools

The photomap is

correctly scaled for

accurate

measurements

Objects are shown in

their correct real-

world position

45

Why Use nearmap

• Less time driving

• Less time on site

• Reduce fleet costs and environmental impact

• Eliminates existing spend on imagery

• Integrates with enterprise systems

46

nearmap and AWS

Q. What does nearmap use AWS for?

A. Everything

• Production

• Development and QA

• Corporate systems

• Image archival

47

AWS Features Used

EC2 - both Reserved and Spot Instances

S3

Glacier

RDS

Route53

CloudFormation

CloudWatch

IAM

SES

Direct Connect

4 June, 2013

Image Data Storage

6 Terabytes of new image data per month

We have 4+ years of existing image data

S3: 150 Terabytes and counting

Glacier: 300 Terabytes and counting

4 June, 2013

50

#2

Block Storage

●●○

Amazon EBS Elastic Block Storage

Ephemeral storage is on every EC2 instance

EBS volumes attach to EC2 instances

EBS volumes persist and can be attached to other instances

Multiple drives on one Amazon EC2 Instance

10 GB

1TB

IOPS Provisioned

Workloads on EBS Provisioned IOPS

P-IOPS is designed to run transactional applications that require high and

consistent IO such as:

• Relational databases

• NoSQL databases (e.g. MongoDB)

• High performance file systems

• Productivity applications (e.g. Microsoft Exchange)

• Enterprise applications (e.g. CRM, ERP)

EBS Snapshots

Take snapshots of EBS volumes and keep them in S3

Use Amazon EBS When You Need

• Filesystem for an instance NTFS, ExtFS, RAID, LVM

• Long-term persistent storage

• Ability to change data frequently

• Access to raw, unformatted block-level storage

#3

Connect AWS Storage with On-Premises

Environments

●●●

Access both via SDKs and via third-party software

S3 Glacier

Internet Web

Services API

HTTP(S)

Via SDK Via 3rd parties

Common Data Storage Challenges and the AWS Opportunity

Internet Web Services API

HTTP(S)

AWS Direct Connect

Block File

Customer Data

Center

Storage Use Cases

Archive Backup Disaster

Recovery

Next Generation Enterprise Storage

S3 Glacier

Common Data Storage Challenges and the AWS Opportunity

Primary

Block

Storage

Primary

File

Storage

Archival

Storage Disk-Based

Backup

Storage

Tape

Infrastructure

&

Management

Replicated

Storage for

Disaster

Recovery

Offsite

Locations

Geo-

Resilience

Next Generation: Gateway & Cloud Storage

Reduce

SAN

Footprint

Reduce

NAS

Footprint

Eliminate

Architectur

e

Hardware

& Software

Eliminate

Backup

Hardware

& Software

Eliminate

Tape Recover in

Cloud or

any Data

Center

Eliminate

Secondary

Data

Centers

Reduce Complexity. Reduce Cost.

Next Generation Enterprise Storage - Benefits

Common Data Storage Challenges and the AWS Opportunity

S3 & Glacier - Designed to work with other AWS Services

Storage Gateway – Connect On-Prem with the AWS Cloud

1. Local, low-latency access to the most frequently used files while storing all data in Amazon S3 (Cached-Volumes) Or 2. Scheduled off-site backups to Amazon S3 for on-premises data (Stored-Volumes)

Use AWS Storage Gateway When You Need To

• Backup your data

• Synchronize data

• Export data for migration

• Set up a departmental fileshare

• Conduct simple disaster recovery

AWS Storage Options

Object Storage Block Storage

Connect AWS Storage with On-Premises Environments

+ +

Performance-Oriented

Cost-Oriented

IT’S ALL ABOUT

CHOICE

Related Content

Web resources:

• http://aws.amazon.com/s3/

• http://aws.amazon.com/ebs

• http://aws.amazon.com/glacier/

• http://aws.amazon.com/digital-media/

72

Thank You.

Shares customer submitted photos

Foursquare streams hundreds of millions of application logs

into Amazon S3 for analytics

Forced to choose between saving their source data or saving

their results. With Amazon Glacier, they can keep both

Real Clear Politics served up 20x normal demand with

S3 Websites