understanding aws storage options (stg101) | aws re:invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Choosing the Right Data Storage Solution

Joe Lyons – Sr. Manager Global Storage

November 13, 2013

AWS Storage Options

• Scalable Storage

• Inexpensive Archive Storage

• Persistent Direct Attached Storage

• Turn-key gateway solution

• Customer Presentation: iMemories

We are constantly producing more data

From all types of industries

#1 Object Storage

●○○○○

AMAZON S3 SIMPLE STORAGE SERVICE

99.999999999% Durability

Trillions Of Unique Customer Objects

Q4 2006

Q1 2007

Q2 2007

Q3 2007

Q4 2007

Q1 2008

Q2 2008

Q3 2008

Q4 2008

Q1 2009

Q2 2009

Q3 2009

Q4 2009

Q1 2010

Q2 2010

Q3 2010

Q4 2010

Q1 2011

Q2 2011

Q3 2011

Q4 2011

Q1 2012

Q2 2012

Q3 2012

Q4 2012

Q1 2013

Q2 2013

Q3 2013

1.5 Million+ Peak Transactions Per Second

Storage Tiers: Buckets + Unlimited Objects

Reduced Redundancy Option 99.99% saves ~20%

Spotify adds over 20,000 tracks a day - RRS

Amazon S3 Website: Static Content

Amazon S3 continuous cost reduction

• 16 Price reductions since launch

• TCO: On-premises vs. Amazon S3

– Can be challenging for some customers

– We can help!

1PB Raw Storage

800TB Usable Storage

600TB Allocated Storage

400TB Written Application Storage

Amazon S3 -

Only actual

usage is

charged

RAW Storage at On perm vs. Cloud

Storage

Use Amazon S3 When You Need

• Unlimited storage capacity

• High durability

• Storage for backups

• Single Origin Store with Delivery via Amazon CloudFront

AMAZON GLACIER LOW-COST ARCHIVING SERVICE

1c Per GB / Month

$120 Per TB / Year

99.999999999% Durability

3-5 Hours Data Retrieval

STORAGE COSTS

VS

RETRIEVAL COSTS

Use Amazon Glacier When You Need

• Inexpensive/Long-term archiving

• Unlimited storage capacity

• Eliminated Tape Museums

• Eliminate Tech Refresh

• High durability

Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE

Lifecycle Rule

Archive Recovery Process with Tape

+ Days or Weeks

Archive Recovery Process with AWS &

Amazon Glacier

$$

Hours

Glacier S3 EC2

/HPC

CloudFront Generating

Business

Value

Use Amazon S3 and Amazon Glacier When You Need

• HSM in the cloud

• Archive data from Amazon S3/RRS to Amazon Glacier by policy

• Delete data from Amazon Glacier by policy

#2 Block Storage

●●○○○

#3 SYNC VOLUMES

●●●○○

AMAZON EBS ELASTIC BLOCK STORAGE

Ephemeral Storage

10 GB 1TB

IOPS Provisioned

4000

IOPS

Amazon EBS Snapshots

Use Amazon EBS When You Need

• Long-term persistent storage

• Data changes frequently

• Block storage for your databases – Provisioned IOPS volumes

• Filesystem for an instance NTFS, ExtFS, RAID, LVM…

• Access to raw, unformatted block-level storage

AWS STORAGE GATEWAY

AWS Storage Gateway

What is AWS Storage Gateway?

45 Amazon Confidential

Integrates on-prem IT environments with Cloud storage for departmental and remote office backup and DR

Utilizes a virtual appliance that sits in customer datacenter

Exposes compatible iSCSI interface on front end

Stores primary data on-AWS in Amazon S3 or on-premise with data backed-up to Amazon S3 as Amazon EBS snapshots

Solution Overview

Run apps in the cloud using your uploaded data –

HPC/Hadoop/Analytics

2. DR

1. Offsite Backup

3. Data Mirroring

AWS Storage Gateway works with your existing backup

application and moves your data into Amazon S3 as EBS

snapshots

Run AWS Storage Gateway in Amazon EC2 and access

snapshots up to 32 TB in size

AWS Storage Gateway – Cached Volumes will help you

create Storage Volumes in cloud and keep most recent

accessed data locally [Reduce SAN footprint for File Shares] 4. Dept. File Share

VTL version of AWS Storage Gateway would help customer

move data in the form of Virtual Tapes from on premise to

Amazon S3 (VTL – Virtual Tape Library) and then to Amazon

Glacier (VTS – Virtual Tape Shelf)

5. Archive/Glacier

IT’S ALL ABOUT

CHOICE PERFORMANCE-ORIENTED

COST-ORIENTED

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Understanding AWS Storage Options

Tamara Monson, iMemories, Inc.

November 13, 2013

Our Story: History • iMemories was founded in 2006

• Cloud was in its infancy – AWS officially launched in 2006: Amazon

S3

• Handling physical assets: – VHS, Betamax, MiniDV tapes

– 16mm, 8mm film reels

– Photos, slides

• Using on-premises equipment: – VCRs, decks

– Film projectors

– Photo and slide scanners

• Focus was digitization for web/online viewing, DVD creation, archiving

2006

2008

20

10

2012

2013

iMemories' Growth

Our Story: Success • Growing infrastructure

• Reliability not as high as desired

• Less-than-ideal customer experience

• Tripled full-time resources to manage

• Costs for static, infrequently accessed assets much higher than for more-frequently accessed assets

• Rate of growth driving need for infrastructure elasticity

2006

2008

20

10

2012

2013

iMemories' Growth

8TB

0.75PB

1.5PB

Our Story: Need

Use Case

• Video and photo

converted and digital

source files

• Short-term local copy

• Access infrequently

– Disc creation

– Additional encodings

Requirements

• Durable, redundant,

secure

• Keep forever

• No or low operational

costs

• No new hardware

• Off-premises

Evaluating Options: Calculating TCO

• Operations

– Percentage FTE

– Facilities and power

– Maintenance

• Hardware

– Life span

– Number of copies

– Raw GB vs. usable GB vs.

consumed GB

0

0.2

0.4

0.6

0.8

1

1.2

Time

Raw Usable Consumed

Evaluating Options: Comparisons

Amazon S3 Cloud Files

Cloud Amazon Glacier

iMemories Evolution

Architecture 1.0

• All storage and processing on-premises (circa 2006)

• Centered on physical assets, devices, servers

• Expensive: scale by adding more

• Non-ideal customer experience

Architecture 2.x

• Solve scale challenges

• Reduce costs

• Best possible customer

experience

Storage 1.0

• On-premises

• ~1.5 PB capacity

• Limitations of scale:

– Add new hardware

– Space and power

File Mgmt

iMemories Scottsdale Data center

Digitization,

Encoding

Servers Storage Volume

Storage 2.0: Amazon Glacier

AWS Cloud

File Mgmt

iMemories Scottsdale Data Center

Digitization,

Encoding

Servers Storage Volume

Glacier

Manager

Regions

Vaults

Amazon SQS Amazon SNS

Amazon Glacier

Manager

- Interfaces with Amazon SQS

- Tracks restore/submit request

state

- Processes Submit requests in

real-time

- Processes Restore requests

once/hour

- 5% of consumed / 720 hours

- peak hourly rate

- Distribute request across 1-hr

windows

- Uses 10 MB chunks

- Performs audits 3x/week

- One discrepancy

- Zero lost files on Glacier (!!!)

- Generates internal activity reports

Glacier Manager

File Management

Digitization,

Encoding

Servers

2.x: Move to AWS

Regions

Elastic Load

Balancing Video

Files

Availability Zones

Streaming

Servers

Streaming

Server

v-ec2-east1.imemories.com

v-ec2-west2.imemories.com

File Mgmt

iMemories Scottsdale Data Center

Digitization,

Encoding

Servers

Storage

Volume

S3 Manager

AWS Cloud

2.x: Move to AWS

Region

Elastic Load

Balancing

Video

Files Upload

Web App

File Mgmt

iMemories Scottsdale Data Center

API

AWS Cloud

Amazon

SWF

u-ec2.imemories.com Video

Processing

Group

In Closing

• iMemories Story – History

– Success

– Need

• Evaluating Storage Options – Define use case and solution requirements

– Understand TCO

– Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap

• iMemories Evolution – Amazon Glacier is first full production integration

– Migrating more functionality to cloud

– Will always have an on-premises component

Enabling families to share memories in endless ways.

http://www.imemories.com

tamara.monson@imemories.com

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

STG101