edge and hybrid storage - amazon web services€¦ · amazon s3 archived tapes stored in amazon...
TRANSCRIPT
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Everett Dolgner
Senior Storage BD Manager, Global Platform BD - Today Storage Specialist, Solutions Architect EMEA – Starting 2 July 2018
Edge and Hybrid Storage
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data movement
Onl
ine
Off
line
Data security and management
Complete Set of Data Building Blocks
Amazon EFS
Amazon EBS
AWS Snow family
AWS Storage Gateway Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3Transfer Acceleration
Storage Partners
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
Amazon S3
Amazon Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage Gateway Hybrid Storage SolutionsUse standard storage protocols to access AWS storage
AWS StorageGateway
Amazon EBS snapshots
Amazon S3 & S3 –infrequent access
Amazon Glacier
AWS Identity and Access Management (IAM)
Amazon CloudWatch
Files
Volumes
TapesO
n-pr
emis
es
AW
S Cloud
AWS Key Management Service (KMS)
AWS CloudTrail
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A Few of Our Customers
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway Amazon S3 Standard & Infrequent Access
Files(NFS v3 / v4)
Volumes(iSCSI)
Tapes(iSCSI VTL)
AWS Storage Gateway
Data stored in Amazon S3 with• Native S3 management (File)• Snapshots to EBS (Volume)• Archive to Amazon Glacier (Tape)
Service uses an appliance to provide• Protocol conversion and device emulation• Caching (read-through / write-back)• Optimized data transfer
HTTPS
Amazon EC2
Amazon Glacier
Amazon EBS snapshots
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape Gateway
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape Gateway
• Easy cloud adoption: Virtual tape storage in S3 & Glacier accessed via gateway• Secure & efficient: Data encrypted and compressed in-transit and at-rest• Scalable: Unlimited archive capacity; up to 1 PB total tape storage per gateway• Dependable: 3-5-hour tape retrieval from Amazon Glacier with 99.999999999% data durability• Low-cost: Simple licensing, predictable storage costs, and reduced management time
Media ChangerVirtual Tapes
stored inAmazon S3
Archived Tapes in Amazon Glacier
BackupServer
iSCSI
TapeGateway
Tape Gateway Appliance (VM)
Tape Drive
HTTPS
Customer Premises Region
Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
Local Cache
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Drop-in replacement for tape backups• Works with leading common backup applications• Emulates a tape library • Supports existing backup workflows
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backup, archive, and disaster recoveryCost-effective storage in AWS with local or cloud restore
“Tapes are a headache, prone with hardware failures, offsite storage costs, and constant
maintenance needs. Storage Gateway provided the most cost-effective and simple alternative. We even got disaster recovery by
using a bi-coastal data center.”
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Your tape archive today
BackupServer
Customer Premises
Tape Library
Offsite Vault
Backup Disk
Appliance
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Move your tape archive
Media ChangerVirtual Tapes
stored inAmazon S3
Archived Tapes stored in
Amazon Glacier
iSCSI
StorageGatewayService
Tape Gateway VM
Tape Drive
HTTPS
RegionBackupServer
Customer Premises
Tape Library
Offsite Vault
• Deploy Storage Gateway VTL as an archive target• Old archives can expire, or be recalled and written to SGW• Decommission Tape Library when all tapes are expired
Backup Disk
Appliance
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway• Cloud-backed block storage that can be presented on-premises• Tier snapshots or whole volumes to the cloud to reduce SAN/NAS mgt.• Flexible recoveries in-cloud or on-prem. with snapshots and clones• Common use cases: Tiered storage for Windows file servers, backup and
restore, disaster recovery, data migrations
Application Server
Amazon EBS snapshots
Storage GatewayService
Volume Gateway appliance
Volume Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway – Stored Mode
Using Stored Volumes a full copy of the volume is stored on-premises and in the cloud
Application Server
Amazon EBS snapshots
Storage GatewayService
Volume Gateway appliance
Volume Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region
100% of volume stored in AWS & on-prem
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway – Cached Mode
Using Cached Volumes a full copy of the volume is stored in AWS and a portion (hot data) is stored on-premises
Application Server
Amazon EBS snapshots
Storage GatewayService
Volume Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region
100% of volume stored in AWS & on-prem
Volume Gateway VM
virtual volume cache frequent accessed blocks (reduce onsite storage)
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High availability Windows Server storageLocal HA with remote DR
us-west-1Data Center
WindowsClients Storage
Gateway VMMicrosoft DFS
HTTPSiSCSI
HTTPSiSCSI
SMB
Storage Gateway Service
us-east-1
us-west-1
us-east-1
Microsoft DFS
HTTPSiSCSI
Storage Gateway Service
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Flexible storageEasily expand storage capacity on-demand
“Storage Gateway allows us to store backups offsite, create online archives of big files, and easily expand our file servers. With cached volumes, we have the
most used files local, and all of the files are stored safely in AWS. We use Storage Gateway whenever we need
a quick way to have big storage available.”
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway + EBS Snapshots
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway – EBS Snapshots
Snapshots are point-in-time backups of a volumeCreated on-demand or on a configurable scheduleCan be used to create an EBS or Storage Gateway volume
Application Server Amazon EBS
snapshots
Storage GatewayService
Volume Gateway appliance
Volume Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recovery with Volume Gateway
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recovery with EBS Snapshots
AppServer
Storage Gateway Service
Customer Premises Region
Storage Gateway appliance
iSCSI volumeA B
A B
C
C
volume
D
D
snap1 snap2 snap3
A B C D
A B CvolumeA B C
AZEBS
EC2A BC D
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File GatewayStore files as objects in Amazon S3; Access via S3 API or file protocols
• Storage in your S3 buckets with 1:1 mapping from files-to-objects• File metadata stored in object metadata• Full control over how your data is accessed and stored• Use cases: Hybrid cloud data processing (Big Data/Data Lakes), data
distribution, e.g. for content or media, backup & recovery, etc.
Customer Premises
Glacier
HTTPSNFS
v3 / v4.1
Application Server
Objects in your S3 bucket
Storage Gateway appliance
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
s3://{bucket name}/{folder1}/{file1}s3://{bucket name}/{folder1}/{file2}s3://{bucket name}/{folder1}/{file3}s3://{bucket name}/{folder1}/{file4}
s3://{bucket name}/{folder2}/{file1}s3://{bucket name}/{folder2}/{file2}
File GatewayOne-to-one mapping from files-to-objects
nfs://{IP}/{share name}/{folder1}/{file1}nfs://{IP}/{share name}/{folder1}/{file2}nfs://{IP}/{share name}/{folder1}/{file3}nfs://{IP}/{share name}/{folder1}/{file4}
nfs://{IP}/{share name}/{folder2}/{file1}nfs://{IP}/{share name}/{folder2}/{file2}
File Gateway appliance
NFS client Objects in your S3 bucket
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AmazonAthena
Amazon QuickSight
Amazon EMR
AWSLambda
Amazon Rekognition
File GatewayMetadata and workload automation
Metadata and objectinventory
Data cache
Fileshare
File Gateway VM
Amazon CloudWatch
Events
NFS client Objects in your S3 bucket
NotifyWhenUploadedAPI
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File GatewayMetadata and workload automation
Metadata and objectinventory
Data cache
Fileshare
File Gateway VM
NFS client Objects in your S3 bucket
ls /foo/*
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File GatewayMetadata and workload automation
Metadata and objectinventory
Data cache
Fileshare
File Gateway VM
RefreshCacheAPI
Amazon CloudWatch
Events
NFS client Objects in your S3 bucket
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content distribution with File Gateway
Miami regional office
New York regional officeLos Angeles headquarters
Amazon S3 bucket
NFS client (writer)
File Gateway VM
File Gateway VM
NFS client (read-only)
NFS client (read-only)
File Gateway VM
Request upload
notification
1
Amazon CloudWatch
2
3
Client writes files and calls NotifyWhenUploaded API…1 When files are uploaded, the gateway sends a CloudWatch Event…
2
Event triggers RefreshCache API on gateways in the remote offices
3 When RefreshCache completes, the gateway sends a CoudWatch Event notification
4
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enabling cloud workloadsMove data to AWS storage for big data, cloud bursting, or migration
“Storage Gateway has the promise to transform the way we move data into the cloud. The NFS interface lets us easily integrate data files from analytical instruments, and the transparent S3 storage
lets us easily connect our cloud-based applications and leverage the powerful storage capabilities of S3.
With Storage Gateway, we can now unleash the full power of AWS on our instrument data.”
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Seamlessly connect on-premises applications with AWS Cloud storage; No changes to existing applications
• Local cache for low-latency access to data stored durably in AWS
• Supports use cases throughout your cloud journey: Backup & DR, tiered storage, hybrid processing workflows, migration
• Login to your AWS account, download the VM (or launch an EC2 instance), deploy new on-premises storage in minutes
AWS Storage Gateway
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.