Dean Samuels

Technical lessons on how to doBackup and Disaster Recovery in the Cloud

Solution Architect, Amazon Web Services

Agenda

• Prologue The need for Backup and DR

• Customer Story Ausenco Limited

• Lessons Backup and DR with AWS

• Conclusion Q&A

The need for Backup & DR

• Archiving is the process of moving data that is no longer actively used to a separate data storage device for long-term retention. Data archives are indexed and have search capabilities so that files and parts of files can be easily located and retrieved.

• A backup or the process of backing up is making copies of data which may be used to restore the original after a data loss event. The primary purpose is to recover data after its loss, be it by data deletion or corruption. The secondary purpose of backups is to recover data from an earlier time.

• Disaster recovery (DR) is the process, policies and procedures related to preparing for recovery or continuation of technology infrastructure critical to an organization after a natural or human-induced disaster.

Some Definitions

Where is this…. Brisbane, QLD

Source: http://www.abc.net.au/news/specials/qld-floods/

Where is this…. Brisbane, QLD - January 13th 2011

Ausenco

Source: http://www.abc.net.au/news/specials/qld-floods/

Milton, QLD (Lang Park)

Source: http://www.abc.net.au/news/specials/qld-floods/

Milton, QLD (Lang Park) – January 13th 2011

Source: http://www.abc.net.au/news/specials/qld-floods/

Coronation Drive, Milton, QLD

Source: http://www.abc.net.au/news/specials/qld-floods/

Coronation Drive, Milton, QLD – January 13th 2011

Source: http://www.abc.net.au/news/specials/qld-floods/

Backup, Disaster Recovery and Data Tiering with AWS

Mike Latsky

Director, IT Operations

Ausenco

About Ausenco

We provide:Engineering, Procurement, Construction Management andOperational Servicesto the energy, resource, and process industries

Presentation title | DD Month Year | 12

IT Perspective• Engineers generate large 2D and 3D

CAD files

• CAD performance demands close network proximity

• Drawings are Ausenco’s deliverable and MUST be protected

• ASX200 company

• Head office in Brisbane

• 29 Offices in 19 countries

• 3500 Employees

Ausenco Offices

Our IT Environment

• Completely virtualised

• Windows

• ERP - Oracle SaaS

• Email, Lync, SharePoint – MS Office 365 SaaS

• Onsite storage

• 2 x primary data centres

• 1 x secondary data centre

Our Challenge

• Backup – Unreliable

• Disaster Recovery– Sites were exposed

• Local Disk Storage– At capacity

Possible Solutions

An Enterprise Backup Solution?

Secondary Data Centres?

More Disk Storage?

Our initial approach

• Large Integrators

• Leading backup providers

• Leading archiving providers

• Hardware providers

• Data centre hosting providers

• Leading cloud providers

Paradigm Shift

• Why use traditional technologies?

• How can we better leverage the AWS cloud?

• How do we do more with less?

• Address the complete data lifecycle

• Flexible, Scalable, Cost Effective

Disaster Recovery Archiving Backup

Solution

The Result

• Low effort• Incremental rollout• Cloud migration

• Efficient, effective usage of bandwidth• Application/database servers• Regionalised deployment

Our Roadmap

• Wider deployment

• Decommission costly secondary data centre

• Migration of production systems to AWS

Thank-you

Backup and DR with AWS

Backup & DR Lessons

Business and Technical Drivers….

Reduce costs

Slash DR budgets by up to 50%

Consolidate sites

Eliminate the need to run a secondary site

Reduce on-premise

Eliminate 30%+ of on-premise physical equipment

Remove aging technologies

Eliminate tape for backup and archive

The fundamental economic model…

Utility, on-demand datacenter

Primary SiteRouters

Firewalls

Network

Application Licenses

Operating Systems

Hypervisor

Servers

SAN

Primary Storage

Backup

Archive

AWSRouters

Firewalls

Network

Application Licenses

Operating Systems

Hypervisor

Servers

SAN

Snapshot Storage

Backup

Archive

Secondary site costs

Backup and DR with AWS

Backup Lessons

Backup Lessons – My backup should be accessible

Source: http://www.abc.net.au/news/specials/qld-floods/

a.k.a. the pain of physical data transfer

AWS Direct ConnectDedicated bandwidth between

your site and AWS

Amazon Storage GatewayShrink-wrapped gateway for volume

synchronization

AWS Import/ExportPhysical transfer of media into and

out of AWS

Getting data into the cloud

Simple Storage ServiceHighly scalable object storage

1 byte to 5TB in size

99.999999999% durability

Elastic Block StoreHigh performance block storage device

1GB to 1TB in size

Mount as drives to instances with

snapshot/cloning functionalities

GlacierLong term object archive

Extremely low cost per gigabyte

99.999999999% durability

Storage Options

Very fast ‘instance’ disks

Slow, rare accessFast web object storage

Cost ($/GB/month) Performance Durability

Amazon S3 0.095

Amazon Glacier 0.01

AWS Storage Gateway 0.095(+ 125/GW)

Amazon EBS 0.10

Amazon EBS (PIOPS) 0.125

*AWS Oregon pricingStorage Options

Snapshot of local volumes

Restoration from snapshots

Backup Lessons – My backup should be accessible

VPN

Public / AWS Direct Connect

AWS Import/Export

z

Backup Lessons – My backup should be accessible

GW-Cached volumes

GW-stored volumes

“Cool” storage

“Cold”

Backup Lessons – My backup should be accessible

Backup Lessons – My backup should be accessible

• “Infinite” scale with Amazon S3 and Amazon Glacier

• Scale to multiple regions

• Seamless

• No need to provision

• Cost tiers (cheaper at scale)

Backup Lessons – My backup should be able to scale

• SSL Endpoints (Amazon S3 and Amazon Glacier)

• Signed API calls

• Store encrypted files

• Server-side encryption

• Multiple copies across different data centers

• Local/cloud with AWS Storage Gateway

Backup Lessons – My backup should be safe

• Easy to integrate within AWS or Hybrid

• AWS Storage Gateway: Run services on Amazon EC2 (DR)

• Clear costs

• Reduced costs

• I decide redundancy/availability in relation to costs

Backup Lessons – My backup should work with a DR policy

Backup Lessons – My backup should work with a DR policy

• Clear ownership

• Permissions with IAM: Users, groups roles

• Logs

• AWS support

Backup Lessons – Someone should care about it

1. My backup should be accessible

2. My backup should be able to scale

3. My backup should be safe

4. My backup should work with a DR policy

5. Someone should care about it

Backup Lessons

Backup and DR with AWS

DR Lessons

DR is part of a wider set of policies and controls…

DR & business continuity

It’s not an all or nothing thingChoose what needs to failover and what does not

Some things more important than othersSome things will still be working

High availability Backup Disaster recovery

Keep your applications

running 24x7

Make sure your data is safe Get your applications and

data back after a major

disaster

Each set of IT assets will have different requirements…

DR & business continuity

Recovery Time Objective (RTO)

How quickly you need this asset to be

recovered?

e.g. 1min? 15min? 1hr? 4hrs? 1day?

Recovery Point Objective (RPO)

How ‘fresh’ the recovery must be for the

asset?

e.g. zero data loss, 15mins out of date?

Assets will sit on a spectrum of technical complexity…

DR & business continuity

Rebuild when required from offsite backup

Run hot-hot configuration with

auto-failover

Custo

mer

facin

g tra

nsac

tiona

l

web ap

plica

tion

Inte

rnal

com

mun

icatio

ns

syste

m

Daily

sche

duled

proc

esse

s & sy

stem

s

Back

end

repo

rting

syste

m

& dat

abas

e

DR Lessons – You NEED a DR plan in place

DR with High Availability

App DR with Standby

DR Lessons – You NEED a DR plan in place

DR Lessons – Testing your DR

• Dev/test in the cloud is super easy

• Spin up capacity only for the test

• Regularly test your DR

• Cost is minimal

• What about data transfer speed?

s3cmd ls --recursive

s3://datasets.elasticmapreduce/ngra

ms/books/ | awk '{print $4;

sub(/s3:\/\/datasets.elasticmapredu

ce/, "/array", $4); print $4}' |

parallel -j0 -N2 --progress

/usr/bin/s3cmd --no-progress get

{1} {2}Copying 2.4 TB down from 48 hours to 9 hours (5x faster)

DR Lessons – Reducing Costs

• Dev/test in the cloud is super easy

• Spin up capacity only for the test

• Regularly test your DR

• Cost is minimal

• What about data transfer speed?

DR Lessons – You can have different DR solutions

• Easy to integrate existing vendors with DR on AWS

• Approach: One vendor/hybrid/multiple vendors

• One region/multi-regions (if you need geo-diversity)

• Different DR Architectures

Backup & Restore Pilot light

Warm standby in AWS

Multi-site solution in AWS & on-

premise

Pilot light

Stopped instances

Pilot light

Running instances

1. You NEED a DR plan in place

2. Testing your DR

3. Reducing costs

4. You can have different DR solutions

DR Lessons

aws.amazon.comget started on the free tier

Technical lessons on how to doBackup and Disaster Recovery in the Cloud