cloud-computing economies of scale€¦ · cloud-computing economies of scale aws executive...

Cloud-Computing

Economies of Scale

AWS Executive Symposium

James Hamilton, 2009.11.10

VP & Distinguished Engineer

e: [email protected]

w: mvdirona.com/jrh/work

b: perspectives.mvdirona.com

Agenda

Infrastructure Efficiency at Scale

Cloud services really are different

Where does the money go?

Where does the power go?

Cloud Computing Economics

Why utility computing makes sense

economically

Amazon Web Services Specialization

11/10/2009 http://perspectives.mvdirona.com 2

Economies of Scale

Large block h/w purchases significantly more economic

Large weekly purchases offer significant savings

H/W Manufacturers willing & able to do custom designs at scale

Automation & custom s/w investments amortize well at scale

Summary: scale economics strongly in play


Large Service [$13/Mb/s/mth]: $0.04/GB

Medium [$95/Mb/s/mth]: $0.30/GB (7.1x)

2006 comparison of very large service with mid-size: (~1000 servers):

Large Service: $4.6/GB/year (2x in 2 Datacenters)

Medium: $26.00/GB/year* (5.7x)

Large Service: Over 1.000 servers/admin

Enterprise: ~140 servers/admin (7.1x)

Services Different from Enterprises

Enterprise Approach:

Largest cost is people – scales with servers (~100:1 common)

Enterprise interests center around consolidation & utilization

• Consolidate workload onto fewer, larger systems

• Large SANs for storage & large routers for networking

Internet-Scale Services Approach:

Largest costs is server & storage H/W

• Typically followed by cooling, power distribution, power

• Networking varies from very low to dominant depending upon service

• People costs under 10% & often under 5% (>1000+:1 server:admin)

Services interests center around work-done-per-$ (or joule)

Observations:

People costs shift from infrastructure to supporting the business

Expect high-scale service techniques to spread to enterprise

Focus instead on work done/$ & work done/joule


Where Does the Money Go?Assumptions (not an Amazon facility):

Facility: ~$200M for 15MW facility, 82% is power dist & mech (15-year amort.)

Servers: ~$2k/each, roughly 50,000 (3-year amort.)

Average server power draw at 30% utilization: 80%

Server to Networking equipment ratio: 2.5:1 (“Cost of a Cloud” data)

Commercial Power: ~$0.07/kWhr

5http://perspectives.mvdirona.com

Observations:• 62% per month in IT gear of which 44% in servers & storage

• Net gear costs high & only 38% in shell, power, power distribution, & mech.

Details at: http://perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx

& http://perspectives.mvdirona.com/2009/03/07/CostOfACloudResearchProblemsInDataCenterNetworks.aspx11/10/2009

44%

18%

19%

15%

4%Servers

Networking EquipmentPower Distribution & CoolingPower

Other Infrastructure

Monthly Costs

3yr server & 15 yr infrastructure amortization

http://perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx

http://perspectives.mvdirona.com/2009/03/07/CostOfACloudResearchProblemsInDataCenterNetworks.aspx

PUE & DCiEMeasure of datacenter infrastructure efficiency

Power Usage Effectiveness

PUE = (Total Facility Power)/(IT Equipment Power)

Datacenter Infrastructure Efficiency

DCiE = (IT Equipment Power)/(Total Facility Power) * 100%


http://www.thegreengrid.org/en/Global/Content/white-papers/The-Green-Grid-Data-Center-Power-Efficiency-Metrics-PUE-and-DCiE
























Where Does the Power Go?

Assuming above average datacenter at PUE ~1.7

Each watt to server loses ~0.7W in power distribution & cooling

IT load (servers, storage, & networking): 1/1.7=> 59%

• Networking under 4%

Power losses easier to track than cooling:

Power transmission & switching losses: ~8%

Cooling losses remainder:100-(59+8) => 33%

Observations:

Server efficiency & utilization improvements highly leveraged

Cooling costs incredibly high at ~1/3

Net gear not large power consumer


Agenda







economically



Infrastructure at Scale

Datacenter design efficiency

Average datacenter efficiency low with PUE over 2.0 (Source: EPA)

• Many with PUE well over 3.0

High scale cloud services in the 1.2 to 1.5 range

Lower cost & much better for environment

Multiple datacenters

At scale multiple datacenters can be used

• Close to customer

• Cross datacenter data redundancy

• Address international markets efficiently

Avoid massive upfront data cost & years to utilize


H/W Cost & Efficiency Optimization

Service optimized hardware

Custom cloud-scale design teams:

• Dell DCS, SGI (Rackable), ZT Systems, Verari, HP, …

Purchasing power at volume

Supply chain optimization

Shorter chain drives much higher server utilization

• Predicting next week easier than 4 to 6 months out

Less overbuy & less capacity risk

Networking transit costs rewards volume

Cloud services unblocks new business & growth

Remove dependence on precise capacity plan


Investments at Scale

Deep automation only affordable when amortized

over large user base

Lack of automation drives both cost & human error fragility

S/W investments at scale

Massive distributed systems investments such as Amazon

Simple Storage Service & Elastic Block Store hard to justify

without scale

Special Skills with deep focus

Distributed systems engineers, power engineering,

mechanical engineering, server h/w design, networking,

supply chain, 24x7 operations staff, premium support,…


Utilization & EconomicsServer utilization problem

30% utilization VERY good &10% quite common

• Expensive & not good for environment

Solution: pool number of heterogeneous services

• Single reserve capacity pool far more efficient

• Non-correlated peaks & law of large numbers

Pay as you go & pay as you grow model

Don’t block the business

Don’t over buy

Transfers capital expense to variable expense

Apply capital for business investments rather than infrastructure

Charge models drive good application owner behavior

Cost encourages prioritization of work by application developers

High scale needed to make a market for low priority work


Agenda







economically



Amazon Cycle of Innovation

15+ years of operational excellence

Managing secure, highly available, multi-datacenter

infrastructure

Experienced at low margin cycle of innovation:

Innovate

Listen to customers

Drive down costs & improve processes

Pass on value to customers

AWS announced price reduction October 27:

Up to 15% off all EC2 instance families & sizes


AWS ApproachBroad set of services:


“Open the hood” approach

Simple, layerable building block services

Component services are substitutable

Infrastructure Services

SimpleDB

Simple Storage Service

CloudFront

Simple Queue Service

Elastic MapReduce

Relational Database

Service

Elastic Block Store

Premium Support

Virtual Private Cloud

Payments & Billing

Flexible Payment

Services

DevPay

On Demand WorkforceMechanical Turk

Alexa Web ServicesWeb Information Service

Top Sites

Merchant ServicesFulfillment Web Service

AWS Scale

Looking at Quantcast Top 500k sites (11/2009)

53% of cloud hosted sites are using AWS

27% growth in AWS hosted site count Oct to Nov

More sites than all others combined

High growth workloads: data Intensive

computing, commercial HPC, analysis, &

optimization

Scale supports deep investment in automation,

monitoring, operations, & funds faster innovation


Sources:

• http://www.quantcast.com/top-sites-1

• http://www.jackofallclouds.com/2009/11/state-of-the-cloud-november-2009/

AWS Pace of Innovation


• RDS Announced

• High-Memory Instances

• Reduced EC2 Pricing

• AWS Multi-Factor Authentication

• Virtual Private Cloud

• Lower Reserved Instance Pricing

• AWS Security Center

• Reserved Instances in EU

• Elastic MapReduce

• SQS in EU

• New SimpleDB Features

• FPS General Availability

• Lower pricing tiers for CloudFront

• AWS Management Console

• EC2 Reserved Instances

• EC2 with Windows

• EC2 in EU

• AWS Toolkit for Eclipse

• AWS Import/Export

• New CloudFront Features

• Monitoring, Auto Scaling

& Elastic Load Balancing

• Elastic MapReduce in EU

• EBS Shared Snapshots

• SimpleDB in EU

• Monitoring in EU

• Auto Scaling in EU

• Elastic Load Balancing in EU

SummaryBenefit from AWS scale

Higher h/w & infrastructure utilization

Better environmentally

Achieve fundamental cost shift

Convert capital expense to variable cost

Lower operating costs at same time

Adapts to inaccurate capacity plans

Pay as you go & pay as you grow

Gain business agility

Obtain h/w resources for new project or expansion in minutes rather than months

Lower infrastructure cost/risk supporting more innovation

Higher productivity


More InformationAmazon Web Services:• http://aws.amazon.com

Designing & Deploying Internet-Scale Services:http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf

Where does the power go & what to do about it:• http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_AFCOM2009.pdf

Recovery-Oriented Computing:http://roc.cs.berkeley.edu/

http://www.cs.berkeley.edu/~pattrsn/talks/HPCAkeynote.ppt

http://www.sciam.com/article.cfm?articleID=000DAA41-3B4E-1EB7-BDC0809EC588EEDF

Autopilot: Automatic Datacenter Operation:http://research.microsoft.com/users/misard/papers/osr2007.pdf

Perspectives Blog:http://perspectives.mvdirona.com

Email:[email protected]


http://aws.amazon.com/

http://research.microsoft.com/~JamesRH/TalksAndPapers/JamesRH_Lisa.pdf

http://research.microsoft.com/~JamesRH/TalksAndPapers/JamesRH_Lisa.pdf

http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_AFCOM2009.pdf

http://roc.cs.berkeley.edu/

http://www.cs.berkeley.edu/~pattrsn/talks/HPCAkeynote.ppt








http://research.microsoft.com/users/misard/papers/osr2007.pdf

http://perspectives.mvdirona.com/

mailto:[email protected]

cloud-computing economies of scale€¦ · cloud-computing economies of scale aws executive...

Documents