the performance of clouds complete

8/3/2019 The Performance of Clouds Complete

1/67

1

Cloudcomputingperformance

ABitcurrentstudyontheperformanceofcloudcomputingplatforms

June,2010

Sponsored by Neustar Webmetrics


2/67

2


3/67

3

ExecutiveSummaryCloudcomputingisasignificantshiftinthewaycompaniesbuildandrunIT

resources.Itpromisespay-as-you-goeconomicsandelasticcapacity.Everymajor

changeinITforcesITprofessionalstorebalancetheirapplicationstrategyjustlookatclient-servercomputing,theweb,ormobiledevices.Today,cloudcomputing

ispromptingasimilarreconsideringofITstrategy.

Butitsstillearlydaysforclouds.Manyenterprisesareskepticalofon-demand

computing,becauseitforcesthemtorelinquishcontrolovertheunderlying

networksandarchitecturesonwhichtheirapplicationsrun.

Inlate2009,performancemonitoringfirmWebmetricsapproachedustowritea

studyoncloudperformance.Wedecidedtoassessseveralcloudplatforms,across

severaldimensions,usingWebmetricstestingservicestocollectperformancedata.

Overthecourseofseveralmonths,wecreatedtestagentsforfiveleadingcloud

providersthatwouldmeasurenetwork,CPU,andI/Oconstraints.Wealsoanalyzedfivecompaniessitesrunningoneachofthefiveclouds.

Asyoumightimagine,thisresultedinaconsiderableamountofdata,whichwethen

processedandbrowsedforinformativepatternsthatwouldhelpusunderstandthe

performanceandcapacityoftheseplatforms.Thisreportistheresultofthateffort.

Testingacrossseveralplatformsisbyitsverynatureimprecise.Differentclouds

requiredifferentprogrammingtechniques,sonotwotestagentswerealike.Somecloudsuselarge-scaledatastoragethatsoptimizedforquickretrieval;othersrely

ontraditionaldatabases.Asaresult,thedatainthisreportshouldserveonlyasaguidelineforfurthertesting:yourmileagewillvarygreatly.


4/67

4

AcknowledgementsFirstandforemost,thisresearchwouldnothavebeenpossiblewithout

Webmetrics/Neustar.Thecompanyfundedthedevelopmentofthetestingagents,

allowedustousetheirsystemsfordatacollection,andunderwrotethecostofgeneratingthestudy.Theydidsowithoutconstraintsoreditorialinput,inthehopes

ofcontributingtoongoingdiscussionsandindustrydialogueaboutcloudperformance.Thiskindofaltruismisrare,andweregratefulfortheirassistance.

Thisresearchwasateameffort.

EricPackmanandPeteTaylorworkedhardtodevelop,deploy,andmaintainmultiplesoftwareagentsacrossseveralclouds,andtocollectandanalyzethe

data,lookingforanomaliesandaddressingmonitoringissues.

LennyRachitskyandShirinRejaliofWebmetricssupportedtheideaofindependentresearchthatwouldhelptheITcommunity,andfundedthis

work. SeanPowercontinuestobeagreatcolleagueandco-authoronallthingsto

dowithwebmonitoring.

IanRae,DanKoffler,andtheteamatSyntenicofferedfirst-handcloudexperienceandassistedwithanalysisandearlyfeedback.

Finally,cloudexperts,includingShlomoSwidler,RandyBias,JeremyEdberg,andmanyotherendusersprovidedinvaluableinsight.


5/67

5

ContentsExecutiveSummary......................................... ............................................ ............................................. 3

Acknowledgements........................... ............................................. ............................................ ............... 4

Contents .......................................... ............................................ ............................................. ...................... 5Thestateofwebperformance .......................................... ............................................. ...................... 6

Elementsoflatency............................................. ............................................ ..................................... 6Ashifttowardscomposeddesigns............................. ............................................. ...................... 6

Thereasonsperformancematters............................................ ............................................ ........ 7

Thestateofcloudcomputing ............................................ ............................................ ....................... 8Whatdowemeanbyclouds? ....................................... ............................................. ...................... 8

Theuncertaintyofasharedresource...................................... ............................................ ........ 9

Theproblemwithcomputingfaraway........................................... ............................................ 9

Cloudarchitectures............................................. ............................................ .................................. 10

Thebigquestions............................... ............................................. ............................................ ............ 11

Testingmethodology............................................................. ............................................. ................... 11Testlimitations...... ............................................ ............................................ .......................................... 12

Testresults ........................................... ............................................. ............................................ ............ 14

Realwebsitetestshigh-levelmetrics.................... ............................................. ................... 14

Agenttestshigh-levelmetrics .......................................... ............................................ ............ 15

Performancehistograms ......................................... ............................................ ........................... 17Theperformanceofindividualclouds............................. ............................................. ............ 18

Howdodifferentcloudshandleworkloads? ....................................... .................................. 33Noteworthyobservations ............................................ ............................................ ........................... 40

Conclusions .......................................... ............................................. ............................................ ............ 60

Furtherresearchandreading ........................................... ............................................ .................... 61

PeterVanEijk ......................................... ............................................ ............................................. .... 61Cloudstatus ...................................... ............................................. ............................................ ............ 61Cloudharmony......................... ............................................ ............................................. ................... 62

AlanWilliamsonandCloudkick............. ............................................ .......................................... 63

TheBitsource........................... ............................................ ............................................. ................... 64

Cloudtestagentcode...................................... ............................................ .......................................... 65

Simpleobjects....................................................... ............................................. .................................. 65

CPUtest ...................................... ............................................ ............................................ .................... 65

I/Otest......................... ............................................. ............................................ .................................. 65


6/67

6

ThestateofwebperformanceThisreportfocusesprimarilyontheperformanceofcloudcomputing.While

websiteperformanceoptimizationhascomealongwayfromtheearlydaysofstatic

sites,therearestillmanyreasonsthatapplicationsareslow.

Elementsoflatency

Weblatencycomesfromfourmainfactors:

Servicediscoveryinvolvesfindingthewebsite.ThisisusuallyaDomainNameService(DNS)lookupinwhichtheURLofthewebsiteisresolvedtoan

IPaddress.Theremaybeotherdelaysinthisprocessforexample,whena

siteredirectstheclienttoanothersite.Cloudschangehowlookupshappen,

becausetheymayredirectvisitorstodifferentdestinations.

Networklatencyisthetimespenttravellingacrossanetwork.Thisisa

functionoftwobasicthings:Theround-triptimebetweenthebrowserandthewebserver;andthenumberofround-tripsthatarerequiredtoloada

page.Apagewithfewobjectswilltakefarlesstimetoload,evenoveraslowerlink,thanapagewithmanycomponentsonit.

Processinglatency,orhosttime,istheworktheserverhastodowhenpreparingcontentforthebrowser.Thisistheprimaryfocusofourstudy,since

cloudcomputingchangeshowtheserverworks.Hostlatencymaycomefrom

simplyrespondingtoarequest;fromcomputationallyintensivecalculations;

fromretrievingdatafromothersourcessuchasadatabase;orfrom

connectingtoback-endsystemsbehindtheserveritself.

Client-sidelatencyisthetimethebrowsertakestoassembleandpresentthewebpagecontent.Whileclient-sidelatencyisanincreasinglyimportant

componentofwebmonitoringinmodernwebsites,clientsdontcare(much)whethertheserverisaphysicalmachineoracloudsowewont

concentrateonthiskindofdelayinthisreport.

Whatsmore,inamashupsitethatcontainsembeddedcomponentsfrom

elsewheresuchasJavaScriptforanalytics,orthird-partyembeddedDIVtagsthe

browsermustretrieveadditionalcomponentsfromelsewhere.

Ashifttowardscomposeddesigns

Cloudcomputingisaloadedterm.Therearetwoimportantwaystolookatclouds:

asatechnology,andasabusinessmodel.

Clouds-as-a-technologyrelyonseveralinnovationsvirtualization,automation,

self-serviceprovisioning,chargebacks,service-centriccomponents,andsoonto

improvetheefficiencyofITwhilereducingthetimeittakestodeployapplications.

Ontheotherhand,clouds-as-a-businessrelyonnewwaysforthirdpartyproviders

toofferITondemand.Becauseoftheseparationandefficiencythatsnowpossible,

serviceproviderscanrentavirtualmachineforanhouratatime.Theyreableto


7/67

7

achieveeconomiesofscalethroughautomation,andasaresultarecheaperthan

dedicatedinfrastructureformanyapplications.

Aconsequenceofthesetechnologiesandbusinessmodelsistheemergenceofthe

developerasthecentralroleinIT.Todaysdevelopercanspinupmachineseasily,withoutlongpurchasecyclesortheburdenofadministration.Shecanexperiment

withseveralversionsoftheapplicationforonlyafewpenniesanhour.Andshecanscaleupanddownwithdemand.

Attherootofthisistheideaofacomposeddesignanapplicationthatconsistsof

manysub-applicationsstitchedtogether.Anapplicationmightrelyonavirtual

machineasacomputingresource,butuseanotherservicetostorelargeobjects;

anothertoqueuemessagesforprocessing;andanothertoquicklystoreandretrieve

key-valuepairdata.Thisisadifferentwayofbuildingapplicationsfromearlier,

moremonolithicapproaches.

Thereasonsperformancematters

Wevealwayswantedfastapplications.Itsonlyrecently,however,thatweveknownwhy.StudiesfromGoogle,Microsoft,ShopzillaandStrangeloopNetworksall

provethatfasterwebsitesimproveeverymeasureofonlinesuccess.Visitorswho

haveabetteronlineexperiencesearchmore,buymorethings,stayonsitesmore,

andsoon.SearchengineslikeGooglefactorpageloadtimeintotheirrankingof

pages,whichinturnaffectssearchresultplacementakeyfactorindrivingtraffic

toasite.

Therearefourmaintypesofsite;eachbenefitsfromperformanceinitsownway.

E-commercesites,whichmaketheirmoneyfromtransactions,seeanincreaseinconversionrates(thepercentageofpeoplewhobuysomething)

aswellasinquantitypurchased. Collaborationsites,whichencouragevisitorstocreateandsharecontent,

seevisitorsthataremorelikelytoreturn,aclearsignofengagement.They

alsoplacehigherinsearchrankings,markingthemasanauthorityona

subject.

Software-as-a-Servicecompanieshavemoreproductiveusers,andfacefewerSLArefundsduetodelaybothofwhichreducechurnandencourage

upsellingtoadditionalservices.

Mediasites,whichmaketheirmoneyfromadvertisingandsponsorship,findthatvisitorsstayonthesiteforlonger,andviewmorepages,whentheir

pagesloadquickly.Thismeansmoreexposurestopay-per-viewadvertisingandmorechangesforpay-per-clickadvertisingtowork.

Regardlessofwhatkindofsiteyourerunning,youshouldcareaboutperformance.

Ifyoureconsideringcloudcomputing,youneedtoknowhowthatmovewillaffect

yourbusiness.


8/67

8

ThestateofcloudcomputingThetermcloudcomputinghasonlybeenaroundafewyears,andtheresbeena

lotofconfusionaroundwhatacloudisandisnt.Giventhehugeitjustworks

appealofclouds,manyprovidersandvendorshaveclothedthemselvesinthecloudmantleinthehopesofrevitalizingtheirweb-basedapplication.

Whatdowemeanbyclouds?

Asaresult,clouddefinitionsare,well,cloudy.Justtobeclear,hereswhatwemeanbyclouds:

Privateandpublic

Cloudsmayrunon-premise(sometimesknownasprivateclouds)oronathird-partysinfrastructure(publicclouds.)Ifyourerunningaprivatecloud,youcontrol

thebaremetalonwhichthecloudruns,aswellasthenetworkandarchitecture.Whilethecloudynatureofavirtualized,on-demandenvironmentmightmake

thingshardertodiagnose,yourenotriskingmuchbyadoptingthismodel.

Weregoingtofocusonpubliccloudsforthisreport.

Infrastructure,Platform,andSoftware

Acloudisallaboutseparation.Thedividinglinebetweenyouandyourprovideris

themostimportantaspectofacloudoffering.Whiletheresstillconsiderable

varietyamongproviders,theindustryhassettledonthreebasicofferings.

AnInfrastructure-as-a-Service(IaaS)modeloffersvirtualmachines.Youcopyyourmachinetothecloud,andthatmachinecanrunwhateveranormal

machinewould.Youreinchargeoftheoperatingsystem,theapplication

stack,andsoon.ExamplesofthismodelincludeAmazonWebServicesEC2,theRackspaceCloud,andTerremarkscloud.

APlatform-as-a-Service(PaaS)modelofferscodeexecution.Youcopyyourcodetothecloud,anditruns.Youdontseeindividualmachinesyou

justworryaboutthecodebutyoumayhavelimitedoptionsforyour

environment.InSalesforcesForce.comcloud,youwriteyourcodeina

programminglanguagecalledApex;GooglesAppEngineallowsonlyJava

andPython,andrequiresthatyouuseGooglesstoragemodel,Bigtable,to

storeandretrievedata.

ASoftware-as-a-Service(SaaS)offeringoffersanapplication.Yousimply

addcontentGoogleApps,Netsuite,Freshbooks,andHotmailareallexamplesofaSaaSapplication.Whilemanypeopleconsiderthisacloud,

itsnotaplatformonwhichthingsarebuilt,sowewontincludeitinthis

report.

Inotherwords,forthisstudywerelookingexclusivelyatpublicPaaSandIaaS

clouds.


9/67

9

Theuncertaintyofasharedresource

Almostalltheconcernsaroundpubliccloudcomputinghavetodowithashared

resource.Youmaybeworriedaboutneighboringcloudtenantswithbadsecurity

practices.Youmightfearthatotherapplicationswillconsumeallavailable

resources.Orperhapsyouthinkthatthecloudproviderhaschosenanunderlying

architecturethathelpsthem,butwontmakethemostofyourapplication.

Alloftheseconcernsarelegitimate.Thepactyouremakingwithapubliccloud,for

betterorworse,isthattheadvantagesofelasticityandpay-as-you-goeconomicsoutweighanyproblemsyoullface.Butevenwithinpublicclouds,therearemany

factorsthatwillaffecthowyoufare.

Take,forexample,IaaSversusPaaSmodels.IaaSapplicationsgiveyoumorecontrol,

becauseyoucanchoosetheoperatingsystem,programminglanguages,andtoolsat

yourdisposal.IaaScloudswetestedcoulddoanorderofmagnitudemoreCPU-

intensivecomputationthanPaaSclouds.Ontheotherhand,whenmanyrequests

happenedatonce,theIaaScloudsgotslowwhilethePaaScloudshandledthe

trafficjustfine.

ThisisbecauseIaaSisstillinfrastructure.Youneedtospinupadditionalmachines

tohandleload.InaPaaSenvironment,theresnoupperlimittoprocessingbecause

yourenotrunningononemachine.Infact,PaaSproviderslikeSalesforceandHerokuhavetocreatefalseceilings(calledgovernors)tostoponecustomerscode

fromconsumingtheentiresystem.

Theproblemwithcomputingfaraway

Evenifapubliccloudwereidenticaltoaprivateone,thesimplefactthatitsfurther

awayfromyouchangesthings.AsMicrosoftsJimGraypointedoutin2003,comparedtothecostofmovingbytesaround,everyotheraspectofcomputingis

virtuallyfree.Thatmeansyouregoingtohavetothinkabouthowtogetcontentintoacloud.

Evenifyouregettingtothecloudrelativelyquickly,notallstoragesystemsarethe

same.Inourresearch,ittooknearlythreedaystoinsertdataintoGooglesBigtable

storagemodel.Thisslow-insert,rapid-retrievalmodelisacharacteristicof

Figure1:GovernorlimitreachedonaForce.comapplication


10/67

10

Googlescomputingsystem,whichallowsveryfastsearchesacrosstheentire

Internet.Indeed,onceourdatawasinBigtable,querieswereveryfast.

Humansalsoexpectresponsiveness.Ifyourebuildinganinternalapplicationfor

peopleinyourbuilding,acloudmayintroducedelaysimplybecauseitsnotinthesamebuildingasyourusers.Ontheotherhand,ifyourebuildinganapplicationfor

usersthroughouttheworld,thenGooglesAppEnginehasover20pointsofpresenceandmayspeedthingsup;similarly,AmazonsCloudfrontofferingcanspeedupthedeliveryofstaticcontenttoyourusers.

Cloudarchitectures

Theapplicationyourebuildingwilldictatethecloudmodelyouadopt.:

Ifyourebuildingasimplethree-tieredwebsite,cloudsareasimplechoice,andyoucanlaunchapre-configuredLAMPstackinanIaaSenvironment.

Ifyoursiteexperiencesspikytraffic,andyourewillingtoedityourcode,youmightconsideraPaaSmodeltohandlescaleandpayonlyforCPUcycles.

Ifyouredoingdataminingacrossalargedataset,thenyoullwantaframeworklikeHadoopthatcanprocessthingsinparallel.

Ifyouwantamessagingplatform,thenyoumaywantamessagequeueservice.

Ifyouretryingtobroadcastmediatomanydestinations,youlllikelyinvolveacontentdeliverynetwork.

Intheend,youhavearchitecturalcontrolovercloudsbutitsatamuchhigherlevel.Ratherthanworryingaboutwhichprocessesarerunningonanindividual

server,youreselectingwhichcloudsandcloudservicestouseasyouassembleyour

application.


11/67

11

ThebigquestionsForourresearch,wewanttoanswerafewbasicquestionsaboutclouds:

Howdodifferentcloudsperformforwebusers?

Whatarethedifferencesacrosscloudsforspecificfunctionsnetworkdelivery,computation,andback-endI/O?

Howmuchisoneclouduseraffectedbyitsneighbors?

HowdoIaaSandPaaScloudsvaryinperformance?

Howdodifferentplatformshandlespikesinconcurrentrequests?

Howvariable,orpredictable,arespecificcloudplatforms?

TestingmethodologyToanswerthesequestions,wechosefivecloudplatformsthreeIaaSandtwoPaaS.Foreach,wecreatedatestapplicationthatcouldexercisethevariouselementsof

latencyinwhichwewereinterested:

Aone-pixelGIF,totestrawresponsetimeandcaching.

A2-MByteobject,totestnetworkthroughputandcongestion.

ACPU-intensivetask(repeatedlycalculatingasinefunction)thatwouldconsumeprocessingcapacity.

AnI/Ointensivetask(searchingthroughadatabaseforaspecificrecord,thenclearingthecache)tomeasureback-endsystemsandresource

contention.

Wealsochosefivecompanieswhosewebbusinessesusedeachcloudplatform,

accordingtothefollowingcriteria:

Theyhadtobereputable,real-worldcompaniesratherthanexperimentsor

personalsites

Thesiteshadtobedynamic,ratherthanstatic

Theyhadtobecodedthatis,notsimplyparameterizedversionsofturnkeySaaSapplicationsthathadbeenskinnedtomatchaparticularbrand

Despitethepopularityofcloudcomputing,thiswasntaseasyasitsounds.Many

websitesthatclaimtoberunninginthecloudareactuallycustomer-orpartner-

portalfront-endsforSaaSapplications,whileothersarelittlemorethanstatic

marketingwebsites.

Foreachsite,wetestedasingleobject(usuallyaone-pixelGIForafavicon.ico

image)andafullpageload.Thismeantthateachcloudhadfourteentestsfive

realsitestestedtwice,andfouragentsites.


12/67

12

ThesetestswererunfrommutiplelocationsworldwideusingWebmetricstesting

service.WegeneratedtheanalysisofperformancefrombothWebmetricsreports

andpost-processingofthelogfilesgeneratedbytheWebmetricsservice.

Thegoalofthisresearchisnttorecommendaparticularcloud;

indeed,ourresearchshowsthatdifferentplatformsworkbetter

fordifferentapplicationtypes.Wevenamedthefivecloud

platformsonwhichwetested,butwevehiddenthenamesof

theproductionapplicationsthatwererunningoneachcloud.

TestlimitationsFirstandforemost:thisisnotascientific,repeatabletest.Theverynatureofcloudsis

thattheyreinconsistentandmultitenant,andsubjecttoseasonalanddaily

variances.Consider:

Therearehugedifferencesbetweenthewaysdataisstoredincloud

platforms.Someusedatabases,othersusemassivelyparalleldatastorage

Somecloudsarebusy,andweresharingmachineswithothers;onotherclouds,wemayhavetheluxuryofamachinetoourselves.Wedontknow.

Thewayaparticularfunction(suchasasinecalculation)isperformedvariesfromcloudtocloud.

Thephysicalmachineonwhichavirtualmachineisrunningwillvaryfromtesttotest.

FactorsbeyondthecloudproviderscontrolInternetcongestion,DDOSattacks,problemswithDNSmaychangetheresults.


13/67

13

Muchofthedelaywithinanapplicationmaybetheresultofsomethingthedeveloperdoes;wesawseveraltestedsiteschangeperformance,andgo

down,duringthetest.

Wepurposelyswitchedbetweensequentialandsimultaneoustestmodesduringthedatacollectionperiodinordertounderstandtheimpactof

contentiononresources.Thisisareasonablesimulationofawebsitethatsfacingchangingtrafficpatterns:sometimes,visitsarespreadout;othertimes,theyhappenallatonce.

PaaSplatformsartificiallylimitthenumberofinstructionswecanruninaperiodoftime,whichmeanswehadtoreducethenumberofsine

calculationsperformedsoitsnotaccuratetosaythatPaaSisfasterthan

IaaSfromthisdata,becausethePaaSplatformisactuallydoingfarlesswork.

Infact,whenwetriedthesamenumberofcomputationsonPaaSandIaaS

platforms,theIaaSplatformwassomuchfasterthattheresultswerent

easilymeasurable.

Nevertheless,theresultsarevaluablepreciselybecausetheyshowthevarianceanduncertainty,andhelpusunderstandwhattolookforacrossclouds.


14/67

14

TestresultsWellfirstlookathigh-levelresultsacrossplatformsandtesttypes,thenlookat

individualtesttypes,individualclouds,andtrendsamongwebsitesrunningoneach

cloud.

Realwebsitetestshigh-levelmetrics

First,letslookathowourfivecloudshandledthereal-worldsitesrunningonthem.

Thisgivesusaroughideaofthehealthofeachplatform,butwedontknowanythingaboutthecodetheyrerunningabaddevelopercouldgiveagoodcloud

terribleresults.

Throughoutthisreport,wellusehistogramstorepresentthe

performancemeasurementswecollected.Whileaveragesare

nicetolookat,ahistogramliketheonebelowismore

meaningful.Itshowsthenumberofteststhatoccurredwithaparticularperformancelevelforexample,inthechartbelow,

weseethatroughly27%ofalltestsofthefivesitesrunningon

Salesforce.comtooklessthanasecondtocomplete.

Salesforce

GoogleRackspace

AmazonTerremark

0.00%

10.00%

20.00%

30.00%

40.00%


15/67

15

Agenttestshigh-levelmetrics

Foramorecontrolledanalysisofcloudperformance,weneedtomeasureknown

workloads.Herearetheresultsforourfouragenttests.

Thecloudsallhandledthesmallimagewell.PaaScloudsweremoreefficientat

deliveringthelargeobject,possiblybecauseoftheirabilitytodistributeworkload

outtocachingtiersbetterthananindividualvirtualmachinecando.

EvenwithinPaaSclouds,CPUandI/Ovariedwidely.

Heresadatatableofaverageloadtimesacrossallfivecloudsforatwo-dayperiodinearlyApril.

Salesforce Google Rackspace Amazon Terremark

1-pixel GIF (light test) 0.11s 0.25 0.18 0.23 0.23

2-MByte GIF (heavy test) 0.50 1.97 3.25 4.41 5.00

CPU Test 8.13 1.63 2.16 10.03 3.75

IO Test 6.26 2.03 3.33 19.46 12.35

Averagescanbemisleading,however,assomecloudsexperiencedfargreaterlatency.Heresthemedianlatency:

1-pixelGIF

2MBGIF

CPUIO

0

5

10

15

20

Seconds

Averagelatencyoffouragentsacrossivecloud

platforms


16/67

16

Andheresthemodeoflatency:

Sometimesvarianceisasimportantaslatency,however,sowelookedatthestandarddeviationacrosseachcloudtest:

1-pixelGIF

2MBGIF

CPUIO

0

5

10

15

20

25

Seconds

Medianlatency

1-pixelGIF

2MBGIF

CPUIO

0

2

4

6

8

10

Seconds

Modeoflatency


17/67

17

NotethatIaaScloudsconsistentlyshowgreatervariancethanPaaSclouds;andthat,

becauseitsmeasuringnetworkperformance,the2MByteGIFtestvariessignificantlywhentestedfrommanylocationsontheInternet.Inotherwords,the

variancehereislargelyduetogeography:WANlatencyvariesthemost.

Performancehistograms

Abetterwaytounderstandlatencyistolookatahistogramofperformance,which

showshowmanytestsexperiencedaparticularleveloflatency.Thefollowingchart

combinesallfourkindsoftestsincenetwork,serverresponsiveness,computation,andbackendI/Oareallcomponentsofanapplicationtoshowhoweachcloud

compares.

1-pixelGIF

2MBGIF

CPUIO

0

1

2

3

4

5

6

Seconds

Standarddeviationoflatency


18/67

18

Theperformanceofindividualclouds

Nowletslookateachcloudproviderindetail.

SalesforceGoogleRackspace

AmazonTerremark

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

01

2

3 4 56

78

910+

Percentoftests

Delayinseconds

Performancebycloud,allfouragents


19/67

19

Salesforcesforce.com

ThisisaPaaScloudfeaturingitsownprogramminglanguage(Apex.)Itsdesigned

primarilyforextendingtheSalesforce.comSaaSCRM,anddevelopershaveaccessto

theirCRMsdatastructuressuchascontactsandsalesfunnels.

HerestheperformancehistogramforthefivesitesrunningonForce.com.


20/67

20

HereshowSalesforcehandledthevariouscomputingtaskswegaveit.

Andtheperformancehistogramforthefourtasks:

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

0

2

4

6

8

10

12+

Percentageoftests

Secondsoflatency

FivesitesonSalesforcecloud


21/67

21

1-pixel

2MBCPU

IO

0

10000

20000

30000

40000

01

23

45

67 8

910+

Testcount

Delayinseconds

Salesforcetestproile


22/67

22

GoogleAppEngine

GooglescloudplatformisaPaaS.DeveloperscanuseeitherJavaorPython,andare

billedforcomputationused.TheirapplicationsmustusetheGoogle-specificdatastore,whichallowsscalabilityandfaulttolerance.

HereshowthefivesitesrunningonGoogleAppEnginedidduringthetest:

HerestheperformancehistogramforthefivesitesrunningonAppEngine:


23/67

23

ThisishowGoogleAppEnginehandledourfourtestagents:

Andherestheperformancehistogram:

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%

0

2

4

68

10

12+

Percentageoftests

Secondsoflatency

FivesitesonGoogleapp

engine


24/67

24

1-pixel

2MBCPU

IOTotals

0

10000

20000

30000

40000

50000

60000

01

23

45

6 78

910+

Testcount

Delayinseconds

Googletestproile


25/67

25

RackspaceCloud

Rackspace.comisanestablishedmanagedserviceprovider,offeringbaremetal

hosting,managedhosting,andIaaSandPaaSclouds.HereshowthefivecompanieswhosesitesrunonRackspacefaredduringourtest.

Herestheperformancehistogramofthewebsitesweevaluated.Notethat

performanceisfairlyconsistentacrossthetestperiod,butthatonesiteisslowcomparedtoallothersandaffectsresultssignificantly.


26/67

26

WhenitcametoourtestsrunningonRackspace,hereswhatperformancelooked

like:

HerestheperformancehistogramforthefourtestfunctionsonRackspace.

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%

0

2

4

6

8

10

12+

Percentageoftests

Secondsoflatency

FivesitesonRackspaceCloud


27/67

27

TheCPUtestsufferedfromtwospikesseeninthetestperiod.

1-pixel

2MBCPU

IOTotals

0

10000

20000

30000

40000

01

23

45

6 78

910+

Testcount

Delayinseconds

Rackspacetestproile


28/67

28

AmazonWebServices

Amazonistheclearmarketleader,andformanytechnologiststheirmodelis

synonymouswithcloudcomputing.MostcustomersusetheEC2virtualmachinesandS3storage,butotherservicesarentasbroadlyadopted.

HereswhatthefivesitesrunningonAmazonlookedlike:

HerestheperformancehistogramforthefivesitesrunningonAWS:


29/67

29

Asforourfourtests,hereshowtheycomparedduringthetest:

Herestheperformancehistogramforthefourtestsinthesameperiod:

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%

0

2

4

68

10

12+

Percentageoftests

Secondsoflatency

FivesitesonAmazonWeb

Services


30/67

30

1-pixel

2MBCPU

IOTotals

0

10000

20000

30000

40000

50000

01

23

45

6 78

910+

Testcount

Delayinseconds

Amazontestproile


31/67

31

TerremarkvCloud

Unlikemostpublicclouds,whichrelyonopensourceXenserverforvirtualization,

TerremarkscloudofferingisbasedonVMWaresvirtualizationtechnology.HereshowfiveTerremarkcustomersfaredduringthetestperiod:

Herestheperformancehistogramforthefivesites:

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

0

2

4

6

8

10

12+

Percentageoftests

Secondsoflatenc

y

FivesitesonTerremarkCloud


32/67

32

Hereshowourfourtestagentsperformed:

Thisistheperformancehistogramforthosetests:

1-pixel2MB

CPUIO

0

10000

20000

30000

01

23

45

67

89

10+

Testcount

Delayinseconds

Terremarktestproile


33/67

33

Howdodifferentcloudshandleworkloads?

Nowletslookattheagentswevecreatedforeachcloud.Heresahigh-levelviewof

howwellthecloudshandledeachtypeoftest.

1-pixelGIF2MBGIF

CPUIO

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

01

23

45

67

89

10+

Percentoftests

Delayinseconds

Performancebytesttype,allclouds


34/67

34

1-pixelGIF

Herearetheresultsfortheretrievalofaone-pixelGIFbyeachoftheclouds.

Herestheperformancehistogramforeachcloud.Thisisameasureofhowwellthe

cloudcachesstaticcontentandrespondsquicklytorequests.

SalesforceGoogle

RackspaceAmazon

Terremark0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

0 1 2 3 4 5 6 7 8 910+

Perce

ntoftests

Delayinseconds

Latency,1-pixelGIF


35/67

35

2-MByteGIF

Thesecondtestevaluatesnetworkthroughput.Hereswhatitlookedlikeacrossa

monthoftesting.

Herestheperformancehistogramforeachcloudrunningthetest.

Salesforce

GoogleRackspace

AmazonTerremark

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

0 1 2 3

4 5 6 7 8 910+

Percentoftests

Delayinseconds

Latency,2MBGIF


36/67

36

CPUtest

Thethirdtestlooksatprocessingcapacity,askingthecloudplatformtocompute

1,000,000sineandsumoperations.

NotethatthemeasurementsforSalesforce.comscloudmustnotbecomparedto

othersdirectly;becauseofgovernorsonthePaaSplatformthatlimitthenumberof

instructionsthatcanbecarriedout,thiscloudonlyexecuted100,000operationsonetenthofthosetheothercloudswereprocessing.

Herestheperformancehistogramforthetest:


37/67

37

AmazonsprocessingoftheCPUtestswasslow,butwechosethesmallestvirtual

machineavailablefromtheirservicecatalog;biggermachineswouldhavehandled

thismorequickly.

WealsolookedaterrorratesfortheCPUtest:

Errorsseen,CPUtest Mar15-Apr15

Uptime Errors Successes

Salesforce 99.96% 11 29106

Google 99.99% 3 32212

Rackspace 99.93% 20 32470

Amazon 100.00% 0 30371

Terremark 100.00% 0 31426

SalesforceGoogle

RackspaceAmazon

Terremark0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

0 1 2 3 4 5 6 7 89 10+

Percentof

tests

Delayinseconds

Latency,CPUtest

Salesforce Google Rackspace Amazon Terremark


38/67

38

I/Otest

Inthefourthtest,wesearchastoragesystemforaspecificstring.Herestheresult

ofthatprocessovertime;thisisthemostrevealingintermsofinterestingdata.

NotethetwospikesinlatencywithinRackspace;themassivedropinboth

TerremarkandAmazon,followedbyanincrease;andthegradualgrowthindelay

withinGooglesAppEngine.

Thedrop,andsubsequentincrease,arecausedbyswitchingthetestingmodelfrom

sequential(wheretestshappenoneafteranother)tosimultaneous(wherealltestsoccuratonce.)BoththeTerremarkandAmazonIaaSagentsbecomemuchslower

whentestsaresimultaneousbecausetheyreunabletoscaleautomatically;

however,Rackspace,whichisalsoanIaaSmodel,wasntaffectedbythechange.

Wellcoverthisinmoredetailbelow.

Herestheperformancehistogram:


39/67

39

ThePaaSprovidersdidwell,largelybecauseoftheirsharedstoragemodelthatis

optimizedforlargedatasetsacrossmanymachines;Rackspacescloudperformed

verywellbuthadoccasionalslow-downs.

SalesforceGoogle

RackspaceAmazon

Terremark0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

0 1 2 3 4 5 6 7 89 10+

Percentof

tests

Delayinseconds

Latency,I/Otest


40/67

40

NoteworthyobservationsAsweconductedthisresearch,wesawmanyanomaliesandinterestingpatterns.

Hereareafewofthem.


41/67

41

GradualincreaseinlatencyonGoogleAppEnginesCPU

Overthecourseofthetest,wesawasignificantincreaseintheCPUlatencyof

GooglesAppEngine.WhileGooglesservicewasstillfasterthanotherplatformsatthisspecifictask,wesawthedelayroughlydoubleinthistime.


42/67

42

SpikesinRackspacesCPUperformance

RackspacesCPUtestsencounteredtwosignificanspikesduringthetest,when

latencyjumpedfromarespectable2secondstothreetimesasmuch.


43/67

43

AproblemwithasubsetofcustomersinanIaaS

Inthischart,weseeseveralRackspaceCloudcustomersexperiencingaslow-down

atthesametime.Onebecomescompletelyunavailable,whiletwoothersareslow.Ourstatictestagentremainsfastthroughoutthetest,suggestingthatthisisan

internalproblemaffectingseveralsitessimultaneously.


44/67

44

HowbigaproblemisWANlatency?

Inthischart,wecompareWANperformanceforourI/Otest.WANlatencyisonlya

fractionofthetotaldelayhere.Asawhole,theresaroughlytwo-seconddifferencebetweenthefastestandslowesttestingpointfromwhichwetestedtheagent.


45/67

45

SpikyI/Operformanceandoutages

Inthischart,weseetheRackspaceI/Otestvaryingsignificantly,whileoneofthe

Rackspacecustomerswerewatchingalsogoesofflinesuggestingthataresourceproblemonthebackendaffectedus,andalsoaffectedacompanywhoseapplication

isI/Odependentthatwesharedthecloudwith.


46/67

46

HighvariancewithinSalesforcescloud

ThischartshowsperformanceontheSalesforcecloud.Earlyon,CPUisveryslow;

thenitrecovers,andproceedstogetsloweragain.Duringthatfirstspike,othersitesoccupyingthePaaSplatformalsobecomespikyandslow.


47/67

47

Theimpactofavailabilityzones

ThisdatawaspresentedbyEranShir,CTOofadvertisingserviceDapper,at

Web2ExpoSanFrancisco.Weincludeitheretoemphasizetheissueswithavailabilityzonesandperformance.

InAmazonsmodel,customerscanchooseanavailabilityzonefromwhichtheirapplicationwillbeserved.Thisispartlyforcompliancereasonssocustomers

knowwhatlegislationappliestotheirdataandpartlytoprovideseparationfor

failoverandSLAs.Butallzonesarenotthesame.Considerthelatencyofan

applicationrunninginAvailabilityZoneUS-East-1A:

ComparethistoUS-East-2A:

IntheEasternzone,performanceoftenspikesto2secondsormore;intheWestern

zone,itsconsistentlyfast.Thiskindofperformancedegradationisessentialto

watch.


48/67

48

WANlatencyvariance

Heresagoodexampleofperformancedegradationfromonepartoftheworld(the

WestCoast)thatdoesntaffectanother.


49/67

49

Slow-downsaffectingallsitesononeday

ThischartofFebruary23showsmanysitesrunningonSalesforcescloudall

slowingdownconcurrently.


50/67

50

Slow-downsaffectingasinglesite

Degradationdoesntalwaysaffecteveryoneonasite.Hereareseveralapplications

runningonAppEngine,whereonlyoneofthemhasaproblem.


51/67

51

WhenthewholeInternetisslow

Somedays,manycloudsseeaproblemwhichislikelyaresultofintermittent

issueswiththeInternetasawhole.


52/67

52

Moneyspentonclouds

Forthedurationofthetests,hereshowmuchwespentoneachplatform.Amount Cloud

$127.69 Amazon Web Services (EC2 and EBS)

4.71 Google App Engine

61.44 Rackspace.com

0.00 Force.com

102.87 Terremark

NotethatwedidnthitanyoftheratelimitsfortheSalesforceforce.comcloud,so

wedidntpayforthecapacityweusedaspartofthetestbutbecauseof

force.comsgovernors,wewereperformingonly10%oftheCPUoperationsoftheotherclouds.


53/67

53

IOcontentiononRackspace

Duringouranalysis,wesawwidechangesinI/OperformanceAtonepoint,56356

rsec/smeantonlya74.26%utilizationofthesystem;atanother,584rsec/smeant

100.00%utilization.Thisisaclearsignthatthesystemiscompetingwithother

processesforresources.Heresthedataforthetwoperiods:

Device rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz

avgqu-sz

await svctm %util

Period 1 Sda1 0.00 0.00 673.27 0.00 56356.44 0.00 83.71 4.07 6.04 1.10 74.26

Period 2 Sda1 0.00 0.00 19.00 0.00 584.00 0.00 30.74 2.95 147.37 52.63 100.00


54/67

54

Simultaneousversusspreadouttests

Whenwebeganthetesting,welaunchedseveralrequestsfromseveralcities

simultaneously.ThismeantthatthetestapplicationshadtodealwiththreeorfourusersatexactlythesamemomentandourCPU-andI/O-intensivetestsdidnot

dealwiththiswellinanInfrastructure-as-a-Servicemodel.

Later,weswitchedtestingtospreadrequestsoutmoreevenly(thesequentialsetting),andtheIaaSpatformsdealtwiththeloadfarbetter.Considerthefollowing

chart,whichshowslatencyoftheI/OtestonanIaaScloudbeforeandafterthe

change:


55/67

55

Nowconsiderthesametestvolume,forthesameperiod,onaPaaScloud.

RememberthatPaaShasnonotionofamachinethesystemscalestohandleload

automatically.


56/67

56

Whiletheressomevariance,itsfarlessthanintheIaaSexampleabove.Heresa

graphofthreecloudplatformsbeforeandaftertheswitch:thetwoIaaSplatforms

(TerremarkandAmazon)farefarbetterwhentherequestsarespreadout,while

thePaaSplatform(Google)barelychanges.


57/67

57

HeresRackspacesperformanceacrossthisperiod,showingtheimprovementin

I/Olatencybyspreadingtherequestsout.


58/67

58

Spikesinasinglecloud

ThefollowingchartshowstheI/Operformanceacrossallcloudproviders.Wecan

seethatI/Oononeofthemhasspikedsignificantly,butthatothercloudsseemtoalsohavesporadicslow-downs.

HeresthesameperiodwithinWebmetricsreportingsystem.


59/67

59

VariancewithinGoogleAppEngine

Thischartshowsathree-dayperiodduringwhichAppEnginesperformancevaried

significantly.Whilethesmallimagetestwasrelativelyconsistentinitsperformance,allthreeothertestsspikedandthereappearedtobecorrelationbetweenI/O,

network,andCPUlatency.


60/67

60

ConclusionsFirstandforemost:theresalottowatch.Cloudscanfailinmanyunexpectedways;

herearesomeofthelessonswevelearned.

Watchyourneighbors.Weveseengoodevidencethatseveralcloudapplicationsslowdownatonce,soyoulldefinitelybeaffectedbyothers

usingthesamecloudasyou.

Understandtheprofileofyourcloud.Thehistogramsshownheredemonstratethatdifferentcloudsaregoodatdifferenttasks.Youllneedto

choosethesizeofyourvirtualmachinesintermsofCPU,memory,andso

oninordertodelivergoodperformance.

Youneedanagentontheinside.Whenyouplanamonitoringstrategy,youneedcustomcodethatexercisedback-endfunctionssoyoucantriagethe

problemquickly.

ChoosePaaSorIaaS. Ifyourewillingtore-codeyourapplicationtotakeadvantageofbigdatasystemslikeBigtable,youcanscalewellbychoosing

aPaaScloud.Ontheotherhand,ifyouneedindividualmachines,youllhave

tobuildelasticityintoyourIaaSconfiguration.

Bigdatadoesntcomeforfree.Usinglarge,shardeddatastoresmightseem

nice;butittakestimetoputthedatainthere,whichmaynotbeappropriate

foryourapplicationsusagepatterns.

Monitorusageandgovernors.InPaaS,ifyouexceedyourratelimits,youruserswillgeterrors.

Troubleshootinggetsharder.YouneeddataontheInternetasawhole,yourcloudproviderasawhole,andyourindividualapplicationsvarious

tiers,inordertoproperlytriagetheproblem.Whenyouwererunningondedicatedinfrastructure,youdidnthavetospendasmuchtimeeliminating

third-partyissues(suchascontentionforsharedbandwidthorI/Oblocking.)

PaaSmeansyoureinthesamebasket.Wenoticedthatifyoureusinga

PaaS,whenthecloudgetsslow,everyonegetsslow.WithIaaS,theresmore

separationoftheCPUandtheserversresponsivenessbutyourestill

contendingforsharedstorageandnetworkbandwidth.

Watchseveralzones.Whenyourelyonavailabilityzonestodistributethe

riskofanoutage,youllalsoneedtodeployadditionalmonitoringtocomparethosezonestooneanother.


61/67

61

FurtherresearchandreadingTheresalreadyagoodbodyofresearchoncloudperformanceinthepublicdomain.

Inpreparingthisstudy,weconsultedmanyonlineresources;herearesomeofthe

mostrelevantones:

PeterVanEijk

Peterusedavarietyofapproachestodeducethesizeandscaleofcloudofferings,

andtolookatchangesintheirperformanceovertime.

OneofthemostnotableaspectsofPetersresearchwastherateatwhichcloud

offeringsareimproving.Hefoundthatinasinglemonth,AmazonsCloudfrontunderwenttwosignificantperformanceimprovements:

Figure2:PeterVanEijk'sresearchintoAmazonCloudfrontperformancefromNYC

HisresultswerepresentedattheComputerMeasurementGroupsannualmeetingandareavailableonSlideshare1.

Cloudstatus

NowpartofVMWare,Hyperichasavarietyofagentsthatcollectcloud-specific

metricsacrossseveralpublicclouds.2

1 http://www.slideshare.net/pveijk/cloud-encounters-sept-2009-for-cmg-

dec-6

2 http://www.cloudstatus.com


62/67

62

Figure3:AsampleofHyperic'sCloudstatusdashboard

WhatsnotablehereisthateachcloudhasuniquethingstomeasureonAmazon,

forexample,itmightincludeSimpleDBorSimpleMQdata,whileonGoogleitmight

includeBigtablelookuptimes.

Cloudharmony

Overatwo-monthperiod,Cloudharmonypaidenduserstoruntheirtestingtool

andmeasurethespeedofclouds.3

Figure4:AggregateresultsofcloudperformancecollectedbyCloudharmony

3 http://blog.cloudharmony.com/2010/02/cloud-speed-test-results.html


63/67

63

Theprimaryfocusofthetestingwasend-userthroughput,ratherthanwebpage

latency,buttheywereabletotestmanydifferentplatformsduringthetest.

Cloudharmonysclient-sidemonitoringtoolcanexercisemanycloudfunctionsas

partofitstestingroutine.

Figure5:Aone-timetestofcloudstorageandplatformsbyCloudharmony'sagent

AlanWilliamsonandCloudkickAllanWilliamsonpublisheddataonoversubscriptionandcontentionwithinanIaaScloud,4referencingdatagatheredbyCloudkick.5Thedataindicatesincreasing

congestionwithinAmazonsresources.TheseissuesseemtoplagueAmazonsUS-

Eastavailabilityzone,somethingweveheardfromseveralsources.

4 http://alan.blog-city.com/has_amazon_ec2_become_over_subscribed.htm5 https://www.cloudkick.com/blog/2010/jan/12/visual-ec2-latency/


64/67

64

Figure6:NetworklatencyincreasinginAmazon'sUS-Eastavailabilityzone

Whilenetworkcongestionisonlyonefactorthatcanleadtopoorenduser

experience,itsanimportantone.Ascloudsgrowinpopularity,itsimportanttorememberthattheyreasharedresourceandtoensurethattheproviderisscaling

infrastructurecapacitycommensuratewithdemand.

TheBitsource

TheBitsourcedidanindependentcomparisonofRackspaceandAmazon6to

measureperformancewithinthecloudsthemselves.

Figure7:TheBitsource'scomparisonofcompilationtimeacrosscloudvirtualmachines

ThiscomparisonfocusedmoreonapplyingtraditionalbenchmarkssuchascompilationorI/Ooperationsacrosstwosystems.

6 http://www.thebitsource.com/featured-posts/rackspace-cloud-servers-

versus-amazon-ec2-performance-analysis/


65/67


66/67

66

$chandle = mysql_pconnect("localhost", "root","barfly")or die("ERROR: Connection Failure to Database");

mysql_select_db("MainObj", $chandle)or die ("Database not found.");

$query="select ip_addr, comment,value from vote where comment like'%OIYDGOAPSL%'";$result = mysql_db_query("MainObj", $query) or die("ERROR: Failed

Query");$result = mysql_query($query);echo "IO
";// Check result// This shows the actual query sent to MySQL, and the error. Useful fordebugging.if (!$result) {

$message = 'ERROR: ' . mysql_error() . "
";$message .= 'Whole query: ' . $query;die($message);

}

while ($row = mysql_fetch_assoc($result)) {echo $row['ip_addr'];

echo "
";}

mysql_free_result($result);// this is to flush the cache for next time or we're effectivelycheating the load Dom0system("/sbin/fc");?>

/sbin/fcassumessuperuserprivilegestoflushalldiskcache.ThiswasdoneattheendofthecodetopreventotherloadissuesastheOSrecoversalittlefromhaving

thebufferssuddenlyallmarkeddirty.Thisforcesthenextwebhittonotbeableto

searchthroughtheentiredatabasefromcache,butratherforcesDISKIOrohappenasthoughthiswasthefirstqueryever.

Thecodeforfc.cis:

#include main(){

setreuid(0,0);system("echo '3' > /proc/sys/vm/drop_caches");

}

Toprovethisworks,useiostat-x1andgovisitthewebpageIO.phpafewtimes,

you'llseealltheIOloadaneweachtime.RenameIO.phpandyou'llnotetheentire

DBfitsintoRAM,andthere'snoIOtodiskanymore.

ThecodeforPaaSI/OtestsvariesbyPaaSplatformsignificantly.BothForce.com

andGoogleAppEngineaddvisibleandinvisibledataperrow.Force.comaddsauthor,creator,andothermetadatasuchthat524krowsisabout1.1GB,whereas

it'sonly190megsonamysqldatabase.WhilewecantflushthecacheinaPaaS

cloud,thefactthatwererunningafullscanofagigabyteofdatainashared

environmentsuggeststhatitslesslikelytobecached.


67/67

Googlesinsertslowtoqueryfastphilosophymeansthatdatawasinsertedata

rateof31,160300-byterowsperCPU-hour.Wequicklyburnedthroughthe6.5free

CPU-hoursGoogleoffersitsusers,andintheend,tooknearly3daystopushallof

thetestdataintoAppEnginesBigtablestorage.

GoogleAppEnginedoesnotsupportsearchingforasubstringinastringfield.As

such,theusualsearchforthe6-bytestringfoundonlyinrow11692(whichwasthebasisfortheI/Otest)isreplacedbyasearchfortheentirefield(all255bytes).Thisisessentiallystillafullindexscan,asthiscolumnisnotindexednormally,and

shouldbegiveortakeaboutthesameamountofaggregateIOloadasalltheother

tests.

the performance of clouds complete

Documents