KaushikVeeraraghavan,JustinMeza,DavidChou,Wonho Kim,SoniaMargulis,ScottMichelson,RajeshNishtala,DanielObenshain,Dmitri

Perelman,andYeeJiun SongFacebookInc.

2

Webserver

Eachuserrequesttoucheshundredsofsystems

NewsfeedNewsfeed

DBDB

CacheCache

PYMLPYML

AdsAds

SearchSearch

EverstoreEverstore

ScribePtail

CoefficientCoefficient

LaserLaser

ScribeScribe

Theworkloadisconstantly evolving

3

Manyproducts Growinguserbase

• Facebook:3dailyreleases

• Instagram:cont.release

Rapidsoftwarechange

Goals

• Howmanymachinesdoeseachsoftwaresystemneed?

• Canweservepeakload?

• Areweoperatingefficiently?

4

Commonapproachestocapacitymanagement

• Loadmodeling:simulatehowsystembehavesathighload• Loadtesting:benchmarkusingsyntheticworkloads

5

Webserver

NewsfeedNewsfeed

DBDB

CacheCache

PYMLPYML

AdsAds

SearchSearch

EverstoreEverstore

ScribePtail

CoefficientCoefficient

LaserLaser

ScribeScribe

6

Liveusertrafficisthemostrepresentativeworkload

• Accuratedistributionofreads&writes

• Donotneedacustomtestsetup

7

• Directliveusertrafficattarget

Livetrafficloadtestsmeasurepeakservingcapacity

8

Livetrafficloadtestsmeasurepeakservingcapacitysafely

• Monitorhealthmetrics• Responselatency• Servererror

• Resetloadwhenthresholdsarehit

Roadmap

• Krakenmeasurespeakservingcapacityatallscales• Asinglewebserver• Asinglecluster• Anentiregeographicalregion

• Krakenidentifiesbottleneckslimitingutilization• Loadimbalance• Networksaturation

• ChallengesindeployingKraken

9

FrontendCluster

FrontendCluster

Clusterweight

Region

Region

Region

Region

Edgeweight

Webserverweight

DNS

EdgePOP

EdgePOP

EdgePOP

ServiceCluster

BackendCluster

Search

Newsfeed

Krakenusesweightstorouterequests

Krakenmeasuresawebserver’speakservingcapacity

11

• Peakwebservercapacity:175requestspersecond(RPS)• Productiontarget:90%utilizationi.e.,157RPS

175

Krakenmeasuresacluster’speakservingcapacity

• Maxclustercapacity=(webservercapacity)*(num.webserversincluster)12

90%

Krakenmeasuresaregion’speakservingcapacity

13• Wenowserve20%moreuserswiththesameinfrastructure

74%

90%

2015 2016

Inefficientloadbalancinglimitsutilization

14

Networksaturationlimitsutilization

15

Challenge:non-linearresponsetotrafficshifts

16

Webserver

NewsfeedNewsfeed

DBDB

CacheCache

PYMLPYML

AdsAds

SearchSearch

EverstoreEverstore

ScribePtail

CoefficientCoefficient

LaserLaser

ScribeScribe

?

Challenge:howcanwefosterexperimentation?

• Setconservativethresholds

• Communicatewidelyabouttests

• Encouragecollaboration• Monitoring• Failuremitigationstrategies

17

Conclusion

• We’verun50+regional,1000+clusterlivetrafficloadtestsin3years

• Krakenhashelpedusidentifyhundredsofbottlenecksandverifyfixes

• Wecannowserve20%moreuserswiththesameinfrastructure

18

Kraken:assumptionsandcaveats

• Statelessservers

• Routablerequests

• Loadimpactsdownstreamsystems

19

Kraken:usertrafficmanagement

WebLB

FrontendCluster

PoP

DNS

FrontendCluster

ServiceLB

ServiceCluster

PoP

Region

PoP

Region

Region

Region

PoPs Datacenterregions Datacenter

Edgeweight Clusterweight

Serverweight Serverweight

20

Kraken:livetrafficloadtests

WebLB

FrontendCluster

HealthMonitor

FeedbackControl

TrafficShifter

EdgePOP

DNS

MeasurehealthIncrease/resetloadUpdateweights

Kraken

21

HealthmetricsforsystemsaffectedbywebloadServicetype Metrics

Webservers CPUutilization,latency,errorrate,fractionofoperationalservers

Aggregator–leaf CPUutilization,errorrate,responsequality

Proxygen softwareL7loadbalancer CPUutilization,latency,connections,retransmitrate,Ethernetutilization,memorycapacityutilization

Memcache Latency,objectleasecount

TAO CPUutilization,writesuc- cess rate,readlatency

Batch processor Queuelength,exceptionrate

Logging Errorrate

Search CPUutilization

Servicediscovery CPUutilization

Messagedelivery CPUutilization

22

Continuousrunsmeasuringawebserver’scapacity

23

175

SomelessonsfromathousandKrakentests

• SimplicityiskeytoKraken’ssuccess.

• Identifyingtherightperformance,errorrateandlatencymetricstotrackisdifficult.

• Cheapsolutions,likeallocatingcapacityorfixingmisconfiguration,areoftenmoreimpactfulthanprofile-basedtuningorsystemredesign.

24