a case study in metrics-driven devops

Download A Case Study in Metrics-Driven DevOps

Post on 28-Jan-2018

54 views

Category:

Software

3 download

Embed Size (px)

TRANSCRIPT

  1. 1. DT2 Session 6/9/1611:30AM ACaseStudyinMetrics-DrivenDevOps Presentedby: BenVining ElectronicArts Broughttoyouby: 350CorporateWay,Suite400,OrangePark,FL32073 888---268---8770904---278---0524-info@techwell.com-http://www.techwell.com/
  2. 2. BenVining ElectronicArts BenViningisadevelopmentmanagerofaDevOpsteamatVancouver-based ElectronicArts.BenhasworkedongamesforEAfranchisessuchasFIFA,NHL, UFC,andPlantsvs.Zombies.Inapreviouslife,hewasabusinesssystemsanalyst withJELD-WENWindows&DoorsinAustralia,helpingtodevelop,implement, andmaintaintheirin-houseERPsystem.Benbelievesinworkingsmart,taking timetocelebratewins,andlearningfrommistakes.ThisisBen'sfirsttime presentingataconference,sopleasegoeasyonhim.Youcanfindhimon LinkedIn.
  3. 3. 5/30/16 1 A Case Study in Metrics- Driven DevOps BEN VINING | ELECTRONIC ARTS DEVOPS WEST | JUNE 2016Overview WHYYOUREHERE Youreinterestedinusing metricstoimproveyour teams: Performance ReputaDon Happiness WHATTOEXPECT IntroducDon TheCaseStudy ExampleObjecDves BigBenets&Key Takeaways QuesDonTime(attheend)
  4. 4. 5/30/16 2 IntroducKon BEN VINING | ELECTRONIC ARTS IntroducKon Ben Vining 3years B.Sciencein InformaDon Systems Universityof NewSouth Wales Sydney,AU 8years Business Systems Analyst JELD-WEN Windows& Doors GoldCoast,AU 3years Development Manager ElectronicArts Vancouver,CA
  5. 5. 5/30/16 3 IntroducKon Electronic Arts (EA) ElectronicArtsInc.isaleadingglobalinteracDveentertainmentsoeware company.EAdeliversgames,contentandonlineservicesforInternet-connected consoles,personalcomputers,mobilephonesandtablets. The Case Study THE TEAM | WHERE WE WERE | BECOMING METRICS-DRIVEN | WHERE WE ARE NOW
  6. 6. 5/30/16 4 Development & Release Engineering (DRE) Weprovidebuild&testautomaDon servicestoourcustomers. Ourcustomersaregame,content,and engineteamsatEA. 70stain6locaDonsaroundthe globe. Ourvision:EAdeveloperscanworkto theirfullestpotenDal,becauseour systemstakecareofeverythingelse. Ourmission:makelifesimplerfor developers. Overall Team Structure Sub-TeamLeads Leadership SeniorLeadership Group DD&TD DDs& TDs Canada USA Europe Engine Infra.
  7. 7. 5/30/16 5 Sub-Team Structure Projects Sub-TeamLeads Canada 1xLarge 4x Medium 4x Medium 6xSmall Project Structure Eachcustomeristreatedasaunique project Manyteam-membersareassignedto eachproject Eachteam-memberisassignedto manyprojects FrequencyofinteracDons(meeDngs, reports,communicaDons,etc.)is dependantonsizeandcultureofthe customerteam PointofContact Leads AddiDonalSEs
  8. 8. 5/30/16 6 Where We Where WearenowinthefourthyearofourapproachtometricscollecDon& reporDng. Beforethat: OurautomaDonsystemswerentreliableenough.Oneinvejobswouldfail toproducemeaningfulresults WedidntknowwhereourDmewent.Wehadnovisibilityonwhichprojects demandedthemosteortorhowlongwespentonsupport Itwasnotuncommonforengineerstoleavetoworkonagameteam Wewereasmallerteamworkingacrossthreestudiosandaselectrangeof projects Becoming Metrics-Driven EvaluaDon&EvoluDon Leadership ImplementaDon&OperaDon Sub-TeamLeads Design SeniorLeadership
  9. 9. 5/30/16 7 Design CorePillars ObjecDvesDeniDons ClassicaDon DataPoints Storage SummaryReport Design Core Pillars Healthy Reliable Ecient Responsible Predictable Responsive InnovaDve Respected
  10. 10. 5/30/16 8 Design ObjecKve DeniKons DescripDon Source Intent Target Name ObjecKve AutomaKon Reliability Descrip0on:Percentageof automaDonrunsthataresuccessful Source:Statsarepulledfromthe automaDonsystemweekly Intent:OurautomaDonsystemis closelymonitoredandmaintained Target:97% Note:failuresduetothegamecode countasasuccesstous AutomaDon Reliability
  11. 11. 5/30/16 9 ObjecKve Time Logged Descrip0on:Percentageofavailable DmeloggedintheDckeDngsystem Source:Statsarepulledfromthe DckeDngsystemweekly Intent:Weknowwherewespend ourDme Target:100%(ofsixhoursaday). Note:noneedtolog:meagainst mee:ngsandotherinterrup:ons TimeLogged ObjecKve Team Health Descrip0on:Averageresultsper teammemberfromtheTeam HealthSurvey Source:Surveyiscompletedtwice peryear Intent:Ourteamishealthy&happy Target:4/5 TeamHealth
  12. 12. 5/30/16 10 ObjecKve External RecogniKon Descrip0on:NumberofDmesour teamispraisedbycustomers(atthe seniorleadershiplevel) Source:Casesaremanuallyrecorded asrequired Intent:Ourbrandiswell-respected aroundthecompany Target:4peryear ExternalRecogniDon Design ObjecKve ClassicaKon Customer Facing Internal OperaDons Strategic Sub-Team
  13. 13. 5/30/16 11 Design Data Points Baseline Target Current Previous Currentvs.Target Currentvs.Previous Design Storage TherawdataiscomingfrommulDplesources(automaDonsystem,DckeDng system,surveys,etc.) WeuseExcelasthecentralstoragelocaDonforourmetrics Eachmetrichasatablethatstorestherawdata Separateentryforeachsub-team,eachperiod Wehaveconsideredusingothertools,butExceldoesthejobandweareall familiarwithusingit
  14. 14. 5/30/16 12 Design Summary Report DisplaystheaggregatedresultsforeachobjecDvewiththedatapointsas discussedearlier Baseline Target Current Previous Currentvs. Target Currentvs. Previous Customer Reliability 95.0% 97.0% 98.1% 95.3% 1.1% 2.8% TicketAck. 8.0 4.0 4.5 4.8 0.5 -0.3 InternalOps TeamHealth 3.8 4.0 4.2 3.8 0.2 0.4 ImplementaKon & OperaKon Training GatherBaselines SprintProcess DataCollecDon FocusedReporDng
  15. 15. 5/30/16 13 ImplementaKon Training Geteveryoneontheteamon-board Whywearedoingthis? ExpectaDons Workowchanges Systemchanges ImplementaKon Gather Baselines Beforewestartedregularlyrecordingthemetrics,werstgatheredthe baselinestounderstandwherewearenow Determiningthebaselinewastrickyforcertainmetrics: Inaccuratetracking Notrackingatall Nofocusonit Dataisthere,butcollecDngitisinecient SDll,havingthesebaselinesmadeiteasytoshowimprovementsearlyinthe process
  16. 16. 5/30/16 14 ImplementaKon Sprint Process Sub-teamrunsaweeklysprint FibonacciSequence(1,2,3,5)foresDmaDngcomplexityoftasks Tasksinthesprintcovertherangeofprojectsthatsub-teamisresponsiblefor Onlycommittasksthatwepromisetonish WereaDevOpsteamberealisDcinhowmuchworkwecommitto Combinedsprintisinternal,i.e.notsharedwithcustomers ImplementaKon Sprint Admin (Time) Name Days Hours/Day LoggedHours Angus 4.5 6 29 Julia 4 6 22
  17. 17. 5/30/16 15 ImplementaKon Sprint Admin (Tasks) Name Task Points Status Angus TASK-1 3 Complete Angus TASK-2 3 UnderesDmated Julia TASK-3 5 De-prioriDzed Julia TASK-4 2 Complete OperaKon Data CollecKon Warning:metricsdatacollecDonintroducesoverhead Weautomatewhatwecan,butthedatacomesfrommulDplesourcesandis manuallyentered ItsimportantforeachobjecDvetohaveaclearowner ObjecDveowners:justgetinanddoit.Onceyouletitsliponeweek,itseasyto letitslipagain.Andagain
  18. 18. 5/30/16 16 OperaKon Focused ReporKng TheresultsofeachautomaDonrunarestoredinacentraldatabase. ThisallowsustocloselymonitortheresultswithfocusedreporDngvia: InteracDveReports TVDashboard DailyAutomatedEmails Reliability ReporKng InteracKve Reports AutomaDonReliability Success GameFail InfraFail Failure Count FailureDescrip0on 70 GAMEcompileerror 20 INFRAnodiscspace 10 INFRAnetworkerror
  19. 19. 5/30/16 17 Reliability ReporKng TV Dashboards Perceived 100% Tests 93.1% Builds 98.1% Reliability ReporKng Automated Emails 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ProjectA ProjectB ProjectC ProjectD TestReliability-Yesterday Success Failure
  20. 20. 5/30/16 18 EvaluaKon & EvoluKon Sub-TeamReviews QuarterlyUpdates AnnualRefresh EvaluaKon Sub-Team Reviews Leadershipmeetsregularly,rotaDngamongstthesub-teams Comparecurrentresultsvs.previous&target WhatacDonshavebeentakensincethepreviousreview? WhatacDonswillbetakeninthefuture? Focusisnotnecessarilyonthenumbers,butmoreontheacDons
  21. 21. 5/30/16 19 EvaluaKon Quarterly Updates Anupdateonthetrendofkeymetrics ispresentedintheteamsquarterly townhallmeeDng Followed-upwithemail 91% 92% 93% 94% 95% 96% 97% 98% 99% 100% Q1 Q2 Q3 Q4 AutomaDonReliability CAN USA EUR Target EvoluKon Annual Refresh Towardstheendoftheyear,theteamhastheopportunitytoupdatethe objecDves: AddnewobjecDves RemovestaleobjecDves MoveobjecDvesbetweencore&sub-team Modifytargets Re-baseline
  22. 22. 5/30/16 20 Where We Are Now WearenowinthefourthyearofourapproachtometricscollecDon& reporDng. Improvementsweveseen: OurautomaDonsystemsarereliable.Weregularlyhitourtargetsand97%of runsproducemeaningfulresultstothecustomer WeknowwhereourDmegoes.WecantrackDmespentperproject,type (supportvs.value-add),orarea(builds,test,infrastructure,reporDng,etc.) Theteamishappyandconsistentlyscoresover4/5intheteamhealthsurvey TheteamhasexpandedtoworkonaddiDonalprojectsinotherstudios Example ObjecKves OVERVIEW | DETAIL
  23. 23. 5/30/16 21 Overview of Example ObjecKves AutomaDonReliability TimeLogged TeamHealth ExternalRecogniDon CustomerSaDsfacDon BudgetVariance TicketAcknowledgement SupportTicketClosure TimeinWork Overhang PlannedWork SupportWork ObjecKve Customer SaKsfacKon Descrip0on:Averageresultsper respondentfromtheCustomer SaDsfacDonSurvey Source:Surveyiscompletedtwice peryear Intent:Ourcustomersarehappy Target:4/5 CustomerSaDsfacDon
  24. 24. 5/30/16 22 ObjecKve Budget Variance Descrip0on:Percentagethatweare withinourbudget Source:Statsarepulledfromthe nancesystemquarterly Intent:WesDcktoourbudget Target:0% BudgetVariance ObjecKve Ticket Acknowledgement Descrip0on:AverageDmecustomer createdDcketsspendinthenew state Source:Statsarepulledfromthe TickeDngsystemweekly Intent:Ourcustomersarepromptly respondedto Target:4hours Ticket Acknowledgement
  25. 25. 5/30/16 23 ObjecKve Support Ticket Closure Descrip0on:AverageDmeittakesto closesupportDckets Source:Statsarepulledfromthe TickeDngsystemweekly Intent:Brokenthingsarexed promptly Target:4days SupportTicket Closure ObjecKve Time in Work Descrip0on:AverageDmeDckets spendintheinprogressstate Source:Statsarepulledf

Recommended

View more >