T16 Performance Testing Thursday, May 3rd, 2018 1:30 PM

Embedding Performance Engineering into the CI/CD Pipeline

Presented by:

Anjeneya Dubey

McGraw-Hill Education

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888-- -268- - -8770 ·· 904- --278-- -0524 - info@techwell.com - http://www.stareast.techwell.com/

Anjeneya Dubey McGraw-Hill Education Anjeneya Dubey is the director of performance engineering for McGraw-Hill Education, a learning science company that delivers personalized learning experiences. His responsibilities include ensuring that every product built is high performing, highly scalable, highly available, highly reliable, and fault tolerant. In his past five years with McGraw-Hill, Anjeneya has built automated performance engineering frameworks that detect performance and scalability issues early on in a fast-paced agile environment. Previously he was a technology consultant, focused on providing enterprise quality and performance engineering solutions. Anjeneya has worked with large institutions to set-up enterprise performance and quality engineering solutions.

3/13/18

1

Embedding Performance Engineering Into Continuous Integration & Continuous Delivery Pipeline

By – Anjeneya Dubey

LittleContextaboutMcGraw-HillEducationandMe

AnjeneyaDubeyDirectorofPerformanceEngineeringAnjeneya.dubey@mheducation.com

SoftwareengineeringPerformanceEngineeringCapacityEngineeringInfrastructurePlanningandimplementationAWSCloudArchitecture&OperationsSiteReliabilityEngineering

2

3/13/18

2

Agenda

• ContinuousIntegrationandContinuousDelivery• WhatdoesitmeantoincludeperformanceengineeringintotheCICDPipeline• Challenges• Whatdidwedotoincludeperformanceengineeringinthepipeline

• Processchanges• Performancetesttypes• TestEnvironmentmanagement• TestDatamanagement• ToolsandTechnologiesweuse• Pass/failDecisionMaking

• SelfServicePerformanceEngineering• UsingAIinproduction• Do’sandDon’ts• Summary

3

• AutomatedbuildprocessandbuildverificationtestsforeachenvironmentinContinuousintegration• ExtendContinuousintegrationbyrapidlydeployingcapabilitiestouserstogaincompetitiveadvantage• Reducetestcycletime&timetomarket• Highlyautomatedtesting&release/roll-back• Quickerautomateddecisionmaking&feedbackloop

ContinuousIntegration/ContinuousDelivery

4

3/13/18

3

EmbeddingPerformanceintothePipelineYourpipelineascode• Dev->Test->Prod• Dev->Test->Performance->Prod

Whatdoesitmean?AddingPerformanceenvironmentintothepipelinemeansthatnowtheperformancetestsareblockingyourcodepromotion

Dev Test Production

CodePromotion

Feedback

Performance

5

Challenges- Cultural

• Performanceisanafterthought• Isnotpartoftheagileteams• Isnotpartofthequalityteams• Donotgetincludedintheagileceremonies• Createawarenessonperformancetasks• Empowerdevtotest

6

3/13/18

4

Challenges- Technical

• Automatingtheperformancetestingandanalysis• ReducingTimetoprepareandexecutetest• Quicklyreactingtoperformancemetrics• AutomaticPass/Fail• Scalingtheloadtesttoolforvarietyoftests• Keepthetestingenv/dataconsistent• WhattoShiftleftwhattoshiftright• Costofrunningperformancetestoneverybuild

7

Howdowedoit@MHE?

8

3/13/18

5

Processchanges

• Makenonfunctional/Performancerequirementaspartofthefunctionalrequirement• APIcontracts• Includeperformanceaspartofdefinitionofdoneforsprints• Cleardefinitionofperformancereadyproduct• DiscussPerformanceresultsaspartofthesprintdemoswithallstakeholders

9

PerformanceRequirementsWorkflow

Storieswithacceptancecriteriathatincludesclearperformancerequirements• APIXmusthandleloadofxxtransactionspersecwith95%ileresponsetimeas100ms• AllStoriesmustbeevaluatediftheyrequireperformancecriteria• Performancetestsshouldbecreatedtovalidatethecriteriawithinthesprints• PoorPerformance=FunctionalBug

10

3/13/18

6

TypicalPEProcess

11

TestEnvironment

• Useproductionlikeperformanceenv• Spinuponlywhenyouruntesttosavecost• RefreshDBsfortestdatamanagement

12

3/13/18

7

• Canexpandandcontract- Autoscaling• Infrastructureascode– Terraform,Puppet• Creating&destroyingenvs atease• Createparallelenvs forparallelexecutions

Cloudmakesiteasier

13

Spinupparallelenvs forparallelexecutions

PerfEnv A

PerfEnv B

• Productioncapacityinstance

• ProtocolLevelfullloadtest

• UIperformancetestusingfunctionaltestscripts• Scaleddownenv

• Stress/capacitytest

PerfEnv C

• Testingoutsidethepipeline

• Troubleshooting• Benchmarking/baselining

tests

14

3/13/18

8

PerformanceTesttypesinCICD

• UserExperience- Browsersideperformance• Loadtests• Capacity/StressTests

15

SingleuserperformanceGoodUX=CustomerHappyHowdowemeasurethat?• Collectsingleuserbrowsersideresponsetimes• Leveragefunctionaltestscripts(selenium)• Createscenariosthatyouwanttomeasurethroughourselfserviceautomationframework• AllMethodsinthescriptshavethesnippetthatcollectstheresponsetimes• Executedfromvariousgeolocations• Usabletimevslastbyte• CollectingHAR&Creatingvideosofthetestsforofflineanalysis• UploadthedatatoS3• MHEPerformancePlatformtakesoverfromthere

16

3/13/18

9

Loadtests

• Fullloadtests• ScaledDowntests• Stresstesttofindcapacity

17

FeatureFlags

Whattodowhenyoufindperformanceissues?• Blocktherelease• TurnOffthefeaturethatcreatestheperformanceissue

18

3/13/18

10

TestDatamanagement

• Makeourtestsselfcontained• Create&destroydataaspartofthetestasmuchaspossible

• Fortheonesyoucantcreateduringthetestyoucreateaspartoftheenvironmentbuildout• SpinupparallelAuroraRDSwithpreseededtestdatatospeedupenv buildout

19

Tools&Technologiesweuse

20

3/13/18

11

PerformanceEngineeringPlatform

• Singularplatformtomanageperformancelifecycleforallofourproducts• PowersCICDforPerformanceengineering• Centralrepoforallmetrics• Dynamicthresholds• Passfaildecisionmaking• PowersSelfServicePerformanceEngineering

21

PEPlatformOverview

CollectorServiceAggregatorService

CentralRepository

Reporting&AlertingService

Developer

22

3/13/18

12

PEPlatform– Performancetesttypes

23

Trending– Performancegraphforeachbuild

Build3383on2/28isfailedfor

anAPI

Build3362on2/26isbrokenforanAPIastheresponsetime

degradedalmost30%

24

3/13/18

13

ContainerizeJMeter

• WeuseJMeterheavilyforthealltheCICDtesting• Distributedloadtesting– weneed1master&Nnumberofslavestogeneratehugeload• ScalingtheJMeterforthousandsofuserswasachallenge• Dockerize JMetergivesthescaleneeded• Speedsuptheprovisioning• Partoftheinfraascode– whichmeanswhenthecodegetsdeployedautomaticallyJMeterfarmgetsprovisionedwherethetestgetsexecuted

25

AutomatedPass/Fail

Basedon3basicrules

• Simple&Easy• Implementable• Dependentonthroughput,responsetimesandsystemKPIs

26

3/13/18

14

Thresholdsforpass/fail

• StaticBusinessresponsetimesSLAs• Dynamicuserexperience/APIlevelResponsetimesthresholds• DynamicSystemResourceutilizationthresholds• BasedonhistoricaltrendforeachAPIandalertsifitdeviatedfromlastntests• AllowsseparatethresholdforeachAPI• Doesn’tallowslippageevenwithinthecontract

27

Self-ServicePerformanceEngineering

• Youdon’tneedtobeperformanceengineertoruntest• Automatetheentireperformancecycle• ScriptCreationthroughaUI• ExecutetestaspartofCICDorExecuteitondemandthroughvoiceenabledAlexaorachatBot• AnalysisthroughAPMandMHEbuiltPerformancePlatform• AutomatedNotificationthroughHipchat/Email/PagerDuty• AutomatedDefectcreationwithdetailsinjira

Tester Devops

Developer

PerformanceEngineer

28

3/13/18

15

Self-ServicePerformanceEngineeringTest

Creation

CICD Alexa ChatBots

APM MHEPEPlatform

Hipchat Email Pagerduty

Execution

Analysis

Notificatio

nDe

fect

Jira29

Notifications

• Automateddefectcreation

• Summaryofthetestresult

• APMdashboardlinkswithdrilldown

• Automatedrealtimehipchatnotifications

• WithJiralinkanddetails

30

3/13/18

16

ShiftRight- Anomalydetection

• TwitterAnomalyDetection• Twitter’sBreakoutDetection• PearsonCorrelationAlgorithm• K-MeansClustering• NewRelicRadar

31

Do’s&Don’ts

• Startwithsimple• Perfectitlater• Removefalsepositives- Getitrightfromthebeginning• KnowyourapplicationsKPIs• Runparallel tests• Runcontinuoustests

• Don’trunbenchmark&endurancetestinCI• Dont removethefailingteststopassthroughCD• Don’tkeepincreasingthethresholdstopasstests• Don’treinventyourPEframeworkratherseehowyoucanleverageyourexistingtoolsandframeworkinCICD

32

3/13/18

17

Summary

• IncludeperformanceengineeringinyourCICDpipeline• Automateautomate&automate• Makeyourtestsrepeatable• Collectmetricsalongtheway• Avoidfalsepositives• Keepanalysis&decisionmakingsimple• Empowerdevs totest

33

Questions?

34