explore the state of open source performance testing in continuous delivery pipelines with...
TRANSCRIPT
World®’16
ExploretheStateofOpenSourcePerformanceTestinginContinuousDeliveryPipelineswithBlazeMeter ChiefScientistAndreyPokhilkoAndreyPokhilko - ChiefScientist,BlazeMeter - CATechnologies
DO5X54E
DEVOPS
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
ContinuousIntegrationhasrapidlybecomethedefactostandardinsoftwaredevelopmentprocesses.Whilefunctionalandunittestinghaveprovedrelativelyeasytointegrateintotheseprocesses,performancetestinghastypicallyraisedmorechallenges.Inthistalk,AndreywillexaminekeyapproachestointegratingperformancetestingintoContinuousDeliveryworkflows,includinghowtoovercomethekeychallengesencounteredineachmethod.
Ultimately,atruecontinuousdeliveryprocessreliesupontheactualizationoffullyautomated,configurationdrivenandversion-basedworkflows.YouwillgetanunderstandingonhowtoevolveyourCIsystemtofacilitatethis.
AndreyPokhilkoCATechnologiesChiefScientist,BlazeMeter
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ImportanceofCIandCD
§ Machinetimecostsnothing,humantimepriceless
§ De-factowinningpractice
§ MostadvancedteamsgowithCD
§ Alotofteamsarestillinprocessofadoptingit
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TriggeringandPreparations
§ Whentodothejob
§ VCScheckout+dependenciescheckout
§ Buildingproject(compilingetc.)
§ Putresultingpackagesintorepos
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
QualityControl
1. Staticcodeanalysis
2. Unittests
3. Functionaltests
4. Performancetests
needs deployment
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Deployment
§ ToQAenvironment(forfurthermanualtests)
§ Tostagingenvironment
§ Toproductionenvironment
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
OneWayRoad(Driverless)
VCSCheckout+
DependenciesBuild Unit
Tests
FunctionalTests
PerfromanceTests
Deployment
Filter Filter Filter Filter
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
VCS-DrivenPipeline
§ NaturalevolutionofCIsystems
§ Branchingandpullrequests
§ Jenkins2.0pipelines
§ TaurusToolaspartofthisapproach
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Challenge#1:TestEnvironment
§ Applicationsarecomplex§ Lotsofdependencies§ Third-partysystems
Databases
Microservices
Third-parties
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Challenge#2:TimeConsuming
§ Preparations§ Alotoffunctionaltests§ Performancetestsarenaturallylong
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Challenge#3:DebuggingCIJobs
§ Evolvingtestcomplexity§ Debuggingandtroubleshooting§ Buildhistoryisavalue
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Challenge#4:ResultsAnalysis
§ Richreportingneeded§ JenkinsUIhasitslimits§ AutomateddecisionmakingforCD§ Collaborationfornon-techpeople
wut???
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Overcome:TestEnvironment
§ Containershelp(youhavetousecontainers:)§ Youdon’tneedtobeasrealistic
≈
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Overcome:TimeConsuming
§ Preparewhatyoucanupfront- inthenight§ Reusewhatyoucan§ Shorttestscanrevealalotofthings§ Don’ttrytoputspikeandendurancetestsintoCI§ Parallelizetests,Jenkins2.0orTaurushelps
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Overcome:DebuggingandImproving
§ TaurusTool,thehandheldpieceofCI
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Overcome:DebuggingandImproving
LoadTestJob
LocalDebugging
23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Overcome:Reporting&ResultConsumption
§ Purpose-builtserviceswithintegrations§ Overview&StatusinJenkins
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Overcome:DecisionMaking
§ Deploytostagingisgreatwin§ Havepass/failcriteria,includingAPMinfo§ It’soktohavedeployasseparatejob(forUAT,toavoid
disruptions)
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
WhyUseJenkinsforTesting?
§ Multi-stepprocess§ Longprocess§ Repetitiveprocess
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
StartSimple
LoadTest1
LoadTest2
LoadTest3
Center of Excellence
Developer
Operations
QA
Maintain Jobs
28 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
BecomeContinuous
§ Periodicjobshavemanyapplications§ Havevalidfailurecriteria(thresholds)§ Usenon-blockingdownstreamjobs
29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
PursueContinuousDelivery
FastFeedbackDeveloper/
DevOps
Build+xUnit
FunctionalTests
LoadTests
Deploy Staging
SlowerFeedback
30 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
THINGSWEPUTINTOCI
BESTPRACTICE(FORNOW)
CHALLENGESOFTESTINGINCI
OVERCOMINGCHALLENGES
1
2
3
4
JENKINSFORPERFORMANCETESTING
SOMEUSEFULPLUGINS
5
6
34 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TaurusTool
§ Whynotplugin,butcommand-linetool§ UniversalplugfortestsinsideJenkins§ TaurusdesignedtoworkwithotherJenkinsplugins
gettaurus.org
35 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
FinalSummary
WehavetotestinsideCI
Therearewaystostartitsimple
Therearetools&pluginstohelp
Jenkins2.0pipelinesFTW!
37 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
RecommendedSessions
SESSION# TITLE DATE/TIME
DO5T57SCaseStudy:HowIMovedTheNFLfromWaterfalltoAgilein4Months 11/16/2016at03:45pm
DO5X55SCaseStudy:WalmartBrasil—SeasonalPeakTrafficReadiness 11/16/2016at04:30am
DO5X56SNext-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagement 11/17/2016at04:30pm
38 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
MustSeeDemos
ModernizeAppDeliveryIntegratedCDTheater5- DOV501P
DeliverTestDataFasterTestDataManagerTheater5- DOV511P
DeliverBetterAppsServiceVirtualizationTheater5- DOV507P
OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P