test driving an organisation - anztb driving an organisation v2_0.pdf · test execution • need a...
TRANSCRIPT
![Page 1: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/1.jpg)
Test Driving an Organisation
Paul O’Keeffe
Shaun Loh
![Page 2: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/2.jpg)
We Have a Problem • Coreapplications• Businessideasslowtogettomarket
• Manualtestingtimesdominatedthereleasecycle
• Longtailtestcyclesdelayingrelease
• Defectdetectionlateintheprocess
• Somedefectsgettingtoproduction
• Multipleindependenttestteamssharinglittle
• Lackoftrustintestsduetoinconsistentcoverageandexecution
![Page 3: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/3.jpg)
Quality / Speed Delivery • Qualityinitiative• Morphedinto“SpeedDelivery”
• Cycletimeisperhapsthemostimportantmetricinsoftwaredevelopment
• Reducethetimebetweenchangerequestandproductiondelivery
![Page 4: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/4.jpg)
![Page 5: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/5.jpg)
GUI Test Automation • Webapplications
• Desktopapplication
• Tightcouplingbetweenapplications• Initiallybuildasafetynet
• Thenbuildoutapplicationcoverage
• BusinesslogicintheGUIswasrifesowecouldn’tsensiblytestattheAPIlevelorevenattheHTTPlevel
![Page 6: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/6.jpg)
What Test Pyramid? • Thetestpyramidminusthebaseandmiddle;◦ andthetop
• Webuildthetop,becausethebaseandmiddlecouldn’tbe
• Codewasnotdesignedtobetestable◦ Frameworks!
• Swisscheesepyramid
• Normally,buildingafewhappydaysscenariosatthetopofthepyramidistherightapproach,butwithoutagoodbaseormiddle,fragilityisworse
• Somanythingswouldbreaktests• Who/whattoblame?
![Page 7: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/7.jpg)
What Could Possibly Go Wrong? • Infrastructure• Environment
• Data• Applications
• Everything…
• StabilisingaGUItestautomationsuiteisincrediblyhardwork
![Page 8: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/8.jpg)
Infrastructure • Cloud(AWS)
• Butproductionstillonpremise,sodifferent
• Longlivedsnowflakes• Provisioningwasgraduallybeingautomated
• Bothfortestingandapplicationinfrastructure
![Page 9: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/9.jpg)
Controllable Environment • Naildownenvironmentonbuildagentsprecisely
• Spreadoutintonailingdowneverythingexternalwhichischangingand/orcouldcauseproblems• Hardwarespecs• Manualtesterssharingenvironments• Batchjobs• Softwareupdates
• Limitedduetoownershipbyotherteams• Separationofacceptancetestingfromdevelopmentteams
![Page 10: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/10.jpg)
Data • Don’trelyonacopyofproductiondata• Becauseyoucan’trelyonit
• Alwaysgeneratethedatayouneedforatest• Productiondataandbatchjobstakingtoolongtoruninunderspecifiedenvironments
![Page 11: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/11.jpg)
Test Isolation • Creatinganewmemberforeachtest
• Naturalisolation• Datawascompletelyscopedbymember
• Ensureseachtestiscompletelyisolatedfromanyother
![Page 12: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/12.jpg)
Applications (with Real Users) • Usersareapologistsforpoorapplicationbehavior• Robotsaren’t
• Realusersseemedtohavebeentrainedtohaveahightoleranceforslowapplicationresponses
• Synchronousbusinessprocessesbuiltontopofasynchronoussystems
• Usersjustrefreshuntilthingshavefinished• Realusersworkedaroundapplicationerrorsliketheevery10thindexproblem
• Doesn’tmaketestautomationanyeasier,though
![Page 13: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/13.jpg)
Application Driver Layer • Applicationmodels
• Pagemodels
• Growthisasyouneeditforthetestsyou’vewritten• Don’tputahardcaponittoosoon◦ Cucumber/Gherkin◦ Commercialtools◦ Otheropensourcetools(whoshallremainnameless)
• Domain,notUI
![Page 14: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/14.jpg)
Application Driver Layer • Mosttoolsputahardcaponapplicationmodelling,usuallyatthepagelevel
• Needmoreflexibility.Noteverythingisshapedlikeacucumber
• Renewalsmodelisagoodexample
• Coderetainstheflexibility,sinceithasthenecessarymechanismsforabstraction
• Askyourselfwhetheryourtoolcancompetewithcodeonthisfront?
![Page 15: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/15.jpg)
Page Models • Disappointinglackoffreetoolstodobasicpagemodellingascode
• LearnedalotabouthowtodogoodpagemodellinginJavaandC#
• SomeadvancesonthecurrentlyacceptedstateoftheartIMHO
• DesignofpagemodelDSLpreventedrun-timeerrorsandenabledautocompletion
• Projectcomingsoonwithsomeofourdesigns
• DSLsforAPIsandGUIs.Sometimesthesameonesowecouldswitchthetwoeasily
![Page 16: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/16.jpg)
Asynchronous Behaviour • Nofixeddelays(sleeps!)• Pollingloops
• Events
![Page 17: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/17.jpg)
Weapon of Mass Destruction
![Page 18: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/18.jpg)
Effect on Systems Under Test • Weaponofmassdestruction
• Unexpectedloads/greaterscale• Runningourtestscausedahigherloadthanproduction• Duetohaving30buildagentsallrunningtestsfasterthannormalusers
• Buttestenvironmentsaremuchscaleddownversionsofproductiontocutcosts
• Singleserverratherthanclusters(multicastproblem)
• Datasizes?• Insignificantcomparedtoproduction• Butgrowthovertimewithoutresetcausedissues
![Page 19: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/19.jpg)
Impact • Manualtestingisusuallyprettylowimpact
• Automatedtestingishighimpact
• ContinuousIntegrationisevenmoreimpact
• Existingsystemsmayonlybedesignedtohandlemanualtests
• Testingperformance/scaledemandsmaybehigherthanproduction!!!• Especiallyifyouwantfastdevelopmentcycles
• Forcerestoftheorganisationtochange/improve
![Page 20: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/20.jpg)
Test Execution • NeedatoolwiththerightgranularityforGUItestautomation.
• Runeachtestandcollectresultswouldbeaboutright.
• CIservertocoarse.Runalltestsinonebuild.• Testgridtoofine.RuneachUIactionremotelyfromacentralcontroller.
• BrokethingsupbyhavingabuildpertestonCIserver,butnotwellsupported.
![Page 21: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/21.jpg)
Continuous Integration Server • Asopposedtothepracticeofcontinuousintegration• Supportforlongrunningacceptancetests
• Scalabilityofconfiguration• Newgeneration:pipelineascode(oratleastbetterconfiguration)◦ GUIshamperthis
![Page 22: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/22.jpg)
MSHTA • TheUncannyValleyofwebapplications• MSHTAapplicationsareapaintotest
• Nearlyawebappbutnotquitesoyouneedspecialtools• Frameworkdidn’tallowapplicationtoberuninabrowserjustfortesting
• WouldbegoodtohaveafreetoolwhichcoulddothisaswellasSeleniumdoesforweb
• MSHTAhavesomedifferences• Oftenmoredialogwindowwizards• Lockeddownbrowser• Fewercontrols
• Testautomationforcesorganisationawayfromthisstyleofapplication
![Page 23: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/23.jpg)
Recording Evidence • Videorecording>• Sequenceofscreenshots>
• Finalscreenshot>• Logsonly>
• Nothing
• Someproblemsarehardtoreproducejustbyrunningthetestagain
• Observability
• Convincingevidenceforotherswithintheorganization• Otherbenefits
![Page 24: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/24.jpg)
Licensing • Permachinelicensingschemesarepainfulfordesktoptestdevelopment,butmoresoforagridofbuildagentsinthecloud
• Donotbuytoolswithlicenceswhichdiscouragethenumberofpotentialenvironments
• Sitelicencesarefine
![Page 25: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/25.jpg)
Results • Shortenedreleasecyclesbymanyweeks
• Increasedconfidencetoperformreleases,wheretheymightnototherwisehavebeenattempted
• Butotherchangesareslow
![Page 26: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/26.jpg)
New State
![Page 27: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/27.jpg)
Test Driving the Organisation • TDDawholeorganisation• Conway'sLaw,sortof• Canmakemovesandreorganisemoreeasilywiththesafetynet
• Removalofredundantdelivery/testingphases
• Smallenvironments
• Fastercycletimes• Eachchangemakesthisalmostmorepossible• Loweroverhead• Costsaving• Earliertesting• ShiftLeft
![Page 28: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/28.jpg)
Opportunity: Data Load • Builttoolsformanualtesterstosetupdataforexploratorytesting
• Alsoforothermanualtestswehadn’tyetautomated
• Removeddependenceonproductiondataandfindingappropriateexamples
• Wherepossible,usedAPIsfordataloadinourtestsetup
• Chickenandegg• Nolongeranyneedforproductiondata,sowechangedtheorganisationinawaywhichsolvedourownproblemandseveralothers
• LearningandDevelopment(trainingdata)
• PusheddevelopmentteamstoopenupAPIs
![Page 29: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/29.jpg)
Automate (Almost) Everything • AnalysistoolslikeDashyandHelpy.• Don’tjustautomateyourtests,automateyouranalysisandanythingelserepetitiveyouhavetodo.
![Page 30: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/30.jpg)
Test Execution • Runyourtestsallthetime?• Tradeoffs
• Atleastwhenanythingchanges,whichinourcasewasprettymuchallthetime
![Page 31: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/31.jpg)
Deployment and Other Automation • Startedpushingthingstowardsautomateddeployments
• Wewouldhelpoutwiththis,sincewehadautomationskills
• Softwaredevelopmentskillsofteamflowedoverintosmallproductdevelopment
• RobotProcessAutomation(RPA)
![Page 32: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/32.jpg)
Think Different • Developerviewoftestautomation
• Fresheyes
• Embracecode
• Improvedqualityoftestcode
• Bringtestinganddevelopmentclosertogether
![Page 33: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/33.jpg)
Way of Working • Agile,butonlywithinoursmallpartoftheSDLC
• Collaborative
• Pairingondemand◦ Moarpairing!!!◦ Particularlywhereincreaseskillsorshareknowledge,wheretheyareunevenlydistributedwithintheteam
• Honestretrospectives• Continuousimprovementactionsactuallyactedupon
• Attemptedtoinfluenceotherteams
• Asfaraswecouldgowithintheconstraintsoftheorganization
![Page 34: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/34.jpg)
Invest in your People • Super-interns• Andotherswithcomplementaryskillsfromelsewhereintheorganisation• Limitedbudget• Negotiatedtogetreallygoodinternsandthentrainthemwell
• Peopledon'tneedtobeanexactmatchforrequiredskills• Someprogrammingabilityiswelcome,butnoneedforlots• CancomefromotherITbackground• Goodtohavealternativeperspectives(e.g.support)
• Needtobekeentolearn• Cangetfantasticleverageatlowcost.Wedid.
![Page 35: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/35.jpg)
Invest in your People • Axesoflearning/growth• Maturesoftwaredevelopment• Automatedtesting• Enterprisetraining/constraints
• Highlycollaborativeenvironment/team• Workwithotherswithdevelopment/testing/domain/applicationexperience• Balanceofperhaps50%ofteamalreadyexperienced
![Page 36: Test Driving an Organisation - ANZTB Driving an Organisation v2_0.pdf · Test Execution • Need a tool with the right granularity for GUI test automation. • Run each test and collect](https://reader033.vdocuments.site/reader033/viewer/2022042912/5f46fb8a18fed80f0c3adf29/html5/thumbnails/36.jpg)
Invest in Your People • Freedomtofailandlearn• Psychologicalsafety• Givespeopleconfidencetotrynewthingsearlier
• Handsonimmediately
• Internsbecamethementors
• Growexistingpeoplefromtheorganisation• Graftthemontoanewplanttoreignitetheirmotivationandgrownewskills
• Chocolate