exploring time­aware test suite prioritization ?· example – after prioritization but,...

Download Exploring Time­Aware Test Suite Prioritization ?· Example – after prioritization But, retesting…

Post on 16-Nov-2018

213 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • MaryLouSoffaUniversityofVirginia

    Collaborators:KristenWalcott

    GregoryM.Kapfhammer,AlleghenyCollege

    ExploringTimeAwareTestSuitePrioritization

  • Regressiontesting

    Softwareconstantlymodified Bugfixes Additionoffunctionality

    Afterchanges,regressiontestingruntestcaseintestsuiteandprovidemore

    Providesconfidencemodificationscorrect Helpsfindnewerror

    Largenumberoftestcasescontinuestogrow Weeks/monthstorunentiretestsuite Costshighcostofmaintenance

  • Reducingcostregressiontesting

    Toreducecost,donotrunalltestcasesprioritizetestsi.e.,reorderthem

    TestPrioritizationTechniques Originalorder Basedonfaultdetectionability Analysistodeterminewhattestcasesaffectedbychange

    andorder Randomselectionordertestsrandomly Reverseruntestsinreverseorder

  • Exampleafterprioritization

    But,retestingusuallyhasatimebudgetbasedontime,wastheaboveorderthebestorder?

    Contribution:Atestprioritizationtechniquethatintelligentlyincorporatesthetesttimebudget

    Timebudget

    T1Time:3

    T2

    Time:10

    T3Time:9

    T4

    Time:12

    T5

    Time:3

    T6

    Time:5

    T7

    Time:3

  • FaultMatrixExample

    XXXT6XXT5

    XXXT4XXT3

    XT2XXXXXXXT1

    f8f7f6f5f4f3f2f1FAULTS/TESTCASE

    Givenmodifiedprogram,have6testcases

    Assumeaprioriknowledgeoffaults,f

  • TestSuiteFaultsandTime

    O.7543T60.7543T50.7543T40.66732T31.011T20.77897T1

    avgfaults/min

    Timecosts

    #faultsTestsvaryaccordingtothetimeoverheadandtheirabilitytorevealfaults

    GOAL:Whentesting,findasmanyfaultsassoonaspossible

  • FaultawarePrioritizationTimelimit12minutes

    OriginalOrder

    T1Time:9

    Faults:7

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    T2Time:1

    Faults:1

    T6Time:4

    Faults:3

    T6Time:4

    Faults:3

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    T2Time:1

    Faults:1

    T1Time:9

    Faults:7

    Faultbasedorder

    7faultsfoundin9minutes

  • NavetimeBasedprioritization

    OriginalOrder

    8faultsin12minutes

    T6Time:4

    Faults:3

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    T2Time:1

    Faults:1

    T1Time:9

    Faults:7

    Navetimebasedorder

    T6Time:4

    Faults:3

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    T2Time:1

    Faults:1

    T1Time:9

    Faults:7

  • AveragePercentFaultDetectionBasedPrioritization

    OriginalOrderT6

    Time:4

    APFD:8.8

    T5Time:4

    APFD:0.8

    T4Time:4

    APFD:0.8

    T3Time:3

    APFD:0.7

    T2Time:1

    APFD:1.0

    T1Time:9

    APFD:.8

    T6Time:4

    Faults:3

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    T2Time:1

    Faults:1

    T1Time:9

    Faults:7

    APFD

    7faultsin10minutes

  • IntelligentTimeawareprioritization

    Originalorder

    T6Time:4

    Faults:3

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    T2Time:1

    Faults:1

    T1Time:9

    Faults:7

    8faultsin11minutes

    T5Time:4

    Faults:3

    T4Time:4

    Faults:3

    T3Time:3

    Faults:4

    IntelligentTimeawareprioritization

    T6Time:4

    Faults:3

    T1Time:9

    Faults:7

    T2Time:1

    Faults:1

  • ComparingTestPrioritization

    Intelligentschemeperformsbetterfindingmostfaultsinshortesttime

    Considerstestingtimebudgetandoverlappingfaultdetectionoftest

    TimeawareprioritizationrequiresheuristicsolutiontoNPcomplete

    Usegeneticalgorithm Fitnessfunctionbasedoncodecoverageforabilitytofind

    faultsandtime

  • InfrastructureTestSuite

    CoverageCalculator

    NewTestsuite

    FitnessValueProducer Genetic

    Algorithm

    TestReorder

    ProgramUnderTest(P)

    TestTransformer

    Programcoverageweight

    Crossoverprobability

    Testadequacycriteria%oftestsuiteexecutiontime

    Addition/deletionproperties

    Maximum#iterations

    MutationprobabilityNumbertuplesperiteration

  • FitnessFunction

    Sincefaultinformationunknown,usemethodandblockcoveragetomeasuretestsuitepotential

    Coverageisaggregatedforentiretestsuite Testprioritizationfitnessmeasures

    ThepercentageofPscodethatiscoveredbyTi ThetimeatwhicheachtestcasecoverscodewithinP

    canusepercentagesofcodecoverage

  • Changetheorderoftestcases

    Developsmallertestsuitesbasedonoperatorsthatchange Order Testcasesincluded

    Fitnessevaluationdeterminesgoodnessofthechangedsuite.

  • CrossoverOperator

    Varytestprioritizationsbyrecombinationatarandomlychosencrossoverpoint

  • AdditionandDeletionOperators

    OperatorsEntiretestsuite

    SelectedTestsuite

    Addoperator

    Deleteoperator

  • MutationOperators

    Anotherwaytoaddvariationtocreatenewpopulation

    Testcasesaremutated replacedbyanunusedtestcase Swaptestcasesifnounusedtestcase

  • ExperimentGoalsandDesign

    DetermineiftheGAproducedprioritizations,onaverage,outperformaselectedsetofotherprioritizations

    Identifyoverheadtimeandspaceassociatedwiththecreationoftheprioritizedtestsuite

  • Experiments

    Blockormethodcoverage Order

    Initialorder Reverseorder Randomorder Faultawareprioritization

  • ExperimentalDesign

    GNU/LinuxWorkstation1.80GHzIntelPentiumand1GBofmainmemory

    UsedJUnittoprioritizetestcases Seededfaults:25%,50%,75%of40faults UsedEmmatocomputecoveragecriteria 2Casestudies

    Gradebook JDependtraversedirectoriesofJavaclass

    files

  • TestAdequacyMetrics

    Methodcoverage Consideredcoveredwhenentered

    Basicblockcoverage Asequenceofbytecodeinstructionswithoutanyjumpsor

    jumptargets Consideredcoveredwhenentered

    Howmuchofthecodehasbeenexecutedused100%

  • APFDResultsforBlockandMethodCoverage

    11%betterGradebook

    13%betterJDepend

  • PrioritizationEfficiency

    13.8hours

    8.3hours

    Time(s)

    Spacecostsinsignificant

  • Gradebook:IntelligentvsRandom

  • JDdepend:IntelligentvsRandom

  • Comparisonswithotherorders

    Experimentstocomparewithothertypesofprioritizations

    Original Reverse Faultaware(impossibletoimplement) Timeaware

  • APFDMetric

  • Gradebook:AlternativePrioritizations

    0.70.90.50.04300.750.70.90.40.1200.750.90.90.50.3100.750.70.80.30.3300.500.70.90.20.2200.500.70.90.10.04100.500.60.50.00.9300.250.40.70.20.9200.250.40.70.20.6100.25

    GAFaultaware

    ReverseInitialFiPi

  • Results

    Comparisonof Original Faultaware(impossibletoimplement) Reverse Gradebook

    120%betterthanoriginal Timeawarebetterthanoriginal

    JDepend Producedbetterresults

  • TechniqueEnhancements

    Makefitnesscalculationfaster Eliminatethemajorityofcoveragecoveroverlapby

    reducingthetestsuite Recordcoverageonapertestbasis

    Distributeexecutionoffitnessfunction Exploittestexecutionhistoriesandfavorteststhathave

    recentlyrevealedfaults Terminatethegeneticalgorithmwhenitachievesfitness

    equivalenttopreviousprioritizations

  • ConclusionsandFutureWork

    Contribution:atestprioritizationtechniquethatincludesthetestingtimebudget

    Timeawareprioritizationcanyielda120%improvementinAPFDwhencomparedtoalternativeprioritizations

    Differentheuristicsanalysis

  • Papertoappear

    InternationalSymposiumonSoftwareTestingandAnalysis(ISSTA)

    July,2006