AdvancesInComputers,101

FIRSTEDITION

AtifMemonCollegePark,MD,USA

TableofContents

Coverimage

Titlepage

Copyright

Preface

ChapterOne:SecurityTesting:ASurveyAbstract1Introduction

2SoftwareTesting3SecurityEngineering

4SecurityTesting5SecurityTestingTechniques

6ApplicationofSecurityTestingTechniques7Summary

Acknowledgments

ChapterTwo:RecentAdvancesinModel-BasedTestingAbstract1Introduction

2MBTOverview3CurrentPenetrationofMBTinIndustry

4LanguagesforMBTModelingActivities5TechnologiesforModel-BasedAutomatedTestGeneration

6Model-BasedSecurityTesting7ConclusionandFutureChallenges

ChapterThree:OnTestingEmbeddedSoftwareAbstract

1Introduction2TestingEmbeddedSoftware

3CategorizationofTestingMethodologies4Black-BoxAbstraction

5Grey-BoxAbstraction6White-BoxAbstraction7FutureDirections

8ConclusionAcknowledgment

ChapterFour:AdvancesinWebApplicationTesting,2010–2014

Abstract1Introduction

2WebApplications3ChallengestoWebApplicationTesting

4WebApplicationTesting,2010–20145Conclusion

ChapterFive:ApproachesandToolsforAutomatedEnd-to-EndWebTestingAbstract

1Introduction2Capture-ReplayWebTesting

3ProgrammableWebTesting4TestCaseEvolution

5AnalysisoftheApproaches6OvercomingtheLimitationsoftheExistingApproaches

7Conclusions

AuthorIndex

SubjectIndex

ContentsofVolumesinthisSeries

Copyright

Preface

Prof.AtifM.MemonPh.D.,CollegePark,MD,USA

ThisvolumeofAdvancesinComputersisthe101stinthisseries.Thisseries,whichhasbeencontinuouslypublishedsince1960,presentsineachvolumefourtosevenchaptersdescribingnewdevelopmentsinsoftware,hardware,orusesofcomputers.

This101stvolumeisthesecondinaminiseriesofvolumesbasedonthetheme“AdvancesinSoftwareTesting.”TheneedforsuchathematicminiseriescameupwhenIwasteachingmygraduateclass“FundamentalsofSoftwareTesting,”inwhichstudentswereaskedtostudyandreportonrecent(years2010–15)advancesinvarioustopicssurroundingsoftwaretesting.Theyfailedtofindup-to-datesurveypapersonalmostalltopics.Inthisminiseries,Ihaveinvitedleadersintheirrespectivefieldsofsoftwaretestingtowriteaboutrecentadvances.Inthefirstvolumeintheminiseries(Volume99),wefocusedoncombinatorialtesting,constraint-basedtesting,automatedfaultlocalization,automaticblack-boxtesting,andtestingaccesscontrol.

Volume101focusesonfiveimportanttopics.InChapter1,entitled“SecurityTesting:ASurvey,”Feldereretal.provideanoverviewofrecentsecuritytestingtechniques.Theyfirstsummarizetherequiredbackgroundoftestingandsecurityengineering.Then,theydiscussthebasicsandrecentdevelopmentsofsecuritytestingtechniquesappliedduringsecuresoftwaredevelopment,ie,model-basedsecuritytesting,code-basedtestingandstaticanalysis,penetrationtestinganddynamicanalysis,aswellassecurityregressiontesting.Theyillustratesecuritytestingtechniquesbyadoptingthemforanexamplethree-tieredweb-basedbusinessapplication.

InChapter2,entitled“RecentAdvancesinModel-BasedTesting,”Uttingetal.provideanoverviewofthefieldofmodel-basedtesting(MBT),particularly,therecentadvancesinthelastdecade.TheygiveasummaryoftheMBTprocess,themodelinglanguagescurrentlyusedbyvariouscommunitieswhopracticeMBT,thetechnologiesusedtogeneratetestsfrommodels,andbestpractices,suchastraceabilitybetweenmodelsandtests.TheyalsobrieflydescribeseveralfindingsfromarecentsurveyofMBTusersinindustry,outlinetheincreasinglypopularuseofMBTforsecuritytesting,anddiscussfuturechallengesforMBT.

InChapter3,“OnTestingEmbeddedSoftware,”Banerjeeetal.describetheuniquechallengesassociatedwithtestingembeddedsoftware,whichisspecializedsoftwareintendedtorunonembeddeddevices.Asembeddeddeviceshaveexpandedtheirreachintomajoraspectsofhumanlives,fromsmallhandhelddevices(suchassmartphones)toadvancedautomotivesystems(suchasantilockbrakingsystems),thecomplexityofembeddedsoftwarehasalsogrown,creatingnewchallengesfortesting.Inparticular,embeddedsoftwarearerequiredtosatisfyseveralnonfunctionalconstraints,inadditiontofunctionality-relatedconstraints.Suchnonfunctionalconstraintsmayinclude(butnotlimitedto)timing/energyconsumption-relatedconstraintsorreliabilityrequirements.

Additionally,embeddedsystemsareoftenrequiredtooperateininteractionwiththephysicalenvironment,obtainingtheirinputsfromenvironmentalfactors(suchastemperatureorairpressure).Theneedtointeractwithadynamic,oftennondeterministicphysicalenvironment,furtherincreasesthechallengesassociatedwithtestingembeddedsoftware.Theauthorsdiscussadvancesinsoftwaretestingmethodologiesinthecontextofembeddedsoftware.Theyintroducekeychallengesintestingnonfunctionalpropertiesofsoftwarebymeansofrealisticexamples.Theyalsopresentaneasy-to-follow,classificationofexistingresearchworkonthistopic.

Theimportanceoftestautomationinwebengineeringcomesfromthewidespreaduseofwebapplicationsandtheassociateddemandforcodequality.Testautomationisconsideredcrucialfordeliveringthequalitylevelsexpectedbyusers,sinceitcansavealotoftimeintestingandithelpsdeveloperstoreleasewebapplicationswithfewerdefects.Themainadvantageoftestautomationcomesfromfast,unattendedexecutionofasetoftestsaftersomechangeshavebeenmadetoawebapplication.Moreover,modernwebapplicationsadoptamultitierarchitecturewheretheimplementationisscatteredacrossdifferentlayersandrunondifferentmachines.Forthisreason,end-to-endtestingtechniquesarerequiredtotesttheoverallbehaviorofwebapplications.Inthelastyears,severalapproacheshavebeenproposedforautomatedend-to-endwebtestingandthechoiceamongthemdependsonanumberoffactors,includingthetoolsusedforwebtestingandthecostsassociatedwiththeiradoption.InChapter4,“AdvancesinWebApplicationTesting,2010–14,”SampathandSprenkleprovidebackgroundonwebapplicationsandthechallengesintestingthesedistributed,dynamicapplicationsmadeupofheterogeneouscomponents.Theythenfocusontherecentadvancesinwebapplicationtestingthatwerepublishedbetween2010and2014,includingworkontest-casegeneration,oracles,testingevaluation,andregressiontesting.Throughthistargetedsurvey,theyidentifytrendsinwebapplicationtestingandopenproblemsthatstillneedtobeaddressed.InChapter5,entitled“ApproachesandToolsforAutomatedEnd-to-EndWebTesting,”Leottaetal.provideacomprehensiveoverviewofautomatedend-to-endwebtestingapproachesandsummarizethefindingsofalong-termresearchprojectaimedatempiricallyinvestigatingtheirstrengthsandweaknesses.

Ihopethatyoufindthesearticlesofinterest.Ifyouhaveanysuggestionsoftopicsforfuturechapters,orifyouwishtobeconsideredasanauthorforachapter,Icanbereachedatatif@cs.umd.edu.

CHAPTERONE

SecurityTesting

ASurvey

MichaelFelderer*;MatthiasBüchler†;MartinJohns‡;AchimD.Brucker‡;RuthBreu*;AlexanderPretschner†*UniversityofInnsbruck,Innsbruck,Austria†TechnischeUniversitätMünchen,Munich,Germany‡SAP,Karlsruhe,Germany

AbstractIdentifyingvulnerabilitiesandensuringsecurityfunctionalitybysecuritytestingisawidelyappliedmeasuretoevaluateandimprovethesecurityofsoftware.Duetotheopennessofmodernsoftware-basedsystems,applyingappropriatesecuritytestingtechniquesisofgrowingimportanceandessentialtoperformeffectiveandefficientsecuritytesting.Therefore,anoverviewofactualsecuritytestingtechniquesisofhighvaluebothforresearcherstoevaluateandrefinethetechniquesandforpractitionerstoapplyanddisseminatethem.Thischapterfulfillsthisneedandprovidesanoverviewofrecentsecuritytestingtechniques.Forthispurpose,itfirstsummarizetherequiredbackgroundoftestingandsecurityengineering.Then,basicsandrecentdevelopmentsofsecuritytestingtechniquesappliedduringthesecuresoftwaredevelopmentlifecycle,ie,model-basedsecuritytesting,code-basedtestingandstaticanalysis,penetrationtestinganddynamicanalysis,aswellassecurityregressiontestingarediscussed.Finally,thesecuritytestingtechniquesareillustratedbyadoptingthemforanexamplethree-tieredweb-basedbusinessapplication.

KeywordsSecuritytesting;Securitytestingtechniques;Model-basedsecuritytesting;White-boxsecuritytesting;Black-boxsecuritytesting;Penetrationtesting;Securityregressiontesting;Securityengineering;Softwaretesting;Survey

1IntroductionModernITsystemsbasedonconceptslikecloudcomputing,location-basedservices,orsocialnetworkingarepermanentlyconnectedtoothersystemsandhandlesensitivedata.Theseinterconnectedsystemsaresubjecttosecurityattacksthatmayresultinsecurityincidentswithhighseverityaffectingthetechnicalinfrastructureoritsenvironment.Exploitedsecurityvulnerabilitiescancausedrasticcosts,eg,duetodowntimesorthemodificationofdata.Ahighproportionofallsoftwaresecurityincidentsiscausedbyattackerswhoexploitknownvulnerabilities[1].Animportant,effective,andwidelyappliedmeasuretoimprovethesecurityofsoftwarearesecuritytestingtechniqueswhichidentifyvulnerabilitiesandensuresecurityfunctionality.

Softwaretestingisconcernedwithevaluationofsoftwareproductsandrelatedartifactstodeterminethattheysatisfyspecifiedrequirements,todemonstratethattheyarefitforpurposeandtodetectdefects.Securitytestingverifiesandvalidatessoftwaresystemrequirementsrelatedtosecuritypropertieslikeconfidentiality,integrity,availability,authentication,authorization,andnonrepudiation.Sometimessecuritypropertiescomeasclassicalfunctionalrequirements,eg,“useraccountsaredisabledafterthreeunsuccessfulloginattempts”whichapproximatesonepartofanauthorizationpropertyandisalignedwiththesoftwarequalitystandardISO/IEC9126[2]definingsecurityasfunctionalqualitycharacteristic.However,itseemsdesirablethatsecuritytestingdirectlytargetstheabovesecurityproperties,asopposedtotakingthedetouroffunctionaltestsofsecuritymechanisms.ThisviewissupportedbytheISO/IEC25010[3]standardthatrevisesISO/IEC9126andintroducesSecurityasanewqualitycharacteristicwhichisnotincludedinthecharacteristicfunctionalityanymore.

WebapplicationsecurityvulnerabilitiessuchasCross-SiteScriptingorSQLInjection,whichcanadequatelybeaddressedbysecuritytestingtechniques,areacknowledgedproblems[4]withthousandsofvulnerabilitiesreportedeachyear[5].Furthermore,surveysaspublishedbytheNationalInstituteofStandardsandTechnology[6]showhighcostofinsecuresoftwareduetoinadequatetestingevenonaneconomiclevel.Therefore,supportforsecuritytesting,whichisstilloftenconsideredasa“blackart,”isessentialtoincreaseitseffectivenessandefficiencyinpractice.Thischapterintendstocontributetothegrowingneedforinformationonsecuritytestingtechniquesbyprovidinganoverviewofactualsecuritytestingtechniques.Thisisofhighvaluebothforresearcherstoevaluateandrefineexistingtechniquesandpractitionerstoapplyanddisseminatethem.Inthischapter,securitytestingtechniquesareclassified(andalsothediscussionthereof)accordingtotheirtestbasiswithinthesecuresoftwaredevelopmentlifecycleintofourdifferenttypes:(1)model-basedsecuritytestingisgroundedonrequirementsanddesignmodelscreatedduringtheanalysisanddesignphase,(2)code-basedtestingandstaticanalysisonsourceandbytecodecreatedduringdevelopment,(3)penetrationtestinganddynamicanalysisonrunningsystems,eitherinatestorproductionenvironment,aswellas(4)securityregressiontestingperformedduringmaintenance.

Thischapterprovidesacomprehensivesurveyonsecuritytestingandisstructuredasfollows.Section2providesanoverviewoftheunderlyingconceptsonsoftwaretesting.

Section3discussesthebasicconceptsofsecurityengineeringandthesecuresoftwaredevelopmentlifecycle.Section4providesanoverviewofsecuritytestinganditsintegrationinthesecuresoftwaredevelopmentlifecycle.Section5discussesthesecuritytestingtechniquesmodel-basedsecuritytesting,code-basedtestingandstaticanalysis,penetrationtesting,anddynamicanalysisaswellassecurityregressiontestingindetail.Section6discussestheapplicationofsecuritytestingtechniquestothreetieredbusinessapplications.Finally,Section7summarizesthischapter.

2SoftwareTestingAccordingtotheclassicdefinitioninsoftwareengineering[7],softwaretestingconsistsofthedynamicverificationthataprogramprovidesexpectedbehaviorsonafinitesetoftestcases,asocalledtestsuite,suitablyselectedfromtheusuallyinfiniteexecutiondomain.Thisdynamicnotionoftesting,socalleddynamictesting,evaluatessoftwarebyobservingitsexecution[8].Theexecutedsystemiscalledsystemundertest(SUT).Moregeneralnotionsoftesting[9]consistofalllifecycleactivities,bothstaticanddynamic,concernedwithevaluationofsoftwareproductsandrelatedartifactstodeterminethattheysatisfyspecifiedrequirements,todemonstratethattheyarefitforpurposeandtodetectdefects.Thisdefinitionalsotakesstatictestingintoaccount,whichcheckssoftwaredevelopmentartifact(eg,requirements,design,orcode)withoutexecutionoftheseartifacts.Themostprominentstatictestingapproachesare(manual)reviewsand(automated)staticanalysis,whichareoftencombinedwithdynamictesting,especiallyinthecontextofsecurity.Forsecuritytesting,thegeneralnotionoftestingcomprisingstaticanddynamictestingisthereforefrequentlyapplied[10–12],andthusalsointhischaptertestingcomprisesstaticanddynamictesting.

Afterrunningatestcase,theobservedandintendedbehaviorsofaSUTarecomparedwitheachother,whichthenresultsinaverdict.Verdictscanbeeitherofpass(behaviorsconform),fail(behaviorsdonotconform),andinconclusive(notknownwhetherbehaviorsconform)[13].Atestoracleisamechanismfordeterminingtheverdict.Theobservedbehaviormaybecheckedagainstuserorcustomerneeds(commonlyreferredtoastestingforvalidation),againstaspecification(testingforverification),Afailureisanundesiredbehavior.Failuresaretypicallyobserved(byresultinginverdictfail)duringtheexecutionofthesystembeingtested.Afaultisthecauseofthefailure.Itisastaticdefectinthesoftware,usuallycausedbyhumanerrorinthespecification,design,orcodingprocess.Duringtesting,itistheexecutionoffaultsinthesoftwarethatcausesfailures.Differingfromactiveexecutionoftestcases,passivetestingonlymonitorsrunningsystemswithoutinteraction.

Testingcanbeclassifiedutilizingthethreedimensionsobjective,scope,andaccessibility[14,15]showninFig.1.

FIGURE1 Testingdimensionsobjective,scope,andaccessibility.

Testobjectivesarereasonorpurposefordesigningandexecutingatest.Thereasoniseithertocheckthefunctionalbehaviorofthesystemoritsnonfunctionalproperties.FunctionaltestingisconcernedwithassessingthefunctionalbehaviorofanSUT,whereasnonfunctionaltestingaimsatassessingnonfunctionalrequirementswithregardtoqualitycharacteristicslikesecurity,safety,reliabilityorperformance.

ThetestscopedescribesthegranularityoftheSUTandcanbeclassifiedintocomponent,integration,andsystemtesting.Italsodeterminesthetestbasis,ie,theartifactstoderivetestcases.Componenttesting(alsoreferredtoasunittesting)checksthesmallesttestablecomponent(eg,aclassinanobject-orientedimplementationorasingleelectroniccontrolunit)inisolation.Integrationtestingcombinescomponentswitheachotherandteststhoseasasubsystem,thatis,notyetacompletesystem.Systemtestingchecksthecompletesystem,includingallsubsystems.Aspecifictypeofsystemtestingisacceptancetestingwhereitischeckedwhetherasolutionworksfortheuserofasystem.RegressiontestingisaselectiveretestingtoverifythatmodificationshavenotcausedsideeffectsandthattheSUTstillcomplieswiththespecifiedrequirements[16].

Intermsofaccessibilityoftestdesignartifacts,wecanclassifyingtestingmethodsintowhite-andblack-boxtesting.Inwhite-boxtesting,testcasesarederivedbasedoninformationabouthowthesoftwarehasbeendesignedorcoded[7].Inblack-boxtesting,testcasesrelyonlyontheinput/outputbehaviorofthesoftware.Thisclassificationisespeciallyrelevantforsecuritytesting,asblack-boxtesting,wherenooronlybasicinformationaboutthesystemundertestisprovided,enablestomimicexternalattacksfromhackers.Inclassicalsoftwaretesting,arelatedclassificationoftestdesigntechniques[17]distinguishesbetweenstructure-basedtestingtechniques(ie,derivingtestcasesfrominternaldescriptionslikeimplementationcode),specification-basedtestingtechniques(ie,derivingtestcasesfromexternaldescriptionsofsoftwarelikespecifications),andexperience-basedtestingtechniques(ie,derivingtestcasesbasedonknowledge,skills,andbackgroundoftesters).

Theprocessoftestingcomprisesthecoreactivitiestestplanning,design,implementation,execution,andevaluation[9].AccordingtoRefs.[18]and[9],testplanningistheactivityofestablishingorupdatingatestplan.Atestplanincludesthetestobjectives,testscope,andtestmethodsaswellastheresources,andscheduleofintendedtestactivities.Itidentifies,amongstothers,featurestobetestedandexitcriteriadefiningconditionsforwhentostoptesting.Coveragecriteriaalignedwiththetestedfeaturetypesandtheappliedtestdesigntechniquesaretypicalexitcriteria.Oncethetestplanhasbeenestablished,testcontrolbegins.Itisanongoingactivityinwhichtheactualprogressiscomparedagainsttheplanwhichoftenresultsinconcretemeasures.Duringthetestdesignphasethegeneraltestingobjectivesdefinedinthetestplanaretransformedintotangibletestconditionsandabstracttestcases.Fortestderivation,specifictestdesigntechniquescanbeapplied,whichcanaccordingtoISO/IEC/IEEE29119[17]beclassifiedintospecification-based,structure-based,andexperience-basedtechniques.Testimplementationcomprisestaskstomaketheabstracttestcasesexecutable.Thisincludestaskslikepreparingtestharnessesandtestdata,providingloggingsupportorwritingtestscriptswhicharenecessarytoenabletheautomatedexecutionoftestcases.Inthetestexecutionphase,thetestcasesarethenexecutedandallrelevantdetailsoftheexecutionareloggedandmonitored.Inmanualtestexecution,testingisguidedbyahuman,andinautomatedtestingbyaspecializedapplication.Finally,inthetestevaluationphasetheexitcriteriaareevaluatedandtheloggedtestresultsaresummarizedinatestreport.

Inmodel-basedtesting(MBT),manuallyselectedalgorithmsautomaticallyandsystematicallygeneratetestcasesfromasetofmodelsofthesystemundertestoritsenvironment[19].Whereastestautomationreplacesmanualtestexecutionwithautomatedtestscripts,MBTreplacesmanualtestdesignswithautomatedtestdesignsandtestgeneration.

3SecurityEngineeringInthissection,wecoverbasicconceptsofsecurityengineeringaswellasanoverviewofthesecuresoftwaredevelopmentlifecycle.

3.1BasicConceptsSecuritytestingvalidatessoftwaresystemrequirementsrelatedtosecuritypropertiesofassetsthatincludeconfidentiality,integrity,availability,authentication,authorization,andnonrepudiation.Thesesecuritypropertiescanbedefinedasfollows[20]:

•Confidentialityistheassurancethatinformationisnotdisclosedtounauthorizedindividuals,processes,ordevices.

•Integrityisprovidedwhendataisunchangedfromitssourceandhasnotbeenaccidentallyormaliciouslymodified,altered,ordestroyed.

•Availabilityguaranteestimely,reliableaccesstodataandinformationservicesforauthorizedusers.

•Authenticationisasecuritymeasuredesignedtoestablishthevalidityofatransmission,message,ororiginator,orameansofverifyinganindividual’sauthorizationtoreceivespecificcategoriesofinformation.

•Authorizationprovidesaccessprivilegesgrantedtoauser,program,orprocess.

•Nonrepudiationistheassurancethatnoneofthepartnerstakingpartinatransactioncanlaterdenyofhavingparticipated.

Securityrequirementscanbeformulatedaspositiverequirements,explicitlydefiningtheexpectedsecurityfunctionalityofasecuritymechanism,orasnegativerequirements,specifyingwhattheapplicationshouldnotdo[10].Forinstance,forthesecuritypropertyauthorizationaspositiverequirementscouldbe“Useraccountsaredisabledafterthreeunsuccessfulloginattempts,”whereasanegativerequirementcouldbeformulatedas“Theapplicationshouldnotbecompromisedormisusedforunauthorizedfinancialtransactionsbyamalicioususer.”Thepositive,functionalviewonsecurityrequirementsisalignedwiththesoftwarequalitystandardISO/IEC9126[2]definingsecurityasfunctionalqualitycharacteristic.Thenegative,nonfunctionalviewissupportedbytheISO/IEC25010[3]standardthatrevisesISO/IEC9126andintroducesSecurityasanewqualitycharacteristicwhichisnotincludedinthecharacteristicFunctionalityanymore.

Anassetisadataitem,orasystemcomponentthathastobeprotected.Inthesecuritycontext,suchanassethasoneormultiplesecuritypropertiesassignedthathavetoholdforthatasset.

Afaultisatextualrepresentationofwhatgoeswronginabehavioraldescription.Itistheincorrectpartofabehavioraldescriptionthatneedstobereplacedtogetacorrectdescription.Sincefaultscanoccurindeadcode—codethatisneverexecuted—andbecausefaultscanbemaskedbyfurtherfaults,afaultdoesnotnecessarilyleadtoanerror.

Attheotherside,anerrorisalwaysproducedbyafault.Afaultisnotnecessarilyrelatedtosecuritypropertiesbutisthecauseoferrorsandfailuresingeneral.

Avulnerabilityisaspecialtypeoffault.Ifthefaultisrelatedtosecurityproperties,itiscalledavulnerability.Avulnerabilityisalwaysrelatedtooneormoreassetsandtheircorrespondingsecurityproperties.Anexploitationofavulnerabilityattacksanassetbyviolatingtheassociatedsecurityproperty.Sincevulnerabilitiesarealwaysassociatedwiththeprotectionofanasset,thesecurityrelevantfaultisusuallycorrelatedwithamechanismthatprotectstheasset.Avulnerabilityeithermeansthat(1)theresponsiblesecuritymechanismiscompletelymissing,or(2)thesecuritymechanismisinplacebutisimplementedinafaultyway.

Anexploitisaconcretemaliciousinputthatmakesuseofthevulnerabilityinthesystemundertest(SUT)andviolatesthepropertyofanasset.Vulnerabilitiescanoftenbeexploitedindifferentways.Oneconcreteexploitselectsaspecificassetandaspecificproperty,andmakesuseofthevulnerabilitytoviolatethepropertyfortheselectedasset.

Athreatisthepotentialcauseofanunwantedincidentthatharmsorreducesthevalueofanasset.Forinstance,athreatmaybeahacker,poweroutages,ormaliciousinsiders.Anattackisdefinedbythestepsamaliciousorinadvertentlyincorrectlybehavingentityperformstotheendofturningathreatintoanactualcorruptionofanasset’sproperties.Thisisusuallydonebyexploitingavulnerability.

Securityaspectscanbeconsideredonthenetwork,operatingsystem,andapplicationlevel.Eachlevelhasitsownsecuritythreatsandcorrespondingsecurityrequirementstodealwiththem.Typicalthreatsonthenetworklevelaredistributeddenial-of-serviceornetworkintrusion.Ontheoperatingsystemlevel,alltypesofmalwarecausethreats.Finally,ontheapplicationlevelthreatstypicalthreatsarerelatedtoaccesscontrolorareapplicationtypespecificlikeCross-SiteScriptingincaseofwebapplications.Alllevelsofsecuritycanbesubjecttotests.

Securitytestingsimulatesattacksandemploysotherkindsofpenetrationtestingattemptingtocompromisethesecurityofasystembyplayingtheroleofahackertryingtoattackthesystemandexploititsvulnerabilities[21].Securitytestingrequiresspecificexpertisewhichmakesitdifficultandhardtoautomate[22].Byidentifyingrisksinthesystemandcreatingtestsdrivenbythoserisks,securityvulnerabilitytestingcanfocusonpartsofasystemimplementationinwhichanattackislikelytosucceed.

Risksareoftenusedasaguidingfactortodefinesecuritytestprocesses.Forinstance,PotterandMcGraw[22]considertheprocessstepscreatingsecuritymisusecases,listingnormativesecurityrequirements,performingarchitecturalriskanalysis,buildingrisk-basedsecuritytestplans,wieldingstaticanalysistools,performingsecuritytests,performingpenetrationtestinginthefinalenvironment,andcleaningupaftersecuritybreaches.AlsotheOpenSourceSecurityTestingMethodologyManual(OSSTMM)[23]andtheOWASPTestingGuide[10]takerisksintoaccountfortheirproposedsecuritytestingactivities.

3.2SecureSoftwareDevelopmentLifeCycleTestingisoftenstartedverylateinthesoftwaredevelopmentlifecycleshortlybeforeitisdeployed.Ithasturnedoutthatthisisaveryineffectiveandinefficientpractice.Oneofthebestmethodstopreventsecuritybugsfromappearinginproductionapplicationsistoimprovethesoftwaredevelopmentlifecyclebyincludingsecurityineachofitsphases,therebyextendingittoasecuresoftwaredevelopmentlifecycle.Asoftwaredevelopmentlifecycleisaseriesofsteps,orphases,thatprovideamodelforthedevelopmentandlifecyclemanagementofanapplicationorpieceofsoftware.Itisastructureimposedonthedevelopmentofsoftwareartifacts.Agenericsoftwaredevelopmentlifecyclemodelconsideringtestingasanorthogonaldimensioncomprisesthephasesanalysis,design,development,deployment,andmaintenance[10].Eachphasedeliversspecificartifacts,ie,theanalysisphaseresultsinrequirements,designprovidesdesignmodels,developmentdeliverscode,deploymentresultsinarunningsystem,andfinallyallartifactsaremaintained.

Asecuresoftwaredevelopmentlifecycletakessecurityaspectsintoaccountineachphaseofsoftwaredevelopment.Acrucialconceptwithinthesecuresoftwaredevelopmentlifecycleisrisk.Ariskisthelikelihoodofanunwantedincidentanditsconsequenceforaspecificasset[24].Takingintoaccountthenegativenatureofmanysecurityrequirements,theconceptofriskcanbeemployedtodirecttheselectionorapplicationofsecuritycounter-measuresliketesting[22,24].Inallphasesofthesecuresoftwaredevelopmentprocess,butespeciallyatthedesignlevel[25],riskanalysesprovideeffectivemeanstoguidesecuritytestingandthusdetectfaultsandvulnerabilities.

MajorsecuritydevelopmentprocessesaretheSecurityDevelopmentLifecycle(SDL)[26]fromMicrosoftandtheOpenSoftwareAssuranceMaturityModel(OpenSAMM)[27]fromOWASP.

MicrosoftsSDLisanestablishedsecuritylifecycleforsoftwaredevelopmentprojectspursuingthefollowingmajorprinciples[26]:

•SecurebyDesign:Securityisabuilt-inqualityattributeaffectingthewholesoftwarelifecycle.

•SecuritybyDefault:Softwaresystemsareconstructedinawaythatpotentialharmcausedbyattackersisminimized,eg,softwareisdeployedwithleastnecessaryprivilege.

•SecureinDeployment:softwaredeploymentisaccompaniedbytoolsandguidancesupportingusersand/oradministrators.

•Communications:softwaredevelopersarepreparedforoccurringthreatscommunicatingopenlyandtimelywithusersand/oradministrators.

TheSDLiscomposedofsecuritypracticesattachedwiththemajoractivitiesofasoftwarelifecycle,ie,requirements,design,implementation,verification,anddeploymentincaseofSDL,whichareextendedbythetwoactivitiestrainingandresponse.Forinstance,thesecuritypractice“establishsecurityrequirements”isattachedtorequirements

analysis,“usethreatmodeling”todesign,“performstaticanalysis”toimplementation,“performfuzztesting”toverification,and“certifyreleaseandarchive”torelease.

SimilartotheSDL,OpenSAMMattachessecuritypracticestocoreactivities,ie,governance,construction,verification,anddeploymentincaseofOpenSAMM,withinthesoftwaredevelopmentlifecycle.Forinstance,verificationincludesthesecuritypracticesdesignreview,codereview,aswellas(dynamic)securitytesting.

Inparticular,OpenSAMMattacheseachsecuritypracticewiththreematuritylevelsandastartingpointofzero:

•Level0:Implicitstartingpointrepresentingtheactivitiesinthepracticebeingunfulfilled

•Level1:Initialunderstandingandad-hocprovisionofsecuritypractice

•Level2:Increaseefficiencyand/oreffectivenessofthesecuritypractice

•Level3:Comprehensivemasteryofthesecuritypracticeatscale

Foreachsecuritypracticeandmaturitylevel,OpenSAMMdoesnotonlydefinetheobjectivesandactivities,butalsogivessupporttoachievethisparticularlevel.Thiscomprisesassessmentquestions,successmetrics,costs,andpersonnelneededtoachievethetargetedmaturitylevel.

4SecurityTestingInthissection,wecoverbasicconceptsofsecuritytestingandtheintegrationofsecuritytestinginthesecuresoftwaredevelopmentlifecycle.

4.1BasicConceptsSecuritytestingistestingofsecurityrequirementsrelatedtosecuritypropertieslikeconfidentiality,integrity,availability,authentication,authorization,andnonrepudiation.

Securitytestingidentifieswhetherthespecifiedorintendedsecuritypropertiesare,foragivensetofassetsofinterests,correctlyimplemented.Thiscanbedonebytryingtoshowconformancewiththesecurityproperties,similartorequirements-basedtesting;orbytryingtoaddressknownvulnerabilities,whichissimilartotraditionalfault-based,ordestructive,testing.Intuitively,conformancetestingconsiderswelldefined,expectedinputs.Ittestsifthesystemsatisfiesthesecuritypropertieswithrespecttothesewell-definedexpectedinputs.Incontrast,addressingknownvulnerabilitiesmeansusingmalicious,nonexpectedinputdatathatislikelytoexploittheconsideredvulnerabilities.

Asmentionedintheprevioussection,securityrequirementscanbepositiveandfunctional,explicitlydefiningtheexpectedsecurityfunctionalityofasecuritymechanism,ornegativeandnonfunctional,specifyingwhattheapplicationshouldnotdo.Thisclassificationofsecurityrequirementsalsoimpactssecuritytesting.Forpositivesecurityrequirements,classicaltestingtechniquescanbeapplied,whereasfornegativesecurityrequirements(acombinationof)additionalmeasureslikeriskanalyses,penetrationtesting,orvulnerabilityknowledgebasesareessential.ThisclassificationisalsoreflectedinbyclassificationintheliteratureasprovidedbyTian-yangetal.[11]aswellasbyPotterandMcGraw[28].

AccordingtoTian-yangetal.[11]twoprincipalapproachescanbedistinguished,ie,securityfunctionaltestingandsecurityvulnerabilitytesting.Securityfunctionaltestingvalidateswhetherthespecifiedsecurityrequirementsareimplementedcorrectly,bothintermsofsecuritypropertiesandsecuritymechanisms.Securityvulnerabilitytestingaddressestheidentificationofunintendedsystemvulnerabilities.Securityvulnerabilitytestingusesthesimulationofattacksandotherkindsofpenetrationtestingattemptingtocompromisethesecurityofasystembyplayingtheroleofahackertryingtoattackthesystemandexploititsvulnerabilities[21].Securityvulnerabilitytestingrequiresspecificexpertisewhichmakesitdifficultandhardtoautomate[28].Byidentifyingrisksinthesystemandcreatingtestsdrivenbythoserisks,securityvulnerabilitytestingcanfocusonpartsofasystemimplementationinwhichanattackislikelytosucceed.

PotterandMcGraw[28]distinguishbetweentestingsecuritymechanismstoensurethattheirfunctionalityisproperlyimplemented,andperformingrisk-basedsecuritytestingmotivatedbyunderstandingandsimulatingtheattacker’sapproach.Testingsecuritymechanismscanbeperformedbystandardtestorganizationswithclassicalfunctionaltesttechniques,whereasrisk-basedsecuritytestingrequiresspecificexpertiseand

sophisticatedanalysis[28].

Forsecurityvulnerabilitytestingapproaches,ShahriarandZulkernine[29]proposesevencomparisoncriteria,ie,vulnerabilitycoverage,sourceoftestcases,testcasegenerationmethod,testinglevel,testcasegranularity,toolautomationaswellastargetapplications.Toolautomationisfurtherrefinedintothecriteriatestcasegeneration,oraclegeneration,andtestcaseexecution.Theauthorsclassify20informallycollectedapproachesaccordingtothesecriteria.Themainaimofthecriteriaissupportforsecuritypractitionerstoselectanappropriateapproachfortheirneeds.ThereforeShahriarandZulkernineblendabstractcriterialikesourceoftestcasesortestcasegenerationmethodwithtechnologicalcriterialiketoolautomationortargetapplications.

Differingfromclassicalsoftwaretesting,whereblack-andwhite-boxtestdesignarenowadaysconsideredverysimilar,ie,inbothcasestestingproceedsfromabstractmodels[8],thedistinctionisessentialforsecuritytesting.White-boxtestingperformstestingbasedoninformationabouthowthesoftwarehasbeendesignedorcoded,andthusenablestestingfromaninternalsoftwareproducerpointofview[7].Black-boxtestingreliesonlyontheinput/outputbehaviorofthesoftware,andthusenablestomimicexternalattacksfromhackers.Theclassificationintowhite-andblack-boxtestingisalsopointedoutbyBachmannandBrucker[12],whoadditionallyclassifysecuritytestingtechniquesduetoexecution(dynamicvsstatictesting)andautomation(manualvsautomatedtesting).

Inaddition,duetothecriticalroleofnegativesecurityrequirements,classicaltestingwhichfocusesontestingfunctionalrequirementsandsecuritytestingdiffer.Itseemsdesirablethatsecuritytestingdirectlytargetssecurityproperties,asopposedtotakingthedetouroffunctionaltestsofsecuritymechanisms.Astheformerkindof(nonfunctional)securitypropertiesdescribeallexecutionsofasystem,testingthemisintrinsicallyhard.Becausetestingcannotshowtheabsenceoffaults,animmediatelyusefulperspectivedirectlyconsiderstheviolationoftheseproperties.Thishasresultedinthedevelopmentofspecifictestingtechniqueslikepenetrationtestingthatsimulatesattackstoexploitvulnerabilities.Penetrationtestsaredifficulttocraftbecausetestsoftendonotdirectlycauseobservablesecurityexploits,andbecausethetestersmustthinklikeanattacker[28],whichrequiresspecificexpertise.Duringpenetrationtesting,testersbuildamentalmodelofsecurityproperties,securitymechanisms,andpossibleattacksagainstthesystemanditsenvironment.Specifyingsecuritytestmodelsinanexplicitandprocessableway,resultsinamodel-basedsecuritytestingapproach.Insuchanapproach,securitytestmodelsprovideguidanceforthesystematicspecificationanddocumentationofsecuritytestobjectivesandsecuritytestcases,aswellasfortheirautomatedgenerationandevaluation.

(Functional)testingnormallyfocusesonthepresenceofsomecorrectbehaviorbutnottheabsenceofadditionalbehavior,whichisimplicitlyspecifiedbynegativerequirements.Testingroutinelymisseshiddenactionandtheresultisdangeroussideeffectbehaviorsthatshipwithasoftware.Fig.2illustratesthissideeffectnatureofmostsoftwarevulnerabilitiesthatsecuritytestinghastocopewith[30].

FIGURE2 Mostfaultsinsecuritymechanismsarerelatedtomissingorincorrectfunctionality,mostvulnerabilitiesarerelatedtounintendedside-effectbehavior(adaptedfromThompson[30]).

Thecirclerepresentsanapplication’sintendedfunctionalityincludingsecuritymechanisms,whichisusuallydefinedbytherequirementsspecification.Theamorphousshapesuperimposedonthecirclerepresentstheapplication’sactual,implementedfunctionality.Inanidealsystem,thecodedapplicationwouldcompletelyoverlapwithitsspecification,butinpractice,thisishardlyeverthecase.Theareasofthecirclethatthecodedapplicationdoesnotcoverrepresentstypicalfunctionalfaults(ie,behaviorthatwasimplementedincorrectlyanddoesnotconformtothespecification),especiallyalsoinsecuritymechanisms.Areasthatfalloutsideofthecircularregionrepresentunindentedandpotentiallydangerousfunctionality,wheremostsecurityvulnerabilitieslay.ThemismatchbetweenspecificationandimplementationshowninFig.2leadingtofaultsinsecuritymechanismsandvulnerabilitiescanbereducedbytakingsecurityandespeciallysecuritytestingaspectsintoaccountearlyandinallphasesofthesoftwaredevelopmentlifecycleasdiscussedinSection4.2.

4.2SecurityTestingintheSecureSoftwareDevelopmentLifecycleAsmentionedbefore,testingwithinthesecuritylifecycleplaystheroletovalidateandverifysecurityrequirements.Duetothenegativenatureofmanysecurityrequirementsandtheresultingbroadrangeofsubordinaterequirements,alsotestingactivitiescoverabroadrangeofscopesandemployedmethods.Inkeepingwithresearchandexperience,itisessentialtotaketestingintoaccountinallphasesofthesecuresoftwaredevelopmentlifecycle,ie,analysis,design,development,deployment,aswellasmaintenance.Thus,

securitytestingmustbeholisticcoveringthewholesecuresoftwaredevelopmentlifecycle[12].Inconcreteterms,Fig.3showsarecommendeddistributionofstaticanddynamictestingeffortsamongthephasesofthesecuresoftwaredevelopmentlifecycleaccordingtoRef.[10].Itshowsthatsecuritytestingshouldbebalancedoverallphases,withafocusontheearlyphases,ie,analysis,design,andimplementation.

FIGURE3 ProportionoftesteffortinsecuresoftwaredevelopmentlifecycleaccordingtoRef.[10].

Toprovidesupportfortheintegrationofsecuritytestingintoallphasesofthesecuresoftwaredevelopmentprocess,majorsecuritydevelopmentprocesses(seeSection3.2),considertheintegrationoftesting.IntheSecurityDevelopmentLifecycle(SDL)[26]fromMicrosoftpracticeswithstronginterferencewithtestingeffortsarethefollowing:

•SDLPractice#2(Requirements):EstablishSecurityandPrivacyRequirements

•SDLPractice#4(Requirements):PerformSecurityandPrivacyRiskAssessments

•SDLPractice#5(Design):EstablishDesignRequirements

•SDLPractice#7(Design):UseThreatModeling

•SDLPractice#10(Implementation):PerformStaticAnalysis

•SDLPractice#11(Verification):PerformDynamicAnalysis

•SDLPractice#12(Verification):PerformFuzzTesting

•SDLPractice#13:ConductAttackSurfaceReview

•SDLPractice#15:ConductFinalSecurityReview

InOpenSAMM[27]fromOWASP,theverificationactivityincludesthesecuritypracticesdesignreview,codereview,aswellas(dynamic)securitytesting.

TheOWASPTestingGuide[10]andtheOWASPCodeReviewGuide[31]providea

detailedoverviewofthevarietyoftestingactivitiesofwebapplicationsecurity.WhiletheTestingGuidehasafocusonblack-boxtesting,theCodeReviewGuideisawhite-boxapproachfocusingonmanualcodereview.Overall,theTestingGuidedistinguishes91differenttestingactivitiessplitinto11subcategories(ie,informationgathering,configurationanddeploymentmanagementtesting,identitymanagementtesting,authenticationtesting,authorizationtesting,sessionmanagementtesting,datavalidationtesting,errorhandling,cryptography,businesslogictesting,aswellasclientsidetesting).ApplyingsecuritytestingtechniquestowebapplicationsiscoveredinSection6.

TheOWASPtestingframeworkworkflow,whichisalsocontainedintheOWASPTestingGuide,containschecksandreviewsofrespectiveartifactsinallsecuresoftwaredevelopmentphases,creationofUMLandthreatmodelsintheanalysisanddesignphases,unitandsystemtestingduringdevelopmentanddeployment,penetrationtestingduringdeployment,aswellasregressiontestingduringmaintenance.Propersecuritytestingrequiresamixoftechniquesasthereisnosingletestingtechniquethatcanbeperformedtoeffectivelycoverallsecuritytestingandtheirapplicationwithintestingactivitiesatunit,integration,andsystemlevel.Nevertheless,manycompaniesadoptonlyonesecuritytestingapproach,forinstancepenetrationtesting[10].

Fig.4abstractsfromconcretesecuritytestingtechniquesmentionedbefore,andclassifiesthemaccordingtotheirtestbasiswithinthesecuresoftwaredevelopmentlifecycle.

FIGURE4 Securitytestingtechniquesinthesecuresoftwaredevelopmentlifecycle.

Model-basedsecuritytestingisgroundedonrequirementsanddesignmodelscreatedduringtheanalysisanddesignphase.Code-basedtestingandstaticanalysisisbasedonsourceandbytecodecreatedduringdevelopment.Penetrationtestinganddynamicanalysisisbasedonrunningsystems,eitherinatestorproductionenvironment.Finally,securityregressiontestingisperformedduringmaintenance.Wealsoapplythisclassificationtostructurethediscussionofsecuritytestingtechniquesinthefollowingsection.

5SecurityTestingTechniquesThissectiondiscussesthesecuritytestingtechniquesmodel-basedtesting,code-basedtestingandstaticanalysis,penetrationtestinganddynamicanalysisaswellasregressiontestingindetail.Foreachtestingtechnique,basicconceptsaswellascurrentapproachesarecovered.

5.1Model-BasedSecurityTestingInmodel-basedtesting(MBT)manuallyselectedalgorithmsautomaticallyandsystematicallygeneratetestcasesfromasetofmodelsofthesystemundertestoritsenvironment[19].MBTisanactiveareaofresearch[32,33]andoffersbigpotentialtoimprovetestprocessesinindustry[14,19,34].Itsprospectivebenefitsincludeearlyandexplicitspecificationandreviewofsystembehavior,bettertestcasedocumentation,theabilitytoautomaticallygenerateuseful(regression)testsandcontroltestcoverage,improvedmaintenanceoftestcasesaswellasshorterschedulesandlowercosts[19].

Process.TheprocessofMBTconsistsofthreemainstepsintegratedintotheoverallprocessoftestdesign,execution,andevaluation.(1)AmodeloftheSUTand/oritsenvironmentisbuiltfrominformalrequirements,existingspecificationdocuments,oraSUT.TheresultingmodeloftheSUTdedicatedtotestgenerationisoftencalledtestmodel.(2)Iftheyareexecutable,oneexecutiontraceofsuchamodelactsasatestcase:inputandexpectedoutputforanSUT.Becausethereareusuallyinfinitelymanyandinfinitelylongexecutiontraces,modelscanthereforebeusedtogenerateaninfinitenumberoftests.Tocutdowntheirnumberandlength,testselectioncriteriaareapplied.Theseguidethegenerationoftests.(3)Oncethetestmodelandthetestselectioncriteriaaredefined,asetoftestcasesisgeneratedfromthetestmodelasdeterminedbythechosentestselectioncriteria.Testgenerationistypicallyperformedautomatically.ThegeneratedtestcasesaretracesofthemodelandthusingeneralatahigherlevelthantheeventsoractionsofanSUT.Therefore,thegeneratedtestcasesarefurtherrefinedtoamoreconcreteleveloradaptedtotheSUTtosupporttheirautomatedexecution.

Model-basedsecuritytesting.Model-basedsecuritytesting(MBST)isanMBTapproachthatvalidatessoftwaresystemrequirementsrelatedtosecurityproperties.Itcombinessecuritypropertieslikeconfidentiality,integrity,availability,authentication,authorization,andnonrepudiationwithamodeloftheSUTandidentifieswhetherthespecifiedorintendedsecurityfeaturesholdinthemodel.

BothMBTandMBSThaveincommon,thattheinputartifactisamodelandnottheSUT.ThereforetheabstractiongapbetweenthemodelandtheSUThastobeaddressed.Inparticular,anidentified(security)issueatthemodelleveldoesnotautomaticallyconfirman(security)issueattheSUT.ThereforeanadditionalstepisneededtomapanabstracttestcasetoanexecutabletestcasethatcanbeexecutedontheSUT.

Selectioncriteria:“Good”testcases.Arguably,“good”testcasesdetectpotential,ratherthanactual,defectswithgoodcosteffectiveness[35].Potentialdefectsneedtobe

describedbydefecthypotheses.Inordertoturnthesehypothesesintooperationaladequacycriteria[36],theyneedtobecapturedbysomeformofexplicitdefectmodel[35,37,38].Oneformofdefectisafault,understoodastherootcauseofanincorrectsystemstate(error)orincorrectsystemoutput(failure).Asweshowbelow,vulnerabilitiescanbeunderstoodasfaults.

Inadditiontoexplicitmodelsof(thefunctionalityof)thesystemundertest,model-basedsecuritytestingusuallymakesuseofoneormoreofthethreefollowingmodelsfortestselection:properties,vulnerabilities,andattackers.Modelsofanattackerencodeanattacker’sbehavior:thedatatheyneed,thedifferentstepstheytake,thewaytheycraftexploits.Attackermodelscanbeseenasmodelsoftheenvironmentofasystemundertest,andknowledgeaboutatargetedvulnerabilityusuallyisleftimplicit.Incontrast,modelsofvulnerabilitiesexplicitlyencodeweaknessesinasystemoramodelofthesystem.Inthissense,theycanbeseenasfaultsthatareusedforthegenerationof“good”testcasegeneration(seeabove).Finally,propertiesdescribedesiredcharacteristicsofamodel,oranimplementation,andtheyincludeconfidentiality,availability,integrity,andsoon.Modelsofpropertiesareneededtodescribethepropertiesofanassetthataresupposednottobeviolated:theydescribewhatexactlythesecuritytestertargets,andwhatanexploitissupposedtoexploit.

Itisnoteworthythatallformsofsecuritytesting,model-basedornot,alwaysworkwithanimplicitorexplicithypothesisaboutapotentialvulnerability.

VulnerabilitiesasFaults.Frequently,asareactiontoknownrelevantthreats,assetsareprotectedbyexplicitsecuritymechanisms.Mechanismsincludeinputsanitization,AddressSpaceLayoutRandomization(ASLR),encryptionofpasswordfiles,butalsointrusiondetectionsystemsandaccesscontrolcomponents.Mechanismsarecomponentsofasystemandcanalwaysbesyntacticallyrecognized:thereisapieceofcode(themechanism)thatissupposedtoprotecttheasset;orthereisnosuchpieceofcode.Avulnerabilityisaspecialkindoffaultwithsecurityimplications.Itisdefinedastheabsenceofacorrectlyfunctioningmechanism.Thiscanmeanboth(1)thatthereisnomechanismatall(eg,noinputsanitizationtakesplacewhichcanleadtobufferoverflowsorSQLInjections)and(2)thatthemechanismdoesnotworkcorrectly,ie,ispartiallyorincorrectlyimplemented,forinstance,ifanaccesscontrolpolicyisfaulty.

Securitytestingcanthenbeunderstoodinthreeseeminglydifferentways:(1)totestifspecificsecuritypropertiesofanassetcanbeviolated(propertiesandpropertymodels);(2)totestthefunctionalityofamechanism(attackermodels);and(3)todirectlytrytoexploitavulnerability(vulnerabilitymodels).Theboundariesareblurred,however:Withtheabovedefinitionofvulnerabilitiesastheabsenceofeffectivelyworkingdefensemechanisms,andtheobservationthatattackermodelsalwaysinvolveimplicitorexplicithypothesesonvulnerabilities,activities(2)and(3)areclosetoidentical.Inpractice,theyonlydifferintermsoftheperspectivethatthetestertakes:themechanismorthevulnerability.Becausetheabovedefinitionalsobindsvulnerabilitiesto—possiblyunspecified—assets,thegoalofactivities(2)and(3)alwaysisactivity(1).Ithenceseemshardtoprovideacrispconceptualdistinctionbetweenthethreeactivitiesof(1)testing

securityproperties,(2)testingsecuritymechanisms,and(3)testingforvulnerabilities.

Classificationofmodel-based(security)testing.SeveralpublicationshavebeenpublishedthatproposetaxonomiesandclassificationsofexistingMBT[32,33]andMBSTapproaches[39,40].WewillfocusontheclassificationproposedbySchieferdeckeretal.[40]consideringdifferentperspectivesusedinsecuringasystem.TheauthorsclaimthatMBSTneedstobebasedondifferenttypesofmodelsanddistinguishthreetypesofinputmodelsforsecuritytestgeneration,ie,architecturalandfunctionalmodels,threat,faultandriskmodels,aswellasweaknessandvulnerabilitymodels.ArchitecturalandfunctionalmodelsoftheSUTareconcernedwithsecurityrequirementsandtheirimplementation.Theyfocusontheexpectedsystembehavior.Threat,faultandriskmodelsfocusonwhatcangowrong,andconcentrateoncausesandconsequencesofsystemfailures,weaknessesorvulnerabilities.Weaknessandvulnerabilitymodelsdescribeweaknessesorvulnerabilitiesbythemselves.

Inthefollowing,weexemplarydescribeselectedapproaches,thatmakeuseofdifferentmodelsaccordingtotheclassificationofSchieferdeckeretal.

5.1.1AModel-BasedSecurityTestingApproachforWebApplicationsAnapproachthatmakesuseoffunctional,fault,andvulnerabilitymodelsaccordingtoSchieferdeckeretal.ispresentedbyBüchleretal.[41].Theypublishedasemiautomaticsecuritytestingapproachforwebapplicationsfromasecuremodel.Theauthorsassumethereisaformalmodel forthespecificationoftheSystemunderTest(SUT).Thismodelissecureasitdoesnotviolateanyofthespecifiedsecuritygoals(eg,confidentialityandauthenticity).Thus,amodel-checkerwillreport forallsecuritypropertiesφdefiningthesecuritygoalsofthemodel.Themodelisbuiltusingabstractmessagesthataredefinedbythemodeler.Thesemessagesrepresentcommonactionsauserofthewebapplicationcanperform.Theideaisthattheseabstractmessagesaresenttotheservertotellitwhichactionstheclientwantstoperform,eg,logintothewebapplication,viewprofilesofdifferentusers,deleteprofiles,updateprofiles,andsoon.Thus,themodelerdoesnotcareaboutdetailsatthebrowser/protocollevelbutonlyaboutabstractmessagesthatrepresentwebapplicationactions.

Tomakeuseofsuchasecuremodel,Büchleretal.[41]definesemanticmutationoperatorsthatrepresentcommon,well-knownvulnerabilitiesatsourcecodelevel.Semanticmutationoperatorsareanabstractionthatthesevulnerabilitiessothattheycanbeinjectedintothemodel.Afterhavingappliedamutationoperatortoanoriginalmodel,themodelcheckermayprovideatracefromthismutatedmodelthatviolatesasecurityproperty.Thistraceiscalledanattacktracebecauseitshowswhichsequenceofabstractmessageshavetobeexchangedinordertoleadthesystemtoastatewherethesecuritypropertyisviolated.Sinceabstractattacktracesareatthesamelevelofabstractionastheinputmodel,theyneedtobeinstantiatedtoturnthemoperational.Theapproachproposesamultistepinstantiationsincewebapplicationsareusuallyaccessedviaabrowser.Inthefirststep,abstractmessagesaretranslatedintoabstractbrowseractions.Thesecondstepis

amappingfromthesebrowseractionstoexecutableAPIcallstomakethemoperationalinabrowser.Finally,atestexecutionengineexecutestheoperationalizedtestcasesontheSUTtoverify,iftheimplementationofthemodelsuffersfromthesamevulnerabilityasreportedbythemodelcheckerattheabstractlevel.

5.1.2AModel-BasedFrameworkforSecurityPolicySpecification,Deployment,andTestingMouelhietal.[42]proposeanapproachbasedonarchitectural,functional,andfaultmodelsandfocusonsecuritypolicies.Theyproposeamodel-basedapproachforthespecification,deployment,andtestingofsecuritypoliciesinJavaapplications.Theapproachstartswithagenericsecuritymeta-modeloftheapplication.Itcapturesthehighlevelaccesscontrolpolicyimplementedbytheapplicationandisexpressedinadedicateddomain-specificlanguage.Beforesuchamodelisfurtherused,themodelisverifiedtocheckthesoundnessandadequacyofthemodelwithrespecttotherequirements.Afterwardsthemodelisautomaticallytransformedtopolicydecisionpoints(PDP).SincesuchPDPsareusuallynotgeneratedfromscratchbutarebasedonexistingframeworks,theoutputofthetransformationis,forinstance,anXACML(ExtendedAccessControlMarkupLanguage)filethatcapturesthesecuritypolicy.ThistransformationstepisessentialinMBTsinceanidentifiedsecurityissueatmodelleveldoesnotautomaticallyimplythesameissueatimplementationlevel,nordoesamodelwithoutsecurityissuesautomaticallyimplythesameontheimplementation.Mouelhietal.makeuseofmutationsatthemodelleveltoensurethattheimplementationconformstotheinitialsecuritymodel.Anexistingtestsuiteisexecutedonanimplementationgeneratedfromamutatedsecuritymodel.Ifsuchmutantsarenotdetectedbytheexistingtestsuite,itwillbeadaptedtocoverthemutatedpartofthesecuritymodelaswell.Finallythetestobjectiveistocheckthattheimplementation(securitypolicy)issynchronizedwiththesecuritymodel.

5.1.3Risk-BasedSecurityTestingInthefollowing,weconsiderapproachesthatarebasedonriskmodels.Risk-basedtestingingeneralisatypeofsoftwaretestingthatexplicitlyconsidersrisksofthesoftwaresystemastheguidingfactortosolvedecisionproblemsinallphasesofthetestprocess,ie,testplanning,design,implementation,execution,andevaluation[1,43,44].Itisbasedontheintuitiveideatofocustestingactivitiesonthoseareasthattriggerthemostcriticalsituationsforasoftwaresystem[45].Thepreciseunderstandingofrisksaswellastheirfocusedtreatmentbyrisk-basedtestinghasbecomeoneofthecornerstonesforcriticaldecisionswithincomplexsoftwaredevelopmentprojectsandrecentlygainedmuchattention[44].Lately,theinternationalstandardISO/IEC/IEEE29119SoftwareTesting[17]ontestingtechniques,processes,anddocumentationevenexplicitlyconsidersrisksasanintegralpartofthetestplanningprocess.Inthefollowing,wedescribethreerisk-basedapproachestosecuritytestinginmoredetail.

Grossmannetal.[46]presentanapproachcalledRisk-BasedSecurityTestingthatcombinesriskanalysisandrisk-basedtestdesignactivitiesbasedonformalizedsecurity

testpatterns.TheinvolvedsecuritytestpatternsareformalizedbyusingaminimaltestdesignstrategieslanguageframeworkwhichisrepresentedasaUMLprofile.Sucha(semi-)formalsecuritytestpatternisthenusedastheinputforatestgeneratoraccompaniedbythetestdesignmodeloutofwhichthetestcasesaregenerated.TheapproachisbasedontheCORASmethod[24]forriskanalysisactivities.Finally,atoolprototypeispresentedwhichshowshowtocombinetheCORAS-basedriskanalysiswithpattern-basedtestgeneration.

Botellaetal.[47]describeanapproachtosecuritytestingcalledRisk-BasedVulnerabilityTesting,whichisguidedbyriskassessmentandcoveragetoperformandautomatevulnerabilitytestingforwebapplications.Risk-BasedVulnerabilitytestingadaptsmodel-basedtestingtechniquesusingapattern-basedapproachforthegenerationoftestcasesaccordingtopreviouslyidentifiedrisks.Forriskidentificationandanalysis,theCORASmethod[24]isutilized.Theintegrationofinformationfromriskanalysisactivitieswiththemodel-basedtestgenerationapproachisrealizedbyatestpurposelanguage.Itisusedtoformalizesecuritytestpatternsinordertomakethemusablefortestgenerators.Risk-BasedVulnerabilityTestingisappliedtosecuritytestingofawebapplication.

Zechetal.[48,49]proposeanewmethodforgeneratingnegativesecuritytestsfornonfunctionalsecuritytestingofwebapplicationsbylogicprogrammingandknowledgeengineering.Basedonadeclarativemodelofthesystemundertest,ariskanalysisisperformedandusedforderivationoftestcases.

5.2Code-BasedTestingandStaticAnalysisManyvulnerabilitiescanonlybedetectedbylookingatthecode.Whiletraditionallynotunderstoodasatestingtechnique,staticanalysisoftheprogramcodeisanimportantpartofanysecuritydevelopmentprocess,asitallowstodetectvulnerabilitiesatearlystagesofthedevelopmentlifecyclewherefixingofvulnerabilitiesiscomparativelycheap[50].InMicrosoft’sSDL[26],SASTispartoftheimplementationphasetohighlightthatthistechniqueshouldbeappliedassoonasthefirstlineofcodeiswritten.Notethatinthissection,weonlydiscusspurelystaticapproaches,ie,approachesthatdonotrequireanexecutabletestsystem.Thus,wediscusshybridapproaches,ie,approachesthatcombinestaticanalysiswithdynamictesting(suchasconcolictesting)inSection5.3.

Codereviewscaneitherbedonemanuallyorautomated.Thelatterisoftencalledstaticcodeanalysis(SCA)orStaticApplicationSecurityTesting(SAST).Moreover,wecaneitheranalyzethesourcecode(ie,thecodethatwaswrittenbyadeveloper)oftheprogramorthecompiledsourcecode(ie,binariesorbyte-code).Astheyarecloselyrelated,wediscussthemnotseparately.Fromasoftwarevendor’sperspectivewhoisaimingatbuildingsecuresoftware,theanalysisonthesourcecodeispreferredoverabinaryanalysis,asthesourcecodeanalysisismorepreciseandcanprovidedetailedrecommendationstodevelopersonhowtofixavulnerabilityonthesourcecodelevel.

5.2.1ManualCodeReview

Manualcodereviewistheprocessbywhichanexpertisreadingprogramcode“line-by-line”toidentifyvulnerabilities.Thisrequiresexpertiseinthreeareas:theapplicationarchitecture,theimplementationtechniques(programminglanguages,frameworksusedtobuildtheapplication),aswellassecurity.Thus,agoodmanualcodereviewshouldstartwithathreatmodeloratleastaninterviewwiththedeveloperstogetagoodunderstandingoftheapplicationarchitecture,itsattacksurface,aswellastheimplementationtechniques.Afterthis,theactualcodereviewcanstartinwhichcodeis,guidedbytheidentifiedattacksurface,manuallyanalyzedforsecurityvulnerabilities.Finally,theresultsoftheanalysisarereportedbacktodevelopmenttofixtheidentifiedvulnerabilitiesaswellastoeducatearchitectsanddeveloperstopreventsimilarissuesinthefuture.Overall,manualcodereviewsareatediousprocessthatrequiresskill,experience,persistence,andpatience.

5.2.2StaticApplicationSecurityTestingAutomatedstaticprogramanalysisforfindingsecurityvulnerabilities,alsocalledStaticApplicationSecurityTesting(SAST)[51],isanattempttoautomatedcodereviews:inprinciple,aSASTtoolanalysestheprogramcodeofasoftwarecomponent(eg,anapplicationoralibrary)automaticallyandreportspotentialsecurityproblems(potentialvulnerabilities).Thislimitsthemanualefforttoreviewingthereportedproblemsand,thus,increasesthescalability(ie,theamountofprogramcodethatcanbeanalyzedinacertainamountoftime)significantly.Moreover,ontheonehand,SASTtools“encapsulate”mostoftherequiredsecurityexpertiseand,thus,theycan(andshould)beusedbydevelopersthatarenotnecessarilysecurityexperts.Ontheotherhand,SASTtoolsonlyreportvulnerabilitiestheyarelookingforand,thus,thereisstillaneedforasmallteamofexpertsthatconfigurestheSASTtoolscorrectly[52,53].

Forcomputingthesetofpotentialsecurityproblemsinaprogram,aSASTtoolmainlyemploystwodifferenttypesofanalysis:

1.SyntacticcheckssuchascallinginsecureAPIfunctionsorusinginsecureconfigurationoptions.AnexampleofthisclasswouldbeananalysisofJavaprogramsforcallstojava.util.random(whichdoesnotprovideacryptographicallysecurerandomgenerator).

2.Semanticchecksthatrequireanunderstandingoftheprogramsemanticssuchasthedatafloworcontrolflowofaprogram.Anexampleofthisclasswouldbeananalysischeckingfordirect(notsanitized)data-flowsfromanprograminputtoaSQLstatement(indicatingapotentialSQLInjectionvulnerability).

AsSASTtoolsworkonoverapproximationsoftheactualprogramcodeaswellasapplyheuristicschecks,theoutputofaSASTtoolisalistofpotentialsecurityvulnerabilities.Thus,oreachfinding,anhumanexpertisnecessarytodecide:

•Ifthefindingrepresentsavulnerability,ie,aweaknessthatcanbeexploitedbyanattacker(truepositive),and,thus,needstobefixed.

•Ifthefindingcannotbeexploitedbyanattacker(falsepositive)and,thus,doesnotneedtobefixed.

Similarly,ifanSASTtooldoesnotreportsecurityissues,thiscanhavetworeasons:

•Thesourcecodeissecure(truenegative)

•Thesourcecodehassecurityvulnerabilitybutduetolimitationsofthetool,thetooldoesnotreportaproblem(falsenegative).

ThereareSASTtoolsavailableformostofthewidelyusedprogramminglanguage,eg,FindBugs[54]thatisabletoanalyzesJavabytecodeand,thus,cananalyzevariouslanguagesrunningontheJavaVirtualMachine.TherearealsospecializestechniquesforJavaprograms(eg,[55]),orC/C++(eg,[56])aswellasapproachesthatworkonmultiplelanguages(eg,[57]).Forasurveyonstaticanalysismethods,wereferthereaderelsewhere[51,58].Moreover,wediscussfurtherstaticanalysistechniquesinthecontextofasmallcasestudyinSection6.

BesidesthefactthatSASTtoolscanbeappliedveryearlyinthesoftwaredevelopmentlifecycleaswellasthefactthatsourcecodeanalysiscanprovidedetailedfixrecommendations,SASThasoneadditionaladvantagesovermostdynamicsecuritytestingtechniques:SASTtoolscananalyzeallcontrolflowsofaprogram.Therefore,SASTtoolsachieve,comparedtodynamictestapproaches,asignificanthighercoverageoftheprogramundertestand,thus,produceasignificantlowerfalsenegativerate.Thus,SASTisaveryeffectivemethod[59]fordetectingprogrammingrelatedvulnerabilitiesearlyinthesoftwaredevelopmentlifecycle.

5.3PenetrationTestingandDynamicAnalysisIncontrasttowhite-boxsecuritytestingtechniques(seeSection5.2),black-boxsecuritytestingdoesnotrequireaccesstothesourcecodeorotherdevelopmentartifactsoftheapplicationundertest.Instead,thesecuritytestisconductedviainteractionwiththerunningsoftware.

5.3.1PenetrationTestingAwell-knownformofblack-boxsecuritytestingisPenetrationTesting.Inapenetrationtest,anapplicationorsystemistestedfromtheoutsideinasetupthatiscomparabletoanactualattackfromamaliciousthirdparty.Thismeans,inmostsettingstheentitythatisconductingthetesthaspotentiallyonlylimitedinformationaboutthesystemundertestandisonlyabletointeractwiththesystem’spublicinterfaces.Hence,amandatoryprerequisiteforthisapproachisa(near)productiveapplication,thatisfeaturecompleteandsufficientlyfilledwithdata,sothatallimplementedworkflowscanbeexecutedduringthetest.Penetrationtestsarecommonlydoneforapplicationsthatareopenfornetworkedcommunication.

TheNISTTechnicalGuidetoInformationSecurityTestingandAssessment[60]partitionsthepenetrationtestingprocessinfourdistinctphases(seeFig.5):

FIGURE5 Phasesofapenetrationtest[60].

1.Planning:Noactualtestingoccursinthisphase.Instead,importantsideconditionsandboundariesforthetestaredefinedanddocumented.Forinstance,therelevantcomponentsoftheapplicationsthataresubjectofthetestaredeterminedandthenature/scopeofthetobeconductedtestsandtheirlevelofinvasiveness.

2.Discovery:Thisphaseconsistsofasteps.First,allaccessibleexternalinterfacesofthesystemundertestaresystematicallydiscoveredandenumerated.Thissetofinterfacesconstitutesthesystem’sinitialattacksurface.Thesecondpartofthediscoveryphaseisvulnerabilityanalysis,inwhichtheapplicablevulnerabilityclassesthatmatchtheinterfaceareidentified(eg,Cross-SiteScriptingforHTTPservicesorSQLInjectionforapplicationswithdatabasebackend).Inacommercialpenetrationtest,thisphasealsoincludesthecheckifanyofthefoundcomponentsissusceptibletopubliclydocumentedvulnerabilitieswhichiscontainedinprecompiledvulnerabilitydatabases.

3.Attack:Finally,theidentifiedinterfacesaretestedthroughaseriesofattackattempts.Intheseattacks,thetestersactivelyattemptstocompromisethesystemviasendingattackpayloads.Incaseofsuccess,thefoundsecurityvulnerabilitiesareexploitedinordertogainfurtherinformationaboutthesystem,widentheaccessprivilegesofthetesterandfindfurthersystemcomponents,whichmightexposeadditionalinterfaces.Thisexpandedattacksurfaceisfedbackintothediscoveryphase,forfurtherprocessing.

4.Reporting:Thereportingphaseoccurssimultaneouslywiththeotherthreephasesofthepenetrationtestanddocumentsallfindingsalongwiththeirestimatedsevereness.

5.3.2VulnerabilityScanningIngeneralpenetrationtestsareacombinationofmanualtestingthroughsecurityexpertsandtheusageofblack-boxvulnerabilityscanners.Black-boxwebvulnerabilityscannersareaclassoftoolsthatcanbeusedtoidentifysecurityissuesinapplicationsthroughvarioustechniques.Thescannerqueriestheapplication’sinterfaceswithasetofpredefinedattackpayloadsandanalysestheapplication’sresponsesforindicatorsiftheattackwassuccessfuland,ifthisisnotthecase,hintshowtoaltertheattackinthesubsequenttries.Bauetal.[4]aswellasAdametal.[61]provideoverviewsofrecentcommercialandacademicblack-boxvulnerabilityscanners.

5.3.3DynamicTaintAnalysis

Animportantvariantofblack-boxtestingisananalysistechniquecalledtaintanalysis.Asignificantportionoftoday’ssecurityvulnerabilitiesarestring-basedcodeinjectionvulnerabilities[62],whichenabletheattackertoinjectsyntacticcontentintodynamicallyexecutedprogrammingstatements,which—inthemajorityofallcases—leadstofullcompromiseofthevulnerableexecutioncontext.ExamplesforsuchvulnerabilitiesincludeSQLInjection[63]andCross-SiteScripting[64].Suchinjectionvulnerabilitiescanberegardedasinformationflowproblems,inwhichunsanitizeddatapathsfromuntrustedsourcestosecuritysensitivesinkshavetobefound.Toachievethis,awellestablishedapproachis(dynamic)datatainting.Untrusteddataisoutfittedwithtaintinformationonruntime,whichisonlycleared,ifthedatapassesadedicatedsanitizationfunction.Ifdatawhichstillcarriestaintinformationreachesasecuritysensitivesink(eg,anAPIthatconvertsstringdataintoexecutablecode),theapplicationcanreactappropriately,forinstancethroughaltering,autosanitizationthedataorcompletelystoppingthecorrespondingprocess.Iftainttrackingisutilizedinsecuritytesting,themainpurposeistonotifythetesterthatinsecuredataflows,thatlikelyleadtocodeinjection,exist.Unlikestaticanalysis,thatalsotargetstheidentificationofproblematicdataflows,dynamictaintanalysisisconductedtransparentlywhiletheapplicationundertestisexecuted.Forthis,theexecutionenvironment,eg,thelanguageruntime,hastobemadetaintaware,sothattheattachedtaintinformationoftheuntrusteddataismaintainedthroughthecourseofprogramexecution,sothatitcanreliablybedetectedwhentainteddataendsupinsecuritysensitivesinks.

5.3.4FuzzingFuzzingorfuzztestingisadynamictestingtechniquethatisbasedontheideaoffeedingrandomdatatoaprogram“untilitcrashes.”Itwaspioneeredinthelate1980sbyBartonMillerattheUniversityofWisconsin[65].Sincethen,fuzztestinghasbeenproventobeaneffectivetechniqueforfindingvulnerabilitiesinsoftware.Whilethefirstfuzztestingapproacheswherepurelybasedonrandomlygeneratedtestdata(randomfuzzing),advancesinsymboliccomputation,model-basedtesting,aswellasdynamictestcasegenerationhaveleadtomoreadvancedfuzzingtechniquessuchasmutation-basedfuzzing,generation-basedfuzzing,orgray-boxfuzzing.

Randomfuzzingisthesimplestandoldestfuzztestingtechnique:astreamofrandominputdatais,inablack-boxscenario,sendtotheprogramundertest.Theinputdatacan,eg,besendascommandlineoptions,events,orprotocolpackets.Thistypeoffuzzingin,inparticular,usefulfortesthowaprogramreactsonlargeorinvalidinputdata.Whilerandomfuzzingcanfindalreadyseverevulnerabilities,modernfuzzersdohaveadetailedunderstandingoftheinputformatthatisexpectedbytheprogramundertest.

Mutation-basedfuzzingisonetypeoffuzzinginwhichthefuzzerhassomeknowledgeabouttheinputformatoftheprogramundertest:basedonexistingdatasamples,amutation-basedfuzzingtoolsgeneratednewvariants(mutants),basedonaheuristics,thatitusesforfuzzing.thereareawiderangeofmutation-basedfuzzingapproachesavailablefordifferentdomains.Werefertheinterestedreaderelsewherefordetails[66,67].

Generation-basedfuzzingusesamodel(oftheinputdataorthevulnerabilities)forgeneratingtestdatafromthismodelorspecification.Comparedtopurerandom-basedfuzzing,generation-basedfuzzingachievesusuallyahighercoverageoftheprogramundertest,inparticulariftheexpectedinputformatisrathercomplex.Again,fordetailswerefertheinterestedreaderelsewhere[68,69].

Advancedfuzzingtechniquescombineseveralofthepreviouslymentionedapproaches,eg,useacombinationofmutation-basedandgeneration-basedtechniquesaswellasobservetheprogramundertestandusetheseobservationsforconstructingnewtestdata.Thisturnsfuzzingintoagray-boxtestingtechniquethatalsoutilizessymboliccomputationthatisusuallyunderstoodasatechniqueusedforstaticprogramanalysis.Probablythefirstandalsomostsuccessfulapplicationofgray-boxfuzzingisSAGEfromMicrosoft[70,71],whichcombinessymbolicexecution(astaticsourcecodeanalysistechnique)anddynamictesting.Thiscombinationistodayknownasconcolictestingandinspiredseveraladvancedsecuritytestingeg,[72,73],aswellasfunctionaltestapproaches.

5.4SecurityRegressionTestingDuetoeverchangingsurroundings,newbusinessneeds,newregulations,andnewtechnologies,asoftwaresystemmustevolveandbemaintained,oritbecomesprogressivelylesssatisfactory[74].Thismakesitespeciallychallengingtokeepsoftwaresystemspermanentlysecureaschangeseitherinthesystemitselforinitsenvironmentmaycausenewthreatsandvulnerabilities[75].Acombinationofregressionandsecuritytestingcalledsecurityregressiontesting,whichensuresthatchangesmadetoasystemdonotharmitssecurity,arethereforeofhighsignificanceandtheinterestinsuchapproacheshassteadilyincreased[76].Regressiontestingtechniquesensurethatchangesmadetoexistingsoftwaredonotcauseunintendedeffectsonunchangedpartsandchangedpartsofthesoftwarebehaveasintended[77].Asrequirements,designmodels,codeoreventherunningsystemcanbechanged,regressiontestingtechniquesareorthogonaltothesecuritytestingtechniquesdiscussedintheprevioussections.

YooandHarman[78]classifyregressiontestingtechniquesintothreecategories:testsuiteminimization,testcaseprioritizationandtestcaseselection.

Testsuiteminimizationseekstoreducethesizeofatestsuitebyeliminatingredundanttestcasesfromthetestsuite.

Testcaseprioritizationaimsatorderingtheexecutionoftestcasesintheregressiontestsuitebasedonacriterion,forinstance,onthebasisofhistory,coverage,orrequirements,whichisexpectedtoleadtotheearlydetectionoffaultsorthemaximizationofsomeotherdesirableproperties.

Testcaseselectiondealswiththeproblemofselectingasubsetoftestcasesthatwillbeusedtotestchangedpartsofsoftware.Itrequirestoselectasubsetofthetestsfromthepreviousversion,whicharelikelytodetectfaults,basedondifferentstrategies.Mostreportedregressiontestingtechniquesfocusonthisregressiontestingtechnique[78].The

usualstrategyistofocusontheidentificationofmodifiedpartsoftheSUTandtoselecttestcasesrelevanttothem.Forinstance,theretest-alltechniqueisonenaivetypeofregressiontestselectionbyreexecutingalltestsfromthepreviousversiononthenewversionofthesystem.Itisoftenusedinindustryduetoitssimpleandquickimplementation.However,itscapacityintermsoffaultdetectionislimited[79].Therefore,considerableamountofworkisrelatedtothedevelopmentofeffectiveandscalableselectivetechniques.

Inthefollowing,wediscussavailablesecuritytestingapproachesaccordingtothecategoriesminimization,prioritization,andselection.TheselectedapproachesarebasedonasystematicclassificationofsecurityregressiontestingapproachesbyFeldererandFourneret[76].

5.4.1TestSuiteMinimizationTestsuiteminimizationseekstoreducethesizeofatestsuitebyeliminatingtestcasesfromthetestsuitebasedonagivencriterion.Currentapproachesonminimization[80–82]addressvulnerabilities.

Tothetal.[80]proposeanapproachthatappliesautomatedsecuritytestingfordetectionofvulnerabilitiesbyexploringapplicationfaultsthatmayleadtoknownmalware,suchasvirusesorworms.Theapproachconsidersonlyfailedtestsfromthepreviousversionrevealingfaultsforreruninanewsystemversionafterfaultfixing.

Heetal.[81]proposeanapproachfordetectingandremovingvulnerabilitiesforminorreleasesofwebapplications.Intheirapproach,onlystrong-associationlinksbetweenpagesfromapreviousversion,optimizedthroughiterations,areselectedforexplorationofwebpagesthatcontainvulnerabilities.

Finally,Garvinetal.[82]proposetestingofself-adaptivesystemforalreadyknownfailures.Theauthorsavoidreproducingalreadyknownfailuresbyconsideringonlythosetestsforexecutionthatexercisetheknownfailuresinpreviousversions.

5.4.2TestCasePrioritizationTestcaseprioritizationisconcernedwithrightorderingoftestcasesthatmaximizesdesirableproperties,suchasearlyfaultdetection.Also,currentapproachestoprioritization[83–85]addressonlyvulnerabilities.

Huangetal.[83]proposeaprioritizationtechniqueforsecurityregressiontesting.Theirtechniquegathershistoricalrecords,whosemosteffectiveexecutionorderisdeterminedbyageneticalgorithm.

Yuetal.[84]proposefault-basedprioritizationoftestcases,whichdirectlyutilizestheknowledgeoftheircapabilitytodetectfaults.

Finally,Viennotetal.[85]proposeamutablerecord-replaysystem,whichallowsarecordedexecutionofanapplicationtobereplayedwithamodifiedversionoftheapplication.Theirexecutionisprioritizedbydefiningasocalledd-optimalmutablereplay

basedonacostfunctionmeasuringthedifferencebetweentheoriginalexecutionandthemutablereplay.

5.4.3TestCaseSelectionTestcaseselectionapproacheschooseasubsetoralltestcasestotestchangedpartsofsoftware.Asforclassicalregressiontesting[78],alsoforsecurityregressiontestingmostapproachesfallintothiscategory[76].Theseapproachestestboth,securitymechanismsandvulnerabilities.Severalsubset-basedapproaches[86–90]andretest-allapproaches[91–94]havebeenproposed.

Feldereretal.[86]provideaUML-basedapproachforregressiontestingofsecurityrequirementsoftypeauthentication,confidentiality,availability,authorization,andintegrity.Tests,representedassequencediagrams,areselectedbasedontestrequirementschanges.Kassabetal.[87]proposeanapproachtoimproveregressiontestingbasedonnonfunctionalrequirementsontologies.Testsareselectedbasedonchangeandimpactanalysisofnonfunctionalrequirements,suchassecurity,safety,performance,orreliability.Eachtestlinkedtoachangedormodifiedrequirementisselectedforregressiontesting.Anisettietal.[88]proposeanapproachforprovidingtestevidenceforincrementalcertificationofsecurityrequirementsofservices.Thisapproachisbasedonchangedetectionintheservicetestmodel,whichwilldetermineifnewtestcasesneedtobegenerated,orexistingonestobeselectedforreexecutionontheevolvedservice.Huangetal.[89]addresssecurityregressiontestingofaccesscontrolpolicies.Thisapproachselectstestsiftheycoverchangedelementsinthepolicy.Finally,Hwangetal.[90]proposethreesafecoverage-basedselectiontechniquesfortestingevolutionofsecuritypolicies,eachofwhichincludesasequenceofrulestospecifywhichsubjectsarepermittedordeniedtoaccesswhichresourcesunderwhichconditions.Thethreetechniquesarebasedontwocoveragecriteria,ie,(1)coverageofchangedrulesinthepolicy,and(2)coverageofdifferentprogramdecisionsfortheevolvedandtheoriginalpolicy.

Vetterlingetal.[91]proposeanapproachfordevelopingsecuresystemsevaluatedundertheCommonCriteria[95].Intheirapproach,testscoveringsecurityrequirementsarecreatedmanuallyandrepresentedassequencediagrams.Incaseofachange,newtestsarewrittenifnecessaryandthenalltestsareexecutedonthenewversion.Brunoetal.[92]proposeanapproachfortestingsecurityrequirementsofwebservicereleases.Theserviceusercanperiodicallyreexecutethetestsuiteagainsttheserviceinordertoverifywhethertheservicestillexhibitsthefunctionalandnonfunctionalrequirements.Kongsli[93]proposestheuseofsocalledmisusestoriesinagilemethodologies(forinstance,extremeprogramming)toexpressvulnerabilitiesrelatedtofeaturesand/orfunctionalityofthesysteminordertobeabletoperformsecurityregressiontesting.Theauthorsuggeststhattestexecutionisidealforagilemethodologiesandcontinuousintegration.Finally,Qietal.[94]proposeanapproachforregressionfault-localizationinwebapplications.Basedontwoprograms,areferenceprogramandamodifiedprogram,aswellasinputfailingonthemodifiedprogram,theirapproachusessymbolicexecutiontoautomaticallysynthesizeanewinputthatisverysimilartothefailinginputanddoesnotfail.Onthisbasis,the

potentialcause(s)offailurearefoundbycomparingcontrolflowbehaviorofthepassingandfailinginputsandidentifyingcodefragmentswherethecontrolflowsdiverge.

6ApplicationofSecurityTestingTechniquesInthissection,wemakeaconcreteproposalonhowtoapplythesecuritytesttechniques(andthetoolsimplementingthem)toasmallcasestudy:abusinessapplicationusingathreetieredarchitecture.WefocusonsecuritytestingtechniquesthatdetectthemostcommonvulnerabilitytypesthatweredisclosedintheCommonVulnerabilitiesandExposures(CVE)index[5]overtheperiodofthelast15years(seeFig.6).Thisclearlyshowsthatthevastmajorityofvulnerabilities,suchasXSS,bufferoverflows,arestillcausedbyprogrammingerrors.Thus,wefocusinthissectiononsecuritytestingtechniquesthatallowtofindthesekindofvulnerabilities.

FIGURE6 Numberofentiresinthecommonvulnerabilitiesandexposures(CVE)indexbycategory.

Forinstance,weexcludetechniquesforensuringthesecurityoftheunderlyinginfrastructuresuchasthenetworkconfiguration,as,eg,discussedinRefs.[96,97]aswellasmodel-basedtestingtechniques(asdiscussedinSection5.1)thatareinparticularusefulforfindinglogicalsecurityflaws.Whileaholisticsecuritytestingstrategymakesuseofallavailablesecuritytestingstrategies,werecommendtoconcentrateeffortsfirstontechniquesforthemostcommonvulnerabilities.Furthermore,wealsodonotexplicitlydiscussretestingafterchangesofthesystemundertest,whichisaddressedbysuitable(security)regressiontestingapproaches(asdiscussedinSection5.4).

6.1SelectionCriteriaforSecurityTestingApproachesWhenselectingaspecificsecuritytestingmethodandtool,manyaspectsneedtobetakenintoaccount,eg:

•Attacksurface:differentsecuritytestingmethodsfinddifferentattackandvulnerabilitytypes.Manytechniquesarecomplementarysoitisadvisabletousemultiplesecuritytestingmethodstogethertoefficientlydetectarangeofvulnerabilitiesaswideaspossible.

•Applicationtype:differentsecuritytestingmethodsperformdifferentlywhenappliedtodifferentapplicationtypes.Forexample,amethodthatperformswellagainstamobileapplicationmaynotbeabletoperformaswellagainstthree-tierclient-serverapplications.

•Performanceandresourceutilization:differenttoolsandmethodsrequiredifferentcomputingpoweranddifferentmanualefforts.

•Costsforlicenses,maintenance,andsupport:tousesecuritytestingtoolsefficientlyinalargeenterprise,theyneedtobeintegratedinto,eg,bugtrackingorreportingsolutions—oftentheyprovidetheirownserverapplicationsforthis.Thus,buyingasecuritytestingtoolisusuallynotaone-timeeffort—itrequiresregularmaintenanceandsupport.

•Qualityofresults:differenttoolsthatimplementthesamesecuritytestingtechniqueprovideadifferentqualitylevel(eg,intermsoffixrecommendationsorfalsepositivesrates).

•Supportedtechnologies:securitytestingtoolsusuallyonlysupportalimitednumberoftechnologies(eg,programminglanguages,interfaces,orbuildsystems).Ifthesetoolssupportmultipletechnologies,theydonotnecessarysupportallofthemwiththesamequality.Forexample,asourceanalysistoolthatsupportsJavaandCmightworkwellforJavabutnotaswellforC.

Inthefollowing,wefocusonthefirsttwoaspects:theattacksurfaceandtheapplicationtype.Thesetwoaspectsare,fromasecurityperspectivethefirstonestoconsiderforselectingthebestcombinationsofsecuritytestingapproachesforaspecificapplicationtype(product).Inasubsequentstep,theotherfactorsneedtobeconsideredforselectingaspecifictoolthatfitstheneedsoftheactualdevelopmentaswellastheresourceandtimeconstraints.

6.2AThree-TieredBusinessApplicationInthischapter,weuseasimplemultitieredbusinessapplication,eg,forbookingbusinesstravels,asarunningexample.Thisarchitectureis,onthefirsthand,verycommonforlargerapplicationsand,ontheotherhand,coversawidevarietyofsecuritytestingchallenges.

Letusassumethatwewanttoplanthesecuritytestingactivitiesforbusinessapplicationthatisseparatedintothreetiers:

•Firsttier:Afront-endthatisimplementedasrich-clientusingmodernwebdevelopmenttechniques,ie,HTML5andJavaScript,eg,usingframeworkssuchasAngularJSorJQuery.

•Secondtier:Atypicalmiddle-tierimplementedinJava(eg,usingJavaServletshostedinanapplicationserversuchasApacheTomcat).

•Thirdtier:Athird-partydatabasethatprovidespersistencyforthebusinessdatathatisprocessedinthesecondtier.

Fig.7illustratesthisexamplearchitecturewherethedottedverticallinesmarkthetrustboundariesoftheapplication.

FIGURE7 Architectureofexampleapplication.

6.2.1TheFirstTier:WebApplicationsWebapplicationsarepredominantlyaffectedbycodeinjectionvulnerabilities,suchasSQLInjection[63]orCross-SiteScripting(XSS)[64].ThisisshowninFig.6:asitcanbeseen,besidesbufferoverflows(andsimilarmemorycorruptionvulnerabilities),whichingeneraldonotoccurinWebapplications,injectionvulnerabilitiesrepresentthevastmajorityofreportedissues.

Inconsequence,securitytestingapproachesforWebapplicationsareprimarilyconcernedwithdetectionofsuchcodeinjectionvulnerabilities.Intheremainderofthissection,weutilizethefieldofWebapplicationsasacasestudyinsecuritytestingandcomprehensivelylistacademicapproachesinthisarea.

6.2.1.1Code-BasedTestingandStaticAnalysisAsintroducedinSection5.2,staticanalysisofsourcecodeisapowerfultooltodetectvulnerabilitiesinsourcecode.Staticanalysisallowstoanalyzethecodeofagivenapplicationwithoutactuallyexecutingit.Todetectinjectionvulnerabilitieswithsuchanapproach,inmostcasesthestaticanalysisattemptstoapproximatethedataflowswithintheexaminedsourcecode.Thisway,dataflowsfromsourcesthatcanbecontrolledbytheattacker(eg,theincomingHTTPrequest)intosecuritysensitivesinks(eg,APIsthatcreatetheHTTPresponseofsendSQLtothedatabase).

DifferentmechanismsandtechniquesproposedintheprogramanalysisliteraturecanbeappliedtoWebapplications.Theactualimplementationofsuchananalysiscanutilizedifferenttechniquessuchasmodelchecking,data-flowanalysis,orsymbolicexecution,typicallydependingonthedesiredprecisionandcompleteness.Threedifferentpropertiesarerelevantfortheanalysisphase.First,thestaticanalysisofthesourcecodeitselfcanbeperformedondifferentlevels,eitheronlywithinagivenfunction(intraproceduralanalysis)ortheinteractionoffunctionscanbeanalyzedaswell(interproceduralanalysis).Second,theexecutioncontextcanbeexaminedindetailorneglected,whichthentypicallyreducestheprecisionoftheanalysis.Athirdaspectofstaticanalysisdealswiththewayhowtheflowofdataorcodeisanalyzed.

OneofthefirsttoolsinthisareawasWebSSARI[98],acodeanalysistoolforPHPapplicationsdevelopedbyHuangetal.basedonaCQual-liketypesystem[99,100].

PixyisastatictaintanalysistoolpresentedbyJovanovicetal.forautomatedidentificationofXSSvulnerabilitiesinPHPwebapplications[101,102].Thetool

performsaninterprocedural,context-sensitive,andflow-sensitivedataflowanalysistodetectvulnerabilitiesinagivenapplication.Combinedwithaprecisealiasanalysis,thetooliscapableofdetectingavarietyofvulnerabilities.Inasimilarpaper,LivshitsandLamdemonstratedhowacontext-sensitivepointeraliasanalysistogetherwithstringanalysiscanberealizedforJava-basedwebapplications[55].

XieandAikenintroducedastaticanalysisalgorithmbasedonso-calledblockandfunctionsummariestodetectvulnerabilitiesinPHPapplications[103].Theapproachperformstheanalysisinbothanintraproceduralandaninterproceduralway.DahseandHolzrefinedthisapproachanddemonstratedthatafine-grainedanalysisofbuilt-inPHPfeaturesandtheirinteractionsignificantlyimprovesdetectionaccuracy[104,105].

WassermanandSuimplementedastaticcodeanalysisapproachtodetectXSSvulnerabilitiescausedbyweakorabsentinputvalidation[106].Theauthorscombinedworkontaint-basedinformationflowanalysiswithstringanalysispreviouslyintroducedbyMinamide[107].

AtoolforstaticanalysisofJavaScriptcodewasdevelopedbySaxenaetal.[108].ThetoolnamedKudzuemployssymbolicexecutiontofindclient-sideinjectionflawsinWebapplications.Jinetal.employstaticanalysis[109]tofindXSSvulnerabilitieswithinHTML5-basedmobileapplications.Interestingly,theyfoundnewwaystoconductXSSattackswithinsuchappsbyabusingthespecialcapabilitiesofmobilephones.

FurtherstaticanalysisapproachestodetectinjectionvulnerabilitieshavebeenproposedbyFuetal.[110],WassermannandSu[111],andHalfondetal.[112]

6.2.1.2DynamicAnalysisandBlack-BoxTestingAcomplementaryapproachisdynamicanalysis,inwhichagivenapplicationisexecutedwithaparticularsetofinputsandtheruntimebehavioroftheapplicationisobserved.Manyoftheseapproachesemploydynamictainttrackingandconsistoftwodifferentcomponents.Adetectioncomponenttoidentifypotentiallyvulnerabledataflowsandavalidationcomponenttoeliminatefalsepositives.

SecuBat[113]isageneralpurposeWebvulnerabilityscannerthatalsohasXSSdetectioncapabilities.Toachieveitsgoals,SecuBatemploysthreedifferentcomponents:acrawlingcomponent,anattackcomponent,andananalysiscomponent.Wheneverthecrawlingcomponentdiscoversasuspiciousfeature,itpassesthepageunderinvestigationtotheattackcomponent,whichthenscansthispageforwebforms.Ifaformisfound,appropriatepayloadsareinsertedintothefieldsoftheformandsubmittedtotheserver.Theresponseoftheserveristheninterpretedbytheanalysiscomponent.Snuck[114]isanotherdynamictestingtool.Inafirststep,SnuckusesalegitimatetestinputtodynamicallydetermineapossibleinjectionandthecorrespondingcontextviaXPATHexpressions.Basedonthedeterminedcontext,thetoolchoosesasetofpredefinedattackpayloadsandinjectsthemintotheapplication.

WhileSecuBatandSnuckfocusonserver-sidesecurityproblems,FlashOver[115]focusesonaclient-sideproblem.Morespecifically,FlashOverdetectsclient-sidereflectedXSSin

AdobeFlashapplets.Itdoessobydecompilingthesourcecodeandstaticallydetectingsuspicioussituations.Thenitconstructsanattackpayloadandexecutestheexploitviadynamictesting.

Recently,Bauetal.[4]andDoupeetal.[61]preparedcomprehensiveoverviewsoncommercialandacademicblack-boxvulnerabilityscannersandtheirunderlyingapproaches.

6.2.1.3TaintTrackingAsdiscussedpreviously,mostWeb-specificsecurityvulnerabilitiescanpredominantlyberegardedasinformationflowproblems,causedbyunsanitizeddatapathsfromuntrustedsourcestosecuritysensitivesinks.Hence,tainttracking(seeSection5.3.3)isawell-establishedmethodtodetectinjectionvulnerabilities.

DynamictainttrackingwasinitiallyintroducedbyPerlin1989[116]andsincethenhasbeenadoptedfornumerousprogramminglanguagesandframeworks[117].Subsequentworksdescribefinergrainedapproachestowardsdynamictaintpropagation.Thesetechniquesallowthetrackingofuntrustedinputonthebasisofsinglecharacters.Forinstance,Nguyen-Tuongetal.[118]andPietraszekandVandenBerghe[119]proposedfinegrainedtaintpropagationforthePHPruntimetodetectvariousclassesofinjectionattacks,suchasSQLInjectionorXSS.Basedondynamictaintpropagation,SuandWassermann[120]describeanapproachthatutilizesspecificallycraftedgrammarstodeterministicallyidentifySQLinjectionattempts.

ThefirsttainttrackingpaperthataimedatautomaticallygeneratingCross-SiteScriptingattackswasauthoredbyMartinetal.[121].ThepresentedmechanismexpectsaprogramadheringtotheJavaServletspecificationandataint-basedvulnerabilityspecificationasaninput,andgeneratesavalidCross-SiteScriptingorSQLInjectionattackpayloadwiththehelpofdynamictainttrackingandmodelchecking.Whilethisapproachrequiresasecurityanalysttomanuallywriteavulnerabilityspecification,Kieyzunetal.focusonthefullyautomaticgenerationoftaint-basedvulnerabilitypayloads[122].Furthermore,theyextendthedynamictaintingtothedatabase.Hence,asopposedtothefirstapproach,thisapproachisabletoalsodetectserver-sidepersistentXSSvulnerabilities.

Besidesserver-sideinjectionvulnerabilities,Webapplicationsarealsosusceptibletoinjectionproblemsontheclient-side,ie,Cross-SiteScriptingproblemscausedbyinsecureJavaScriptexecutedinthebrowser[123].Throughmakingthebrowser’sJavaScriptenginetaint-aware,thisvulnerabilityclasscanbedetectedduringtesting:Lekiesetal.implementedabrowser-basedbyte-leveltaint-trackingenginetodetectreflectedandpersistentclient-sideXSSvulnerabilities[124].Byleveragingthetaintinformation,theirapproachiscapableofgeneratingapreciseXSSpayloadmatchingtheinjectioncontext.AsimilarapproachwastakenbyFLAX[125],whichalsoemploystainttrackinginthebrowser.Insteadofusinganexploitgenerationtechnique,FLAXutilizesasink-awarefuzzingtechnique,whichexecutesvariationsofapredefinedlistofcontext-specificattackvectors.

6.2.2TheSecondTier:Java-BasedServerApplicationsAlsothesecondtier,ie,theJava-basedbusinessapplication,ispredominantlyaffectedbycodeinjectionvulnerabilities,suchasSQLInjection[63]orCross-SiteScripting[64].Evenifwesecurethefront-endagainsttheseattacks,wecannotrelyontheseprotectionasanattackermightcircumventthefront-endbyattackingthesecondtierdirectly(eg,usingWSDL-based,RESTful,orOData-basedinterfaced).Thus,weneedtoapplythesamestandardsforsecuredevelopmentalsotothesecondtier.

6.2.2.1Code-BasedTestingandStaticAnalysisTherearemanystaticanalysisapproachesavailableforlanguagestraditionallyusedforserversystems,eg,C/C++orJava;bothintheacademicworldaswellascommercialofferings.Commercialtoolsareatleastavailablesince10years[57,126]andusedwidelysinceatleast5years[52,53].Duetospacereason,weonlydiscussafewselectedworksforJava:mostnotability,FindBugs[54]isa“ready-to-run”toolthatfindsalargenumberofpotentialsecurityvulnerabilitiesinJavaprogramsand,moreover,canbeconsideredthefirstwidelyusedstaticanalysistoolforJavathatincludedsecuritychecks.Alreadyin2005,Livshitspresentedamethodbasedontaintpropagationforfindingdata-flowrelatedvulnerabilitiesinJavaprograms[55].Trippetal.[127]improvedthisideatomakeitapplicabletolargescaleprogramsbyusingaoptimizedpointeranalysisasprerequisiteforbuildingthecallgraph.Analternativetousingacall-graph-basedtaintinganalysis,therearealsoapproaches,suchas[128]thatarebasedonslicing.Moreover,therearetwodifferentOpenSourceframeworksavailablethatforbuildingownstaticanalysistoolsforJava:Wala[129],whichitselfiswritteninJava,andSwaja[130],whichiswritteninOCaml.

6.2.2.2PenetrationTestingandDynamicAnalysisFromanimplementationperspective,themostimportantdynamictestingapproachis—besidesmanualpenetrationtesting—fuzzing.Asweimplementthesecondtier,wefocusongray-boxtestingapproachesthatcombinewhite-andblack-boxtesting.ThemostwellknownandindustriallyprovenapproachisSAGE[71],whichisusedbyMicrosoft.BesidesSAGE,therearealsootherapproachesthatsharethesamebasicconcept:usinginstrumentationorstaticanalysesoftheimplementationtoimprovethequality,efficiency,oreffectivenessofdynamicsecuritytesting.Forexample,Bekraretal.[72]usesabinarytaintanalysistoincreasethefuzztestingcoveragewhile[131]usessymbolicexecutiontoachievethesamegoal.Gray-boxfuzzingisalsosuccessfullyappliedtocommercialoperatingsystem[132].

Whenusingagray-boxtestingtechniqueonamultitierarchitecture,onecandecidetoonlytestthesecondtierinisolationortotestalltiers“atonce.”Inthelattercase,onlyselectedtiersmightbesubjecttotainttrackingorsymbolicexecution,asalreadysuchapartialcoveringoftheimplementationwithwhite-boxtechniquesallowstoimprovethetestingresults.Moreover,evenpartialapplicationofwhite-boxtechniqueshelpstodiagnosetherootcauseofanvulnerabilityand,thus,usuallyhelpstominimizethetime

requiredforfixingavulnerability.

Untilrecently,thusgray-boxtestingapproacheswherenotgenerallyavailableincommercialsecuritytestingsolutions.Thishaschanged:firstvendorsarestaringtointegratesimilartechniquesintheirtools,usingtheInteractiveSecurityApplicationTesting(ISAT).

6.2.3TheThirdTier:Back-EndSystemsInourexample,weassumethattheback-endsystem(ie,thedatabase)issuppliedbyathird-partyvendor.Therefore,weonlyhaveaccesstothesystemsasblack-box.Still,weneedtosecuritytestitasweareimplementingstoredprocedures(SQL)ontopofitaswellasneedtoconfigureitssecurity(eg,toensureproperaccesscontrolandasecureandtrustworthycommunicationtothesecondtier).Moreover,wemightwanttoassesstheoverallimplementation-levelsecurityoftheexternallydevelopedproduct.

6.2.3.1SecurityTestingtheImplementationForincreasingourconfidenceinthesecurityofthedatabase,athird-partycomponent,itself,wecanapplymanualpenetrationtestingaswellasfuzzingforcheckingforbufferoverflowsintheinterfacesthatareexposedviathenetwork.Asweassumethatweonlyhaveaccesstothebinary,wecanonlyuseblack-boxfuzzerssuchaspresentedbyWooetal.[133]orgray-boxfuzzersthatarebasedonbinaryanalysis,eg,[134].

Moreover,allcodethatwedevelopontopofthecoredatabase,ie,storedprocedureswritteninanSQL-dialectsuchasPL/SQLshouldbeanalyzedforinjectionattacksusingdedicatedstaticanalysisapproachessuchasRef.[135](alsomanycommerciallyavailablestaticanalysistoolssupporttheanalysisofSQLdialects).Moreover,wecanapplyseveraldynamicsecuritytestingapproachesthatspecializeontestingSQLInjections:Appeltetal.[136]presentamutation-basedtestingapproachforfindingSQLInjectionswhileWangetal.[137]useamodel-basedfuzzingapproach.ForanexperiencereportonusingfuzzersforfindingSQLInjectionsseeRef.[138].

6.2.3.2SecurityTestingtheConfigurationFinally,evenifallsoftwarecomponentsareimplementedsecurely,westillneedtoensurethattheactualconfigurationusedduringoperationsisalsosecure.Whilewetouchthistopiconlyverybriefly,wewanttoemphasizethatitisimportanttotestthesecureconfigurationofthecommunicationchannels,theaccesscontrolofalltiers,aswellastokeepthesystemsuptodatebypatchingknownvulnerabilities.TestingthesecurityofthecommunicationchannelsincludesapproachesthatcheckthecorrectvalidationofSSL/TLScertificates(eg,Frankencerts[139])aswellasprotocolfuzzerssuchasSNOOZE[140]orSECFUZZ[141].Fortestingthecorrectaccesscontrol,variousmodel-basedapproaches(eg,[142–144])havenbeenappliedtocasestudiesofdifferentsize.Finally,toolslikeNessus[145]thatrathereasilyallowtoscannetworksforapplicationswithknownvulnerabilitiesand,thus,applicationsthatneedtobeupdatedorpatched.

7SummaryInthischapter,weprovidedanoverviewofrecentsecuritytestingtechniquesandtheirpracticalapplicationincontextofathree-tieredbusinessapplication.Forthispurpose,wefirstsummarizedtherequiredbackgroundonsoftwaretestingandsecurityengineering.Testingconsistsofstaticanddynamiclifecycleactivitiesconcernedwithevaluationofsoftwareproductsandrelatedartifacts.Itcanbeperformedonthecomponent,integration,andsystemlevel.Withregardtoaccessibilityoftestdesignartifactswhite-boxtesting(ie,derivingtestcasesbasedondesignandcodeinformation)aswellasblack-boxtesting(ie,relyingonlyoninput/outputbehaviorofsoftware)canbedistinguished.Securitytestingvalidatessoftwaresystemrequirementsrelatedtosecuritypropertiesofassetsthatincludeconfidentiality,integrity,availability,authentication,authorization,andnonrepudiation.Securityrequirementscanbepositiveandfunctional,explicitlydefiningtheexpectedsecurityfunctionalityofasecuritymechanism,ornegativeandnonfunctional,specifyingwhattheapplicationshouldnotdo.Duetothenegativenatureofmanysecurityrequirementsandtheresultingbroadrangeofsubordinaterequirements,itisessentialtotaketestingintoaccountinallphasesofthesecuresoftwaredevelopmentlifecycle(ie,analysis,design,development,deploymentaswellasmaintenance)andtocombinedifferentsecuritytestingtechniques.

Foradetaileddiscussionofsecuritytestingtechniquesinthischapter,wethereforeclassifiedthemaccordingtotheirtestbasiswithinthesecuresoftwaredevelopmentlifecycleintofourdifferenttypes:(1)model-basedsecuritytestingisgroundedonrequirementsanddesignmodelscreatedduringtheanalysisanddesignphase,(2)code-basedtestingandstaticanalysisonsourceandbytecodecreatedduringdevelopment,(3)penetrationtestinganddynamicanalysisonrunningsystems,eitherinatestorproductionenvironment,aswellas(4)securityregressiontestingperformedduringmaintenance.Withregardtomodel-basedsecuritytesting,weconsideredtestingbasedonarchitecturalandfunctionalmodels,threat,faultandriskmodels,aswellasweaknessandvulnerabilitymodels.Concerning,code-basedtestingandstaticanalysiswetookmanualcodereviewsaswellasstaticapplicationsecuritytestingintoaccount.Withregardtopenetrationtestinganddynamicanalysis,weconsideredpenetrationtestingitself,vulnerabilityscanning,dynamictaintanalysis,aswellasfuzzing.Concerningsecurityregressiontesting,wediscussedapproachestotestsuiteminimization,testcaseprioritization,andtestcaseselection.Toshowhowthediscussedsecuritytestingtechniquescouldbepracticallyapplied,wediscusstheirusageforathree-tieredbusinessapplicationbasedonawebclient,anapplicationserver,aswellasadatabasebackend.

Overall,thischapterprovidedabroadoverviewofrecentsecuritytestingtechniques.Itfulfillsthegrowingneedforinformationonsecuritytestingtechniquestoenabletheireffectiveandefficientapplication.Alongtheselines,thischapterisofvaluebothforresearcherstoevaluateandrefineexistingsecuritytestingtechniquesaswellasforpractitionerstoapplyanddisseminatethem.

AcknowledgmentsTheworkwassupportedinpartbytheresearchprojectsQELaB—LivingModelsforOpenSystems(FFG822740)andMOBSTECO(FWFP26194-N15).

References[1]SchieferdeckerI.,GrossmannJ.,SchneiderM.Model-basedsecuritytesting.In:Proceedings7thWorkshoponModel-BasedTesting.2012.

[2]ISO/IEC.ISO/IEC9126-1:2001softwareengineering—productquality—Part1:qualitymodel.2001.

[3]ISO/IEC.ISO/IEC25010:2011systemsandsoftwareengineering—systemsandsoftwarequalityrequirementsandevaluation(SQuaRE)—systemandsoftwarequalitymodels.2011.

[4]BauJ.,BurszteinE.,GuptaD.,MitchellJ.Stateoftheart:automatedblack-boxwebapplicationvulnerabilitytesting.In:2010IEEESymposiumonSecurityandPrivacy(SP).IEEE;2010:332–345.

[5]MITRE,Commonvulnerabilitiesandexposures,http://cve.mitre.org.

[6]NIST.Theeconomicimpactsofinadequateinfrastructureforsoftwaretesting.2002(availableatwww.nist.gov/director/planning/upload/report02-3.pdf[accessedApril7,2015]).

[7]BourqueP.,DupuisR.,eds.GuidetotheSoftwareEngineeringBodyofKnowledgeVersion3.0SWEBOK.IEEE;2014.http://www.computer.org/web/swebok.

[8]AmmannP.,OffuttJ.IntroductiontoSoftwareTesting.Cambridge,UK:CambridgeUniversityPress;2008.

[9]ISTQB.Standardglossaryoftermsusedinsoftwaretesting.ISTQB;2012Version2.2,Tech.Rep.

[10]OWASPFoundation,OWASPTestingGuidev4,https://www.owasp.org/index.php/OWASP_Testing_Project(accessedMarch11,2015).

[11]Tian-yangG.,Yin-shengS.,You-yuanF.Researchonsoftwaresecuritytesting.WorldAcad.Sci.Eng.Technol.2010;69:647–651.

[12]BachmannR.,BruckerA.D.Developingsecuresoftware:aholisticapproachtosecuritytesting.DatenschutzundDatensicherheit(DuD).2014;38(4):257–261.

[13]ISO/IEC.Informationtechnology—opensystemsinterconnection—conformancetestingmethodologyandframework.1994(internationalISO/IECmulti-partstandardNo.9646).

[14]UttingM.,LegeardB.PracticalModel-BasedTesting:AToolsApproach.SanFrancisco,CA:MorganKaufmannPublishersInc.2007.0123725011.

[15]ZanderJ.,SchieferdeckerI.,MostermanP.J.Model-BasedTestingforEmbeddedSystems.CRCPress;2012;vol.13.

[16]IEEE.IEEEstandardglossaryofsoftwareengineeringterminology.Washington,

DC:InstituteofElectronicalandElectronicsEngineers;1990(IEEEStd610121990).

[17]ISO.ISO/IEC/IEEE29119softwaretesting.2013(availableathttp://www.softwaretestingstandard.org/[accessedApril7,2015]).

[18]IEEE.IEEEstandardforsoftwareandsystemtestdocumentation.2008(IEEEStd829-2008).

[19]SchieferdeckerI.Model-basedtesting.IEEESoftw.2012;29(1):14–18.

[20]CommitteeonNationalSecuritySystems.NationalInformationAssuranceGlossary.Tech.Rep.2010;4009:CommitteeonNationalSecuritySystems.

[21]ArkinB.,StenderS.,McGrawG.Softwarepenetrationtesting.IEEESecur.Priv.2005;3(1):84–87.

[22]PotterB.,McGrawG.Softwaresecuritytesting.IEEESecur.Priv.2004;2(5):81–85.

[23]HerzogP.Theopensourcesecuritytestingmethodologymanual3.2010.http://www.isecom.org/research/osstmm.html(accessedApril11,2015).

[24]LundM.S.,SolhaugB.,StolenK.Model-DrivenRiskAnalysis.Springer;2011.

[25]VerdonD.,McGrawG.Riskanalysisinsoftwaredesign.IEEESecur.Priv.2004;2(4):79–84.

[26]HowardM.,LipnerS.TheSecurityDevelopmentLifecycle:SDL:AProcessforDevelopingDemonstrablyMoreSecureSoftware.MicrosoftPress;2006.0735622140.

[27]OWASP,OpepSAMM,http://www.opensamm.org/(accessedMarch30,2015).

[28]PotterB.,McGrawG.Softwaresecuritytesting.IEEESecur.Priv.2004;2(5):81–85.

[29]ShahriarH.,ZulkernineM.Automatictestingofprogramsecurityvulnerabilities.In:33rdAnnualIEEEInternationalComputerSoftwareandApplicationsConference,2009,COMPSAC’09.IEEE;550–555.2009;vol.2.

[30]ThompsonH.H.Whysecuritytestingishard.IEEESecur.Priv.2003;1(4):83–86.

[31]OWASPFoundation,OWASPCodeReviewGuidev1.1,https://www.owasp.org/index.php/Category:OWASP_Code_Review_Project(accessedMarch11,2015).

[32]Dias-NetoA.C.,TravassosG.H.Apicturefromthemodel-basedtestingarea:concepts,techniques,andchallenges.Adv.Comput.2010;80:45–120.

[33]UttingM.,PretschnerA.,LegeardB.Ataxonomyofmodel-basedtestingapproaches.Softw.Test.Verif.Reliab.2012;22(2):297312.

[34]GrieskampW.,KicillofN.,StobieK.,BrabermanV.Model-basedqualityassuranceofprotocoldocumentation:toolsandmethodology.Softw.Test.Verif.

Reliab.2011;21(1):55–71.

[35]PretschnerA.Defect-basedtesting.In:IOSPress;2015:.DependableSoftwareSystemsEngineering.http://www.iospress.nl/book/dependable-software-systems-engineering/.

[36]ZhuH.,HallP.A.V.,MayJ.H.R.Softwareunittestcoverageandadequacy.ACMComput.Surv.0360-03001997;29(4):366–427.

[37]MorellL.J.Atheoryoffault-basedtesting.IEEETrans.Softw.Eng.0098-55891990;16(8):844–857.

[38]PretschnerA.,HollingD.,EschbachR.,GemmarM.Agenericfaultmodelforqualityassurance.In:MoreiraA.,SchätzB.,GrayJ.,VallecilloA.,ClarkeP.,eds.Model-DrivenEngineeringLanguagesandSystems.Berlin:Springer;978-3-642-41532-687–103.LectureNotesinComputerScience.2013;vol.8107.

[39]FeldererM.,AgreiterB.,ZechP.,BreuR.Aclassificationformodel-basedsecuritytesting.In:TheThirdInternationalConferenceonAdvancesinSystemTestingandValidationLifecycle(VALID2011).2011:109–114.

[40]SchieferdeckerI.,GrossmannJ.,SchneiderM.Model-basedsecuritytesting.In:Proceedings7thWorkshoponModel-BasedTesting.2012.

[41]BüchlerM.,OudinetJ.,PretschnerA.Semi-automaticsecuritytestingofwebapplicationsfromasecuremodel.In:2012IEEESixthInternationalConferenceonSoftwareSecurityandReliability(SERE).IEEE;2012:253–262.

[42]MouelhiT.,FleureyF.,BaudryB.,TraonY.Amodel-basedframeworkforsecuritypolicyspecification,deploymentandtesting.In:Proceedingsofthe11thInternationalConferenceonModelDrivenEngineeringLanguagesandSystems,MoDELS’08,Toulouse,France.Berlin:Springer;2008:978-3-540-87874-2537–552.

[43]GerrardP.,ThompsonN.Risk-Basede-BusinessTesting.ArtechHousePublishers;2002.

[44]FeldererM.,SchieferdeckerI.Ataxonomyofrisk-basedtesting.Int.J.Softw.ToolsTechnol.Transf.2014;16(5):559–568.

[45]WendlandM.-F.,KranzM.,SchieferdeckerI.Asystematicapproachtorisk-basedtestingusingrisk-annotatedrequirementsmodels.In:TheSeventhInternationalConferenceonSoftwareEngineeringAdvances,ICSEA2012.2012:636–642.

[46]GrossmannJ.,SchneiderM.,ViehmannJ.,WendlandM.-F.Combiningriskanalysisandsecuritytesting.In:Springer;2014:322–336.LeveragingApplicationsofFormalMethods,VerificationandValidation.SpecializedTechniquesandApplications.

[47]BotellaJ.,LegeardB.,PeureuxF.,VernotteA.Risk-basedvulnerabilitytestingusingsecuritytestpatterns.In:Springer;2014:337–352.LeveragingApplications

ofFormalMethods,VerificationandValidation.SpecializedTechniquesandApplications.

[48]ZechP.,FeldererM.,KattB.,BreuR.Securitytestgenerationbyanswersetprogramming.In:2014EighthInternationalConferenceonSoftwareSecurityandReliability.IEEE;2014:88–97.

[49]ZechP.,FeldererM.,BreuR.Securityriskanalysisbylogicprogramming.In:Springer;2014:38–48.RiskAssessmentandRisk-DrivenTesting.

[50]GallaherM.P.,KroppB.M.Theeconomicimpactsofinadequateinfrastructureforsoftwaretesting.NationalInstituteofStandards&Technology;2002Tech.Rep.PlanningReport02-03.

[51]ChessB.,WestJ.SecureProgrammingwithStaticAnalysis.firsted.Boston,MA:Addison-WesleyProfessional;2007.9780321424778.

[52]BesseyA.,BlockK.,ChelfB.,ChouA.,FultonB.,HallemS.,Henri-GrosC.,KamskyA.,McPeakS.,EnglerD.Afewbillionlinesofcodelater:usingstaticanalysistofindbugsintherealworld.Commun.ACM.0001-07822010;53:66–75.

[53]BruckerA.D.,SodanU.Deployingstaticapplicationsecuritytestingonalargescale.In:KatzenbeisserS.,LotzV.,WeipplE.,eds.GISicherheit2014,LectureNotesinInformatics.GI;978-3-88579-622-091–101.2014;vol.228.

[54]AyewahN.,HovemeyerD.,MorgenthalerJ.D.,PenixJ.,PughW.Experiencesusingstaticanalysistofindbugs.IEEESoftw.2008;25:22–29(specialissueonsoftwaredevelopmenttools,September/October(25:5)).

[55]LivshitsV.B.,LamM.S.Findingsecurityerrorsinjavaprogramswithstaticanalysis.In:Proceedingsofthe14thUsenixSecuritySymposium.2005:271–286.

[56]EvansD.Staticdetectionofdynamicmemoryerrors.SIGPLANNot.0362-13401996;31:44–53.

[57]ChessB.,McGrawG.Staticanalysisforsecurity.IEEESecur.Priv.1540-79932004;2(6):76–79.

[58]PistoiaM.,ChandraS.,FinkS.J.,YahavE.Asurveyofstaticanalysismethodsforidentifyingsecurityvulnerabilitiesinsoftwaresystems.IBMSyst.J.0018-86702007;46(2):265–288.

[59]ScandariatoR.,WaldenJ.,JoosenW.Staticanalysisversuspenetrationtesting:acontrolledexperiment.In:Proceedingsofthe24thIEEEInternationalSymposiumonSoftwareReliabilityEngineering.IEEE;2013:1–10.

[60]ScarfoneK.,SouppayaM.,CodyA.,OrebaughA.Technicalguidetoinformationsecuritytestingandassessment.NationalInstituteofStandardsandTechnology(NIST);2008SpecialPublication800-115.

[61]DoupeA.,CovaM.,VignaG.WhyJohnnycan’tpentest:ananalysisofblack-

boxWebvulnerabilityscanners.In:DIMVA2010.2010.

[62]M.Johns,Codeinjectionvulnerabilitiesinwebapplications—Exemplifiedatcross-sitescripting,Ph.D.thesis,UniversityofPassau,2009.

[63]HalfondW.G.,ViegasJ.,OrsoA.AclassificationofSQL-injectionattacksandcountermeasures.In:ProceedingsoftheIEEEInternationalSymposiumonSecureSoftwareEngineering.IEEE;2006:65–81.

[64]GrossmanJ.,HansenR.,PetkovP.,RagerA.CrossSiteScriptingAttacks:XSSExploitsandDefense.Syngress;2007(SethFogie).

[65]MillerB.P.,FredriksenL.,SoB.AnempiricalstudyofthereliabilityofUNIXutilities.Commun.ACM.0001-07821990;33(12):32–44.

[66]RawatS.,MounierL.Offset-awaremutationbasedfuzzingforbufferoverflowvulnerabilities:fewpreliminaryresults.In:FourthInternationalIEEEConferenceonSoftwareTesting,VerificationandValidation,ICST2012,Berlin,Germany,21-25March,2011WorkshopProceedings.2011:531–533.

[67]DucheneF.,RawatS.,RichierJ.,GrozR.KameleonFuzz:evolutionaryfuzzingforblack-boxXSSdetection.In:FourthACMConferenceonDataandApplicationSecurityandPrivacy,CODASPY’14,SanAntonio,TX,USA,March03-05,2014.2014:37–48.

[68]YangD.,ZhangY.,LiuQ.BlendFuzz:amodel-basedframeworkforfuzztestingprogramswithgrammaticalinputs.In:11thIEEEInternationalConferenceonTrust,SecurityandPrivacyinComputingandCommunications,TrustCom2012,Liverpool,UnitedKingdom,June25-27,2012.2012:1070–1076.

[69]ZhaoJ.,WenY.,ZhaoG.H-Fuzzing:anewheuristicmethodforfuzzingdatageneration.In:Proceedingsofthe8thIFIPInternationalConferenceNetworkandParallelComputingNPC2011,Changsha,China,October21-23,2011.2011:32–43.

[70]GodefroidP.,LevinM.Y.,MolnarD.A.Automatedwhiteboxfuzztesting.In:ProceedingsoftheNetworkandDistributedSystemSecuritySymposium,NDSS2008,SanDiego,California,USA,10thFebruary-13thFebruary2008.TheInternetSociety;2008.

[71]GodefroidP.,LevinM.Y.,MolnarD.A.SAGE:whiteboxfuzzingforsecuritytesting.Commun.ACM.2012;55(3):40–44.

[72]BekrarS.,BekrarC.,GrozR.,MounierL.Ataintbasedapproachforsmartfuzzing.In:2012IEEEFifthInternationalConferenceonSoftwareTesting,VerificationandValidation,Montreal,QC,Canada,April17-21,2012.2012:818–825.

[73]AbdelnurH.J.,StateR.,FestorO.AdvancedfuzzingintheVoIPspace.J.Comput.Virol.2010;6(1):57–64.

[74]LehmanM.M.Software’sfuture:managingevolution.IEEESoftw.

1998;15(1):40–44.

[75]FeldererM.,KattB.,KalbP.,JürjensJ.,OchoaM.,PaciF.,TranL.M.S.,TunT.T.,YskoutK.,ScandariatoR.,PiessensF.,VanoverbergheD.,FourneretE.,GanderM.,SolhaugB.,BreuR.Evolutionofsecurityengineeringartifacts:astateoftheartsurvey.Int.J.Secur.Softw.Eng.2014;5(4):48–97.

[76]FeldererM.,FourneretE.Asystematicclassificationofsecurityregressiontestingapproaches.Int.J.Softw.ToolsTechnol.Transf.2015;1–15.

[77]LeungH.K.N.,WhiteL.Insightsintoregressiontesting(softwaretesting).In:ProceedingsConferenceonSoftwareMaintenance1989.IEEE;1989:60–69.

[78]YooS.,HarmanM.Regressiontestingminimisation,selectionandprioritisation:asurvey.Softw.Test.Verif.Reliab.2010;1(1):121–141.

[79]FourneretE.,CantenotJ.,BouquetF.,LegeardB.,BotellaJ.SeTGaM:generalizedtechniqueforregressiontestingbasedonUML/OCLmodels.In:2014EighthInternationalConferenceonSoftwareSecurityandReliability(SERE).2014:147–156.

[80]TóthG.,KőszegiG.,HornákZ.Casestudy:automatedsecuritytestingonthetrustedcomputingplatform.In:Proceedingsofthe1stEuropeanWorkshoponSystemSecurity,EUROSEC’08,Glasgow,Scotland.ACM;2008:978-1-60558-119-435–39.

[81]HeT.,JingX.,KunmeiL.,YingZ.Researchonstrong-associationrulebasedwebapplicationvulnerabilitydetection.In:2ndIEEEInternationalConferenceonComputerScienceandInformationTechnology,2009,ICCSIT2009.2009:237–241.

[82]GarvinB.J.,CohenM.B.,DwyerM.B.Usingfeaturelocality:canweleveragehistorytoavoidfailuresduringreconfiguration?In:Proceedingsofthe8thWorkshoponAssurancesforSelf-adaptiveSystems,ASAS’11,Szeged,Hungary.ACM;2011:978-1-4503-0853-324–33.

[83]HuangY.-C.,PengK.-L.,HuangC.-Y.Ahistory-basedcost-cognizanttestcaseprioritizationtechniqueinregressiontesting.J.Syst.Softw.0164-12122012;85(3):626–637.http://www.sciencedirect.com/science/article/pii/S0164121211002780(novelapproachesinthedesignandimplementationofsystems/softwarearchitecture).

[84]YuY.T.,LauM.F.Fault-basedtestsuiteprioritizationforspecification-basedtesting.Inf.Softw.Technol.2012;54(2):179–202.http://www.sciencedirect.com/science/article/pii/S0950584911001947.

[85]ViennotN.,NairS.,NiehJ.Transparentmutablereplayformulticoredebuggingandpatchvalidation.In:ProceedingsoftheEighteenthInternationalConferenceonArchitecturalSupportforProgrammingLanguagesandOperatingSystems,ASPLOS’13,Houston,Texas,USA.ACM;2013:127–138.

[86]FeldererM.,AgreiterB.,BreuR.Evolutionofsecurityrequirementstestsforservice-centricsystems.In:EngineeringSecureSoftwareandSystems:ThirdInternationalSymposium,ESSoS2011.Springer;2011:181–194.

[87]KassabM.,OrmandjievaO.,DanevaM.Relational-modelbasedchangemanagementfornon-functionalrequirements:approachandexperiment.In:2011FifthInternationalConferenceonResearchChallengesinInformationScience(RCIS).2011:1–9.

[88]AnisettiM.,ArdagnaC.A.,DamianiE.Alow-costsecuritycertificationschemeforevolvingservices.In:2012IEEE19thInternationalConferenceonWebServices(ICWS).2012:122–129.

[89]HuangC.,SunJ.,WangX.,SiY.SelectiveregressiontestforaccesscontrolsystememployingRBAC.In:ParkJ.H.,ChenH.-H.,AtiquzzamanM.,LeeC.,KimT.-h.,YeoS.-S.,eds.AdvancesinInformationSecurityandAssurance.Berlin:Springer;978-3-642-02616-470–79.LectureNotesinComputerScience.2009;vol.5576.

[90]HwangJ.,XieT.,ElKatebD.,MouelhiT.,LeTraonY.Selectionofregressionsystemtestsforsecuritypolicyevolution.In:Proceedingsofthe27thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering.ACM;2012:266–269.

[91]VetterlingM.,WimmelG.,WisspeintnerA.Securesystemsdevelopmentbasedonthecommoncriteria:thePalMEproject.In:Proceedingsofthe10thACMSIGSOFTSymposiumonFoundationsofSoftwareEngineering,SIGSOFT’02/FSE-10,Charleston,SouthCarolina,USA.ACM;2002:129–138.

[92]BrunoM.,CanforaG.,PentaM.,EspositoG.,MazzaV.Usingtestcasesascontracttoensureservicecomplianceacrossreleases.In:BenatallahB.,CasatiF.,TraversoP.,eds.Service-OrientedComputing—ICSOC2005.Berlin:Springer;978-3-540-30817-187–100.LectureNotesinComputerScience.2005;vol.3826.

[93]KongsliV.Towardsagilesecurityinwebapplications.In:Companiontothe21stACMSIGPLANSymposiumonObject-OrientedProgrammingSystems,Languages,andApplications,OOPSLA’06,Portland,Oregon,USA.NewYork,NY:ACM;2006:1-59593-491-X805–808.

[94]QiD.,RoychoudhuryA.,LiangZ.,VaswaniK.DARWIN:anapproachtodebuggingevolvingprograms.ACMTrans.Softw.Eng.Methodol.2012;21(3):19:1–19:29.

[95]ISO/IEC.ISO/IEC15408-1:2009informationtechnology—securitytechniques—evaluationcriteriaforITsecurity—part1:introductionandgeneralmodel.2009.

[96]BjørnerN.,JayaramanK.CheckingcloudcontractsinMicrosoftAzure.In:Proceedingsofthe11thInternationalConferenceDistributedComputingandInternetTechnologyICDCIT2015,Bhubaneswar,India,February5-8,2015.2015:21–32.

[97]BruckerA.D.,BrüggerL.,WolffB.FormalFirewallConformanceTesting:AnApplicationofTestandProofTechniques.Softw.Test.Verif.Reliab.2015;25(1):34–71.

[98]HuangY.-W.,YuF.,HangC.,TsaiC.-H.,LeeD.-T.,KuoS.-Y.Securingwebapplicationcodebystaticanalysisandruntimeprotection.In:InternationalConferenceontheWorldWideWeb(WWW),WWW’04,NewYork,NY,USA.NewYork,NY:ACM;2004:1-58113-844-X40–52.

[99]FosterJ.S.,FähndrichM.,AikenA.Atheoryoftypequalifiers.SIGPLANNot.1999;34(5).

[100]FosterJ.S.,TerauchiT.,AikenA.Flow-sensitivetypequalifiers.SIGPLANNot.2002;37(5).

[101]JovanovicN.,KruegelC.,KirdaE.Pixy:astaticanalysistoolfordetectingwebapplicationvulnerabilities(shortpaper).In:IEEESymposiumonSecurityandPrivacy,SP’06.Washington,DC:IEEEComputerSociety;2006:0-7695-2574-1258–263.

[102]JovanovicN.,KruegelC.,KirdaE.Precisealiasanalysisforstaticdetectionofwebapplicationvulnerabilities.In:WorkshoponProgramminglanguagesandanalysisforsecurity,PLAS’06,Ottawa,Ontario,Canada.NewYork,NY:ACM;2006:1-59593-374-327–36.

[103]XieY.,AikenA.Staticdetectionofsecurityvulnerabilitiesinscriptinglanguages.In:USENIXSecuritySymposium.179–192.2006;vol.15.

[104]DahseJ.,HolzT.Staticdetectionofsecond-ordervulnerabilitiesinwebapplications.In:Proceedingsofthe23rdUSENIXSecuritySymposium.2014.

[105]DahseJ.,HolzT.Simulationofbuilt-inPHPfeaturesforprecisestaticcodeanalysis.In:ISOC-NDSS.2014.

[106]WassermannG.,SuZ.Staticdetectionofcross-sitescriptingvulnerabilities.In:ICSE’08,Leipzig,Germany.NewYork,NY:ACM;2008:978-1-60558-079-1171–180.

[107]MinamideY.Staticapproximationofdynamicallygeneratedwebpages.In:InternationalConferenceontheWorldWideWeb(WWW).2005.

[108]SaxenaP.,AkhaweD.,HannaS.,MaoF.,McCamantS.,SongD.AsymbolicexecutionframeworkforjavaScript.In:IEEESymposiumonSecurityandPrivacy,SP’10.Washington,DC:IEEEComputerSociety;2010:978-0-7695-4035-1513–528.

[109]JinX.,HuX.,YingK.,DuW.,YinH.,PeriG.N.CodeinjectionattacksonHTML5-basedmobileapps:characterization,detectionandmitigation.In:21stACMConferenceonComputerandCommunicationsSecurity(CCS).2014.

[110]FuX.,LuX.,PeltsvergerB.,ChenS.,QianK.,TaoL.AstaticanalysisframeworkfordetectingSQLinjectionvulnerabilities.In:31stAnnual

InternationalComputerSoftwareandApplicationsConference,2007,COMPSAC2007.IEEE;87–96.2007;vol.1.

[111]WassermannG.,SuZ.Soundandpreciseanalysisofwebapplicationsforinjectionvulnerabilities.In:ProceedingsofProgrammingLanguageDesignandImplementation(PLDI’07),SanDiego,CA.2007.

[112]HalfondW.G.J.,OrsoA.,ManoliosP.Usingpositivetaintingandsyntax-awareevaluationtocounterSQLinjectionattacks.In:Proceedingsofthe14thACMSIGSOFTInternationalSymposiumonFoundationsofSoftwareEngineering.ACM;2006:175–185.

[113]KalsS.,KirdaE.,KruegelC.,JovanovicN.SecuBat:awebvulnerabilityscanner.In:InternationalConferenceontheWorldWideWeb(WWW),WWW’06,Edinburgh,Scotland.NewYork,NY:ACM;2006:1-59593-323-9247–256.

[114]d’AmoreF.,GentileM.Automaticandcontext-awarecross-sitescriptingfilterevasion.DepartmentofComputer,Control,andManagementEngineeringAntonioRubertiTechnicalReports;2012Tech.Rep.4.

[115]AckerS.V.,NikiforakisN.,DesmetL.,JoosenW.,PiessensF.FlashOver:automateddiscoveryofcross-sitescriptingvulnerabilitiesinrichinternetapplications.In:ASIACCS.2012.

[116]WallL.,ChristiansenT.,OrwantJ.ProgrammingPerl.thirded.Sebastopol,CA:O’Reilly;2000.

[117]SchwartzE.J.,AvgerinosT.,BrumleyD.Allyoueverwantedtoknowaboutdynamictaintanalysisandforwardsymbolicexecution(butmighthavebeenafraidtoask).In:IEEESymposiumonSecurityandPrivacy.2010.

[118]Nguyen-TuongA.,GuarnieriS.,GreeneD.,ShirleyJ.,EvansD.Automaticallyhardeningwebapplicationsusingprecisetainting.In:20thIFIPInternationalInformationSecurityConference.2005.

[119]PietraszekT.,BergheC.V.Defendingagainstinjectionattacksthroughcontext-sensitivestringevaluation.In:RecentAdvancesinIntrusionDetection(RAID2005).2005.

[120]SuZ.,WassermannG.Theessenceofcommandinjectionattacksinwebapplications.In:ProceedingsofPOPL’06.2006.

[121]MartinM.,LamM.S.AutomaticgenerationofXSSandSQLinjectionattackswithgoal-directedmodelchecking.In:USENIXSecuritySymposium,SEC’08,SanJose,CA.Berkeley,CA:USENIXAssociation;2008:31–43.

[122]KieyzunA.,GuoP.J.,JayaramanK.,ErnstM.D.AutomaticcreationofSQLinjectionandcross-sitescriptingattacks.In:ICSE’09.Washington,DC:IEEEComputerSociety;2009:978-1-4244-3453-4199–209.

[123]KleinA.DOMBasedCrossSiteScriptingorXSSoftheThirdKind.2005.http://www.webappsec.org/projects/articles/071105.shtml.

[124]LekiesS.,StockB.,JohnsM.25millionflowslater-large-scaledetectionofDOM-basedXSS.In:ACMConferenceonComputerandCommunicationsSecurity(CCS).2013.

[125]SaxenaP.,HannaS.,PoosankamP.,SongD.FLAX:systematicdiscoveryofclient-sidevalidationvulnerabilitiesinrichwebapplications.In:ISOC-NDSS.TheInternetSociety;2010.

[126]McGrawG.SoftwareSecurity:BuildingSecurityIn.Addison-WesleyProfessional;2006.0321356705.

[127]TrippO.,PistoiaM.,FinkS.J.,SridharanM.,WeismanO.TAJ:effectivetaintanalysisofwebapplications.SIGPLANNot.0362-13402009;44:87–97.

[128]MonateB.,SignolesJ.Slicingforsecurityofcode.In:TRUST.2008:133–142.

[129]WALA,T.J.WatsonLibrariesforAnalysis,http://wala.sf.net.

[130]HubertL.,BarréN.,BessonF.,DemangeD.,JensenT.P.,MonfortV.,PichardieD.,TurpinT.Sawja:staticanalysisworkshopforJava.In:FoVeOOS.2010:92–106.

[131]HallerI.,SlowinskaA.,NeugschwandtnerM.,BosH.Dowsingforoverflows:aguidedfuzzertofindbufferboundaryviolations.In:Proceedingsofthe22ndUSENIXConferenceonSecurity,SEC’13,Washington,DC.Berkeley,CA:USENIXAssociation;2013:978-1-931971-03-449–64.

[132]MazzoneS.B.,PagnozziM.,FattoriA.,ReinaA.,LanziA.,BruschiD.ImprovingMacOSXsecuritythroughgrayboxfuzzingtechnique.In:ProceedingsoftheSeventhEuropeanWorkshoponSystemSecurity,EuroSec’14,Amsterdam,TheNetherlands.NewYork,NY:ACM;2014:978-1-4503-2715-22:1–2:6.

[133]WooM.,ChaS.K.,GottliebS.,BrumleyD.Schedulingblack-boxmutationalfuzzing.In:Proceedingsofthe2013ACMSIGSACConferenceonComputer&CommunicationsSecurity,CCS’13,Berlin,Germany.NewYork,NY:ACM;2013:978-1-4503-2477-9511–522.

[134]LanziA.,MartignoniL.,MongaM.,PaleariR.Asmartfuzzerforx86executables.In:ThirdInternationalWorkshoponSoftwareEngineeringforSecureSystems,2007,SESS’07:ICSEWorkshops2007.2007:77.

[135]BuehrerG.,WeideB.W.,SivilottiP.A.G.UsingparsetreevalidationtopreventSQLinjectionattacks.In:Proceedingsofthe5thInternationalWorkshoponSoftwareEngineeringandMiddleware,SEM’05,Lisbon,Portugal.NewYork,NY:ACM;2005:1-59593-205-4106–113.

[136]AppeltD.,NguyenC.D.,BriandL.C.,AlshahwanN.AutomatedtestingforSQLinjectionvulnerabilities:aninputmutationapproach.In:Proceedingsofthe2014InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2014,SanJose,CA,USA.NewYork,NY:ACM;2014:978-1-4503-2645-2259–269.

[137]WangJ.,ZhangP.,ZhangL.,ZhuH.,YeX.Amodel-basedfuzzingapproachforDBMS.In:20138thInternationalConferenceonCommunicationsandNetworkinginChina(CHINACOM).LosAlamitos,CA:IEEEComputerSociety;2013:426–431.

[138]GarciaR.Casestudy:experiencesonSQLlanguagefuzztesting.In:ProceedingsoftheSecondInternationalWorkshoponTestingDatabaseSystems,DBTest’09,Providence,RhodeIsland.NewYork,NY:ACM;2009:978-1-60558-706-63:1–3:6.

[139]BrubakerC.,JanaS.,RayB.,KhurshidS.,ShmatikovV.UsingfrankencertsforautomatedadversarialtestingofcertificatevalidationinSSL/TLSimplementations.In:Proceedingsofthe2014IEEESymposiumonSecurityandPrivacy,SP’14.Washington,DC:IEEEComputerSociety;2014:978-1-4799-4686-0114–129.

[140]BanksG.,CovaM.,FelmetsgerV.,AlmerothK.C.,KemmererR.A.,VignaG.SNOOZE:towardastatefulnetwOrkprOtocolfuzZEr.In:Proceedingsofthe9thInternationalConferenceInformationSecurityISC2006,SamosIsland,Greece,August30-September2,2006.2006:343–358.

[141]TsankovP.,DashtiM.T.,BasinD.A.SECFUZZ:fuzz-testingsecurityprotocols.In:7thInternationalWorkshoponAutomationofSoftwareTest,AST2012,Zurich,Switzerland,June2-3,2012.2012:1–7.

[142]BertolinoA.,TraonY.L.,LonettiF.,MarchettiE.,MouelhiT.Coverage-basedtestcasesselectionforXACMLpolicies.In:2014IEEESeventhInternationalConferenceonSoftwareTesting,VerificationandValidation,WorkshopsProceedings,March31-April4,2014,Cleveland,Ohio,USA.2014:12–21.

[143]BruckerA.D.,BrüggerL.,KearneyP.,WolffB.Anapproachtomodularandtestablesecuritymodelsofreal-worldhealth-careapplications.In:ACMsymposiumonAccessControlModelsandTechnologies(SACMAT),Innsbruck,Austria.NewYork,NY:ACMPress;2011:978-1-4503-0688-1133–142.

[144]MartinE.Testingandanalysisofaccesscontrolpolicies.In:29thInternationalConferenceonSoftwareEngineering—Companion,2007,ICSE2007.2007:75–76.

[145]RogersR.,RogersR.NessusNetworkAuditing.seconded.Burlington,MA:SyngressPublishing;2008ISBN9780080558653,9781597492089.

MichaelFeldererisaseniorresearcherandprojectmanagerwithintheQualityEngineeringresearchgroupattheInstituteofComputerScienceattheUniversityofInnsbruck,Austria.HeholdsaPh.D.andahabilitationincomputerscience.Hisresearchinterestsincludesoftwareandsecuritytesting,empiricalsoftwareandsecurityengineering,modelengineering,riskmanagement,softwareprocesses,andindustry-academiacollaboration.MichaelFeldererhascoauthoredmorethan70journal,conference,andworkshoppapers.Heworksinclosecooperationwithindustryandalsotransfershisresearchresultsintopracticeasaconsultantandspeakeronindustrialconferences.

MatthiasBüchlerisaPh.D.studentattheTechnischeUniversitätMünchen.Heholdsamaster’sdegreeincomputerscience(InformationSecurity)fromtheSwissFederalInstituteofTechnologyZurich(ETHZ).Hisresearchinterestsincludeinformationsecurity,securitymodeling,securityengineering,securitytesting,domainspecificlanguages,andusagecontrol.

MartinJohnsisaresearchexpertintheProductSecurityResearchunitwithinSAPSE,whereheleadstheWebapplicationsecurityteam.Furthermore,heservesontheboardoftheGermanOWASPchapter.BeforejoiningSAP,MartinstudiedMathematicsandComputerScienceattheUniversitiesofHamburg,SantaCruz(CA),andPassau.Duringthe1990sandtheearlyyearsofthenewmillenniumheearnedhislivingasasoftwareengineerinGermancompanies(includingInfoseekGermany,andTCTrustcenter).HeholdsaDiplomainComputerSciencefromUniversityofHamburgandaDoctoratefromtheUniversityofPassau.

AchimD.Bruckerisaresearchexpert(architect),securitytestingstrategist,andprojectleadintheSecurityEnablementTeamofSAPSE.Hereceivedhismaster’sdegreeincomputersciencefromUniversityFreiburg,GermanyandhisPh.D.fromETHZurich,Switzerland.HeisresponsiblefortheSecurityTestingStrategyatSAP.Hisresearchinterestsincludeinformationsecurity,softwareengineering,securityengineering,andformalmethods.Inparticular,heisinterestedintoolsandmethodsformodeling,building,andvalidatingsecureandreliablesystems.HealsoparticipatesintheOCLstandardizationprocessoftheOMG.

RuthBreuisheadoftheInstituteofComputerScienceattheUniversityofInnsbruck,leadingtheresearchgroupQualityEngineeringandthecompetencecenterQELaB.Shehaslongstandingexperienceintheareasofsecurityengineering,requirementsengineering,enterprisearchitecturemanagementandmodelengineering,bothwithacademicandindustrialbackground.Ruthiscoauthorofthreemonographsandmorethan150scientificpublicationsandservesthescientificcommunityinavarietyoffunctions(egBoardMemberofFWF,theAustrianScienceFund,MemberoftheNISPlatformoftheEuropeanCommission).

AlexanderPretschnerholdsthechairofSoftwareEngineeringatTechnischeUniversitätMünchen.Researchinterestsincludesoftwarequality,testing,andinformationsecurity.Master’sdegreesincomputersciencefromRWTHAachenandtheUniversityofKansasandPh.D.degreefromTechnischeUniversitätMünchen.PriorappointmentsincludeafullprofessorshipatKarlsruheInstituteofTechnology,anadjunctassociateprofessorshipatKaiserslauternUniversityofTechnology,agroupmanager’spositionattheFraunhoferInstituteofExperimentalSoftwareEngineeringinKaiserslautern,aseniorscientist’spositionatETHZurich,andvisitingprofessorshipsattheUniversitiesofRennes,Trento,andInnsbruck.

CHAPTERTWO

RecentAdvancesinModel-BasedTestingMarkUtting*;BrunoLegeard†,‡;FabriceBouquet†;ElizabetaFourneret†;FabienPeureux†,‡;AlexandreVernotte†*UniversityoftheSunshineCoast,QLD,Australia†InstitutFEMTO-ST,UMRCNRS6174,Besançon,France‡SmartestingSolutions&Services,Besançon,France

AbstractThischaptergivesanoverviewofthefieldofmodel-basedtesting(MBT),particularlytherecentadvancesinthelastdecade.ItgivesasummaryoftheMBTprocess,themodelinglanguagesthatarecurrentlyusedbythevariouscommunitieswhopracticeMBT,thetechnologiesusedtogeneratetestsfrommodels,anddiscussesbestpractices,suchastraceabilitybetweenmodelsandtests.ItalsobrieflydescribesseveralfindingsfromarecentsurveyofMBTusersinindustry,outlinestheincreasinglypopularuseofMBTforsecuritytesting,anddiscussesfuturechallengesforMBT.

KeywordsModel-basedtesting;Modelingapproaches;TestgenerationTechnology;Securitytesting

1IntroductionBroadlyspeaking,model-basedtesting(MBT)isaboutdesigningtestsfromsomekindofmodelofthesystembeingtestedanditsenvironment.Inthissense,alltestdesignisbasedonsomementalmodel,socouldperhapsbecalledmodel-basedtesting.Butitiscommon,andmoreuseful,tousethetermmodel-basedtestingtoreferto:

•moreformalmodels(expressedinsomemachine-readable,well-defined,notation);

•moreformaltestgeneration(weareinterestedintestgenerationalgorithmsthatareautomatic,orarecapableofbeingautomated);

•andmoreautomatedexecution(thegeneratedtestsmustbesufficientprecisethattheyarecapableofbeingexecutedautomatically).

Testingisanimportant,butpainfulandcostly,partofthesoftwaredevelopmentlifecycle.Sothepromise,orhope,ofMBTisthatifwecanonlyobtainamodelfromsomewhere(preferablyatzerocost),thenallthosetestswillbeabletobegeneratedautomatically,andexecutedautomatically,inordertofindallthefaultsinthesystem,atgreatlyreducedcostandeffort.

Thatisobviouslyasilverbullet,adreamthatcannotbetrue.ThetruthaboutMBTliessomewherebetweenthatdream,andtheotherextreme:apessimisticdismissalthatitcouldbeofnohelpwhatsoever.ThischapteraimstoshinesomelightonthecurrentrealityofMBT,therangeofpractices,theuseofMBTinindustry,someoftherecentMBTresearchandtooladvancesthathavehappenedinthelastdecade,andnewapplicationareaswhereMBTisbeingapplied.

WefirstsetthescenewithanoverviewofMBT:theprocess,thepeople,therangeofMBTpractices,andabriefhistory.TheninSection3wediscusscurrentusageofMBT,particularlyinindustry,inSection4wediscussrecentadvancesinthelanguagesusedforthetestmodels,inSection5wereviewrecentadvancesinthetestgenerationtechnologies,inSection6wediscusstheuseofMBTforsecuritytesting,whichisarecentgrowthareaintheuseofMBT,andfinallyweconcludeanddiscussfuturechallengesforMBT.

2MBTOverviewMBTreferstotheprocessandtechniquesfortheautomaticderivationoftestcasesfrommodels,thegenerationofexecutablescripts,andthemanualorautomatedexecutionoftheresultingtestcasesortestscripts.

Therefore,thekeytenetsofMBTarethemodelingprinciplesfortestgeneration,thereusabilityofrequirementsmodels,thetestselectioncriteria,thetestgenerationstrategiesandtechniques,andthetransformationofabstracttestsintoconcreteexecutabletests.

TheessenceofMBTistobridgethedomainandproductknowledgegapbetweenthebusinessanalystsandtestengineers.Modelsareexpectedtobetruerepresentationsofbusinessrequirementsandtoassociatethoserequirementswiththedifferentstatesthattheproductwilltakeasitreceivesvariousinputs.Ideally,themodelswillcoverallofthebusinessrequirementsandwillbesufficientlycompletetothusensurenear100%functionalcoverage.

2.1MBTProcessFig.1showsatypicalMBTprocess,whichstartsfromtherequirementsphase,goesthroughmodelingfortestgeneration,andendsintestmanagementandtestautomation.Themainfourstagesofthisprocessare:(1)designingmodelsfortestgeneration;(2)selectingtestgenerationcriteria;(3)generatingthetests;andthen(4)executingthetests,eithermanuallyorautomatically.Webrieflydiscusseachofthesestages.Table1showssomecommonMBTterminology.

FIGURE1 Thetypicalmodel-basedtestingprocess.

Table1TerminologyGlossaryofModel-BasedTestingTermsFollowingISTQBSoftwareRestingGlossaryofRermsv3.0

Term DefinitionMBTmodel Anymodelusedinmodel-basedtesting.Modelcoverage Thedegree,expressedasapercentage,towhichmodelelementsareplannedtobeorhavebeenexercisedbyatestsuite.OfflineMBT Model-basedtestingapproachwherebytestcasesaregeneratedintoarepositoryforfutureexecution.OnlineMBT Model-basedtestingapproachwherebytestcasesaregeneratedandexecutedsimultaneously.Model-basedtesting

Testingbasedonorinvolvingmodels.

Testadaptionlayer

Thelayerinatestautomationarchitecturethatprovidesthenecessarycodetoadapttestscriptsonanabstractleveltothevariouscomponents,configurationorinterfacesoftheSUT.

Testmodel Amodeldescribingtestwarethatisusedfortestingacomponentorasystemundertest.Testselectioncriteria

Thecriteriausedtoguidethegenerationoftestcasesortoselecttestcasesinordertolimitthesizeofatest.

1.Designingmodelsfortestgeneration.Themodels,generallycalledMBTmodels,representtheexpectedbehaviorandsomeprocessworkflowofthesystemundertest(SUT),inthecontextofitsenvironment,atagivenabstractionlevel.Thepurposeofmodelingfortestgenerationistomakeexplicitthecontrolandobservationpointsofthesystem,theexpecteddynamicbehaviororworkflowstobetested,theequivalenceclassesofsystemstates,andthelogicaltestdata.Themodelelementsandtherequirementscanbelinkedinordertoensurebidirectionaltraceabilitybetweenthethreemainartifacts:therequirements,theMBTmodel,andthegeneratedtestcases.MBTmodelsmustbepreciseandcompleteenoughtoallowtheautomatedderivationoftestsfromthesemodels.

2.Selectingsometestselectioncriteria.ThereareusuallyaninfinitenumberofpossibleteststhatcanbegeneratedfromanMBTmodel,sothetestanalysthastoapplysometestselectioncriteriatoselectthepertinenttests,ortoensuresatisfactorycoverageofthesystembehaviors.Onecommonkindoftestgenerationcriteriaisbasedonstructuralmodelcoverage,usingwell-knowntestdesignstrategies[1],forinstanceequivalencepartitioning,processcyclecoverageorpairwisetesting.Anotherusefulkindofteststrategyensuresthatthegeneratedtestcasescoveralltherequirementsandbusinessprocesses,possiblywithmoretestsgeneratedforrequirementsandprocessesthathaveahigherlevelofrisk.Inthisway,MBTcanbeusedtoimplementariskandrequirements-basedtestingapproach.Forexample,foranoncriticalapplication,thetestanalystmaychoosetogeneratejustonetestforeachofthenominalbehaviorsinthemodelandeachofthemainerrorcases;butforthemorecriticalrequirements,thetestanalystcouldapplymoredemandingcoveragecriteriasuchasprocesscycletesting,toensurethatthebusinessesprocessesassociatedwiththatpartoftheMBTmodelsaremorethoroughlytested.

3.Generatingthetests.ThisisafullyautomatedprocessthatgeneratestherequirednumberoftestcasesfromtheMBTmodelsonthebasisofthetestselectioncriteriaconfiguredbythetestanalyst.EachgeneratedtestcaseistypicallyasequenceofSUTactions,withinputparametersandexpectedoutputvaluesforeachaction.Thesegeneratedtestsequencesaresimilartothetestsequencesthatwouldbedesignedmanuallyusinganaction-wordapproach[2].TheyareeasilyunderstoodbyhumansandarecompleteenoughtobedirectlyexecutedontheSUTbyamanualtester.Thepurposeofautomatedtestgenerationistogeneratefullycompleteandexecutabletests:MBTmodelsshouldmakeitpossibletocomputetheinputparametersandtheexpectedresults.Datatablesmaybeusedtolinkabstractvaluesfromthemodelwithconcretetestvalues.Tomakethegeneratedtestsexecutable,afurtherphaseautomaticallytranslateseachabstracttestcaseintoaconcrete(executable)script,usingauser-definedmappingfromabstractdatavaluestoconcreteSUTvalues,andamappingfromabstractoperationstoGUIactionsorAPIcallsoftheSUT.Forexample,ifthetestexecutionisviatheGUIoftheSUT,thentheactionwordscouldbelinkedtothegraphicalobjectmapusingatestrobot.IfthetestexecutionoftheSUTisAPIbased,thentheactionwordsneedtobeimplementedonthisAPI.Thiscouldbeadirectmappingoramorecomplexautomationlayer.TheexpectedresultsforeachabstracttestcasearetranslatedintooraclecodethatwillchecktheSUToutputsanddecideonatestpass/failverdict.ThetestsgeneratedfromMBTmodelsmaybestructuredintomultipletestsuites,andpublishedintostandardtestmanagementtools.MaintenanceofthetestrepositoryisdonebyupdatingtheMBTmodels,thenautomaticallyregeneratingandrepublishingthetestsuitesintothetestmanagementtool.

4.Executingmanuallyorautomaticallythetests.Thegeneratedtestscanbeexecutedeithermanuallyorinanautomatedtestexecutionenvironment.Eitherway,theresultisthatthetestsareexecutedontheSUT,andthattestseitherpassorfail.ThefailedtestsindicatedisparitybetweentheactualSUTresultsandtheexpectedones,asdesignedintheMBTmodels,whichthenneedtobeinvestigatedtodecidewhetherthefailureiscaused

byabugintheSUT,orbyanerrorinthemodeland/ortherequirements.ExperienceshowsthatMBTisgoodatfindingSUTerrors,butisalsohighlyeffectiveatexposingrequirementserrors[3,4],evenbeforeexecutingasingletest(thankstothemodelingphase).Inthecaseofautomatedtestexecution,testcasescanbeexecutedeitheroffline,mostcommonlyused,oronline.Withofflineexecution,thetestcasesarefirstgenerated,andtheninasecondstep,theyareexecutedonthesystemundertest.Withonlineexecution,thetestexecutionresultsinfluencethepathtakenbythetestgeneratorthroughthemodel,sotestcasegenerationandexecutionarecombinedintoonestep.

Thisprocessishighlyincrementalandhelpstomanagetestcaselifecyclewhentherequirementschange.MBTgeneratorsareabletomanagetheevolutionofthetestrepositorywithrespecttothechangeintherequirementsthathavebeenpropagatedtothetestgenerationmodel.

2.2TestRepositoryandTestManagementToolsThepurposeofgeneratingtestsfromMBTmodelsistoproducethetestrepository(seeFig.2).Thistestrepositoryistypicallymanagedbyatestmanagementtool.Thegoalofsuchatoolistohelpintheorganizingandexecutingoftestsuites(groupsoftestcases),bothformanualandautomatedtestexecution.

FIGURE2 Relationshipbetweenrequirementsandtestrepositories.

IntheMBTprocess,thetestrepositorydocumentationisfullymanagedbytheautomatedgeneration(fromMBTmodels):documentationofthetestdesignsteps,requirementstraceabilitylinks,testscriptsandassociateddocumentationareautomaticallyprovidedforeachtestcase.Therefore,themaintenanceofthetestrepositoryisdoneonlythroughthemaintenanceofMBTmodelsandthenregenerationfromthesemodels.

2.3RequirementsTraceability

AkeyelementoftheaddedvalueofMBTistheautomationofbidirectionaltraceabilitybetweenrequirementsandtestcases.Bidirectionaltraceabilityistheabilitytodeterminelinksbetweentwopartsofthesoftwaredevelopmentprocess.ThestartingpointoftheMBTprocessisthevariousfunctionaldescriptionsofthetestedapplication,suchasusecases,functionalrequirements,anddescriptionsofbusinessprocesses.Tobeeffective,requirementstraceabilityimpliesthattherequirementsrepositoryisstructuredenoughsothateachindividualrequirementcanbeuniquelyidentified.Itisdesirabletolinktheserequirementstothegeneratedtests,andtolinkeachgeneratedtesttotherequirementsthatittests.

AbestpracticeinMBT,providedbymostofthetoolsonthemarket,istolinkthemodelelementstotherelatedtestrequirements.TheselinksintheMBTmodelsenabletheautomaticgenerationandmaintenanceofatraceabilitymatrixbetweenrequirementsandtestcases.

2.4ActorsandRolesintheMBTProcessTheMBTprocessinvolvesfourmainroles(seeFig.3).

FIGURE3 Mainrolesinthemodel-basedtestingprocess.

1.Businessanalysts(orsubjectmatterexperts)arethereferencepersonsfortheSUTrequirements,businessprocessesandbusinessneeds.Theyrefinethespecificationandclarifythetestingneedsbasedontheircollaborationwiththetestanalysts.Inagileenvironments,theycontributeindefinitionanddiscussionofuserstoriesandattendsprintmeetingstomakesurethattheevolvinguserstoriesareproperlydevelopedinthemodels.TheirdomainknowledgeandexperienceallowthemtoeasilyunderstanddependenciesbetweendifferentmodulesandtheirimpactontheMBTmodelsandtoprovideusefulinputtotestanalystsduringreviewsofMBTmodels.

2.TestanalystsdesigntheMBTmodels,basedoninteractionwithcustomersandbusinessanalystsorsubjectmatterexperts.Theyusethetestgenerationtooltoautomaticallygenerateteststhatsatisfythetestobjectivesandproducearepositoryoftests.Testanalystsarealsoinchargeofreviewingthemanualtestcasesgeneratedthroughmodelsandvalidatingthecorrectnessandcoverageofthetests.

3.Testengineers(ortesters)areinchargeofmanualexecutionoftests,relyingontheavailableinformationinthetestrepository,whichisgeneratedbythetestanalystsbasedonMBTmodels.

4.Testautomationengineersareinchargeoftheautomatedexecutionoftests,bylinkingthegeneratedteststothesystemundertest.Theinputforthetestautomationengineersisthespecificationoftheadaptationlayerandactionwords,definedinthetestgenerationmodelandtobeimplemented.Thisisdeliveredbythetestanalysts.

Testanalystsareinchargeofthetestrepositoryquality,whichconcernstherequirementscoverageandthedetectionofdefects.Ontheonehand,theyinteractwiththesubjectmatterexperts,whichmakesthequalityoftheirinteractioncrucial.Ontheotherhand,thetestanalystsinteractwiththetestersinordertofacilitatemanualtestexecutionorwiththetestautomationengineerstofacilitateautomatedtestexecution(implementationofkeywords).Thisinteractionprocessishighlyiterative.

2.5CharacteristicsofMBTApproachesThissectionisadaptedfromtheUtting,Pretschner,andLegeardpaperonataxonomyofMBTapproaches[5].Generallyspeaking,taxonomiesinsoftwareengineeringhelpclarifythekeyissuesofafieldandshowthepossiblealternativesanddirections.Theycanbeusedtoclassifytoolsandtohelpuserstoseewhichapproachesandtoolsfittheirspecificneedsmoreclosely.Itisexactlywhatwearedoinghere:weareproposingsevendifferentcharacteristicsthatdescribekeyaspectsofMBT.ThesecharacteristicsfollowthemainphasesoftheMBTprocess(modeling,testgeneration,testexecution).Theyaredefinedbyconceptsthatarelargely,butnotentirely,independentofeachother:forinstance,ifaprojectisconcernedwithcombinatorialtestingofbusinessrulesmanagedbyabusinessrulesmanagementsystem,thisislikelytolimititschoiceofmodelingparadigm.IfthetargetisGUItestingofawebapplication,testselectioncriteriawilllikelybelinkedwiththecoverageoftheGUIoperations.

Fig.4givesanoverviewofsevencharacteristicsofMBTapproaches.

FIGURE4 Asimplifiedtaxonomyofmodel-basedtesting.

2.5.1Modeling:InputonlyorInput–OutputThefirstcharacteristicisthetypeofmodelsusedfortestgeneration,whichcanbereducedtoabinarydecision:dothemodelsspecifyonlytheinputstotheSUT,ordotheyspecifytheexpectedinput–outputbehavioroftheSUT?(Table2).

Table2Input–OutputModelsCharacteristics

Theinput-onlymodelshavethedisadvantagethatthegeneratedtestswillnotbeabletoactasanoracleandareincapableofverifyingthecorrectnessoftheSUTfunctionalbehavior.Inputmodelscanbeseenasmodelsoftheenvironment.Purebusinessprocessmodelsthatrepresentsomeuserworkflows(butnottheexpectedbehavior)areaprominentexample;Domainmodelingwithcombinatorialalgorithmssuchaspairwise,isanotherone.Generatedtestsfrominput-onlymodelsareincompleteandmustbemanuallycompletedbeforeexecution.

Input–outputmodelsoftheSUTnotonlymodeltheallowableinputsthatcanbesenttotheSUT,butmustalsocapturesomeoftheintendedbehavioroftheSUT.Thatis,themodelmustbeabletopredictinadvancetheexpectedoutputsoftheSUTforeachinput,oratleastbeabletocheckwhetheranoutputproducedbytheSUTisallowedbythemodelornot.Input–outputmodelsmakeitpossibletoautomaticallygeneratecompletetests,includinginputparametersandexpectedresultsforeachstep.

2.5.2Modeling:DeterministicorNondeterministicThischaracteristicrelatestothedeterminismvsnondeterminismnatureofthemodel,andhowitissupportedbythetestgenerationprocess(Table3).

Table3Determinist/NondeterministModelsCharacteristics

Nondeterminismcanoccurinthemodeland/ortheSUT.IftheSUTexhibitshazardsinthetimeorvaluedomains,thiscanoftenbehandledwhentheverdictisbuilt(whichmightbepossibleonlyafterallinputhasbeenapplied).IftheSUTexhibitsgenuinenondeterminism,asaconsequenceofconcurrency,forinstance,thenitispossiblethatteststimuliasprovidedbythemodeldependonpriorreactionsoftheSUT.Inthesecases,thenondeterminismmustbecateredforinthemodel,andalsointhetestcases(theyarenotsequencesanymore,butrathertreesorgraphs).

2.5.3Modeling:ParadigmThethirdcharacteristiciswhatparadigmandnotationareusedtodescribethemodel(Table4).Therearemanydifferentmodelingnotationsthathavebeenusedformodelingthebehaviorofsystemsfortestgenerationpurposes.Weretainhereonlythemodelingparadigmusedinthedomainofenterprisesoftware(seetheexcellentbookofPaulC.Jorgensenentitled“Modelingsoftwarebehavior—ACraftman’sApproach”tolearnmoreofthesenotations[6]:

Table4ModelingParadigmCharacteristics

•Activity-BasedNotationssuchasFlowcharts,BPMN,orUMLactivitydiagramsthatallowdefiningsequencesofactionsanddecisionsdescribingaflow.

•State-Based(orPre/Post)Notations.Thesemodelasystemasacollectionofvariables,whichrepresentasnapshotofthestateofthesystem,plussomeoperationsthatmodifythosevariables.

•Transition-BasedNotations.Thesefocusondescribingthetransitionsbetweendifferentstatesofthesystem.Typically,theyaregraphicalnode-and-arcnotations,likefinitestatemachines(FSMs),wherethenodesoftheFSMrepresentthemajorstatesofthesystemandthearcsrepresenttheactionsoroperationsofthesystem.Examplesoftransition-basednotationsusedforMBTincludeUMLStateMachinesandlabeledtransitionsystems.

•DecisionTables.Theyareusedtodescribelogicalrelationships,andareagoodtooltorepresentbusinessrulesinmodelsfortestgeneration.

•StochasticNotations.ThesedescribeasystembyaprobabilisticmodeloftheeventsandinputvaluesandtendtobeusedtomodelenvironmentsratherthanSUTs.Forexample,statisticalmodeling,eg,Markovchainsisusedtomodelexpectedusageprofiles,sothatthegeneratedtestsexercisethatusageprofile.

Inpractice,severalparadigmscanberepresentedinonesinglenotation.Forexample,theUMLnotationoffersbothatransition-basedparadigm,withstatemachinediagrams,andapre–postparadigm,withtheOCLlanguage.ThetwoparadigmscanbeusedatthesametimeinMBTmodelsandcanbecomposedwithbusinessprocessmodelsinBPMN.Thishelpstoexpressboththedynamicbehaviorandsomebusinessrulesondiscretedatatypes,aswellasinterestingbusinessscenarios.

2.5.4TestGeneration:TargetedTestingCategoriesMBTmayaddressseveralcategoriesoftestingwithrespecttothesoftwaredevelopmentlifecycle(testinglevels),linkedtothetypeoftestingactivityandalsolinkedtotheaccessibilityandmodeoftheSUT.Table5givesthevarioussubcharacteristicsandattributesofthisdimension.

Table5TestingCategoriesCharacteristics

2.5.5Testgeneration:TestSelectionCriteriaAsshowninFig.5,testcasesareautomaticallygeneratedfromtheMBTmodelandfromtestselectioncriteria.Inpractice,themodelsalonearenotsufficienttogeneratetestsbecausethousandsofsequences(andthereforetestcases)maybegeneratedbywalkingthroughthemodel.TestselectioncriteriaaredefinedbythetestanalysttoguidetheautomatictestgenerationsothatitproducesagoodtestsuiteonethatfulfillstheprojecttestobjectivesdefinedfortheSUT.Thedefinitionoftestselectioncriteria,foratestingproject,dependsontheprojecttestobjectivesandonanalyzingtherisksassociatedwiththesoftwareinaRisk-and-Requirements-BasedTestingapproach.

FIGURE5 Testselectioncriteria.

Therearetwomainsubcharacteristicstoconsider,whichdefinestwodifferentfamiliesoftestselectioncriteria(Table6):

Table6TestSelectionCriteriaCharacteristics

•Coverage-basedselectioncriteriaThisreferstoaformofstructuralcoverageonthemodelelementstoachievethetargetedtestobjectives.Forexample,ifrequirementsarelinkedtotransitionsinUMLstatemachines,thenatestselectioncriterionmaybetocoverallthetransitionslinkedtoagivensetofrequirements.Thus,requirementscoverageisachievedbyastructuralcoverageoftransitionsofthemodels.

•Scenario-basedselectioncriteria—Explicittestcasespecificationswheretestcasesaregeneratedfromdescriptionsofabstractscenarioscanobviouslybeusedtocontroltestgeneration.Inadditiontothemodel,thetestanalystwritesatestcasespecificationinsomeformalnotation,andthisisusedtodeterminewhichtestswillbegenerated.

Thedifferencebetweenthetwoapproachesisthatforthefirstonethedefinitionofthetestselectioncriterionismainlydonebyspecifyingalevelofcoverageoftheelementsofthetestgenerationmodel.Thisisagreatadvantageintermsofresilienceofthetest

selectioncriteriawithrespecttomodelchanges.Thesecondkindoftestselectioncriteriaismoreprecisebecauseeachscenariocanbeexplicitlydefined,but,becausethedefinitionisdonescenariobyscenario,scenario-basedselectioncriteriaaremorefragilewithrespecttomodelevolution.

Thesetwokindsoftestselectioncriteriaarefurtherrefinedbymorepreciseattributes,particularlycoverage-basedselectioncriteria,whichrefertoalargesetofcoveragecriteriasuchastransition-basedcoverage,data-flowcoverage,decisioncoverageanddatacoverage.Forthescenario-basedselectioncriteria,theattributeisthelanguagetoexpressscenarios.AnexampleofsuchalanguageisUMLsequencediagrams.

2.5.6TestExecution:ManualorAutomatedThegeneratedtestcasesmaybeexecutedmanuallyorautomatically(Table7).ManualtestexecutionconsistsofahumantesterexecutingthegeneratedtestcasesbyinteractingwiththeSUT,followingtheinstructionsinthetestcase(documentedsteps).Automatedtestexecutionconsistsoftranslatingthegeneratedtestcasesintoexecutabletestscriptsofsomeform.Forinstance,ifeachgeneratedtestcaseisjustasequenceofkeywords,itcouldbeexecutedmanuallybythetesters,oratestautomationengineercouldwriteanadaptationlayer(programoralibrary)thatautomaticallyreadsthosekeywordsandexecutesthemonthesystemundertest.Automaticexecutiontypicallyrequiresmorework,todeveloptheadaptationlayer,butsomeofthisoverheadcanbereducediftheadaptorcodeisreusedallalongthetestinglifecycle,forinstanceformanydifferenttestsorforseveraldifferentversionsofthegeneratedtests.

Table7TestExecution—Manual/AutomatedCharacteristics

Inthecaseofautomatedgenerationofmanualtestcases,thismeansthatthemodelsfortestgenerationshouldincludeonewayoranotherthedocumentationofabstractoperationsandattributes.Then,thetestgeneratorhastomanagethepropagationandadaptationofthisinformationinthegeneratedtests.

2.5.7TestExecution:OfflineorOnlineThelastcharacteristicdealswithtestexecutionandtherelativetimingoftestcasegenerationandtestexecution:offlineMBToronlineMBT(Table8).

Table8TestExecution—Offline/OnlineCharacteristics

WithonlineMBT,thetestgenerationalgorithmscanreacttotheactualoutputsoftheSUT.ThisissometimesnecessaryiftheSUTisnondeterministic,sothatthetestgeneratorcanseewhichpaththeSUThastaken,andfollowthesamepathinthemodel.Inthatcase,MBTtoolsinteractdirectlywiththeSUTandtestitdynamically.

OfflineMBTmeansthattestcasesaregeneratedstrictlybeforetheyarerun.Theadvantagesofofflinetestingaredirectlyconnectedtothegenerationofatestrepository.Thegeneratedtestscanbemanagedandexecutedusingexistingtestmanagementtools,whichmeansthatfewerchangestothetestprocessarerequired.Onecangenerateasetoftestsonce,thenexecuteitmanytimesontheSUT(eg,regressiontesting).Also,thetestgenerationandtestexecutioncanbeperformedondifferentmachinesorindifferentenvironments,aswellasatdifferenttimes.Moreover,ifthetestgenerationprocessisslowerthantestexecution,whichisoftenthecase,thenthereareobviousadvantagestodoingthetestgenerationphasejustonce.

2.6ABriefHistoryofMBTTheuseofmodelsfortestgenerationhasbeenanareaofstudyfromthemid-1970s,withtheseminalworkofT.S.Chow,fromtheBellLaboratories,ontestingsoftwaredesignmodeledbyfinite-statemachines[7].Thisworkdescribesanalgorithmforgeneratingtestsfromthisrepresentationofsoftwaredesign.Thispaperinitiatedalongseriesofresearchwork,whichisstillon-goingtoday,onusingfinite-statemachinerepresentations,statechartsandUMLstatediagramsasabasisforautomatedtestgeneration.Thisresearchstreamhasbeenveryproductive,particularlyintestingreactivesystems:ie,proposingalargesetofautomatedtestgenerationalgorithms,definingtestselectioncriteria,establishingconformancetheories,andfinallyprovidingresearchMBTtoolsthatcontinuetoinfluencetheMBTscene.

ThisinterestinMBTincreasedstronglyfrom1990onwards,notonlyintheacademicfieldbyusingalargespectrumofformalsoftwarerequirementsandbehaviorrepresentationsfortestgeneration,butalsointheindustrialfieldasaresponsetoproblemsfoundindesigningandmaintaininglargestatictestsuites.MajorscientificfoundationsofMBTweresetupduringthisperiod,includingtestgenerationalgorithmsforautomatingtraditionalmanualtestdesigntechniquessuchasprocesscycletesting,equivalenceclasses

partitioningorboundaryvalueanalysis[4]andalsocombinatorialtestgenerationtechniquessuchaspairwisetechniques[8].Moreover,theprinciplesofautomatedbidirectionaltraceabilitybetweenrequirementsandtestsweresetupatthistime.

Early2000ssawtheemergenceofMBTasaregulartestingpracticeinthesoftwareindustry.ThisimpliedintegratingMBTwithkeyindustrystandardssuchasUMLandBPMNforthemodelingphase,andwiththeindustrialtestmanagementandtestexecutionenvironmentstosetupacontinuousandsystematictestengineeringprocess.Atthesametime,themethodologyofMBTwasbeingstudiedinordertoclarifythebestpracticestogofromrequirementstomodelsfortestgeneration,formanagingthetestgenerationprocessandtohelpqualityassurance(QA)teamstoadoptMBT.Thisdecadewasalsotheperiodoflife-sizepilotprojectsandempiricalevidenceanalysistoconfirmtheefficiencyofMBTpracticeontestingprojects(eg,[3,4,9]).

MBTisnowinadoptionphaseinthesoftwaretestingindustry.ThenextsectionprovidessomeevidenceaboutthepenetrationofMBTinindustryandtheapplicationdomains.

3CurrentPenetrationofMBTinIndustryThefirstexperimentswithMBTinindustrystartedinthesameperiodastheideaofdrivingtestgenerationfrommodelsappearedinacademia,meaninginthe1970susingFiniteStateMachinemodels[7].ButthetangibleemergenceofMBTinindustrydatesfromtheearly2000swiththecreationofspecializedtoolproviderssupportinganeffectiveuseofMBTsuchasConformiqandSmartesting,andtheavailabilityoftoolsfromlargecompaniessuchasSpecExplorerfromMicrosoft.So,whatmorehashappenedduringthelastdecade?ThereisclearevidencethatMBTisslowlypenetratingthetestingmarket,nothinglikeatornado,butmorelikeaninkblotstrategy.Intherestofthissection,weshallreviewthreemarkersofthisslowbutpalpablepenetration:

•TheMBTUserSurvey2014results;

•TheevolutionofthepresentationattheETSIUserConferenceonMBT(nowUCAATUserConferenceofAdvancedAutomatedTesting)from2010to2014;

•AcertificationofcompetenceforMBT:ISTQBCertifiedTesterModel-BasedTesting,whichisanextensionofthefoundationlevel.

3.1The2014MBTUserSurveyResultsThissurveywasconductedfrommid-June2014toearlyAugust2014[10]tolearnaboutthecurrentmotivations,practicesandoutcomesofMBTinindustry.The2014MBTUserSurveyisafollow-uptoasimilar2011survey,1andhasbeenconductedunderthesameconditionsandsimilardisseminationoftheinformationaboutthesurvey.ThisgivessomeevidenceaboutthepenetrationofMBTinindustry:

•In2014,exactly100MBTpractitionersresponded,vs47in2011,thusagrowthofslightlyover100%canbenoticedwithinthelast3years.

•Inbothcases,therespondentswere90%fromindustry(and10%fromacademia),whichisduetothefactthatthesurveywasexplicitlyrestrictedtopractitioners.

•Giventhewide-spreaddisseminationofthesurveyinprofessionalsocial-networkinggroups,software-testingforums,andconferencesworldwide,andthetypicalresponseratestoexternalsurveyssuchasthis,weestimatethatbetween2%and10%ofthenumberofMBTpractitionersactuallyparticipatedinthesurvey.Thissuggeststhattherearebetween1000and5000MBTactivepractitionersworldwidein2014!

ThenextMBTUserSurveywillbeconductedagainin2017,whichwillgiveanotherindicationofthegrowthofthenumberofactiveMBTpractitioners.

Oneinterestingresultfromthe2014surveyisthelevelofmaturityintheMBTusage.Fig.6showsthat48%oftherespondentsroutinelyuseMBTand52%arestillintheevaluationortrialphase.Onaverage,therespondentshad3yearsexperiencewithMBT.Thisfigureshowsthatweareclearlyinanimmatureareawithalotofnewcomers(morethan50%).Theobviousquestionis:whathappensaftertheevaluationandpilotphases?Dotheuserscontinueandgeneralize,ortheygive-upandstopMBTadoption?Itistoo

earlyforadefinitiveansweronthispoint,butanotherquestionofthe2014MBTUserSurveyprovidesuswithsomeinformationonthelevelofsatisfactionobtainedbytherespondents.

FIGURE6 AtwhatstageofMBTadoptionisyourorganization?

Table9providesacomparisonbetweenexpectationsandobservedsatisfactionlevels,classifiedbythemaintypesofexpectation:cheapertests,bettertests,supporttomanagecomplexityofthetestingproject,supportforcommunicationbetweenprojectstakeholders,andshiftleft(startingtestdesignearlier).

Table9ComparisonBetweenExpectationsandObservedsatisfactionlevel

Theresultsshowthat,forthemajorityofrespondents,MBTgenerallyfulfilstheirexpectations,andtheythereforegetthevaluetheyarelookingfor.

ThispositivevisionisconsistentwiththeanswerstothenextquestiononhoweffectivehasMBTbeenintheirsituation(seeFig.7).AmajorityofrespondentsviewedMBTasausefultechnology:64%foundMBTmoderatelyorevenextremelyeffective,whereasonly13%ratedthemethodasineffective.Morethan70%oftherespondentsstatedthatitisverylikelyorevenextremelylikelythattheywillcontinuewiththemethod.

FIGURE7 HoweffectivedoyouthinkMBThasbeen?

Finally,wemayhavealookatthecurrentdistributionofMBTacrossdifferentareasofindustry,asshowninFig.8.Nearly40%oftherespondentscomefromtheembeddeddomain.EnterpriseITaccountsforanother30%,webapplicationsforroughly20%.OtherapplicationdomainsfortheSUTaresoftwareinfrastructure,communications,andgaming.ThemainlessonlearnedisthatMBTisdistributedoverthemainareasofsoftwareapplications,withanoverrepresentationintheembeddeddomain.

FIGURE8 Whatisthegeneralapplicationdomainofthesystemundertest?

3.2AnalysisoftheMBTUserConferencePresentationsTheMBTUserConferenceisanannualconferencededicatedtoMBTpractitioners,

sponsoredbytheETSI(EuropeanTelecomStandardInstitute).Theconferencestarteditsfirsteditionin2010.In2013,theconferencehasbeenrenamedUCAAT(UserConferenceonAdvancedAutomatedTesting)tobroadenthesubjectarea,butitisstillaconferenceconsistingforthemostpartofexperiencereportsonMBTapplications.Thisisaprofessionalconferencewithmorethan80%ofpeoplecomingfromindustry,andbetween160and220attendeeseachyear,dependingmainlyonthelocation(duetotheattractionforlocalpeople).

Table10providesthelistofapplicationareascoveredduringeachyearoftheconference.Thisconfirmsthelargescopeofapplicationsofmodel-basedtechniquesandtools.Thepresentationsareavailableoneachconferencewebsite(seeURL).

Table10MBTApplicationAreas

TheseconferencesarealsogoodshowcasesofthediversityofMBTdeployment.TheyshowthevarietyofMBTapproaches,inthewaythatMBTtestmodelsareobtained(forinstancebyreusingexistingmodelsordevelopingspecifictestmodels),thenatureofsuchmodels(varietyoflanguagesandnotations),andalsothewaygeneratedtestsareexecuted(offlinetestexecutionoronlinetestexecution).

3.3ISTQBCertifiedTesterMBTextensionAnothermarkerofthepenetrationofMBTinindustryistheupcomingISTQBcertificationformodel-basedtesters.ISTQB®(InternationalSoftwareTestingQualificationsBoard)isanonprofitassociationfoundedin2002.ISTQB®hasdefinedthe“ISTQB®CertifiedTester”certificationscheme,withmorethan380,000certificationsissuedworldwidesincethebeginning.ISTQB®isanorganizationbasedonvolunteerworkbyhundredsofinternationaltestingexperts.

TheISTQB®schemehasthreelevels:foundations(basiclevel),advanced(fortestmanagersandanalysts)andexpert(forexperiencedpractitioners,whichstrideforimprovingthetestingprocessanditsautomation).ThebasiclevelisISTQBCertifiedTesterFoundationLevelwithtwoextensions:oneforagiletesters(testersembeddedinprojectsusinganagilesoftwaredevelopmentprocess)issuedin2014,andoneformodel-basedtesterscurrentlyinBetareviewphase(June2015)andexpectedtobeissuedinOctober2015.

TheMBTISTQBcertificationisorganizedasfollows:

•ISTQBCertifiedTesterFoundationLevelisaprerequisite

•Itconsistsof2daysoftraining

•Itcovers38learningobjectives:

–9atcognitivelevelK1(remember)

–23atcognitivelevelK2(understand)

–6atcognitivelevelK3(apply)

•Thecontentofthesyllabusisstructuredinfivechapters,whicharethefollowing:

–Introductiontomodel-basedtesting,introducesMBTanddiscussesitsbenefits

–MBTModeling,givesanoverviewofmodelinglanguagesusedinMBTandguidelinesforcreatingMBTmodels,asacoreactivityinanMBTapproach.

–SelectionCriteriaforTestCaseGeneration,discussesavarietyofselectioncriteriausedtodrivethetestcasegeneration

–MBTTestExecution,illustratesspecificactivitiesrelatedonexecutingthegeneratedtestcasesonthesystemundertest

–EvaluatinganddeployinganMBTapproach,discussesthechallengesandgivesguidelinesforsuccessfulintroductionofMBTinanenterprise

Aftersuccessfullypassingthecertification,itisexpectedthatanISTQBCertifiedModel-BasedTesterwillhaveacquiredthenecessaryskillstosuccessfullycontributetoMBTprojectsinagivencontext.ThiscertificationwillhelptodisseminatetheMBTapproachinindustry,wheretheISTQBcertificationschemeisverypopular.Ittargetspeoplewhohavereachedafirstlevelofachievementintheirsoftwaretestingcareer,includingTestEngineers,TestAnalysts,TechnicalTestAnalysts,andTestManagers,butalsosoftwaredeveloperswhoarewillingtoacquireadditionalsoftwaretestingskills.

4LanguagesforMBTModelingActivitiesThissectionexpandsonthe“Modelsfortestgeneration”presentedinFig.1,bydiscussingthelanguagesandnotationsusedtobuildMBTmodels.Therearemanyparadigmsandlanguages[5]thatcanbeusedfortheMBTmodelingactivities.ThissectiongivesanoverviewoftheapproachesusedbythreecategoriesofMBTusers:

1.thedevelopersorpeoplenearesttotheimplementationcode.Inthiscase,themodelinglanguageistypicallyadesign-by-contractnotationthatisclosetothecode.Thisdesign-by-contractconceptwaspopularizedbyBertrandMeyerwiththeEiffellanguagein1986.

2.thebusinesspeoplethatusebusinessrulestodescribetheprocessesbeingtested—theytypicallyuseagraphicalnotationsuchasBPMN(withdecisiontables)orUMLtodescribetheMBTmodel.

3.theenduser.Inthiscase,thetesterusesthesameinterfaceastheenduserofthesystembeingtested.Thechallengeishowtorecordandreusethesetests,orgeneralizethemintoamodelofthesystem.

4.1TestingwithContractsThedesign-by-contractapproachisoftenconsideredasunittestingbecauseitisclosetothecodeoftheapplicationbeingtested.Also,therearemanydiscussionsabouttest-driven-development(TDD)vsdesign-by-contract(DbC).Infact,bothtechniquesareusedtohelpimprovethespecificationoftheapplication.ButTDDisforthedevelopmentteamtoanalyzetestcasesbeforecodeproduction,whereasDbCismoregeneralinscope,eventhoughitmayalsobeusedbythedevelopmentteam.Inconcreteterms,TDDassociatesanassertion(trueorfalse)withtheexpectedvaluesreturnedfromacalloraction.ButinDbC,youcanusemorecomplexpredicatestodescribemoresophisticatedproperties,usingahigherlevellanguage.

Infact,therearethreecommonusagesofcontracts.Inthefirst,theprogramandcontractarebothtranslatedintologicformulae,whichcanthenbecheckedforconsistencyusingprooftechniques—butthisisoutsidethescopeofthischapter,whichisaboutmodel-basedtesting.Asecondusageofcontractsistotranslatethecontractsintoruntimeassertionsthatcanbecheckedastheapplicationisexecuted.Thisisausefulformoftesting,butrequiresseparateinputsandsequencesofoperationstodrivetheapplication.Thethirdapproach,calledcontract-basedtesting,wasintroducedbyAichernig[11]andgeneratestestsoftheapplicationfromthecontracts.Inthiscase,thecontractsarebeingusedastheMBTmodelforthetestgeneration,whichisbasedon:

•usingtheinvariantsandpreconditionstogenerateinputdatafortests;

•usingthepostconditionstoprovidetheoracletocomputethetestverdictduringtheexecutionofthetests.

WeshowinTable11,somedevelopmentlanguageswiththeirassociatedcontractlanguages.Allofthesecontractlanguageshavebeenusedfortestgenerationpurposes.

Table11ExampleofLanguage

Inthenextsection,wedescribeahigherlevelnotationforMBTmodels,whichisfurtherawayfromtheapplicationimplementationlevel.

4.2TestingBusinessRulesAspresentedinSection2.6,UMLandBPMNareindustrystandardsusingforthemodelingphase.So,UMLisnaturallyusedtorealizeMBTmodelsanddedicatedtestingprofiles.OftenthisisdoneusingtheUMLTestingProfile,2whichhasbeendefinedbytheObjectModelingGroup(OMG),presentedforthefirsttimein2004[23].

MuchresearchhasbeenperformedtoestablishtherangeofUMLdiagramsthatcanbeusedtodescribethesystemfortesting—twoexamplesareproposedin[24,25].TheproblemisthepreciseinterpretationoftheUMLdiagramstocomputethetests[26].Insomecases,theMBTmodelexpressedusingasinglekindofUMLdiagramisnotsufficientlyprecisetousefortestgenerationpurposes.Ratherthanmakingthatdiagrammorecomplex,itisoftenbettertoaddotherelementsfromotherkindsofUMLdiagrams,suchasusecases[27,28],sequencediagrams[29–32],orinteractiondiagrams[33,34].Thiscombinationoftwo(ormore)differentkindsofdiagrams/notationsistypicallyusedtoallowthetestsequencestobespecifiedusingonenotation,whilethedetailsoftheinputsandoutputsarespecifiedusingadifferentnotation.

AnotherkindofmodelinglanguageusedforitssimplicityisBusinessProcesslanguages,asoutlinedbyReijersetal.[35],whodescribeaninterestingstudyontheusageofthislanguage.Thelanguageisclosertobusinessrulesandallowsinteractionwithallactors.IBMgiveanoverviewin[36]andasimilarapproachisalsousedwithSAP,asdescribedbyMecke[37].Thisapproachisalsousefulfortestingwebservices[38–40].

4.3TestingGUIsTheautomationofthetestgenerationand(usually)testexecutionisoneofthemainattractionsthathelpsthediffusionofMBT.Thedesiredapproachisoftentogenerateteststhatinteractwiththesystemundertestinexactlythesamewayastheenduserdoes.Infact,wewanttotestthesystemusingthesameinterfaces.Oneofthemaindomainstousethistechniqueisweb-basedapplications.ExecutionrobotssuchasSelenium,Watij(fortheJavaversion)orSahiarethemostwellknown.Butthedifficultyisobtainingtheteststhattheexecutionrobotcanexecute.Thetraditionalbasicapproachistorecordtheuseractions,andsimplyreplaytheselater.Morecomplexelementscanbeaddaseventcontexts[41]orparameters.Butsuchrecordedtestsequencesareverylowlevelandfragile[4,page23],soitisdesirabletohavehigherlevelapproachestogeneratingthetestsequences,whichisanareawhereMBTcancontribute.

Inrecentyears,muchresearchhasbeendoneontheanalysisofGUIsforthepurposeoftestgeneration.ApopularapproachisGUI-ripping[42],whereMBTmodelsareconstructedfromtheGUIapplicationautomatically,ratherthanbeingwrittenmanually.Forexample,theGUITARframework[43]automaticallyconstructsatreeofallwidgetsreachableinaGUI,plusanevent-flowgraphofalltheinteractionsbetweenGUIobjects,andthenusesthesemodelstogeneratemanyshorttestsequencesthatfindeachwidgetandexerciseeachevent[44].However,alimitationofGUIrippingisthatthegeneratedtestscontainfairlyweakoraclechecks,suchasjustcheckingthattheapplicationdoesnotcrash.Thisisbecausetheautomaticallygeneratedmodelsdonotcapturedeepsemanticknowledgeoftheexpectedapplicationbehavior.Onepracticalapproachforaddressingthisissueistomanuallyaddstrongeroraclecheckstothetestsaftertheyaregenerated.AnextensionofthisGUI-rippingapproachhasalsobeenappliedtoAndroidapplications[45]andwasshowntooutperformrandomtestingofAndroidapplications.

AnotherlimitationofGUI-rippingapproachesisthatwhenthetestgeneratorreachesawidgetthatrequirescomplexinputvalues,suchasanintegerorstringvalue,itisnoteasytogeneratemeaningfulinputvalues.Themostcommonapproachistogeneraterandomvalues,orrandomlychoosevaluesfromasmallsetofprechosenvalues.AmoresophisticatedsolutiontothisproblemhasbeendevelopedbyArltetal.[46],whichreliesonblack-boxGUI-rippingtestingtoreachaninputwidget,followedbyusingwhite-boxsymbolicexecutionofthecodethathandlesthatinputvaluetodeterminewhatinputvaluestriggerthedifferentpathsthroughthecode.Whenappliedtofouropen-sourceapplications,theyfoundthatthisapproachgeneratedtestswithbettererrordetectionthanrandomgenerationofinputvalues.

TheGUI-rippingapproachtendstogeneratemanyshorttests,whicheachtestjustafeweventswithintheGUI.Aretherewaysofextractingmoreknowledgeoftheapplication,sothatitispossibletogeneratelongerandmoreapplication-specifictestsequences?WebrieflydescribetwoextensionsoftheGUI-rippingapproachthatshowthatthisispossible.

Thefirstistocombineblack-boxandwhite-boxmodelsoftheapplication.ForexampleArltetal.[47]usestaticanalysisofthebytecodeoftheapplicationtodetermine

dependenciesbetweenevents,sothatlongertestsequencescanbegeneratedtoexercisethosedependencies.Thisgeneratesfewertestsequences,andlongertestsequences,butmatchesorimprovestheerrordetectionratesofpureblack-boxGUIripping.Ontheotherhand,theEXSYSTtestgeneratorforinteractiveJavaprograms[48]usescompletelydifferenttechnology(geneticalgorithmstosearchthroughthemassivesearchspaceofalleventsequences)togenerateGUIteststhathavehighcodecoverageoftheapplication.ItdoesthisbymonitoringthebranchcoverageoftheunderlyingapplicationastheGUItestsareexecuted,andusingthatcoveragelevelasthefitnessfunctionforthegeneticsearch.Althoughthesetwotoolsusecompletelydifferenttechnologies,theybothusethetechniqueofobservingtheexecutionoftheapplicationcode(white-box),andconnectingthoseobservationstotheblack-boxmodeloftheGUIinputeventsinordertogeneratesmarterandlongerinputeventsequencesthatcanfindmoreGUIfaults.

Thesecondapproachtoimprovingthedepthofthegeneratedtestsequencesistobasetestgenerationoncommonuser-interfacepatterns[49],suchastheclick-login;enter-username;enter-password;click-submitsequence,followedbythetwoexpectedoutcomesofeitheraninvalidloginmessage,orcorrectloginasshownbytransfertoanotherpage.SuchuseofpatternsgoessomewaytowardsincorporatingmoreoftheapplicationsemanticsintotheMBTmodel,whichenablesricherteststobegenerated.AcomplementaryapproachtoimprovingthesemanticrichnessoftheGUImodelistoinferspecificationsandinvariantsoftheGUIfromthetracesofGUIexecutions.Thisisachallengingtaskgiventheinfinitespaceofpossibletraces,butforexample,AutoInSpec[50]doesthisinagoal-directedway,guidedbycoveragecriteriaandbyatestsuiterepairalgorithm,andthusfindsmoreinvariantsthannongoal-directedapproaches.

AnexampleofanapproachthatusesMBTmodelswithevendeepersemanticknowledgeoftheapplicationisthemodel-driventestingofwebapplicationsbyBolisetal.[51].Thisusesmodelswrittenassequentialnetsofabstractstatemachines(ASM).Themodelsmaybedevelopedspecificallyfortestingpurposes,ormaybeadaptedfromthehigh-leveldevelopmentmodelsoftheapplication.Eitherway,thedevelopmentofsuchmodelsrequiresapplicationknowledgeandmodelingexpertise,butthebenefitofhavingtherichermodelsisthattheycangeneratemorecomprehensiveteststhatincludestrongeroracles.

Finally,allthesedifferentapproachestoautomatedorsemiautomatedGUItestingraisetheinterestingquestion:whichapproachisbest?Lellietal.[52]havemadestepstowardsansweringthisquestionbydevelopingafaultmodelforGUIfaults,validatingthisagainstfaultsfoundinrealGUIs,anddevelopingasuiteofGUImutantsthatcanbeusedtoevaluatetheeffectivenessofGUItestingtools.

5TechnologiesforModel-BasedAutomatedTestGenerationAsshownintheprevioussections,akeypointofMBTisthatthegenerationoftestcasesisperformedfromadeclarativeMBTmodelthatformalizesthebehavioroftheSystemUnderTest(SUT)inthecontextofitsenvironmentandatagivenlevelofabstraction.ThemodelcapturesthecontrolandobservationpointsoftheSUT,itsexpecteddynamicbehavior,thedataassociatedwiththetests,andfinallytheinitialstateoftheSUT.Hence,theMBTmodelispreciseandformalenoughtoenableunambiguousinterpretations,sothatthederivationoftestcasescanbeautomated[53].

Testgenerationtoolshavethetaskofchoosingcertaintracesofexecutions.Inotherwords,themodeldescribesasearchspacecontainingalargesetofexecutiontraces,potentiallyinfinite.Thetestgenerationtoolmustusevariousalgorithmsandheuristics,plusguidancefromtheuser(eg,coveragecriteria,propertiestobeguaranteed,andpatternstobeapplied),toselectasubsetofexecutiontracesthatdefinetestcases.Toreducethecostofsoftwaretesting,andgiveagoodreturnoninvestment,MBTapproachesneedtosupportahighdegreeofautomationtoautomaticallyandsystematicallyderivethetestcases[54].

Toaddressthisgoal,alargenumberofMBTtoolshavebeendevelopedinthepastfewdecades,andalotofresearchworkhasbeeninvestigatedandpublished.Asaresult,asignificantnumberofdifferenttechnologieshaveemergedtosupportMBTautomation.ForverysmallMBTmodelsandsimpleSUTsitispossibletogeneratebasictestsuitesusingsimplealgorithmssuchasrandomwalks,ortheChinesePostmanalgorithm3[55],butformorecomplexmodelswithcomplexdatatypes,testgenerationquicklybecomesadifficultoptimizationproblem.Infact,mostrealtestgenerationproblemsarecomplexenoughtobeundecidable,whichmeansthatnoknownalgorithmcanguaranteetofindanoptimalsolutioninareasonableamountoftime.Forthisreason,MBTtestgeneratorstypicallyuseheuristicsolvers,whichaimtoquicklyfindatestcasethatsatisfiesthedesiredconstraintsandiscloseenoughtotheoptimalsolutiontobeacceptable,eventhoughitmaynotbefullyoptimal.

TheobjectiveofthissectionistogiveanoverviewofthemajortestgenerationtechnologiesusedatpresenttoautomatethederivationoftestcasesfromMBTmodels.Moreprecisely,weintroducethefollowingfourcurrentlypopularandwell-usedtechnologies:SAT/SMTsolving,constraintprogramming,search-basedalgorithms,andmodelchecking.Foreachofthem,wediscussthetypicalbenefits,weaknessesandfuturechallengesthatrelatetotestgenerationissues.

5.1SATandSMTProversInitiallyusedforstaticanalysisandprogramverification,BooleanSATisfiability(SAT)andSatisfiabilityModuloTheory(SMT)solvershavereceivedconsiderableattentionduringthelastdecadetoautomaticallyderivetestcasesfromMBTmodels.ASATprover

isatoolthatcandeterminewhetherthereisasolutionsatisfyingapropositionalformularepresentedbyaBooleanexpression(ie,writtenonlywithBooleanvariables,parenthesesandoperatorsofdisjunction,conjunction,andnegation).ThecentralalgorithmaroundwhichtheSATproversarebuiltiscalledtheDavis–Putnam–Logemann–Lovelandalgorithm[56],DPLLforshort.ItisacompletesolvingalgorithmbasedonpropositionallogicformulaeinConjunctiveNormalFormandusingabacktrackingprocess.

AnSMTproblemisexpressedasalogicalfirst-orderformulacombiningdifferenttheoriessuchasequality,lineararithmetic,andbit-vectors.Inthisway,anSMTproblemisageneralizationofaSATproblemwheretheBooleanvariablesarereplacedbypredicatesusingdifferenttheories.AnSMTproveristhusbasedonaSATprover,whichisusedtosolvethelogicalformulasoffirstorderonaparticulartheorybyusingadedicateddecisionprocedure.Asaconsequence,anSMTproverattemptstodetermineifanSMTproblemhasasolution,and,ifasolutionexists,returnsthatsolutionintheformofavaluationofeachofthevariables.

Nowadays,themostpopularSMTsolversareZ3[57],CVC3[58],CVC4[59],Yices[60]andMathSAT5[61].Z3isanSMTproverdevelopedbyMicrosoftResearchandispartofprofessionaltoolssuchasVisualStudioviathewhiteboxcodeanalyzerPex[62].CVC3,(CooperatingValidityChecker3),anditssuccessorCVC4areacademicproversdevelopedjointlybytheUniversitiesofNewYorkandIowa.YicesisdevelopedbySRIInternational’sComputerScienceLaboratory,andfinally,MathSAT5isthelatestversionoftheMathSatprover,whichhasbeenjointlydevelopedbytheUniversityofTrentoandtheFBK-IRST.

Fortestgenerationpurposes,alltheelementsofthemodelsuchasconditions,assignmentsandstructures(eg,if–then–else)aretranslatedintofirst-orderpredicatesandconjoined.Theresultingformula,calledanSMTinstance,describesalltheinstructionsthatshouldbeexecutedalongatestcaseexecutionpath,includingtheglobalsequenceofSUToperationcallsandthelocalchoiceswithineachofthoseoperations.Thesatisfyinginstancesofthisformulacorrespondtovalidinputvaluesthatenableadesiredtestcasesequencetobeactivated.Thisapproachcanbeusedtocomputeinputdatatoactivateaparticularpathinatargetedoperation,aswellastoprovidesequencesofoperationcalls.

Forexample,Cantenotetal.[63]describeatestgenerationframework,basedonSMTsolving,fromUML/OCLmodel.ThisframeworkisabletotranslatetheUML/OCLmodelintoSMTinstances,andvarioussimulationstrategiesareappliedontheseSMTinstancestocomputetestsequences.TheirpaperproposesandcomparesfivedifferentstrategiestobuildthefirstorderformulasusedbyanSMTinstance.Otherresearchersproposeoptimizationstoimprovethetestgenerationperformance,includingoptimizationsthatexploitthefeaturesoftheSAT/SMTprover,aswellastheformofthefirstorderformulas[64].TheseexperimentspromisetomakeSAT/SMTtestgenerationtechniquesasefficientasotherexistingmethods,especiallywhendealingwithBooleanexpressions.

However,asunderlinedbyCristiáandFrydman’sexperimentalfeedbackabouttestcasegenerationfromZspecificationsusingthetwoSMTproversYicesandCVC3[65],themainweaknessregardingtheuseofSMTproversfortestgenerationpurpose,iscurrently

thelackofnativedecisionproceduresforthetheoryofsets.Thisimpliesthattherepresentationofsetsmustbemappedontosomeothermathematicalstructure(suchasuninterpretedfunctions,arrays,lists,etc.),whichcanresultinseverelybelow-averageperformance.ThisobservationhasbeenrecentlyinvestigatedandconfirmedbyCristiá,RossiandFrydman[66],whereexperimentshaveshownthatCLPsolverswithsetconstraintscanbemoreeffectiveandefficientthanSMTproversforthesekindsofproblems.Thenextsectionintroducessuchconstraint-basedapproaches.

5.2ConstraintLogicProgrammingAconstraintsystemisdefinedbyasetofconstraints(properties)thatmustbesatisfiedbythesolutionoftheproblembeingmodeled.SuchasystemcanberepresentedasaConstraintSatisfactionProblem(CSP)[67].Formally,aCSPisatupleΩ=<X,D,C>whereXisasetofvariables{x1,…,xn},Disasetofdomains{d1,…,dn},wherediisthedomainassociatedwiththevariablexi,andCisasetofconstraints{c1(X1),…,cm(Xm)},whereaconstraintcjinvolvesasubsetXjofthevariablesofX.ACSPΩcanbeseenasaconstraintnetworkwhosenodesarethevariablesofXwiththeirassociateddomainsofD,andwhosearcsaretheconstraintsofC.Itissuchthateachvariableappearinginaconstraintshouldtakeitsvaluefromitsdomain.

Hence,aCSPmodelsNP-completeproblemsassearchproblemswherethecorrespondingsearchspaceistheCartesianproductspaced1×…×dn[68].ThesolutionofaCSPΩiscomputedbyalabelingfunction,whichprovidesasetv(calledvaluationfunction)oftuplesassigningeachvariablexiofXtoonevaluefromitsdomaindisuchthatalltheconstraintsofCaresatisfied.Moreformally,visconsistent—orsatisfiesaconstraintc(X)ofC—iftheprojectionofvonXisinc(X).IfvsatisfiesalltheconstraintsofC,thenΩisaconsistentorsatisfiableCSP.

UsingLogicProgrammingforsolvingaCSPhasbeeninvestigatedformanyyears,especiallyusingConstraintLogicProgrammingoverFiniteDomains,writtenCLP(FD)[69].Thisapproachbasicallyconsistsofembeddingconstraintsatisfactiontechniques[70]intologicprogramminglanguages,byimplementingtheconceptoflogicaldomainvariables,whichtaketheirvaluefromafinitediscretesetofintegers.

Testcasegenerationcanthusbeperformedbysolvingtheconstraintscollectedforeachpathofthemodelusingaconstraintsolver.Theavailabilityofefficientconstraintsolvershasmadeitpossibletoproposelotsofautomatedconstraint-basedapproachesfortesting:tocomputetestdataaswellastocomputesequencesofoperationcalls,includinginputvalues.Tothisend,theconstrainttechniquesareappliedtosolveaconstraintrepresentationoftheMBTmodel.Basically,thisinvolvesmappingeachpathformalizedinthemodeltoasetofconstraints:structures(eg,if–then–elsestatements)definetheconstraintsoftheCSP,bymanipulatingformulasexpressedusingthevariablesoftheMBTmodel.Inthisway,anysolutionsatisfyingtheCSPdefinesthetestvalues(inputandoutputvaluesthatallowustosolvetheoracleproblem)andenablesustoexercisethecorrespondingpaththroughthemodel.

Toillustratethiskindofapproach,wecancite[71],inwhichtheauthorspresentanoriginalapproachandexperimentalresultsaboutusingconstraintsolvingtocomputefunctionaltestcasesforcontrollersforroboticpainters.Thetestingwasdonewithinacontinuousintegrationprocess,sothetestgenerationandexecutionneededtobefast.Thelessonslearntfromthisindustrialexperimentationshowedthatthetestingstrategy,implementedusingthefinitedomainconstraintsolvinglibraryCLP(FD)[72]ofSICStusProlog,isfasterandmoreeffectivethancurrenttestmethodologiescurrentlyusedinthecompany,evenifthisstrategydoesnotensureacompletecoverage(noteverypossibletransition)ofthebehaviorsformalizedinthemodel.

Otherwork[66],usedthe{log}solver[73,74]togeneratetestcasesfromspecificationswrittenintheZnotation,whichisbasedonfirst-orderlogicoverasettheory.Indeed,tohandletheZnotation,the{log}solverisabletomanipulateandcomputeconstraintsolvingonsetsusingnativesetstructuresandprimitiveoperations.Assuch,itcanfindsolutionsoffirst-orderlogicformulasinvolvingset-theoreticoperators,translatedfromZspecificationsto{log}’spredicates.ThefeedbackofthisexperimentsshowedthatsuchaCLPsolveroversetsisabletotackletwocommonproblemswithinMBT:

•theeliminationofunsatisfiabletestobjectives,builtbypartitioningthestatespaceandcollecting(dead)pathconditions;

•improvementofthecomputationaleffectivenessofthereachabilityproblemoffindingstatesverifyingthesatisfiabletestobjectives.

However,regardingscalabilityissues,constraintsolversarenotabletoreasonaboutaprogram’senvironmentandmaybelessscalablewithlarge-scalespecificationthatincludeahighnumberofvariableslinkedbycomplexconstraints.Moreover,whenaCSPhasseveralsolutions,aCLPsolverisnotabletoprioritisethem,sojustreturnsthefirstsolutionfound.Hence,thedesirabilityofthesolutionmaybeincreasedbytheuseofanobjectivefunction.InnovativesolutionstorespondtothisCLPchallengehavebeenaddressedbythesearch-basedresearchandpractitionerscommunity.Wediscussitschallengesinamodel-basedframeworkinthenextsection.

5.3Search-BasedAlgorithmsAsdiscussedpreviously,thedesirabilityofthesolutionmaybeincreasedbytheuseofanobjectivefunction,whichrequiresthesolvertofindaparticularsolution(eg,minimizeormaximizecertainvariables)[39].Tosolvethisoptimizationproblem,aCLPsolverwouldhavetofindallthesolutions,tocomparethemtoeachothertobeabletoselectthebestone.Thiscomputationmayinvolveanextracostsincethesolvermustexploreallthesearchspaceifitcannotbesurenottofindabettersolutionthatthosealreadycomputed.

Metaheuristicalgorithms,suchassearch-basedalgorithms,aimtogeneratetestsandtestinputsusingthisprinciple(thoughmostofthemfocusontestinputgeneration).Thesearch-basedtestgenerationtechniquesdefineafitnessfunction(orasetoffitnessfunctions)toguidethetestgenerationprocess.WithinMBT,severalsearchalgorithmshavebeenproposedfocusingingeneralonevolutionaryalgorithmsordynamicsymbolic

execution.Evolutionaryalgorithmsmimictheprocessofnaturalevolution.GeneticAlgorithms(GAforshort)arethemostcommonlyused,theyareinspiredbyDarwiniantheory:selection,crossoverandmutations.UMLstatemachinesarethemostfrequentlyuseddiagramforGA-basedapproaches[75].

Forinstance,Doungsa-ardetal.[76]viewthetestdataasachromosome,whichrepresentsasequenceoftransitionsinthestatemachine.Theirfitnessfunctionaimstotriggereachtransitionandthusobtainmaximaltransitioncoverageinthediagram.LefticaruandIpate[77]usealsostatediagramsbuttheirgeneticalgorithmisappliedforinputgenerationonachosenpathofthestatemachine.Theyfurtherevaluatedtheirapproachwithtwoothersearchtechniques:simulatedannealingandparticalswarmoptimization.TheoutcomeoftheirevaluationisthatahybridtechniqueusingsimulatedannealingandGAcombinedwithlocalsearchtechniquesenablessignificantimprovementoftheireffectiveness.Alietal.[78],basedontheirpreviousworkin[79],proposeaheuristicapproachincludingGA,(1+1)EvolutionaryAlgorithm(EAforshort)andAlternateVariableMethod(AVMforshort).Theyevaluatedtheirapproachonartificialconstraintsandonanindustrialstudy.Theiroverallexperienceshowedthatthe(1+1)EAwiththeirnovelheuristicsisthemostefficientfortheirstudies.Incontrasttothepreviousworkthatfocusesmostlyontestdatageneration,Shiroleetal.[80]proposedageneticalgorithmtogeneratefeasiblepathsandtestdata.TheyuseEAwithactivitydiagramstoproducetestscenarios.Theyalsoextendedtheirwork,bycouplingthetestgenerationwithgeneticalgorithmstogeneratevalidandinvalidflowsofmessagesequences[81].

Tomitigatestateexplorationproblemsandthemoregeneralcombinatorialexplosionproblems,Albertetal.[82]combinesearch-basedalgorithmswithdynamicsymbolicexecution,whichconsistsofdoingconcreteorsymbolicexecutionsoftheSUTtoderivepathconditionsforfeasiblepaths,inordertosimplifytheCSPthatmustbesolved.ThemostfamoustoolsimplementingthisdynamicsymbolicexecutionstrategyareDART[83],CUTE[84],andPex[85].Thisstrategyismostlyusedforwhite-boxtesting,becauseforblack-boxtesting,itrequirescomputingtestsonlineoverthewholeSUT,whichisoftennoteasilypracticable.Moredetailsaboutthisstrategy,canbefoundinsurveysbyCadarandothers[86,87].

Finally,itshouldbenotedthatdynamicsymbolicexecutionisoftencoupledwithmodel-checkingapproachesratherthanCLPsolving.Thenextsectiondescribeshowmodelcheckingcanbeusedtoperformmodel-basedtestgeneration.

5.4ModelCheckingAmodelcheckerwasinitiallyatoolusedtoperformformalverification[88].Ittakesasinputanautomatonofasystemandatemporallogicpropertytobesatisfied.Themodelcheckervisitsallthereachablestatesoftheautomatonandverifiesforeachofthesestatesthatthetemporallogicpropertyissatisfied.Ifastatedoesnotsatisfytheproperty,itreturnsthepathtoreachit,intheformofasequenceofstatesstartingfromtheinitialstate.Thispaththusdefinesacompletecounterexamplethatillustratestheviolationofthepropertybytheautomaton.

Thesebasicprincipleswereadaptedearlyontoperformmodel-basedtestgeneration[89].Basically,inthiscontext,theautomatonplaystheroleofMBTmodelwhereasthetemporallogicpropertydefinesaparticularcriteriontobecovered.Moreprecisely,thetemporallogicpropertyisexpressedasthenegationofagiventestobjective:inthisway,whenthemodelcheckerfindsastateviolatingtheproperty(ie,satisfyingthetestobjective),itwillreturntherelatedcounterexample,whichthusconstitutesatestcasecoveringthegiventestobjective[90].

Onthebasisofthissimplebutefficientinterpretation,thenextchallengeisforcingthemodelcheckertofindallthepossiblecounterexamplesinordertoachieveagivencoveragecriterionoftheautomaton.Thiscanbedonebygeneratingaseparatetemporallogicproperty(testobjective)foreachstatethatsatisfiesthecoveragecriterion,andthenrunningthemodelcheckingoneachoneofthosetestobjectives.Hence,attheendofthewholeprocess,eachcomputedcounterexampledefinesaspecifictestcase,includingexpectedoutputtoassignthetestverdict.

Obviously,testgenerationtechniquesbasedonmodelcheckinghavebenefitedfromtheincreasingperformanceofmodelcheckersandfromtheimprovedexpressivenessoftheautomataandpropertiestobeverified.Forexample,somemodelcheckersareabletomanipulatetimedautomata[91],likeUPPAAL[92],oruseadedicatedlanguagetoavoiddescribingamodeldirectly,likeMaude[93].

ToillustratesuchMBTapproachesbasedonmodelchecking,wecanforexamplementionthetool-supportedmethodologyproposedin[94].ItconsistsoftranslatinganinitialMBTmodel,expressedasaBusinessProcessModel,intoanalgebraicPetrinetinordertogeneratetestcasesusingmodelcheckingtechniquesonanequivalentdecisiondiagram.Thetestgenerationisdrivenbydedicatedtestintentions,whichgeneratetestcasesincludingtheiroraclesaccordingtothetransitionsystem.Morerecently,anothertoolbox[95],basedonUPPAAL,hasbeendevelopedforgeneratingtestcasesbyapplyingmodelcheckingtechniquesonnetworksoftimedautomata.Finally,itshouldbenotedthat,basedontheUPPAALmodelchecker,anonlinetestingtool,calledUPPAALforTestingReal-timesystemsONline(UPPAALTRON)[96],supportsthegenerationoftestcasesfrommodels,andtheironlineexecutionontheSUT.

Recently,authorsin[97]combinedformalspecificationandMBTapproachesandevaluatedtheirapproachthroughtheEnergyBusstandard.TheyusedtheTGVtool[98]forverifyingformalspecificationinconsistencies.TheyfurtherconstructedanMBTplatformforconformancetestingofEnergyBusimplementations.Thiscombinationoftoolsandapproacheshasbeenappliedforthefirsttimeinamandatorystepofanewindustrialstandardintroduction,forinstanceEnergyBus.

ContrarytotheTGVtool,whichusesdedicatedalgorithmsfortestcasegeneration,theauthorsin[99]expressedthesystemanditspropertiesasbooleanequations,andthenusedanequationlibrarytochecktheequationson-thefly.Themodelcheckerusedonamodelwithfaultsproducescounterexamples,seenasnegativeabstracttestcases.

Thesetechnologies,usingmodel-checkingtechniquestoderivetestcases,appeartobeefficientandwell-usedsolutionstoautomatingMBTapproaches.However,sincemodel

checkersarenotnativelydevotedtotestcasegeneration,thesetechniquessufferfromalackofimprovementswithregardtotestsuitequalityandperformance[100],suchasthenativeimplementationoftestcoveragestrategies.Moreover,theyalsosufferfromthemajorweaknessofmodel-checkingtechniques:statespaceexplosion.Indeed,statespaceexplosionremainsacriticalproblem,evenifsometoolsarenowcapableofhandlingthestatespacesassociatedwithrealisticproblems.Nevertheless,theperformanceofmodelcheckersregularlyincreasesduetoinnovativeapproaches,andsoitmakesMBTapproachesmoreandmorescalable.Amongtheseinnovativeapproaches,wecanmentionsymbolicmodelchecking[101],whichallowstherepresentationofsignificantlylargerstatespacesbyusingorderedbinarydecisiondiagramstorepresentsetsofstatesandfunctionrelationsonthesestates.Finally,boundedmodelchecking[102]isalsoaveryrelevantapproachfortestgeneration[103].Itaimstoacceleratethegenerationofcounterexamplesbytranslatingthemodel-checkingproblemintoaSATproblem,buttoachievethis,itdoesnotperformanexhaustiveverification.

6Model-BasedSecurityTestingSystemsnotconformingtosecurityrequirementsarevulnerableandaremorelikelytosuccumbtosecurityattacksandsystemintrusions.Securitytestingaimstoincreasetheuser’sconfidenceinthesystem,byensuringitsconformancetosecurityrequirements.

Varioustechniquesexisttoaddressthechallengeofincreasingthesystem’ssafetyandsecurityandtherebyincreasetheuser’sconfidenceinthesystem.AccordingtoSchieferdecker,securitytestingtechniquescanbegroupedintofourfamilies[104]:

•vulnerabilityscanning:networkscannersareusedtocheckforactiveportsonthesystemandattempttoaccessthoseports;

•staticapplicationsecuritytesting(SAST):typicallythesystembyte/binary/sourcecodeisanalyzedforknownsecuritythreats;

•monitoring:anapplicationmonitorsandlogstracesoftheinputs/outputsofthesystemduringitsexecution;

•dynamicapplicationsecuritytesting(DAST):consistsofdynamicallycheckingthesecurityrequirements.

DASTtechniquescanbeperformedusingMBTapproachesthatarecustomizedtosecurityfunctionsandsecurityproperties.ThisiscalledModel-BasedSecurityTesting(MBST).MBSTisarelativelynovelandpromisingfieldofresearch.Itisespeciallydedicatedtosystematicandefficientspecificationanddocumentationofsecuritytestobjectives,securitytestcasesandtestsuites,aswellasautomatedorsemiautomatedtestgeneration[105].

AccordingtoTian-Yangetal.[106]wecanconsidertwomaingoalsforsecuritytesting:

•Functionalsecuritytestingaimstoensurethatthesecurityrequirements,bothintermsofsecuritypropertiesandsecurityfunctions,areimplementedcorrectly.

•Whilevulnerabilitytestingaimstoidentifyanddiscoverpotentialvulnerabilities,basedonriskandthreatanalysis,butalsobasedontheinformationnormallygivenbydatabasessuchastheNationalVulnerabilityDatabase(NVD)ortheCommonVulnerabilitiesExposure(CVE)database.

Thus,inthissectionweareinterestedtoprovideanoverviewofModel-BasedSecurityTesting,dividedintothesetwosecuritytestinggoals.Section6.1focusesonthefunctionalsecuritytestingtechniques(MBFST),andSection6.2detailstechniquesforvulnerabilitytesting(MBVT).

6.1Model-BasedFunctionalSecurityTestingModel-BasedFunctionalSecurityTesting(MBFST)aimsononehandtoensurethesecurefunctioningofthesystem,whichforcriticalsystemsismerelydedicatedtosecurity,ontheotherhandtoensurethesecurityrequirements,forexampleconfidentiality,integrity,

availability,authentication,authorizationandnonrepudiation.

TheCNSS(CommitteeonNationalSecuritySystems)intheirglossarydefinesthesesecuritypropertiesasfollows[107]:

•Confidentialityistheassurancethatinformationisnotdisclosedtoentities(users,processes,devices)whicharenotauthorized.

•Integrityisprovidedwhenanentityhasnotbeenmodifiedinanunauthorizedmanner.

•Authenticationisasecuritymeasureofverifyingtheidentityorotherattributesclaimedbyorassumedofanentity(user,process,ordevice),ortoverifythesourceandintegrityofdata.

•Authorizationprovidesaccessprivilegesgrantedtoauser,program,orprocess.

•Availabilityguaranteesthepropertyofbeingaccessibleanduseableupondemandbyanauthorizedentity.

•Nonrepudiationistheassurancethatthesenderofinformationisprovidedwithproofofdeliveryandtherecipientisprovidedwithproofofthesenderidentity,soneithercanlaterdenyhavingprocessedtheinformation.

InMBFSTthemodelsarefunctionalanddescribethesecurityfunctionsandthesystem’sexpectedbehavior.Inaddition,theyareenrichedwithannotations,forexamplestereotypesorhighlevelscenariosthatfocusonexpressingspecificsecuritypropertiesorthesecurityengineer’sexperience,sincesecuritytestingisataskthatrequiresahighdegreeofsecurity-relatedknowledge.Theseannotationsorscenariosguidefurtherthegenerationofsecurityrelatedtests.

MBFSTproposestechniquesandtoolsusingideassuchasselectioncriteria(staticanddynamic),modelchecking,androbustnesstechniques(forinstancemutationtestingorfuzzing).Furthermore,thesetechniquesaddresssecurity-criticaldomainssuchaswebservices,protocols,legacysystems,smartcards,cryptographiccomponents,etc.Allthesedomainsrequirerigoroustestingandvalidation.Moreover,theapproacheshaveshownthattheycanbeappliedtocertificationssuchasCommonCriteria.Thus,MBFSTremainsachallengingandpromisingapproachtosecuritytesting,andthecreationofmoresecureandsafercriticalsystems.

IntheremainingpartofthissectionwediscussfirstthethreemainfamiliesoftechniquesandapproachesinMBFSTandthenwepresentindetailanMBFSTmethodologyandtools,whichhasbeenappliedinindustryfortestingsecuritycomponents.

6.1.1TestSelectionCriteriaTestselectioncriteriaarechosentoguidetheautomatictestgenerationsothatitproducesagoodtestsuite—onethatfulfilsthetestpolicydefinedforthesystemundertest(SUT)[5],asdiscussedinSection2.5.Wecancategorizethemintotwofamilies:staticanddynamic.InthecontextofMBT,staticselectioncriteriaarerelatedtothestructureofthe

modelelements,whereasdynamicselectioncriteriarelatetothedynamicaspectsofthesystem,forinstanceusingtheexpert’sexperience.

InMBFST,mostofthetechniquesbasedonstatictestselectioncriteriafocusonaccess-controlpolicytesting.ForinstanceLeTraonetal.definednewstructuraltestselectioncriteriaforaccesscontrolpolicies[108]andtestgenerationbasedonaccesscontrolmodels[109].Further,theyhavepresentedanapproachfortestgenerationusingacombinatorialtestingtechniquebycombiningroles,permissionsandcontexts[110].Recently,theyworkedonatool-supportedprocessforbuildingaccess-controlMBTmodelsfromcontractsandaccess-rules[111].

Otherworkbasedonstaticselectioncriteriafocusesingeneralonprivacyproperties.Anisettietal.expresstheprivacypropertiesofaservice(P-ASSERT)andgeneratetestcasesbasedonservicemodels,whichisusedfurtherintheircertificationschemefordigitalprivacyofservices[112].

Experienceinindustryshowedthatstatictestselectioncriteriacannotcoverhigh-levelsecurityproperties,sotoensurethesecurityofacriticalapplicationitisnecessarytousedynamiccriteriainordertoproduceteststhatcoversuchproperties.Ingeneral,thetestscenariosareexpressedinadedicatedlanguagethatcanbeeithertextualorgraphical,describingthesequencesofsteps(usuallyoperationcalls)thatcanbeperformed,alongwithpossibleintermediatestatesreachedduringtheunfoldingofthescenario.Wenowbrieflydescribeseveralapproachesthatusedynamictestselectioncriteria.

Malloulietal.providedaformalapproachtointegratetimedsecurityrules,expressedintheNomadlanguage,intoaTEFSMfunctionalspecificationofasystem.ThentheyusetheTestGen-IFtooltogeneratetestcases,whicharelaterexecutedonthesystemusingtclwebtestscripts[113].

AnotherMBFSTapproachistheoneproposedbyJulliandetal.[114].TheygeneratetestcasesbasedonB-modelsandusedynamictestselectioncriteria(alsocalledtestpurposes)forproducingtestobjectives,representedasregularexpressions.

Legeardetal.,forsecuritytestingofcryptographiccomponents,useanapproachbasedondynamictestselectioncriteria:TestPurpose(TP)[115].TheTPlanguageallowsonetoexpresshigh-levelscenariosbasedonsecurityexpertexperienceandithasbeensuccessfullydeployedandusedattheFrencharmycomputersciencedepartment.Moreover,Cabreraetal.havecreatedatextuallanguageforexpressinghigh-leveluserscenariosthattakesintoaccountthetemporalaspectofthesecuritypropertiesthatarewritteninTemporalOCL(TOCL)[116].TOCLhasbeensuccessfullyappliedforCommonCriteriaevaluation.Theselanguages,TPandTOCL,arebothintegratedwithintheSmartestingCertifyItToolandallowuserstoguidethegenerationoffunctionalsecuritytests.Wediscusstheseapproachesindetailinthelastsection.

6.1.2ModelCheckingAcommontechniqueforgeneratingtestsinthesecuritytestingistouseamodelchecker.Themodelcheckerproducesatracebyprovidingacounterexampleforagivenproperty

[117].InthescopeofMBFST,Pellegrinoetal.useamodelcheckerforASLanandASLan++4togenerateabstracttestcasesascounterexamplesforthesecurityproperties.Tocreateconcretetestcasesexecutableonthesystemundertesttheyuseatestadapter[118].

JürjenspresentedanapproachtogeneratetracesbyinjectingfaultsintoUMLmodels,andUMLsecstereotypes,usedforverificationofproperties.Furthermore,Fourneretetal.appliedtheUMLsecverificationtechniqueforsecuritypropertiesinthedomainofasmartcardindustry,andthenbasedontheirtransformationintoaTestPurposeLanguagetheygeneratetestscoveringthesecurityproperties[119].

AnotherapproachproposedbyAichernigetal.usesamodelcheckingtechniqueonanInput/OutputLabelTransitionSystem(IOLTS)modeltogeneratetestcases.Theyfurtherinjectfaultsusingmutationoperators,andgeneratetracesusedastestobjectivesfortheTGVtool[120].Thus,wecanalsoclassifythisasarobustnesstechnique.

6.1.3RobustnessTechniquesManyMBFSTtechniquesaimtoaddressrobustnessissuesofthesystemundertest.Mostofthemconcernprotocoltestingandarebasedonmutationorfuzzingoperators.Webrieflydescribeseveralexamplesofeachofthesemodel-basedtechniquesforrobustnesstesting.

Mutationtestingisquitealargefieldofstudyandhasbeenstudiedoverdecades,asshownbyJiaandHarmanintheirsurvey[121].InthecontextofMBTithasbeenappliedinvariousdomains,suchassecuritypolicies,protocols,etc.

Inthedomainofsecuritypolicies,MartinandXiepresentedamutation-testingframeworkforpolicyspecificationsintheir“eXtensibleAccessControlMarkupLanguage”(XACML).TogeneratetestsfromXACMLpoliciestheysynthesizeinputstoachange-impactanalysistool.LeTraonetal.introducedeightmutationoperatorsfortheOrganization-BasedAccessControlOrBACpolicy[122].

Inthedomainofprotocolrobustness,WimmelandJurjensapplymutationoperatorstoSystemStructureDiagramsinAutoFocus,inordertogeneratetestcasesbasedonattackscenarios[123].

Closetotheirwork,Dadeauetal.[124],introducedanapproachtoverifythesecurityofanactualimplementationofasecurityprotocol.Theygeneratetestcasesbasedonfault-modelswrittenintheHigh-LevelSecurityProtocolLanguage(HLPSL).Theyhavesuccessfullyappliedtheirapproachtoasetofexistingsecurityprotocols.

Model-BasedFuzzingusesknowledgeaboutthemessagestructuretosemanticallygenerateinvalid,unexpectedorrandommessages.Themessagescouldbegeneratedfromscratch—generation-basedfuzzingorasmutantsofvalidmessages,mutation-basedfuzzing.Traditionallyinfuzzing,messagesrelatetotheinvalidinputdataoftests.HenceinthecontextofMBFST,fuzzingconsistsofgeneratinginvalidmessagesequences,referredtoasbehavioralfuzzing.

TheDIAMONDSprojectintroducesnoveltechniquesintheresearchfieldofmodel-basedsecuritytestingandparticularlyinfuzzing[105].Shieferdeckeretal.designedamutation-basedfuzzingapproachthatusesfuzzingoperatorsonscenariomodelsspecifiedbysequencediagrams.Thefuzzingoperatorsperformamutationofthediagramsresultinginaninvalidsequence.Contrarytothepreviouswork,wheretestsareexecutedaftertheirgeneration,referredasofflinetesting(seeSection2),Shieferdeckeretal.useonlinebehavioralfuzzing,whichgeneratestestsatruntime[125].

AnotherworkbyJohanssonetal.,closetotheDIAMONDSproject,developedT-Fuzz,ageneration-basedfuzzingframeworkforprotocolimplementationtestingbasedontheTTCN-3language[126].Thisapproachreliesonprotocolmodelsusedforconformancetesting,byreusingthealreadyexistingtestenvironment.Inaddition,theypresentitssuccessfulapplicationtothevalidationoftheNon-AccessStratum(NAS)protocol,usedintelecommunicationnetworksforcarryingsignalingmessages.

6.1.4AnExampleofFunctionalSecurityTestingTechniqueBasedonDynamicTestSelectionTestingsecuritycomponentsiscomplexandexpensiveduetotheirspecificnature.Ononehand,asecuritycomponentimplementssecurityfunctions,suchascryptographicalgorithms,includingsymmetricandpublic-keycyphers.Ontheotherhand,theinteractionwiththecomponentisallowedthroughanApplicationProgramInterface(API).ThesecryptographicAPIsallowtheexchangeofinformationandaccesstocriticaldatainasecurewayanditiscommontodefinethembyspecificationstoincreasesecurity,forexamplePKCS#115orGlobalPlatform6.

TheCertifyIttool,producedbySmartesting,offersanindustrialsolutionforModel-BasedFunctionalSecurityTesting(MBFST).Thissectiongivesashortillustrationofhowthetoolcombinesstaticanddynamictestselectioncriteriatosupportsecuritytesting.

TheSmartestingMBFSTmethodologyfortestingofsecuritycomponents,considersthespecificationastheentrypoint.ThestaticviewofthesystemisrepresentedusinganMBTmodel,sothestructureoftheAPI,forexamplecryptographic,isrepresentedbyaUMLclassdiagram.Inaddition,thedynamicviewofthesystem(itsbehavior)isrepresentedbyOCLconstraints,whichareusedtodrivetestgeneration.Thegeneratedtestcasesareindependentofanycodeimplementation.Toconcretizeandexecutethetestsonthesystemundertestsdifferentadapterlayerscanbecreatedfordifferentimplementations,eachofthembeingdependentonthesystemundertest.

However,asdiscussedpreviously,thisclassicalMBTapproachtofunctionaltestingisnotsufficienttocoversecurityrequirements.Thus,recentversionsofthetool(eg,CertifyItversion6.3)improvethisprocessintwodirections.First,inordertoaddressthecoverageofthesecurity-specifictestrequirements,itusespropertiestorepresentthesecurityfunctionalaspectsofanapplication.Thesepropertiescouldalsobeusedtoevaluatethestrengthofanalreadyexistingtestsuiteintermsofcoverageofthesecurityrequirements(throughthecoverageoftheproperties).Second,itusestheexpertsknowledgetogeneraterealistictestscenariosthatcanbequitelong,whichareexpectedto

revealmorecomplexweaknessesthansingle-purposetestcasesthatcoverfunctionalrequirements.

Moreprecisely,thetoolhelpstouncoverthepotentiallydangerousbehaviorsresultingfromtheinteractionsoftheapplicationwiththesecuritycomponent.Then,inadditiontofunctionaltestingthattargetsthecoverageofthesystem’sbehavior,itsupportstestingofthesecurityrequirementsforthesystem,bycombiningtwodynamictestselectioncriteria(basedonTOCL[116]andTestPurposes(TP)[115])forgenerationoftesttargetsandteststhatcoverthesecurityrequirements.TheTPlanguageisbasedonregularexpressions,andbycombiningkeywordsitallowsthetestengineerstoconceivescenariosintermsofstatestobereachedandoperationstobecalled[115].TheTOCLlanguageisbasedontemporallogicanditallowstheexpressionoftemporalpropertiesthatarecomposedoftwoartifacts:atemporalpatternandascope.Thescopesaredefinedfromeventsanddelimittheimpactofthepattern.Todefinethesequencesappropriateforexecution,thepatternsareappliedonascopeandtheyaredefinedfromeventandstateproperties,expressedbyOCLconstraints[116].

TheseTOCLandTPapproachesarecomplementary,sincetheycoverdifferenttypesofsecurityrequirements.Ononehand,TPcoverssecurityrequirementsthatarerequiredtoexpressspecificapplicationscenarios,andtheyarenotabletoexpressthetemporalaspectsofprecedenceorsuccessionofevents.Ontheotherhand,withTOCLitispossibletocapturethesetemporalaspectsinthetimeaxis.However,inbothcases,thegeneratedsecuritytestsexerciseasmuchaspossibletheunusualinteractionswiththesecuritycomponent.

TheSmartestingCertifyIttoolfurthermonitorsthetestcoverageofthesecurityrequirementsexpressedinTOCL,andgeneratesnewtestcasesifitisnecessarytoincreasethecoverage[116].Finally,thetoolgeneratesacoveragereportthatensuresthetraceabilitybetweenthespecification,thesecurityrequirementsandthegeneratedtests,sothatthatcoveragereportcanbeusedinaproductcertification.Fig.9depictstheTOCLpluginintegratedwithintheCertifyIttoolandshowsitsfeaturesforcoveragemonitoringandreportgeneration.

FIGURE9 TOCLpluginintegratedintoCertifyIt.

ToillustratetheapproachbasedonTOCLandthetool,weusethespecificationof

PKCS#11.PKCS#11definestheAPICryptokithatoffersaninterfaceformanagingthesecurityandinteroperabilityofsecuritycomponents.Thespecificationdefinesvarioussecurityrequirementsforwhichwewereabletogeneratetestcases,forexample:“Ausercannotverifyasigned-messageusingtheC_VerifyoperationwithoutlogintoCryptoki(usingtheoperationC_Login).”Fromatestingperspective,thisrequirementisinterpretedastheusermustcallaC_LoginoperationbeforecallingC_VerifyInit,whichinitiatestheverificationfunction.TheTOCLlanguageallowsthisrequirementtobeexpressedbytwoproperties:onethatdefinesthenominalcaseandasecondcomplementarypropertythatdefinesthe“flawed”case.

Thefirstpropertydefineswheneveraverificationfunctionisperformedwithsuccess(modelbehavior@CKR:OK,CKRbeingatagtorepresentafunctionreturnvalue),itmustbeprecededbyaloginoperation,performedalsowithasuccess.Wecandistinguishthetemporalpattern(beforethefirstoccurrenceofasuccessfulcallofC_VerifyInitfunction)andthescope(eventuallyasuccessfulcalloftheloginfunctionfollowsthepreviousevent,forinstanceC_VerifyInit).

eventuallyisCalled(C_Login,@CKR:OK)beforeisCalled(C_VerifyInit,@CKR:OK)

Thesecondpropertyexpressesthatwhenauserisloggedout,theusermustgothroughtheloginstatebeforeperforminganymessageverificationfunction.

eventuallyisCalled(C_Login,@CKR:OK)

betweenisCalled(C_Logout,@CKR:OK)andisCalled(C_VerifyInit,@CKR:OK)

EachTOCLpropertyistranslatedintoanautomaton,whichallowsthecoverageofthepropertytobemeasured,andalsosupportsthegenerationofadditionalteststoaugmentthecoverageoftheTOCLproperty.

Measuringthecoverageofapropertyisbasedonmeasuringthecoverageoftheautomatontransitionsbyeachalreadyexistingtest.ThisstepisillustratedinFig.9.Theautomatonalsohasanerrorstate,representedbythestatecontainingacross.Ifthisstateisreachedbyanytestsitmeansthatthepropertyisviolated,whichneedsafurtherinvestigationtodefinewhetherthesecuritypropertyistoorestrictivelywritten,ortheMBTmodelcontainserrors.Inthelattercase,ourexperiencefoundthattheTOCLpropertieshelpindebuggingtheMBTmodel.Indeed,MBTmodelsmaycontainerrors,aswellasthecode,andtheircorrectnessisoftentackledbyresearchersandpractitioners.

Oncethecoverageisevaluated,ifanytransitionsoftheautomatonarenotcovered,CertifyItcanproducetesttargetsbasedontheuncoveredautomatontransitionsandthengenerateadditionalabstracttestcasestoaugmentthepropertycoverage.

6.2Model-BasedVulnerabilitySecurityTestingMBThasprovenitsefficiencywhenitcomestotestsecurityproperties,suchasaccesscontrolpoliciesforinstance.However,alargepartofwhatwecallsecurityisimpliedandusuallynotexplicitlyspecifiedinadocument.ThislargepartisreferredasVulnerabilityTesting,whichconsistsofdefining,identifying,quantifyingandprioritizingthesecurity

holes(vulnerabilities)inasystem,network,orapplicationsoftware.Whereassecuritypropertiestesting(MBFST)isabout“verifyingthatagivensecuritypropertyorpolicyismet,”vulnerabilitytesting(MBVT)ismoreabout“verifyingthatuserscannotuseasysteminawayitwasnotintendedtobeused.”Whilemodel-basedvulnerabilitytestingmayhelpconducttestsateveryinfrastructurelayer(networks,systems,applications),mostpapersfocusonapplication-levelvulnerabilities,typicallyforwebapplications.

Indeed,themosaicoftechnologiesusedincurrentwebapplications(eg,HTML5andJavaScriptframeworks)increasestheriskofsecuritybreaches.Thissituationhasledtosignificantgrowthinapplication-levelvulnerabilities,withthousandsofvulnerabilitiesdetectedanddisclosedannuallyinpublicdatabasessuchastheMITRECVE—CommonVulnerabilitiesandExposures[127].Themostcommonvulnerabilitiesfoundinthesedatabasesarealackofresistancetocodeinjection,suchasSQLInjection(SQLI)orCross-SiteScripting(XSS),whichhavemanyvariants.ThiskindofvulnerabilityfrequentlyappearsinthetoplistofcurrentWebapplicationsattacks.

Application-levelvulnerabilitytestingisfirstperformedbydevelopers,buttheyoftenlacksufficientin-depthknowledgeofrecentvulnerabilitiesandrelatedexploits.Thiskindoftestingcanalsobeperformedbycompaniesspecializedinsecuritytesting,inpenetrationtestingforinstance.Buttheymainlyusemanualapproaches,makingthedisseminationoftheirtechniquesverydifficult,andtheimpactofthisknowledgeverylow.Finally,webapplicationvulnerabilityscannerscanbeusedtoautomatethedetectionofvulnerabilities,butsincetheyoftengeneratemanyfalsepositivesandnegatives,humaninvestigationisalsorequired.

Inthissection,wefirstprovideanoverviewonModel-BasedVulnerabilityTesting,whichcanbegroupedintothreemainfamilies:pattern-basedandattack-model-based,modelchecking,andfuzzingapproaches.Wethenpresentapattern-drivenandMBTapproachforwebapplications,proposedwithintheRASENproject.

6.2.1Pattern-BasedandAttack-Model-BasedApproachesThemajorityofMBTpapershavechosennottorepresentthebehavioroftheSUT,butratherthebehaviorofattackers.Whetherthesearereferredtoasattack-modelsorpatterns,theideaistomodelhowanmaliciousindividualwouldconductanattack,stepbystep.Wenowoutlineseveralrepresentativeexampleofthisapproach.

Blomeetal.[128]describeamodel-basedvulnerabilitytestingtoolcalledVERA,standingfor“VERAExecutestheRightAttacks.”Itisbasedonthefactthatusuallythepresenceofavulnerabilityisaprerequisitetodeployanattack,butactuallyexploitingthisvulnerabilityistimeconsuming.Thisapproachreliesonattackermodels,whichcanbeseenasextensionsofMealyfinitestatemachines.Thesemodels,ifcombinedwiththeback-endoftheapproach,canprovidefullyautomatedtestscripts.Theback-endoftheapproachiscomposedof(i)aninstantiationlibrary,whichisbasicallyalistofnominalandmaliciousvectors,(ii)aconfigurationfilethatcontainssystem-specificinformation(cookiedatasuchasthesessionID,targetURL,specificheaders,etc.),and(iii)anXMLfiledescribingtheattackermodeltobetested.Thisapproachcanaddressalargevarietyof

vulnerabilitytypeslikecodeinjection,sourcedisclosure,fileenumeration,remotefileinclusion(RFI),cross-siterequestforgery(CSRF),amongothers.

BozicandWotawa[129]presentaMBTapproachrelyingonattackpatternstodetectwebapplicationvulnerabilitiessuchasSQLinjectionsandXSS.Anattackpatternisaspecificationofamaliciousattack.RepresentedbyaUMLstatemachine,itspecifiesthegoal,conditions,individualactionsandpostconditionsoftherepresentedattack.TestcasesarecomputedandexecutedbybranchingthroughthestatesofthestatemachineandexecutingthecorrespondingmethodsoftheSUT.Thisapproachhasbeenimplementedasatoolchainusingseveralexistingtools,suchasYakinduforthestatemachinemodeling,Eclipsetoencapsulatetheentiresystem,andWebScarabfortheinterpretationofcommunicationbetweentheWebapplicationandclients,andformanualsubmissionofattacks.Experimentshavebeenconductedonthreevulnerableapplications(DVWA,Mutillidae,andBodgeIt)andonereallifeapplication(WordPressAnchor).SQLIandXSSvulnerabilitieswerefoundonMutillidaeandDVWA,onvarioussecuritylevels.NovulnerabilitywasfoundonWordpressAnchorbecauseanadministratorneedstoapproveeachpostsubmittedbyusers.Itrequiresamoredetailedmodeloftheattack.

Weietal.[130]focusonpenetrationtestcaseinputsandproposeamodel-basedpenetrationtestmethodforSQLinjections.First,theyprovideattackmodelsusingtheSecurityGoalModelnotation,whichisamodelingmethodusedtodescribevulnerabilities,securityproperties,attacks,andsoon.Modelsaregenericanddescribegoalsinatop-downfashion.Atypicalgoalisforinstance“stealsysteminformation,”andismodeledastwosubparts:error-messageutilizingandblindinjection.Hence,eachtop-downpathinamodelrepresentsanattackprocessthatrealizesacertainattackgoal.Eachtop-downsuccessfulattackprocessrepresentstheattackscheme,definedasatriple<OBJ,INP,OUT>,OBJbeingtheattackgoal,INPbeingtheattackinput,andOUTbeingthevulnerableresponseoftheWebapplication.Toperformanactualattack,onemustinstantiatethetestcasemodelaccordingtothefingerprintofthewebapplicationandusecertaincoveragecriteriatogenerateexecutabletestcases.TheauthorscreatedanautomatedwebapplicationSQLinjectionvulnerabilitypenetrationtesttoolcalledNKSIscan:itappliesthewidelyused“crawling-attack-analysis”methodtodetecttheSQLinjectionvulnerabilityinsubjectapplications.TheycomparedtheirtechniquewithpopularscannersIBMAppScanandAcunetix.ResultsshowthatNKSIwasabletodiscovermoreflawsthanthosetwoscanners.

Xuetal.[131]presentanapproachtoautomatethegenerationofexecutablesecuritytestsfromThreatModel-ImplementationDescription(TMID)specifications,whichconsistofthreatmodelsrepresentedasPredicate/Transition(PrT)netsandaModel-ImplementationMapping(MIM)description.Athreatmodeldescribeshowamaliciousindividualmaytriggerthesystemundertesttoviolateasecuritygoal.AMIMdescriptionmapstheindividualelementsofathreatmodeltotheirimplementationconstructs.Abstracttestcases(ie,completeattackpaths)arecomputedintwosteps.Firstareachabilitygraphisgeneratedfromthethreatnet.Itrepresentsallstatesandstatetransitionsreachablefromtheinitialmarking.Thenthereachabilitygraphistransformedtoatransitiontreecontainingcompleteattackpathsbyrepeatedlyexpandingtheleaf

nodesthatareinvolvedinattackpathsbutdonotresultfromfiringsofattacktransitions.ConcretetestcasesarederivedbyautomaticallycomposingtheattackpathsandtheMIMdescription.TheapproachhasbeenimplementedinISTA,aframeworkforautomatedtestcodegenerationfromPredicate/Transitionnets,andexperimentshavebeenconductedontworeal-worldsystems.Itshowsgoodresultswithmostvulnerabilitiesbeingfound(90%),whethertheyareweb-relatedvulnerabilities(XSS,SQLi,CSRF,etc.)orprotocol-basedvulnerabilities(FTP).

Salvaetal.[132]presentaModel-BasedDataTestingapproachforAndroidapplicationsthatautomaticallygeneratestestcasesfromintent-basedvulnerabilities,usingvulnerabilitypatterns.ItspecificallytargetstheAndroidIntentMessagingmechanism,whoseobjectiveistoallowsharingofactionsanddatabetweencomponentsusingcontentproviders,inordertoperformoperations.Theconcernisthatattackersmayexploitthismechanismtopassonpayloadsfromcomponenttocomponent,infectingthewholesystemandmakingtheirattackmoresevere.Thisapproachthereforesearchesfordatavulnerabilitiesinsidecomponents.Theautomatedgenerationoftestcasesreliesonthreeartifacts:vulnerabilitypatterns,classdiagrams,andspecifications.VulnerabilitypatternsarespecializedInput–OutputSymbolicTransitionSystems,whichallowformalexpressionofintent-basedvulnerabilities.Apatternformallyexhibitsintent-basedvulnerabilitiesandhelpstodefinetestverdicts.ClassdiagramsarepartiallygeneratedfromthedecompiledAndroidapplicationundertest,andrepresentAndroidcomponentswiththeirtypesandtheirrelationships.TheytypicallyprovidetheActivities(theseareAndroidcomponentsthatdisplayscreenstoletusersinteractwithprograms)orServicescomposedwithcontentproviders.SpecificationsaregeneratedfromtheAndroidmanifest.Theyexpressthebehaviorofcomponentsafterthereceiptofintentscombinedwithcontent-providerrequests.Testcasegenerationisperformedbycomposingthethreeartifacts.ThismethodhasbeenimplementedinatoolcalledAPSET,andhasbeenappliedtoseveralreallifeapplications.Resultssupporttheeffectivenessofthetool,findingvulnerabilitiesinpopularAndroidapplicationssuchasYouTubeandMaps.

Apattern-drivenandMBTapproachhasbeendevelopedbyVernotteetal.[133]forvariousvulnerabilitytypes,technicalandlogical.TheapproachreliesonattackpatternsandabehavioralmodeloftheSUT.Thetestgeneratorusesattackpatternsasguides,andfollowseachstepintothemodel.Ifeachstephasbeenfulfilled,anabstracttestcaseiscomputed.AmorethoroughpresentationofthisapproachmaybefoundinSection6.2.4.

6.2.2Model-CheckingApproachesTestcasescanalsobeobtainedbyusingamodelchecker.Givenawebsitespecification/model,atypicalmodel-checkingapproachwillinjectfaultsintothemodelanduseamodelcheckertogenerateattacktraces.Varioustechniqueshavebeenproposedtodetecttechnicalvulnerabilities(XSS,SQLI,CSRF,etc.)aswellaslogicalvulnerabilities(authenticationbypass,insecuredirectobjectreferences,etc.).Weshalldiscussthreeexamplesofsuchmodel-checkingapproaches.

Buchleretal.[134]representtheSUTusingasecureAVANTSSARSpecification

Language(ASLan++)model,wherealltracesfulfillthespecifiedsecurityproperties.Alibraryoffaultinjectionoperatorshasbeendeveloped.Thegoalistoapplyafaultinjectionoperatortothemodel,anduseamodelcheckertoreportanyviolatedsecuritygoal.Ifasecuritygoalhasindeedbeenviolated,thereportedtracethenconstitutesanAbstractAttackTrace(AAT).Theattacktracesaretranslatedintoconcretetestcasesbyusingatwo-stepmapping:thefirststepistotranslateanAATintoWAAL(WebApplicationAbstractLanguage)actions,thesecondstepistotranslateWAALactionsintoexecutablecode.Anattackmaybeconductedinafullyautomatedfashion,atthebrowserlevel.Insomespecificcases(disabledinputelements,etc.),atestexpertmayberequiredtocraftHTTPlevelrequestsinordertorecoverfromtheerror.Thisapproachishighlyamenabletofullautomation.

Rocchettoetal.[135]presentaformalmodel-basedtechniqueforautomaticdetectionofCSRFduringthedesignphase.ItisbasedontheASLan++languagetodefinetheseveralentitiesinvolved(client,server)andtheirinteractions.Theclientisusedasanoraclebytheattacker,andthemodeliscenteredaroundthewebserverandextendstheworkofDolev–Yao(usuallyusedforsecurityprotocolanalysis).Togeneratetests,themodelissubmittedtotheAVANTSSARplatform,which,whenaCSRFisfound,returnsanabstractattacktracereportingthelistofstepsanattackerhastofollowinordertoexploitthevulnerability.ThistechniquetakesintoaccountthatthewebservermayhavesomeCSRFprotectioninplace,andwilltrytobypassit.ItwilltypicallylookforCSRFtoken-relatedflaws,forinstanceifthetokensareuniqueforeachclient,andforeachclient/serverinteraction.Ifnoattacktraceisproduced,thespecificationisconsideredsaferegardingCSRF.Theauthorsassumethatattackerscanlistentothenetworkandbuildtheirattackuponthetransactionsbetweenaclientandtheserver.

Felmetsgeretal.[136]presentadvancestowardtheautomateddetectionofapplicationlogicvulnerabilities,combiningdynamicexecutionandmodelcheckinginanovelway.Dynamicexecutionallowsfortheinferenceofspecificationsthatcaptureawebapplicationslogic,bycollectinglikelyinvariants.Alikelyinvariantisderivedbyanalyzingthedynamicexecutiontracesofthewebapplicationduringnormaloperation,andcapturesconstraintsonthevaluesofvariablesatdifferentprogrampoints,aswellasrelationshipsbetweenvariables.Theintuitionisthattheobserved,normalbehaviorallowsonetomodelpropertiesthatarelikelyintendedbytheprogrammer.Modelcheckingisusedwithsymbolicinputstoanalyzetheinferredspecificationswithrespecttothewebapplicationscode,andtoidentifyinvariantsthatarepartofatrueprogramspecification.Avulnerabilityisthereforeanyviolationofsuchaninvariant.ThistechniquehasbeenimplementedinatoolcalledWaler(WebApplicationLogicErrorsAnalyzeR),whichtargetsservlet-basedwebapplicationswritteninJava.Uptonow,Walerdetectsarestrictedsetoflogicflawsandiscurrentlylimitedtoservlet-basedwebapplications,butwasstillabletofindpreviouslyundetectedvulnerabilitiesinreal-lifeapplicationswhileproducingalownumberoffalsepositives.

6.2.3FuzzingApproachesFuzzingisextensivelyusedinvulnerabilitytesting[137]tointroducemalformeddataor

mutatenominalvaluestotriggerflawedcodeinapplications.Fuzzingtechniquesareusuallyverycheaptodeploy,donotsufferfromfalsepositives,butlackanexpected-resultmodelandthereforerelyoncrashesandfailstoassignaverdict.Twomainfuzzingtechniquesexist:mutationbasedandgenerationbased.Mutationfuzzingconsistsofalteringasamplefileordatafollowingspecificheuristics,whilegeneration-basedfuzzerstaketheinputspecificationandgeneratetestcasesfromit.Fuzzingmaybeusedforcraftingmaliciousinputdata[138],orcraftingerroneouscommunicationmessages[139].

TheapproachpresentedbyDuchene[138]consistsofmodelingtheattacker’sbehavior,anddrivingthismodelbyageneticalgorithmthatevolvesSUTinputsequences.Itrequiresastate-awaremodeloftheSUT,eitherderivedfromanASLan++descriptionorinferredfromtracesofvalid/expectedSUTexecution.Thismodelisthenannotatedusinginputtaintdata-flowanalysis,tospotpossiblereflections.ConcreteSUTinputsaregeneratedwithrespecttoanAttackInputGrammarwhichproducesfuzzedvaluesforreflectedSUTinputparameters.ThefitnessfunctiondependsontheobtainedSUToutputfollowingtheinjectionofaconcreteSUTinput.Itcomputestheveracityofaninputbylookingforcorrelations,usingthestringdistancebetweenagiveninputparametervalueandasubstringoftheoutput.Twogeneticoperatorsareused:mutationandcross-over.ItisanefficienttechniquefordetectingXSS,asitgoesbeyondtheclassicalXSSevasionfiltersthatmaynotbeexhaustive.SuchatechniquealsotacklesmultistepXSSdiscoverybyusingamorecomplexstringmatchingalgorithmtogenerateanannotatedFSM,inordertoinspecttheSUTtofindthepossibilitiesofXSSatcertainplaces.

Amodel-basedbehavioralfuzzingapproachhasbeendesignedbyWangetal.[139]todiscovervulnerabilitiesofDatabaseManagementSystems(DBMS).ADBMSdefinesaformatrulethatspecifiespacketformatandabehaviorrulethatspecifiesitssemanticsandfunctionality.Thisapproachisbasedontwomainartifacts.Thefirstartifactisabehavioralmodel,whichincludesfuzzingpatternsandbehavioralsequences.ThisisobtainedfromabehavioranalysisofDBMS(protocolformatanalysis,attacksurfaceanalysis,etc.).Afuzzingpatternexpressesthedatastructureofpackets,theneedsofsecuritytesting,andthedesignstrategyforvulnerabilitydiscovery.AbehavioralsequencedefinesthemessagetransferorderbetweenclientandDBMS.ThesecondartifactisaDBMSFuzzercomposedofatestinstance(adetailedtestscriptbasedonfuzzingpatterns),andafinitestatemachinemodelEXT-NSFSMusedforsemivalidtestcasegenerationbasedonbehavioralsequencesandtestinstances.Theauthorsdescribeageneralframeworkforbehavioralfuzzingthathasbeenimplementedandusedinseveralexperiments.Itallowsforthegenerationofthousandsoffuzzinginstances,anddespiteafewerrorsofanalysisandscript,thetoolwasabletodiscoverbufferoverflowvulnerabilities,10ofwhichwerenotreleasedyet.

6.2.4ExampleofaPattern-DrivenandModel-BasedVulnerabilityTestingApproachTheVESONTIOteamfromtheFEMTO-ST/DISCinstitutehasdesignedapattern-drivenModel-BasedVulnerabilityTestingapproach,proposedwithintheframeworkoftheRASENproject,togenerateandexecutevulnerabilitytestcases.ItcombinesMBTand

fuzzingtechniques,anddrivesthetestgenerationbysecuritytestpatternsresultingfromriskassessment.Thisapproachaimstoimprovetheaccuracyandprecisionofvulnerabilitytesting.Itissupportedbytoolsthatautomatethedetectionofvulnerabilities,particularlyinwebapplications.

Theprocess,showninFig.10,iscomposedofthefourfollowingactivities:

FIGURE10 Pattern-drivenandmodel-basedvulnerabilitytestprocess.

1.TheModelingactivity.AsforeveryMBTapproach,themodelingactivityconsistsofdesigninganMBTmodelthatcanbeusedtoautomaticallygenerateabstracttestcases.ThePMVTapproach,basedontheCertifyIttechnology,requiresamodeldesignedusingtheUML4MBTnotation:UMLclassdiagramsspecifythestaticstructure,whilestatediagramsdescribethedynamicbehavioroftheapplication(notablythenavigationbetweenpages).Toeaseandacceleratethismodelingactivity,aDomainSpecificModelingLanguage(DSML)hasbeendeveloped,calledDASTML,whichallowstheglobalstructureofawebapplicationtobemodeled.Itiscomposedofthreeentities:Page,ActionandDatawithvariouslinkpossibilitiesbetweenthethree.Onlyrelevantinformationtovulnerabilitytest

casegenerationisrepresented,suchastheavailablepages(orscreensincaseofsingle-urlapplications),theavailableactionsoneachpage,andtheuserinputsofeachaction(potentiallyusedtoinjectanattackvector).AnalgorithmperformstheautomaticinstantiationoftheUML4MBTnotationbasedonagivenDASTMLmodel.

2.TheTestPurposedesignactivity.Thisactivityconsistsofformalizingatestprocedurefromeachvulnerabilitytestpattern(vTP)thatthegeneratedtestcaseshavetocover.vTPsprovideastartingpointforsecuritytestcasederivationbygivinginformationonhowtocomputeappropriatevulnerabilitytestcasesdependingonthekindofvulnerability.ThesepatternsaretypicallygatheredfrompublicdatabasessuchasCVEandOWASP,andfromresearchprojectssuchastheITEA2DIAMONDSproject.BecausevTPsareinformalspecificationstheyneedtobetranslatedintoamachine-readablelanguage,toallowtheautomaticcomputationoftestcasesbythegenerationengine.Henceeachproceduretargetingadedicatedvulnerabilityisgivenbyatestpurpose,whichisahigh-levelexpressionthatformalizesatestingobjectivetodrivetheautomatedtestgenerationonthetestmodel.Basically,suchatestpurposecanbeseenasapartialalgorithmthatdefinesasequenceofsignificantstepsthathastobeexecutedbythetestcasescenario.Eachsteptakestheformofasetofoperationsorbehaviorstobecovered,orspecificstatetobereachedonthetestmodel,inordertoassesstherobustnessoftheapplicationundertestwithrespecttothevulnerabilitythatisbeingtested.ThetestpurposelanguagesupportscomplexpatternmodelingbymakinguseofOCLconstraintstodefinespecificstatestoreachanddatatocollect,and“foreach”statementstoiterateoverenumerationliterals(abstractdata)andthusunfoldagiventestpurposeintonumerousabstracttestcases.

3.TheTestGenerationactivity.Thetestgenerationprocessautomaticallyproducesabstractvulnerabilitytestcases,includingtheexpectedresults.Itconsistsofinstantiatingthevulnerabilitytestpurposesonthetestmodeloftheapplicationundertest:thetestmodelandthetestpurposesarebothtranslatedintoelementsanddatadirectlycomputablebythetestgeneratorCertifyIt.TestcasegenerationisperformedbyinstantiatingtheselectedtestpurposesonthebehavioralUML4MBTtestmodelspecifyingthewebapplicationundertest.Notably,testpurposesaretransformedintotesttargets,whicharedefinedbyasequenceofintermediateobjectivesusedbythetestgenerationengine.Thetesttargetsarethenexecutedonthetestmodeltogeneratetheabstracttestcases.Inthisway,eachtestpurposeproducesoneormoreabstracttestcasesthatverifythetestpurposespecification,whilesatisfyingtheconstraintsofthebehavioraltestmodel.

4.TheAdaptation,TestExecutionandObservationactivity.Theabstracttestcasesarefinallyexportedintothetestexecutionenvironment.ThisconsistsofautomaticallycreatingaJUnittestsuite,inwhicheachabstracttestcaseisexportedasaJUnittestcaseskeletonthatembedsthetestsequenceandtheobservationproceduresinordertoautomatetheverdictassignment.However,duringthemodelingactivityalldatausedbytheapplicationismodeledatanabstractlevel.Asaconsequence,testcasesareabstractandcannotbeexecuteddirectlyastheyare.Tobridgethegap,testengineersmustlinktheabstractdatatoconcretedatain

ordertoprovideexecutabletestscripts.Itshouldbeemphasizedthatallabstractoperations(“login,”“register,”“goto_page,”andsoon)areautomaticallyconcretizedusingbasicHTMLUnitprimitives.Asthissequenceofprimitivesisrathergeneric,testengineersmayhavetotweakthegeneratedcodeifthewebapplicationundertestrequiresit.

Insummary,thekeyaspectsofthisapproachare:

1.Theformalizationofvulnerabilitytestpatternsusinggenerictestpurposestodrivethetestgenerationengine;

2.TheuseofaDSMLtoeaseandacceleratethefunctionalmodelingoftheWebapplicationundertest;

3.Thefullautomationofthetestingprocess,includingtestgeneration,testexecutionandverdictassignment.

ThisPMVTapproachhasbeenfoundtobesuitableforarangeofvulnerabilitytypes,includingtechnicalones(XSS,SQLinjections,CSRF)aswellaslogicalones(authenticationbypass,privilegeescalation).

7ConclusionandFutureChallengesThischapterhasreviewedtheadvancesinMBToverthelastfewyears.Itisapparentthattherehavebeenmanyresearchadvancesandtoolimprovements,sothatthereisnowarichvarietyofnotationsandlanguagesusedforexpressingtheMBTmodels.Somestandardchoicesofmodelingnotationsarestartingtoemergeformodelingbusinessapplications(notably,UMLandBPMN),andtheareaofGUIrippinghasbecomeapopularwayofobtainingmodelsfortestingGUIs.

ThepastdecadehasseencontinualimprovementsbothintheMBTalgorithmsusedtogeneratetests,andintheefficiencyandflexibilityoftheunderlyingsolvertechnologiesusedduringtestgeneration.Theseimprovementshavetheincrementaleffectofallowingthetoolstoscaletolargerproblems,andproducingbetterqualitytestsuites.ThescopeofMBTpracticehasalsoexpandedtoincludeawiderangeofdifferentkindsofsecuritytesting,asdiscussedinSection6.

Butwithregardstoindustryadoption,theprogressisslower.Thematurityanalysisofvarioustopicsrelatedtosoftwaretestingintheindustry,bytheTrendsandBenchmarkSwitzerlandReportfrom2013[140],showedthatMBTisplacedintheintroductionphase.Aswehaveseenbefore,thisisstillthecasein2015,evenifsomeevidenceshowsgradualprogressinthedisseminationoftheapproach.

ItisofcourseverydifficulttopredictthefuturepenetrationofMBTintovariousindustryareas.BesidetheMBTUserSurvey2014presentedinSection3.1,weneedmoredetailedstudiestobetterunderstandthekindsofsoftwarewhereMBTismosteffective,andthecurrentstrengthsandweaknessesoftheapproach.Forexample,onerecentstudybySchulzeetal.[141]comparedtheeffortandthebenefitsofmanualtestingvsMBTandfoundthatMBTdetectedsignificantlymorefaults,butrequiredmoretimeup-frontactivities(modelingandimplementingtestinfrastructure).TheyalsonotedthatMBTwasbetteratdetectingfunctionalissues,whereasthemanualtestingwasmoreefficientfordetectingGUIissues.Overall,theyconcludedthat“MBTdetectedmoreissueswithhigherseverityscoreswithlesstestingandanalysiseffortthanmanualtesting.However,itrequiredmoreinitialeffort,whicheventuallywouldpayoffifseveralversionsofthesamesystemwouldbetested”.

ThisstudypointsoutoneofthekeyissuesregardingthepenetrationofMBTinindustry,whichishowtoreduceeffortwhenusingMBT.Ofcourse,intheMBTprocesstheseeffortsaremainlyrelatedtomodelingactivities(ie,creationandmaintenanceoftheMBTmodel).ThisisalsolinkedwiththeadaptationoftheMBTtechnologiestocurrentsoftwaretesterskills(typicallyISTQBFoundationLevelCertifiedTesters),whooftenhavenoorveryfewsoftwaremodelingskills.Consequently,MBTmodelingshouldbeadaptedtobemoreeasilyusedbysuchprofessionaltesters.

TheseaspectsmaybeseenasdriversfortheresearchchallengesfortheMBTresearchcommunity.ThereareseveraldirectionswecanmentionthatdirectlyaddresstheseaspectsofMBTeffortreductionandmodelingsimplification:

FullyorpartiallyautomatethecreationoftheMBTmodel.InthecontextofMBTof

webapplications,andmoregenerally,model-basedGUItesting(seeSection4.3),GUI-rippingtechniquescanautomatetheconstructionofthemodel.Thishastobeextendedtoothertypesoftestingbyanalyzingexistingartifactstoprovidetherightinformation.Onelimitofthisapproachisofcoursetheoracleproblem:extractinginformationfromthesystemundertestcanhelptodeterminesequenceoftestactions.Buttheexpectedresultsneedtobederivedfromanexternalsource,notfromthebuggyapplication.

ProvideDSMLtosimplifythecreationandmaintenanceoftheMBTmodel.Domain-SpecificModelingLanguagesadaptthemodelcreationprocesstothesemanticsoftheapplicationdomain.Domain-specificmodelingallowsusingexistingdomainterminology,withknownsemantics,andfamiliarnotationinaspecificapplicationdomain.Forexample,DSMLmaybeusedforERPMBT,byadaptingthemodelinglanguagetothetargetedbusinessdomain,orinthecontextofaircrafttrafficmonitoringsystems,byspecifyingMBTmodelsusingspecificdomainconceptssuchasplane,navigationpathandmonitoringzone.TheuseofDSMLinMBTmayallowfastermodeldevelopmentandwidermodelaccessibility,ascomparedtotheuseofgeneral-purposemodelinglanguages.

Betterreuseofexistingrequirementsartifacts.Atsystemoracceptancetestinglevels,testingobjectivesarestronglyrelatedtosystemrequirements(particularlyfunctionalrequirements).Requirementsengineeringleadstoalargevarietyofartifacts,whichareoftenveryinformalsuchasuserstoriesorusecasedescriptions[142].Thisleadsalsotostructuredartifactslikebusinessprocessmodelsorinterfacerequirementsspecification.AutomatingthereuseofsuchinformalartifactsmayfacilitateandaccelerateMBTmodelingactivities.Forexample,derivingtestcasesorpartialMBTmodelsfromusecasesmayhelptocapturethebasicflowofeventsandthealternateflowsofevents.

Someotherpracticalchallengesforthefuture,particularlyfortheincreaseduseofMBTinindustry,are:

•theportabilityofmodelsbetweendifferentMBTtools—evenwhentwotoolsusesimilarmodelingnotationssuchasUML,theytendtousedifferentsubsetsofthosenotations,somodelsarenotimmediatelytransferablebetweenMBTtools.Thisislikelytoimprovegraduallyascertainnotationsandtoolsachievemarketdominance;

•thetrainingofnewMBTusersinhowtomodelfortesting,andhowtouseMBTtechniquesinpractice.TheproposedISTQBcertifiedtesterextensionforMBTisexpectedtomakeabigimprovementinthisarea,butlearninghowtodesigngoodMBTmodelsisanontrivialtask,soon-goingtraining,shepherdingandsupportsystemswillbeneeded;

•theurgencyforfurtherstudiesthatcomparetheuseofMBTwithothertestingapproaches.

References[1]LeeC.APractitioner’sGuidetoSoftwareTestDesign.Norwood,MA:ArtechHouse,Inc.2004.

[2]HungQ.,MichaelH.,BrentK.GlobalSoftwareTestAutomation:ADiscussionofSoftwareTestingforExecutives.Cupertino,USA:HappyAbout;2006.

[3]PretschnerA.,PrenningerW.,WagnerS.,KühnelC.,BaumgartnerM.,SostawaB.,ZölchR.,StaunerT.Oneevaluationofmodel-basedtestinganditsautomation.In:Proceedingsofthe27thInternationalConferenceonSoftwareEngineering,ICSE’05,St.Louis,MO,USA.NewYork,NY:ACM;2005:1-58113-963-2392–401.doi:10.1145/1062455.1062529.

[4]UttingM.,LegeardB.PracticalModel-BasedTesting—AToolsApproach.SanFrancisco,CA:MorganKaufmann;2006.0123725011.

[5]MarkU.,AlexanderP.,BrunoL.Ataxonomyofmodel-basedtestingapproaches.Softw.Test.Verif.Rel.1099-16892012;22(5):297–312.doi:10.1002/stvr.456.

[6]JorgensenP.C.ACraftsman’sApproach.firstedBocaRaton,FL:AuerbachPublications;2009.

[7]ChowT.S.Testingsoftwaredesignmodeledbyfinite-statemachines.IEEETrans.Softw.Eng.0098-55891978;4(3):178–187.

[8]Pairwisewebsiteathttp://www.pairwise.org/,2015

[9]WolfgangG.,NicolasK.,KeithS.,VictorB.Model-basedqualityassuranceofprotocoldocumentation:toolsandmethodology.Softw.Test.Verif.Rel.1099-16892011;21(1):55–71.

[10]BinderR.,BrunoL.,AnneK.Model-basedtesting:wheredoesitstand?Commun.ACM.0001-07822015;58(2):52–56.doi:10.1145/2697399.

[11]AichernigB.K.Contract-basedtesting.In:Berlin:Springer;3-540-20527-634–48.FormalMethodsattheCrossroads:FromPanaceatoFoundationalSupport,LectureNotesinComputerScience.2003;vol.2757.

[12]BaudinP.,CuoqP.,FilliâtreJ.-C.,MarchéC.,MonateB.,MoyY.,PrevostoV.ACSL:ANSI/ISOCspecificationlanguageversion1.7.2013.http://frama-c.com/download/acsl-implementation-Fluorine-20130601.pdf.

[13]BarnettM.,LeinoK.R.M.,SchulteW.TheSpec#programmingsystem:anoverview.In:ProceedingsoftheInternationalWorkshoponConstructionandAnalysisofSafe,SecureandInteroperableSmartDevices(CASSIS’04),Marseille,France,LectureNotesinComputerScience,vol.3362,Springer;2004:49–69.

[14]JackyJ.,VeanesM.NModel,online2006.2015.http://nmodel.codeplex.com/(lastaccessMarch2015).

[15]CheonY.,LeavensG.T.Asimpleandpracticalapproachtounittesting:theJMLandJUnitway.In:MagnussonB.,ed.16thEuropeanConferenceonObject-OrientedProgramming,ECOOP2002,LectureNotesinComputerScience,vol.2374,Springer,Berlin;2002:3-540-43759-2231–255.

[16]ZimmermanD.M.,NagmotiR.JMLUnit:thenextgeneration.In:FormalVerificationofObject-OrientedSoftware,LectureNotesinComputerScience.Springer;183–197.2010;vol.6528.

[17]LeavensG.T.,BakerA.L.,RubyC.JML:anotationfordetaileddesign.In:KilovH.,RumpeB.,SimmondsI.,eds.BehavioralSpecificationsofBusinessesandSystems.Boston,MA:KluwerAcademicPublishers;1999:175–188(36references).

[18]GligoricM.,GveroT.,JagannathV.,KhurshidS.,KuncakV.,MarinovD.TestgenerationthroughprogramminginUDITA.In:ICSE(1).2010:225–234.http://doi.acm.org/10.1145/1806799.1806835.

[19]VisserW.,HavelundK.,BratG.P.,ParkS.,LerdaF.Modelcheckingprograms.Autom.Softw.Eng.2003;10(2):203–232.http://dx.doi.org/10.1023/A:1022920129859.

[20]HeideggerP.,ThiemannP.JSConTest:contract-driventestingandpatheffectinferenceforJavaScript.J.Obj.Technol.2012;11(1):1–29.http://dx.doi.org/10.5381/jot.2012.11.1.a6.

[21]MirshokraieS.Effectivetestgenerationandadequacyassessmentforjavascript-basedwebapplications.In:Proceedingsofthe2014InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2014,SanJose,CA,USA.NewYork,NY:ACM;2014:978-1-4503-2645-2453–456.doi:10.1145/2610384.2631832.

[22]EnderlinI.,DadeauF.,GiorgettiA.,BenOthmanA.Praspel:aspecificationlanguageforcontract-basedtestinginPHP.In:ICTSS.2011:64–79.

[23]BakerP.,DaiZ.R.,GrabowskiJ.,HaugenO.,SamuelssonE.,SchieferdeckerI.,WilliamsC.E.TheUML2.0testingprofile.In:Proceedingsofthe8thConferenceonQualityEngineeringinSoftwareTechnology(CONQUEST),Nuremberg,Germany.2004:181–189.

[24]DaiZ.R.,GrabowskiJ.,NeukirchenH.,PalsH.FromdesigntotestwithUML.In:GrozR.,HieronsR.M.,eds.TestingofCommunicatingSystems.Berlin:Springer;978-3-540-21219-533–49.LectureNotesinComputerScience.2004;vol.2978.

[25]SawantV.,ShahK.ConstructionoftestcasesfromUMLmodels.In:ShahK.,LakshmiGortyV.R.,PhirkeA.,eds.TechnologySystemsandManagement.Berlin:Springer;978-3-642-20208-761–68.doi:10.1007/978-3-642-20209-4.CommunicationsinComputerandInformationScience.2011;vol.145.

[26]CantenotJ.,AmbertF.,BouquetF.Testgenerationwithsatisfiabilitymodulotheoriessolversinmodel-basedtesting.Softw.Test.Verif.Rel.1099-

16892014;24(7):499–531.doi:10.1002/stvr.1537.

[27]YueT.,AliS.,BriandL.AutomatedtransitionfromusecasestoUMLstatemachinestosupportstate-basedtesting.In:FranceR.B.,KuesterJ.M.,BordbarB.,PaigeR.F.,eds.ModellingFoundationsandApplications.Berlin:Springer;978-3-642-21469-1115–131.doi:10.1007/978-3-642-21470-7\_9.LectureNotesinComputerScience.2011;vol.6698.

[28]NogueiraS.,SampaioA.,MotaA.Testgenerationfromstatebasedusecasemodels.Form.Asp.Comput.0934-50432014;26(3):441–490.doi:10.1007/s00165-012-0258-z.

[29]PickinS.,JezequelJ.-M.UsingUMLsequencediagramsasthebasisforaformaltestdescriptionlanguage.In:BoitenE.A.,DerrickJ.,SmithG.,eds.IntegratedFormalMethods.Berlin:Springer;978-3-540-21377-2481–500.doi:10.1007/978-3-540-24756-2\_26.LectureNotesinComputerScience.2004;vol.2999.

[30]RountevA.,KaganS.,SawinJ.Coveragecriteriafortestingofobjectinteractionsinsequencediagrams.In:CerioliM.,ed.FundamentalApproachestoSoftwareEngineering.Berlin:Springer;978-3-540-25420-1289–304.doi:10.1007/978-3-540-31984-9\_22.LectureNotesinComputerScience.2005;vol.3442.

[31]TripathyA.,MitraA.Testcasegenerationusingactivitydiagramandsequencediagram.In:AswathaK.M.,SelvaraniR.,KumarT.V.S.,eds.ProceedingsofInternationalConferenceonAdvancesinComputing.India:Springer;978-81-322-0739-9121–129.doi:10.1007/978-81-322-0740-5\_16.AdvancesinIntelligentSystemsandComputing.2012;vol.174.

[32]PanthiV.,MohapatraD.AutomaticTestCaseGenerationUsingSequenceDiagram.In:KumarA.,RamaiahM.S.,KumarT.V.S.,eds.ProceedingsofInternationalConferenceonAdvancesinComputing,AdvancesinIntelligentSystemsandComputing,vol.174,Springer,India;2012:978-81-322-0739-9277–284.doi:10.1007/978-81-322-0740-5\_33.

[33]KumarR.,BhatiaR.K.Interactiondiagrambasedtestcasegeneration.In:KrishnaP.V.,BabuM.R.,AriwaE.,eds.GlobalTrendsinInformationSystemsandSoftwareApplications.Berlin:Springer;978-3-642-29215-6202–211.doi:10.1007/978-3-642-29216-3\_23.CommunicationsinComputerandInformationScience.2012;vol.270.

[34]JenaA.K.,SwainS.K.,MohapatraD.P.TestcasecreationfromUMLsequencediagram:asoftcomputingapproach.In:JainL.C.,PatnaikS.,IchalkaranjeN.,eds.IntelligentComputing,CommunicationandDevices.India:Springer;978-81-322-2011-4117–126.doi:10.1007/978-81-322-2012-1\_13.AdvancesinIntelligentSystemsandComputing.2015;vol.308.

[35]ReijersA.H.,vanWijkS.,MutschlerB.,LeursM.BPMinpractice:whoisdoing

what?In:HullR.,MendlingJ.,TaiS.,eds.BusinessProcessManagement.Berlin:Springer;978-3-642-15617-545–60.doi:10.1007/978-3-642-15618-2\_6.LectureNotesinComputerScience.2010;vol.6336.

[36]JensenS.H.,ThummalapentaS.,SinhaS.,ChandraS.TestGenerationfromBusinessRules.IBMResearchReport;2014Tech.Rep.RI14008.

[37]MeckeC.AutomatedtestingofmySAPbusinessprocesses.In:MeyerhoffD.,LaibarraB.,vanderPouwKraanR.,WalletA.,eds.SoftwareQualityandSoftwareTestinginInternetTimes.Berlin:Springer;2002:978-3-540-42632-5261–279.doi:10.1007/978-3-642-56333-1\_17.

[38]AndreasH.,TobiasG.,VolkerG.,HolgerF.Businessprocess-basedtestingofwebapplications.In:zurMuehlenM.,SuJ.W.,eds.BusinessProcessManagementWorkshops.Berlin:Springer;978-3-642-20510-1603–614.doi:10.1007/978-3-642-20511-8.LectureNotesinBusinessInformationProcessing.2011;vol.66.

[39]AnandS.,BurkeE.K.,ChenT.Y.,ClarkJ.,CohenM.B.,GrieskampW.,HarmanM.,HarroldM.J.,McMinnP.Anorchestratedsurveyofmethodologiesforautomatedsoftwaretestcasegeneration.J.Syst.Softw.0164-12122013;86(8):1978–2001.doi:10.1016/j.jss.2013.02.061.

[40]WangY.,YangN.TestcasegenerationofwebservicecompositionbasedonCP-nets.J.Softw.2014;9(3).http://ojs.academypublisher.com/index.php/jsw/article/view/jsw0903589595.

[41]YuanX.,CohenM.B.,MemonA.M.GUIinteractiontesting:incorporatingeventcontext.IEEETrans.Softw.Eng.g.0098-55892011;37(4):559–574.doi:10.1109/TSE.2010.50.

[42]MemonA.,BanerjeeI.,NguyenB.N.,RobbinsB.ThefirstdecadeofGUIripping:extensions,applications,andbroaderimpacts.In:20thWorkingConferenceonReverseEngineering(WCRE),2013.2013:11–20.doi:10.1109/WCRE.2013.6671275.

[43]MemonA.,NguyenB.N.GUITAR.2015.http://sourceforge.net/projects/guitar/(lastaccessMarch2015).

[44]HacknerD.R.,MemonA.M.TestcasegeneratorforGUITAR.In:Companionofthe30thInternationalConferenceonSoftwareEngineering,ICSECompanion’08,Leipzig,Germany.NewYork,NY:ACM;2008:978-1-60558-079-1959–960.doi:10.1145/1370175.1370207.

[45]AmalfitanoD.,FasolinoA.R.,TramontanaP.,DeCarmineS.,MemonA.M.UsingGUIrippingforautomatedtestingofandroidapplications.In:Proceedingsofthe27thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering,ASE2012,Essen,Germany.NewYork,NY:ACM;2012:978-1-4503-1204-2258–261.doi:10.1145/2351676.2351717.

[46]ArltS.,BorromeoP.,SchfM.,PodelskiA.ParameterizedGUITests.In:Nielsen

B.,WeiseC.,eds.TestingSoftwareandSystems.Berlin:Springer;978-3-642-34690-3247–262.LectureNotesinComputerScience.2012;vol.7641.

[47]ArltS.,PodelskiA.,BertoliniC.,SchafM.,BanerjeeI.,MemonA.M.LightweightstaticanalysisforGUItesting.In:23rdIEEEInternationalSymposiumonSoftwareReliabilityEngineering(ISSRE),2012.IEEE;2012:301–310.

[48]GrossF.,FraserG.,ZellerA.EXSYST:Search-basedGUItesting.In:34thInternationalConferenceonSoftwareEngineering(ICSE),2012.2012:1423–1426.doi:10.1109/ICSE.2012.6227232ISSN0270-5257.

[49]MoreiraR.M.L.M.,PaivaA.C.R.,MemonA.Apattern-basedapproachforGUImodelingandtesting.In:24thIEEEInternationalSymposiumonSoftwareReliabilityEngineering(ISSRE),2013.2013:288–297.doi:10.1109/ISSRE.2013.6698881.

[50]CohenM.B.,HuangS.,MemonA.M.AutoInSpec:usingmissingtestcoveragetoimprovespecificationsinGUIs.In:23rdIEEEInternationalSymposiumonSoftwareReliabilityEngineering(ISSRE),2012.2012:251–260.doi:10.1109/ISSRE.2012.33ISSN1071-9458.

[51]BolisF.,GargantiniA.,GuarnieriM.,MagriE.,MustoL.Model-driventestingforwebapplicationsusingabstractstatemachines.In:GrossniklausM.,WimmerM.,eds.CurrentTrendsinWebEngineering.Berlin:Springer;978-3-642-35622-371–78.LectureNotesinComputerScience.2012;vol.7703.

[52]LelliV.,BlouinA.,BaudryB.ClassifyingandqualifyingGUIdefects.In:IEEEInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST2015).IEEE;April2015:1–10.doi:10.1109/ICST.2015.7102582.

[53]ZhuH.,BelliF.Advancingtestautomationtechnologytomeetthechallengesofmodel-basedsoftwaretesting.J.Inform.Softw.Technol.2009;51(11):1485–1486.

[54]DustinE.,GarrettT.,GaufB.ImplementingAutomatedSoftwareTesting:HowtoSaveTimeandLowerCostsWhileRaisingQuality.Indianapolis,USA:AddisonWesleyProfessional;2009.0-32-158051-6.

[55]ThimblebyH.W.ThedirectedChinesepostmanproblem.Soft.Pract.Exp.2003;33(11):1081–1096.http://dx.doi.org/10.1002/spe.540.

[56]DavisM.,LogemannG.,LovelandD.Amachineprogramfortheorem-proving.Commun.ACM.1962;5(7):394–397.

[57]deMouraL.,BjørnerN.Z3:anefficientSMTsolver.In:14thInternationalConferenceonToolsandAlgorithmsfortheConstructionandAnalysisofSystems(TACAS’08),Budapest,Hungary,LectureNotesinComputerScience,vol.4963,Springer,Berlin;2008:337–340.

[58]BarrettC.,TinelliC.CVC3.In:19thInternationalConferenceonComputerAidedVerification(CAV’07),Berlin,Germany.2007:298–302.

[59]BarrettC.,ConwayC.L.,DetersM.,HadareanL.,JovanovicD.,KingT.,ReynoldsA.,TinelliC.CVC4.In:23rdInternationalConferenceonComputerAidedVerification(CAV’11),Snowbird,UT,USA;2011:171–177.

[60]DutertreB.Yices2.2.In:Berlin:Springer;737–744.Computer-AidedVerification(CAV’14),LectureNotesinComputerScience.2014;vol.8559.

[61]CimattiA.,GriggioA.,SchaafsmaB.,SebastianiR.TheMathSAT5SMTSolver.In:InternationalConferenceonToolsandAlgorithmsfortheConstructionandAnalysisofSystems(TACAS’13),LectureNotesinComputerScience,vol.7795;2013:93–107.

[62]TillmannN.,deHalleuxJ.Pexwhiteboxtestgenerationfor.NET.In:Berlin:Springer;134–153.TestsandProofs(TAP’08),LectureNotesinComputerScience.2008;vol.4966.

[63]CantenotJ.,AmbertF.,BouquetF.TestgenerationwithSMTsolversinmodel-basedtesting.STVR,Softw.Test.Verif.Rel.2014;24(7):499–531.

[64]ArcainiP.,GargantiniA.,RiccobeneE.OptimizingtheautomatictestgenerationbySATandSMTsolvingforBooleanexpressions.In:26thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering(ASE’11).Washington,DC:IEEEComputerSociety;2011:388–391.

[65]CristiáM.,FrydmanC.S.ApplyingSMTsolverstothetesttemplateframework.In:7thWorkshoponModel-BasedTesting(MBT’12),Tallinn,Estonia,ElectronicProc.inTheoreticalComputerScience,vol.80;2012:28–42.

[66]CristiáM.,RossiG.,FrydmanC.S.{log}asatestcasegeneratorforthetesttemplateframework.In:11thInternationalConferenceonSoftwareEngineeringandFormalMethods(SEFM’13),Madrid,Spain,LectureNotesinComputerScience,vol.8137;2013:229–243.

[67]MacworthA.K.Consistencyinnetworksofrelations.J.Artif.Intell.1977;8(1):99–118.

[68]GolombS.W.,BaumertL.D.Backtrackprogramming.J.ACM.1965;12(4):516–524.

[69]vanHentenryckP.,DincbasM.Domainsinlogicprogramming.In:Nat.Conf.onArtificialIntelligence(AAAI’86).1986:759–765.

[70]TsangE.P.K.Foundationsofconstraintsatisfaction.Computationincognitivescience.SanDiego,CA:AcademicPress;1993.978-0-12-701610-8.

[71]MossigeM.,GotliebA.,MelingH.Testingrobotcontrollersusingconstraintprogrammingandcontinuousintegration.Inform.Softw.Technol.2015;57:169–185.

[72]CarlssonM.,OttossonG.,CarlsonB.Anopen-endedfinitedomainconstraintsolver.In:9thInternationalSymposiumonProgrammingLanguages:Implementations,Logics,andPrograms(PLILP’97).London,UK:Springer-

Verlag;1997:191–206.

[73]DovierA.,PiazzaC.,PontelliE.,RossiG.Setsandconstraintlogicprogramming.ACMTrans.Program.Lang.Syst.2000;22(5):861–931.

[74]DovierA.,PiazzaC.,RossiG.Auniformapproachtoconstraint-solvingforlists,multisets,compactlists,andsets.ACMTrans.Comput.Log.2008;9(3):1–30.

[75]ShiroleM.,KumarR.UMLbehavioralmodelbasedtestcasegeneration:asurvey.SIGSOFTSoftw.Eng.Notes.0163-59482013;38(4):1–13.

[76]Doungsa-ardC.,DahalK.,HossainA.,SuwannasartT.TestdatagenerationfromUMLstatemachinediagramsusingGAs.In:InternationalConferenceonSoftwareEngineeringAdvances,ICSEA2007.2007:doi:10.1109/ICSEA.2007.70pp.47–47.

[77]LefticaruR.,IpateF.FunctionalSearch-basedTestingfromStateMachines.In:1stInternationalConferenceonSoftwareTesting,Verification,andValidation,2008.2008:525–528.doi:10.1109/ICST.2008.32.

[78]AliS.,IqbalM.Z.,ArcuriA.ImprovedheuristicsforsolvingOCLconstraintsusingsearchalgorithms.In:GeneticandEvolutionaryComputationConference,GECCO’14,Vancouver,BC,Canada,July12-16,2014.2014:1231–1238.

[79]AliS.,ZohaibIqbalM.,ArcuriA.,BriandL.C.GeneratingtestdatafromOCLconstraintswithsearchtechniques.IEEETrans.Softw.Eng.2013;39(10):1376–1402.

[80]ShiroleM.,KommuriM.,KumarR.TransitionsequenceexplorationofUMLactivitydiagramusingevolutionaryalgorithm.In:Proceedingsofthe5thIndiaSoftwareEngineeringConference,ISEC’12,Kanpur,India.NewYork,NY:ACM;2012:97–100.

[81]ShiroleM.,KumarR.Ahybridgeneticalgorithmbasedtestcasegenerationusingsequencediagrams.In:RankaS.,BanerjeeA.,BiswasK.,DuaS.,MishraP.,MoonaR.,PoonS.-H.,WangC.-L.,eds.ContemporaryComputing.Berlin:Springer;53–63.CommunicationsinComputerandInformationScience.2010;vol.94.

[82]AlbertE.,delaBandaM.J.G.,Gómez-ZamalloaM.,RojasJ.M.,StuckeyP.ACLPheapsolverfortestcasegeneration.TheoryPract.LogicProgram.2013;13:721–735(specialIssue4-5).

[83]GodefroidP.,KlarlundN.,SenK.DART:directedautomatedrandomtesting.In:ACMSIGPLANConferenceonProgrammingLanguageDesignandImplementation(PLDI’05),Chicago,IL,USA.NewYork,NY:ACM;2005:213–223.

[84]SenK.,MarinovD.,AghaG.CUTE:aconcolicunittestingengineforC.In:10thEuropeanSoftwareEngineeringConference(ESEC’05),Lisbon,Portugal.NewYork,NY:ACM;2005:263–272.

[85]TillmannN.,DeHalleuxJ.Pex:whiteboxtestgenerationfor.NET.In:2ndInternationalConferenceonTestsandProofs(TAP’08),Prato,Italy.Berlin:Springer-Verlag;2008:134–153.

[86]CadarC.,GodefroidP.,KhurshidS.,PăsăreanuC.S.,SenK.,TillmannN.,VisserW.Symbolicexecutionforsoftwaretestinginpractice:preliminaryassessment.In:33rdInternationalConferenceonSoftwareEngineering(ICSE’11),Waikiki,Honolulu,HI,USA.NewYork,NY:ACM;2011:1066–1071.

[87]CadarC.,SenK.Symbolicexecutionforsoftwaretesting:threedecadeslater.Commun.ACM.2013;56(2):82–90.

[88]ClarkeE.M.,EmersonE.A.,SistlaA.P.Automaticverificationoffinite-stateconcurrentsystemsusingtemporallogicspecifications.ACMTrans.Program.Lang.Syst.1986;8(2):244–263.

[89]CallahanJ.,SchneiderF.,EasterbrookS.Specification-basedtestingusingmodelchecking.In:SPINWorkshop,RutgersUniversity.1996:1066–1071(Tech.ReportNASA-IVV-96-022).

[90]FraserG.,WotawaF.,AmmannP.E.Testingwithmodelcheckers:asurvey.Softw.Test.Verif.Rel.2009;19(3):215–261.

[91]AlurR.,DillD.L.Atheoryoftimedautomata.Theor.Comput.Sci.1994;126:183–235.

[92]JohanB.,LarsenK.G.,FredrikL.,PaulP.,WangY.UPPAAL–atoolsuiteforautomaticverificationofreal-timesystems.In:WorkshoponVerificationandControlofHybridSystemsIII,no.1066inLectureNotesinComputerScience.Berlin:Springer-Verlag;1995:232–243.

[93]BaeK.,MeseguerJ.ThelineartemporallogicofrewritingMaudemodelchecker.In:Berlin:Springer;208–225.RewritingLogicandItsApplications,LectureNotesinComputerScience.2010;vol.6381.

[94]BuchsD.,LucioL.,ChenA.Modelcheckingtechniquesfortestgenerationfrombusinessprocessmodels.In:Berlin:Springer;59–74.ReliableSoftwareTechnologies,Ada-Europe2009,LectureNotesinComputerScience.2009;vol.5570.

[95]EnoiuE.P.,CauÅevićA.,OstrandT.J.,WeyukerE.J.,SundmarkD.,PetterssonP.Automatedtestgenerationusingmodelchecking:anindustrialevaluation.Softw.ToolsTechnol.Transf.2014;1–19.

[96]LarsenK.G.,MikucionisM.,NielsenB.Onlinetestingofreal-timesystemsusingUPPAAL.In:Berlin:Springer;79–94.FormalApproachestoTestingofSoftware,Linz,Austria,LectureNotesinComputerScience.2004;vol.3395.

[97]Graf-BrillA.,HermannsH.,GaravelH.Amodel-basedcertificationframeworkfortheenergybusstandard.In:AbrahamE.,PalamidessiC.,eds.FormalTechniquesforDistributedObjects,Components,andSystems.Berlin:Springer;

84–99.LectureNotesinComputerScience.2014;vol.8461.

[98]JardC.,JéronT.TGV:theory,principlesandalgorithms:atoolfortheautomaticsynthesisofconformancetestcasesfornon-deterministicreactivesystems.Int.J.Softw.ToolsTechnol.Transf.1433-27792005;7(4):297–315.

[99]KriouileA.,SerweW.Usingaformalmodeltoimproveverificationofacache-coherentsystem-on-chip.In:BaierC.,TinelliC.,eds.ToolsandAlgorithmsfortheConstructionandAnalysisofSystems.Berlin:Springer;708–722.LectureNotesinComputerScience.2015;vol.9035.

[100]FraserG.,WotawaF.,AmmannP.Issuesinusingmodelcheckersfortestcasegeneration.J.Syst.Softw.2009;82(9):1403–1418.

[101]RozierK.Y.Lineartemporallogicsymbolicmodelchecking.Comput.Sci.Rev.2011;5(2):163–203.

[102]BiereA.,CimattiA.,ClarkeE.M.,StrichmanO.,ZhuY.BoundedModelChecking.In:London:Elsevier;117–148.AdvancesinComputers.2003;vol.58.

[103]GentK.,HsiaoM.S.FunctionaltestgenerationattheRTLUsingSwarmIntelligenceandBoundedModelChecking.In:22ndAsianTestSymposium(ATS’13),Yilan,Taiwan.IEEEComputerSociety;2013:233–238.

[104]I.Schieferdecker,J.Gromann,A.Rennoch,Modelbasedsecuritytestingselectedconsiderations(keynoteatSECTESTatICST2011(accessedSeptmber25,2012)),http://www.avantssar.eu/sectest2011/pdfs/Schieferdecker-invited-talk.pdf.

[105]SchieferdeckerI.,GrossmannJ.,SchneiderM.Model-basedsecuritytesting.In:ProceedingsMBT2012.2012:1–12.http://dx.doi.org/10.4204/EPTCS.80.1.

[106]Tian-yangG.,Yin-shengS.,You-yuanF.Researchonsoftwaresecuritytesting.WorldAcad.Sci.Eng.Technol.2010;70:647–651.

[107]CommitteeonNationalSecuritySystems.CNSSInstruction-4009.NationalInformationAssuranceGlossary.2010.

[108]leTraonY.,MouelhiT.,PretschnerA.,BaudryB.Test-drivenassessmentofaccesscontrolinlegacyapplications.In:1stInternationalConferenceonSoftwareTesting,Verification,andValidation,2008.2008:238–247.

[109]MouelhiT.,FleureyF.,BaudryB.,TraonY.Amodel-basedframeworkforsecuritypolicyspecification,deploymentandtesting.In:Proceedingsofthe11thInternationalConferenceonModelDrivenEngineeringLanguagesandSystems,MoDELS’08,Toulouse,France.Berlin:Springer-Verlag;2008:978-3-540-87874-2537–552.

[110]PretschnerA.,MouelhiT.,leTraonY.Model-basedtestsforaccesscontrolpolicies.In:1stInternationalConferenceonSoftwareTesting,Verification,andValidation,2008.2008:338–347.

[111]XuD.,ThomasL.,KentM.,MouelhiT.,LeTraonY.Amodel-basedapproachtoautomatedtestingofaccesscontrolpolicies.In:Proceedingsofthe17thACMSymposiumonAccessControlModelsandTechnologies,SACMAT’12,Newark,NewJersey,USA.NewYork,NY:ACM;2012:978-1-4503-1295-0209–218.

[112]AnisettiM.,ArdagnaC.A.,BezziM.,DamianiE.,SabettaA.Machine-readableprivacycertificatesforservices.In:MeersmanR.,PanettoH.,DillonT.,EderJ.,BellahseneZ.,RitterN.,DeLeenheerP.,DouD.,eds.OntheMovetoMeaningfulInternetSystems:OTM2013Conferences.Berlin:Springer;978-3-642-41029-1434–450.LectureNotesinComputerScience.2013;vol.8185.

[113]MallouliW.,LallaliM.,MammarA.,MoralesG.,CavalliA.Modelingandtestingsecurewebapplications.In:Paris,France:AtlantisPress;207–255.Web-BasedInformationTechnologiesandDistributedSystems,AtlantisAmbientandPervasiveIntelligence.2010;vol.2.

[114]MassonP.-A.,PotetM.-L.,JulliandJ.,TissotR.,DeboisG.,LegeardB.,ChetaliB.,BouquetF.,JaffuelE.,VanAertrickL.,AndronickJ.,HaddadA.Anaccesscontrolmodelbasedtestingapproachforsmartcardapplications:resultsofthePOSÉproject.J.Inform.Assur.Secur.2010;5(1):335–351.

[115]BotellaJ.,BouquetF.,CapuronJ.-F.,LebeauF.,LegeardB.,SchadleF.Model-basedtestingofcryptographiccomponents–lessonslearnedfromexperience.In:SixthIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation,Luxembourg,Luxembourg,March18-22,2013.2013:192–201.

[116]DadeauF.,CastillosK.C.,LedruY.,TrikiT.,VegaG.,BotellaJ.,TahaS.Testgenerationandevaluationfromhigh-levelpropertiesforcommoncriteriaevaluations–theTASCCCtestingtool.In:SixthIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation,Luxembourg,Luxembourg,March18-22,2013.2013:431–438.

[117]BouquetF.,PeureuxF.,AmbertF.Model-basedtestingforfunctionalandsecuritytestgeneration.In:AldiniA.,LopezJ.,MartinelliF.,eds.FoundationsofSecurityAnalysisandDesignVII.Switzerland:SpringerInternationalPublishing;978-3-319-10081-41–33.LectureNotesinComputerScience.2014;vol.8604.

[118]PellegrinoG.,CompagnaL.,MorreggiaT.Atoolforsupportingdevelopersinanalyzingthesecurityofweb-basedsecurityprotocols.In:YenigünH.,Yilmaz,A.UlrichC.,eds.TestingSoftwareandSystems.Berlin:Springer;978-3-642-41706-1277–282.LectureNotesinComputerScience.2013;vol.8254.

[119]FourneretE.,OchoaM.,BouquetF.,BotellaJ.,JürjensJ.,YousefiP.Model-basedsecurityverificationandtestingforsmart-cards.In:SixthInternationalConferenceonAvailability,ReliabilityandSecurity,ARES2011,Vienna,Austria,August22-26,2011.2011:272–279.

[120]AichernigB.K.,WeiglhoferM.,WotawaF.Improvingfault-basedconformancetesting.Electron.NotesTheor.Comput.Sci.1571-06612008;220(1):63–77.

[121]JiaY.,HarmanM.Ananalysisandsurveyofthedevelopmentofmutationtesting.IEEETrans.Softw.Eng.0098-55892011;37(5):649–678.

[122]TraonY.L.,MouelhiT.,BaudryB.Testingsecuritypolicies:goingbeyondfunctionaltesting.In:ISSRE2007,The18thIEEEInternationalSymposiumonSoftwareReliability,Trollhättan,Sweden,5-9November2007.2007:93–102.

[123]WimmelG.,JürjensJ.Specification-basedtestgenerationforsecurity-criticalsystemsusingmutations.In:GeorgeC.,MiaoH.,eds.FormalMethodsandSoftwareEngineering.Berlin:Springer;471–482.LectureNotesinComputerScience.2002;vol.2495.

[124]DadeauF.,HéamP.-C.,KheddamR.Mutation-basedtestgenerationfromsecurityprotocolsinHLPSL.In:HarmanM.,KorelB.,eds.4thInt.Conf.onSoftwareTesting,VerificationandValidation,ICST2011,Berlin,Germany.IEEEComputerSocietyPress;2011:240–248.

[125]SchneiderM.,GromannJ.,SchieferdeckerI.,PietschkerA.Onlinemodel-basedbehavioralfuzzing.In:SixthIEEEInternationalConferenceonSoftwareTesting,VerificationandValidationWorkshops(ICSTW),2013.2013:469–475.

[126]JohanssonW.,SvenssonM.,LarsonU.E.,AlmgrenM.,GulisanoV.T-fuzz:model-basedfuzzingforrobustnesstestingoftelecommunicationprotocols.In:SeventhIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST),2014.2014:323–332.

[127]MITRE.CommonWeaknessEnumeration.2015.http://cwe.mitre.org/(lastaccessedApril2015).

[128]BlomeA.,OchoaM.,LiK.,PeroliM.,DashtiM.T.VERA:aflexiblemodel-basedvulnerabilitytestingtool.In:Proc.ofthe6thInt.ConferenceonSoftwareTesting,VerificationandValidation(ICST’13).Luxembourg:IEEEComputerSociety;2013:471–478.

[129]BozicJ.,WotawaF.Securitytestingbasedonattackpatterns.In:IEEESeventhInternationalConferenceonSoftwareTesting,VerificationandValidationWorkshops(ICSTW),2014.IEEE;2014:4–11.

[130]WeiT.,Ju-FengY.,JingX.,Guan-NanS.AttackmodelbasedpenetrationtestforSQLinjectionvulnerability.In:2012IEEE36thAnnualComputerSoftwareandApplicationsConferenceWorkshops(COMPSACW).IEEE;2012:589–594.

[131]XuD.,TuM.,SanfordM.,ThomasL.,WoodraskaD.,XuW.Automatedsecuritytestgenerationwithformalthreatmodels.IEEETrans.Depend.SecureComput.2012;9(4):526–540.

[132]SalvaS.,ZafimiharisoaS.R.Datavulnerabilitydetectionbysecuritytestingforandroidapplications.In:InformationSecurityforSouthAfrica,2013.IEEE;2013:1–8.

[133]VernotteA.,DadeauF.,LebeauF.,LegeardB.,PeureuxF.,PiatF.Efficient

detectionofmulti-stepcross-sitescriptingvulnerabilities.In:Berlin:Springer;2014:358–377.InformationSystemsSecurity.

[134]BuchlerM.,OudinetJ.,PretschnerA.Semi-automaticsecuritytestingofwebapplicationsfromasecuremodel.In:Proc.ofthe6thInt.ConferenceonSoftwareSecurityandReliability(SERE’12),Gaithersburg,MD,USA.IEEEComputerSociety;2012:253–262.

[135]RocchettoM.,OchoaM.,DashtiM.T.Model-baseddetectionofCSRF.In:Berlin:Springer;2014:30–43.ICTSystemsSecurityandPrivacyProtection.

[136]FelmetsgerV.,CavedonL.,KruegelC.,VignaG.Towardautomateddetectionoflogicvulnerabilitiesinwebapplications.In:USENIXSecuritySymposium.2010:143–160.

[137]KaksonenR.,TakanenA.Testcoverageinmodel-basedfuzztesting.In:ModelBasedTestingUserConference,Tallinn/Estonia.ETSI;2012(invitedtalk).

[138]F.Duchene,Detectionofwebvulnerabilitiesviamodelinferenceassistedevolutionaryfuzzing,Ph.D.thesis,GrenobleUniversity,2014(ph.D.thesis).

[139]WangJ.,GuoT.,ZhangP.,XiaoQ.Amodel-basedbehavioralfuzzingapproachfornetworkservice.In:ThirdInternationalConferenceonInstrumentation,Measurement,Computer,CommunicationandControl(IMCCC),2013.IEEE;2013:1129–1134.

[140]TrendsandBenchmarksReportSwitzerland.Wheredowestandwherearewegoingto?Testing2013.2013.http://www.swissq.it/wp-content/uploads/2013/07/Testing-Trends-and-Benchmarks-2013_Web_En.pdfAvailablefromwww.swissq.it.

[141]SchulzeC.,GanesanD.,LindvallM.,CleavelandR.,GoldmanD.Assessingmodel-basedtesting:anempiricalstudyconductedinindustry.In:CompanionProceedingsofthe36thInternationalConferenceonSoftwareEngineering,ICSECompanion2014,Hyderabad,India.NewYork,NY:ACM;2014:978-1-4503-2768-8135–144.doi:10.1145/2591062.2591180.

[142]WangC.,PastoreF.,GoknilA.,BriandL.,IqbalZ.Automaticgenerationofsystemtestcasesfromusecasespecifications.In:InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA’15,Baltimore,Maryland,USA,July12-17,2015.NewYork,USA:ACM;2015:385–396.doi:10.1145/2771783.2771812(acceptedpaper,tobepublished).

MarkUttingisaSeniorLecturerinICTattheUniversityoftheSunshineCoast.Previously,heworkedasSeniorResearchFellowinsoftwareengineeringatQUTforseveralyears,developingcomputersimulationsoffutureQueenslandElectricityNetworks,andasAssociateProfessorattheUniversityofWaikatoinNewZealand,teachingprogrammingandsoftwareengineering.Hehasalsoworkedinindustry,developingnext-generationgenomicssoftwareandmanufacturingsoftware.Markiscoauthorofthebook“PracticalModel-BasedTesting:AToolsApproach,”aswellasmorethan60publicationsonmodel-basedtesting,verificationtechniquesforobject-orientedandreal-timesoftware,andlanguagedesignforparallelcomputing.

BrunoLegeardProfessorofSoftwareEngineeringattheUniversityofFranche-Comté(France),cofounderandSeniorScientistatSmartestingSolutions&Services,isinternationallyrecognizedasanexpertandawell-knownspeakerinthemodel-basedtestingfield.Hehasgiventalksatnumeroustestingandsoftwareengineeringconferences.Heisexperiencedindeployingmodel-basedtestingsolutionsbothinenterpriseinformationsystemsareaandintheembeddedsystemsfield.B.Legeardwrotetheseminalbook“PracticalModel-BasedTesting—AToolsApproach,”publishedbyMorganandKaufmannin2007,withDr.MarkUtting.HewasalsoacoleaderoftheauthorteamdevelopingtherecentISTQBModel-BasedTestingcertification.HeearnedhisMasterofScienceDegreeinSoftwareEngineeringandhisPhDinComputerSciencefromINSALyon,France.

FabriceBouquetstudiedcomputerscienceandreceivedhisPhDfromtheUniversityofProvence,Francein1999.HeisafullProfessorofSoftwareEngineeringattheUniversityofFranche-Comté,France.Heresearchesthevalidationofcomplexsystemsfromrequirementstomodels,includingoperationalsemantics,testing,modeltransformation,functionalandnonfunctionalproperties,withapplicationsinvehicle,aircraft,smartobjects,andenergy.

ElizabetaFourneretcompletedherPhDatINRIA/UniversityofFranche-Comté,Francein2012inModel-BasedRegressionTestingofCriticalSystems.SheiscurrentlyaResearchEngineerattheDepartmentofComputerScienceattheInstitutFEMTOSTinBesançon,France.Sheisinterestedindevelopingnewsolutionsandtoolsfortestingandtestdatagenerationtowardensuringthesystem’scompliancetogivenrequirements.Inthisframework,herresearchactivitiesfocusonModel-BasedTesting(MBT)approachesthattacklethemodelingandtestingofcriticalsystems,forinstance,systemsinthesatelliteorsmart-cardindustry.

FabienPeureuxreceivedhisPhDinComputerSciencefromtheUniversityofFranche-Comtéin2002,whereheworkssince2003asAssistantProfessoranddoeshisresearchactivitieswiththeFEMTO-STInstitute.Since2005,heisalsoseniorscientificconsultantfortheSmartestingcompany.Hismainexpertiseisfocusedontheautomationofvalidationprocessinthedomainsofsmartcardapplications,informationsystems,andembeddedsoftware,withaparticularinterestinModel-BasedTestingtechniquesandagileapproaches.

AlexandreVernottereceivedhisPhDattheInstitutFemto-ST,Besanconin2015inModel-BasedSecurityTestingforWebapplications.HerecentlyobtainedapostdocpositionattheDepartmentofIndustrialInformationandControlSystemsattheRoyalInstituteofTechnology(KTH)inStockholm,Sweden.Hisresearchcentersonenterprisesystemarchitecturessecurity.Hisinterestsalsoincludethreat,risk,andbehavioralmodeling,Model-BasedTestingandModel-BasedEngineering.

1Forthe2011MBTUserSurvey,seehttp://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf2http://utp.omg.org3TheChinesePostmanalgorithmfindstheshortestpaththatcoversallthetransitionsofafinitestatemachine.4http://www.avantssar.eu/5http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-11-cryptographic-token-interface-standard.htm6http://www.globalplatform.org/

CHAPTERTHREE

OnTestingEmbeddedSoftwareAbhijeetBanerjee*;SudiptaChattopadhyay†;AbhikRoychoudhury**NationalUniversityofSingapore,Singapore†SaarlandUniversity,Saarbrücken,Germany

AbstractForthelastfewdecades,embeddedsystemshaveexpandedtheirreachintomajoraspectsofhumanlives.Startingfromsmallhandhelddevices(suchassmartphones)toadvancedautomotivesystems(suchasanti-lockbrakingsystems),usageofembeddedsystemshasincreasedatadramaticpace.Embeddedsoftwarearespecializedsoftwarethatareintendedtooperateonembeddeddevices.Inthischapter,weshalldescribetheuniquechallengesassociatedwithtestingembeddedsoftware.Inparticular,embeddedsoftwarearerequiredtosatisfyseveralnon-functionalconstraints,inadditiontofunctionality-relatedconstraints.Suchnon-functionalconstraintsmayinclude(butnotlimitedto),timing/energy-consumptionrelatedconstrainsorreliabilityrequirements,etc.Additionally,embeddedsystemsareoftenrequiredtooperateininteractionwiththephysicalenvironment,obtainingtheirinputsfromenvironmentalfactors(suchastemperatureorairpressure).Theneedtointeractwithadynamic,oftennon-deterministicphysicalenvironment,furtherincreasesthechallengesassociatedwithtesting,andvalidationofembeddedsoftware.Inthepast,testingandvalidationmethodologieshavebeenstudiedextensively.Thischapter,however,explorestheadvancesinsoftwaretestingmethodologies,specificallyinthecontextofembeddedsoftware.Thischapterintroducesthereadertokeychallengesintestingnon-functionalpropertiesofsoftwarebymeansofrealisticexamples.Italsopresentsaneasy-to-follow,classificationofexistingresearchworkonthistopic.Finally,thechapterisconcludedwithareviewofpromisingfuturedirectionsintheareaofembeddedsoftwaretesting.

KeywordsNon-functionalpropertytesting;Performancetesting;Energyconsumptionofsoftware;Search-basedsoftwaretesting;Symbolicexecution

1IntroductionOverthelastfewdecades,researchinsoftwaretestinghasmadesignificantprogress.Thecomplexityofsoftwarehasalsoincreasedatadramaticpace.Asaresult,wehavenewchallengesinvolvedinvalidatingcomplex,real-worldsoftware.Inparticular,wearespecificallyinterestedintestingandvalidationofembeddedsoftware.Inthismodernworld,embeddedsystemsplayamajorroleinhumanlives.Suchsoftwarecanbefoundubiquitously,inelectronicsystemssuchasconsumerelectronics(eg,smartphones,mp3players,anddigitalcameras)andhouseholdappliances(eg,washingmachinesandmicrowaveovens)toautomotive(eg,electriccarsandantilockbrakingsystems)andavionicapplications.Softwaredesignedforembeddedsystemshaveuniquefeaturesandconstraintsthatmakeitsvalidationachallengingprocess.Forinstance,unlikeDesktopapplications,thebehaviorofanembeddedsystemsoftendependsonthephysicalenvironmentitoperatesin.Asamatteroffact,manyembeddedsystemsoftentaketheirinputsfromthesurroundingphysicalenvironment.This,however,posesuniquechallengestotestingofsuchsystemsbecausethephysicalenvironmentmaybenon-deterministicanddifficulttorecreateduringthetestingprocess.Additionally,mostembeddedsystemsarerequiredtosatisfyseveralnon-functionalconstraintsuchastiming,energyconsumption,reliability,tonameafew.Failuretomeetsuchconstraintscanresultinvaryingconsequencesdependingupontheapplicationdomain.Forinstance,ifthenatureofconstraintsonthesoftwarearehardrealtime,violationmayleadtoseriousconsequences,suchasdamagetohumanlifeandproperty.Therefore,itisofutmostimportancethatsuchsystemsbetestedthoroughlybeforebeingputtouse.Intheproceedingsections,weshalldiscusssomeofthetechniquesproposedbythesoftwareengineeringcommunitythataretargetedattestingandvalidationofreallife,embeddedsystemsfromvariousapplicationdomainsandcomplexities.However,firstweshallpresentanexample,inspiredfromareallifeembeddedsystem,thatwillgivethereaderanideaonthenatureofconstraintscommonlyassociatedwithembeddedsystems.

Fig.1providestheschematicrepresentationofawearablefalldetectionapplication[1].Suchanapplicationisusedlargelyinthehealthcaredomaintoassistthefrailorelderlypatients.Thepurposeofthesystem,asshowninFig.1,istodetectapotentialfallofitswearerandtoinvokeappropriatesafetymeasures.Inordertodetectafall,thesystemneedstomonitortheuser’smovement.Thistaskisaccomplishedviaanumberofsensors,thatarepositionedatdifferentpartsofthepatient’sbody.Thesesensorsdetectphysicalmotionsandcommunicatetheinformationviawirelesssensornetworks.Inthescenariowhenthesystemdetectsapotentialfallitactivatesappropriatesafetymeasures,suchasinformingthehealthcareprovidersovermobilenetworks.Testingthefall-detectionsystemisessentialtoensureitsfunctionalcorrectness,suchasapotentialfallmustnotgoundetected.However,suchatestingrequirestheinputsfromthesensors.Toproperlytestthesystem,itsdesignersshouldbeabletosystematicallymodeltheinputsfromsensorsandthesurroundingenvironment.

FIGURE1 Awearablefall-detectionapplication.

Apartfromthefunctionalcorrectness,thefall-detectionsystemalsoneedstosatisfyseveralnon-functionalconstraints.Forinstance,thedetectionofafallshouldmeethardtimingconstraints.Intheabsenceofsuchconstraints,therespectivepatientmightgetseriouslyinjured,makingthesystemimpracticaltouse.Moreover,iftheapplicationisdeployedintoabatteryoperateddevice,itsenergyconsumptionshouldbeacceptabletoensureagracefuldegradationofbatterylife.Finally,duetothepresenceofunreliablehardwarecomponents(eg,sensors)andnetworks(eg,sensorandmobilenetworks),theapplicationshouldalsoguaranteethatapotentialfallofthepatientisdetectedwithacceptablereliability.

Non-functionalpropertiesofembeddedsoftware,suchastimingandenergy,areextremelysensitivetotheunderlyingexecutionplatform.Thismakesthetestingprocesscomplicated,astheunderlyingexecutionplatformmaynotbeavailableduringthetimeoftesting.Besides,iftheembeddedsoftwareistargetedatmultipleexecutionplatforms,itsnon-functionalpropertiesneedtobevalidatedforeachsuchplatform.Toalleviatetheseissues,aconfigurablemodelfortheexecutionplatformmightbeusedduringthetestingprocess.Forinstance,suchaconfigurablemodelcancapturethetimingorenergybehaviorofdifferenthardwarecomponents.Buildingsuchconfigurablemodels,however,mayturnoutchallengingduetothecomplexityofhardwareandits(vendor-specific)intellectualproperties.

Overthelasttwodecades,numerousmethodsinsoftwaretestinghavebeenproposed.Theseincluderandomtesting,search-basedtesting,anddirectedtesting(eg,basedonsymbolicexecution),amongseveralothers.Thesetestingmethodologieshavefocusedprimarilyonthevalidationoffunctionalproperties.Validationofnon-functionalsoftwareproperties,havegainedattentiononlyrecently.InthisChapter,weexplorethepotentialofdifferenttestingmethodologiesinthecontextofembeddedsoftware.Foranembeddedsoftware,itsnon-functionalaspectsplayacrucialroleinthevalidationprocess.WeintroducesomesalientpropertiesofvalidatingtypicalembeddedsystemsinSection2.Subsequently,weshallexploretherecentadvancesintestingembeddedsystemsinSection3.Wefirstcategorizealltestingmethodologiesintothreebroadercategories.Suchcategoriesreflectthelevelofabstraction,inwhichembeddedsystemsarevalidated.Inparticular,ourfirstcategorycapturesblack-boxtesting,wherethesystemisabstractedawayandtestinputsaregeneratedviasamplingoftheinputspace.Theremainingcategorieseitheruseanabstractmodelofthesystemortheactualimplementation.Weshalldiscussthatdifferenttestingmachineries(eg,evolutionarytestingandsymbolic

execution)canbeemployedforsuchcategories.Basedonourcategorizationoftestingembeddedsystems,weshallarguethatnosinglecategorycanbedecidedtobesuperiorthanothers.Ingeneral,thechoiceofabstraction,fortestingembeddedsystem,largelydependsontheintentionofthedesigner.Forinstance,ifthedesignerisinterestedindetectingfine-grainedevents(eg,memoryrequestsandinterrupts),itisrecommendedtocarryoutthetestingprocessontheactualimplementation(eg,binarycode).Onthecontrary,testingbinarycodemayrevealnon-functionalbugstoolateinthedesignprocess,leadingtoacompleteredesignofthesoftware.

Throughthischapter,weaimtobringtheattentionofsoftwareengineeringcommunitytowardstheuniquechallengesinvolvedinembeddedsoftwaretesting.Specifically,testingofnon-functionalpropertiesisanintegralpartofvalidatingembeddedsoftware.Inordertovalidatenon-functionalproperties,softwaretestingmethodologiesshouldexplicitlytargettodiscovernon-functionalbugs,suchasthelossofperformanceandenergy.Moreover,inordertotestfunctionalpropertiesofembeddedsoftware,thedesignershouldbeabletosimulatetheinteractionofsoftwarewiththephysicalenvironment.Weshalldiscussseveraleffortsinrecentyearstodiscoverfunctionalaswellasnon-functionalbugsinembeddedsoftware.Inspiteoftheseefforts,numerouschallengesstillexistinvalidatingembeddedsoftware.Forinstance,non-functionalbehaviorsofembeddedsoftware(eg,timeandpower)canbeexploitedtodiscoversecretinputs(eg,secretkeysincryptographicalgorithms).Testingoftimingandenergy-relatedpropertiesisfarfrombeingsolved,nottomentiontheimmaturityoftheresearchfieldtovalidatesecurityconstraintsinembeddedsoftware.Wehopethischapterwillprovidethenecessarybackgroundtosolvetheseexistingchallengesinsoftwaretesting.

2TestingEmbeddedSoftwareAnalogoustomostsoftwaresystems,testingembeddedsoftwareisanintegralpartofthesoftwaredevelopmentlifecycle.Toensuretherobustnessofembeddedsoftware,bothitsfunctionalandnon-functionalpropertiesneedtobeexamined.Inthefollowingdiscussion,weoutlinesomesalientfeaturesthatmakethetestingofembeddedsystemsuniqueandchallenging,comparedtotraditionalsoftwaresystems.

2.1TestingFunctionalPropertiesThefunctionalityofsoftwaresystemscapturethewaysuchsystemsshouldbehave.Therefore,testingfunctionalpropertiesisacriticalphaseforallapplications.Typically,thefunctionalitytestingofsoftwareaimstodiscover“buggy”scenarios.Forinstance,suchbuggyscenariosmaycapturetheviolationofsoftwarebehaviorwithrespecttothespecificationoranimplementationbug(eg,nullpointerdereferenceandassertionfailure).Todiscoverandinvestigateabuggyscenario,thedesignermustbeprovidedwithappropriatetestinputsthattriggertherespectivebug.Therefore,softwaretestingtoolsshouldhaveacleardomainknowledgeoftherelevantinputstothesystem.Forembeddedsoftware,thefunctionalityisoften(partially)controlledbythephysicalenvironment.Suchphysicalenvironmentmightincludeairpressure,temperature,physicalmovement,amongothers.Unfortunately,thephysicalenvironment,whereanembeddedsoftwareiseventuallydeployed,isoftennotpresentduringthetestingtime.Forinstance,considerthefall-detectionapplication,whichwasintroducedintheprecedingsection.Itiscrucialthatthedesignedsoftwareinvokesappropriateactionsaccordingtothemovementofthepatient.Intheactualworkingenvironment,suchmovementsaresampledfromsensorinputs.ConsiderthecodefragmentinFig.2,whichreadsanaccelerometerandtakesactionaccordingly.Thefunctionf(buffer)capturesapredicateonthevaluesreadintothebuffer.Theelsebranchofthecodefragmentexhibitsadivision-by-zeroerrorwhenbuffer[0]=0.Inordertoexecutetheelsebranch,thetestinputmust,additionally,satisfytheconditionf(buffer)=0.Asthevalueofbufferdependsonthephysicalenvironment,theinputsfromtheaccelerometermightoftenneedtobesimulatedviasuitableabstractions.Similarly,forembeddedsoftware,whosefunctionalitymightdependonairpressureortemperature,thetestingprocessshouldensurethattherespectivesoftwareactsappropriatelyindifferentenvironmentalconditions.Ingeneral,tosimulatethephysicalenvironment,thedesignermaypotentiallytakethefollowingapproaches:

FIGURE2 Thedependencyoffunctionalityonthephysicalenvironment.

•Thephysicalenvironment(eg,inputsreadfromsensors)mightbemadecompletelyunconstrainedduringthetimeoftesting.Thisenablesthetestingofsoftwareunderalloperatingconditionsofthephysicalenvironment.However,suchanapproachmightturninfeasibleforcomplexembeddedsoftware.Besides,unconstrainingthephysicalenvironmentmightleadtounnecessarytestingforirrelevantinputs.Suchinputsmayincludesensorreadings(suchas−300Kforairtemperaturereadings)thatmayneverappearintheenvironmentwherethesoftwareisdeployed.

•Thephysicalenvironmentmightbesimulatedbyrandomlygeneratingsyntheticinputs(eg,generatingrandomtemperaturesreadings).However,suchanapproachmayfailtogeneraterelevantinputs.However,liketraditionalsoftwaretesting,search-basedtechniquesmightimprovethesimulationofphysicalenvironmentviaevolutionarymethodsandmetaheuristics.

•Withaclearknowledgeoftheembeddedsoftware,thetestingprocesscanbeimproved.Forinstance,inthefall-detectionsystem,itisprobablynotcrucialtosimulatethemovementforallpossiblemovementangles.Itis,however,importanttotesttheapplicationforsomeinputsthatindicateafallofthepatient(hence,indicatingsafety)andalsoforsomeinputsthatdoesnotcaptureafall(hence,indicatingtheabsenceoffalsepositives).Ingeneral,buildingsuchabstractionsontheinputspaceischallenginganditalsorequiresasubstantialdomainknowledgeoftheinputspace.

Weshallnowdiscusssomenon-functionalpropertiesthatmostembeddedsoftwarearerequiredtosatisfy.

2.2TestingNon-functionalPropertiesIngeneral,mostembeddedsoftwareareconstrainedviaseveralnonfunctionalrequirements.Inthefollowingandfortherestofthechapter,weshallprimarilyconcentrateonthreecrucialpropertiesofembeddedsoftware—timing,energy,andreliability.

2.2.1TimingConstraintsTimingconstraintscapturethecriteriatocompletetaskswithinsometimebudgets.Theviolationofsuchconstraintsmayleadtoacompletefailureoftherespectivesoftware.This,inturn,mayhaveseriousconsequences.Forinstance,considerthefall-detectionapplication.Thecomputationofapotentialfallshouldhavereal-timeconstraints.More

precisely,thetimeframebetweenthesamplingofsensorinputsandtriggeringanalarmingsituationshouldhavestricttimingconstraints.Violationofsuchconstraintsmayleadtothepossibilityofdetectingafalltoolate,hence,makingtherespectivesoftwareimpractical.Therefore,itiscrucialthatthevalidationprocessexplicitlytargetstodiscovertheviolationoftiming-relatedconstraints.Itis,however,challengingtodeterminethetimingbehaviorofanapplication,asthetimingcriticallydependsontheexecutionplatform.Theexecutionplatform,inturn,maynotbeavailableduringthetestingphase.Asaresult,thevalidationoftiming-relatedconstraints,mayofteninvolvebuildingatimingmodeloftheunderlyingexecutionplatform.Suchatimingmodelshouldbeabletoestimatethetimetakenbyeachexecutedinstruction.Ingeneral,buildingsuchtimingmodelsischallenging.Thisisbecause,thetimetakenbyeachinstructiondependsonthespecificinstructionsetarchitecture(ISA)oftheprocessor,aswellasthestateofdifferenthardwarecomponents(eg,cache,pipeline,andinterconnect).ToshowtheinterplaybetweentheISAandhardwarecomponents,letusconsidertheprogramfragmentshowninFig.3.

FIGURE3 Thetiminginterplaybetweenhardwarecomponents(eg,caches)andinstructions.

InFig.3,thetruelegoftheconditionalexecutesanaddinstructionandthefalselegofthebranchexecutesamultiplyinstruction.Letusassumethatwewanttocheckwhetherthiscodefinisheswithinsomegiventimebudget.Inotherwords,wewishtofindoutiftheexecutiontimeofbranchwiththelongerexecutiontimeislessthanthegiventimebudget.Inatypicalprocessor,amultiplicationoperationgenerallytakeslongerthananadditionoperation.However,iftheprocessoremploysacachebetweentheCPUandthememory,thevariablezwillbecachedafterexecutingthestatementz:=3.Therefore,thestatementx:=x*zcanbecompletedwithoutaccessingthememory,buttheprocessormayneedtoaccessthememorytoexecutex:=x+y(tofetchyforthefirsttime).Asaresult,eventhoughmultiplicationisacostlyoperationcomparedtoaddition,inthisparticularscenario,themultiplicationmayleadtoafastercompletiontime.Thisexampleillustratesthatatimingmodelforanexecutionplatformshouldcarefullyconsidersuchinteractionbetweendifferenthardwarecomponents.

Onceatimingmodelisbuiltfortheexecutionplatform,therespectivesoftwarecanbetestedagainstthegiventiming-relatedconstraints.Broadly,thevalidationoftimingconstraintsmayinvolvethefollowingprocedures:

•Thetestingproceduremayaimtodiscovertheviolationofconstraints.Forinstance,let

usassumethatforafall-detectionapplicationtobepractical,thealarmingsituationmustbenotifiedwithin1ms(cf.Fig.1).Suchaconstraintcanbeencodedviatheassertion:assert(time<=1ms),wheretimeisthetimetakenbythefall-detectionapplicationtocomputeapotentialfall.Thevalueoftimecanbeobtainedbyexecutingtheapplicationdirectlyonthetargetedplatform(whenavailable)orbyusingatimingmodelforthesame.Thetestingprocessaimstofindtestinputsthatmaypotentiallyinvalidatetheencodedassertions.

•Itmay,however,turndifficultforadesignertodevelopsuitableassertionsthatcapturetimingconstraints.Insuchcases,shemightbeinterestedtoknowtheworst-caseexecutiontime(WCET)ofthesoftware.Asthenamesuggests,WCETcapturesthemaximumexecutiontimeofanapplicationwithrespecttoallinputs.AccuratelydeterminingtheWCETofanapplicationisextremelychallenging,especiallyduetothecomplexinteractionsacrossdifferentsoftwarelayers(application,operatingsystems,andhardware)andduetotheabsenceof(proprietary)architecturaldetailsoftheunderlyingexecutionplatform.However,WCETofanapplicationcanbeapproximatedviasystematicallytestingthesoftwarewithappropriateinputs.Forinstance,weshalldiscussinSection3abouttheprogressinevolutionarytestingtodiscovertheWCET.

2.2.2EnergyConstraintsLiketiming,energyconsumptionofembeddedsoftwaremayalsoneedcarefulconsideration.Inparticular,iftherespectivesoftwareistargetedforabattery-operateddevice,theenergyconsumptionofthesoftwaremayposeaseriousbottleneck.Forinstance,ifafall-detectionsoftwareisbattery-operated,thepowerdrainedfromthebatteryshouldbeacceptableinawaytotriggerthealarmingsituation.Liketiming,theenergyconsumptionofsoftwareisalsohighlysensitivetotheunderlyingexecutionplatform.Therefore,intheabsenceoftheexecutionplatform,anappropriateenergy-modelneedstobedeveloped.Suchanenergymodelcanbeusedduringthetesttimetoestimatetheenergyconsumptionofsoftwareandtocheckwhetherthesoftwaresatisfiescertainenergyconstraints.Similartotimingconstraints,energyconstraintscanbecapturedsystematicallyviaassertionsorviacomputingtheworstcaseenergyconsumption(WCEC)oftherespectivesoftware.ThecomputationofWCEChassimilarchallengesasthecomputationoftheWCETandtherefore,suchcomputationsmightinvolveapproximationsviasystematicallygeneratingtestinputs.

2.2.3ReliabilityConstraintsAsembeddedsoftwareofteninteractswiththephysicalenvironment,itneedstoreliablycapturethedataacquiredfromthephysicalworld.Usually,thisisaccomplishedviasensors(eg,gyroscopeandaccelerometers),whichinteractswiththesoftwareviacommunicatingthedatafromthephysicalworld.Forinstance,inthefalldetectionapplication,thedatareadviathesensorsaresentviawirelesssensornetwork.Ingeneral,itispotentiallyinfeasibletogetthesensordataaccurately.Thismightbeduetotheinaccuracyofsensorchipsorduetopotentialpacketdropsinthenetwork.Therefore,thereliabilityofdifferentsoftwarecomponentsmayposeaconcernforacriticalembedded

software,suchasafalldetector.Besides,thereliabilityofacomponentanditscosthasnontrivialtrade-offs.Forinstance,amoreaccuratesensor(orareliablenetwork)mightincurhighercost.Overall,thedesignermustensurethattherespectivesoftwareoperateswithanacceptablelevelofreliability.Asanexample,inthefalldetector,thedesignerwouldliketoensurethataphysicalfallisalarmedwithx%reliability.Computingthereliabilityofanentiresystemmightbecomechallengingwhenthesystemconsistsofseveralcomponentsandsuchcomponentsmightinteractwitheachother(andthephysicalworld)inafairlycomplexfashion.

Tosummarize,apartfromthefunctionality,mostembeddedsoftwarehaveseveralnon-functionalaspectstobeconsideredinthetestingprocess.Suchnon-functionalaspectsincludetiming,energy,andreliability,amongothers.Ingeneral,thenon-functionalaspectsofembeddedsoftwaremayleadtoseveralcomplextrade-offs.Forinstance,anincreasedrateofsamplingsensorinputs(whichcapturethedatafromthephysicalworld)mayincreaseenergyconsumption;however,itmightincreasethereliabilityofthesoftwareintermsofmonitoringthephysicalenvironment.Similarly,anaiveimplementationtoimprovethefunctionalitymaysubstantiallyincreasetheenergyconsumptionoritmayleadtothelossofperformance.Asaresult,embeddedsoftwarearerequiredtobesystematicallytestedwithrespecttotheirnon-functionalaspects.Inthenextsection,weshalldiscussseveraltestingmethodologiesforembeddedsoftware,withaspecificfocusontheirnon-functionalproperties.

3CategorizationofTestingMethodologiesReal-timeandembeddedsystemsareusedextensivelyinawidevarietyofapplications,rangingfromautomotiveandavionicstoentertainmentandconsumerelectronics.Dependingontheapplication,theconstraintsapplicableonsuchsystemsmayrangefrommission-criticaltosoft-realtimeinnature.Additionally,embeddedsystemsoftenhavetointeractwiththephysicalenvironmentthatmaybedeterministicornon-deterministic.Suchfactorsimplythatembeddedsystemshavetobedesignedanddevelopedwithvaryingoperationalrequirementsandnosingletestingtechniqueiswellsuitedtoallsystems.Insomescenarios,thesystemundertest(SUT)maybetoocomplextomodelandhence,approximate,yetfastsampling-basedtechniquesaresuitable.Inotherscenarios,wheretheSUThasmission-criticalconstraintsandrequiresthoroughtesting,afine-grainedmodelingofthesystemiscrucial.Inthefollowingparagraphs,weshallcategorizeanddiscusssomeoftheexistingworksontestingembeddedsystems,withaspecificfocusonworksbeingpublishedinthepast5years.Inparticular,wecategorizeallworksintofollowingthreedivisions(asshowninFigure4):

FIGURE4 Classificationofexistingapproachesforembeddedsoftwaretesting.

Black-BoxAbstraction:SuchtechniquesoftenconsidertheSUTasablack-box.Testcasesaregeneratedbysampling,randomizedtestingtechniques.

Grey-BoxAbstraction:SuchtechniquesdonottreattheSUTasablack-box.TheSUTisrepresentedbyamodel,whichcapturesonlytheinformationrelatedtothepropertyofinterest.Testcasesaregeneratedbyexploringthesearchspaceofthemodel.

White-BoxAbstraction:Techniquesinthiscategoryoftenrequirethesourcecodeorbinaryoftheimplementedsystemforthetestingprocess.Inotherwords,thesourcecodeandbinaryservesasthemodelofthesystem.Testcasesaregeneratedbysearchingtheinputspaceoftheimplementedsystem.

Insubsequentsections,weshallelaborateoneachofthecategorizationasdescribedintheprecedingparagraphs.

4Black-BoxAbstractionOneofthemostsimple(butnotnecessarilyeffective)approachesoftestingcomplexsystemsistouniformlysampleitsinputspace.Thegoalofsuchsamplingistogeneratetestinputs.Asexceedinglysimpleassuchamethodmightseem,theeffectivenessofsuchuniform(orunguided)samplingremainsquestionable.Whentestingasystem,ingeneral,theobjectiveistoproducetestinputsthatbearswitnessestofailureofthesystem.Suchafailuremightcapturetheviolationofapropertyofinterest.Besides,suchviolationsshouldbemanifestedwithinacertaintimebudgetfortesting.1Testingapproaches,whicharepurelybasedonuniformrandomsampling,clearlydonotadheretotheaforementionedcriteria.Forexample,considerasystemthatexpectsanintegervalueasaninput.Forsuchasystemuniformrandomsamplingmayblindlycontinuetogeneratetestinputsforeverwithoutprovidinganyinformationaboutthecorrectness(orin-correctness)ofthesystem.However,therewillbesystemsinthewildthataretoocomplextomodel.Suchsystemsrequiresomesortofmechanismbywhichtheycanbetestedtosomeextent.Forsuchsystems,thesamplingbasedtechnique,asdiscussedinthefollowingparagraphs,mightbeuseful.

Theworkin[2,3]proposessamplingbasedtechniquestogeneratefailure-revealingtestinputsforcomplexembeddedsystems.Inparticular,theyfocusongeneratingtestinputsthatleadtoviolationoftiming-relatedproperties.Forthesetechniquestowork,theessentialtiming-relatedpropertiesofthesystemmustbeformulatedviaMetricTemporalLogic(MTL).AnMTLformulacanbe,inabroadway,describedasacompositionofpropositionalaswellastemporaloperators.Commonexamplesofpropositionaloperatorsareconjunction,disjunction,andnegation,whereassomeexampleoftemporaloperatorswouldbeuntil,always,andeventually.Besides,MTLextendsthetraditionallineartemporallogic(LTL)withtimingconstraints.Forinstance,considerourexampleinFig.1.Letusconsiderthatapotentialfallofthepatientmustbereportedwithin100timeunits.SuchacriteriacanbecapturedviathefollowingMTLformula:

fallcapturestheeventofapotentialfallandalarmcapturestheeventtonotifythehealthcareproviders.Besides,thetemporaloperators and◊capturealwaysandeventually,respectively.Oncethetiming-relatedpropertiesofthesystemhavebeenidentifiedandencodedasMTLformulas,thenextstepistoidentifytestinputs(asshowninFig.5),forwhichtheaforementionedformuladonotholdtrue(ie,thesystemfails).

FIGURE5 Overviewofsamplingbasedtest-generationtechniques.

Thecornerstoneofsampling-basedapproachesliesinthedefinitionofametric,asoftencalledrobustnessmetric.Suchametricrepresentsthedistanceofagivenexecutiontrace(oftheSUT,foragiveninput)fromafailurerevealingexecutiontrace.Themetricisdesignedinsuchamannerthatifanexecutiontracehasanegativevaluefortherobustnessmetric,thenitimpliesthattherespectiveexecutionhasleadtoaviolationofsometiming-relatedproperty.Similarly,apositivevalueforarobustnessmetricsignifiesthattheexecutionsatisfiestheMTLformulas.Ingeneral,therobustnessmetricprovidesameasureofhowrobustlyanexecutiontracesatisfiestheencodedMTLformulas.Oncesuchametrichasbeendefined,itneedstobedecidedwhetherthereexistsaninputthatleadstotheviolationofthegivenproperty.Thisdecisionproblemcanbetransformedintoanoptimizationproblem.Forinstance,thisoptimizationproblemmightaimtodiscovertheexecutionwiththelowestrobustnessvalue.Existingworkshavediscussedanumberofwaysofsolvingtheoptimization(minimizingrobustness)problem.Forexample,thetechniqueof[2]usesMonte-Carlosimulationstosolvethisoptimizationproblem.Anobviousdrawbackbeingthatthetechniqueof[2]canonlygiveprobabilisticguaranteestofindfailureinducingtestinputs.Atthesametime,anadvantageofsuchatechniqueistofindexecutionwherethetiming-relatedpropertywastheclosesttobeingviolated.Subsequentworkinthisdirectionhaveexperimentedwithotheroptimizationtechniques,suchas[3]usesCross-entropymethodbasedoptimizationand[4]usesant-colonybasedoptimization,intryingtoimprovetheefficiencyofthetest-generationprocess.

5Grey-BoxAbstractionThisclassoftechniquesworkbycreatinganabstractmodeloftheSUT.AsshowninFig.6,ingeneral,frameworksdiscussedinthiscategoryrequirethreekeycomponentsasfollows:

FIGURE6 Overviewofgrey-boxabstractionbasedtestingtechniques.

•Atechniqueformodelgeneration

•Atechniqueformodelexploration(togeneratetestcases),and

•Anoracleforidentifyingfailure-revealingtests

Oncethepropertyofinteresthasbeenidentified,themodeloftheSUTcanbegeneratedthroughanautomatic,semi-automaticormanualapproaches.Themodelcanbegeneratedbyanalyzingthesystemspecification,thesourcecodeortheenvironment.Thegeneratedmodelisthenexploredusingawidevarietyoftechniques,rangingfromrandomwalkofthemodeltoevolutionaryorgeneticalgorithms.Testoracleisacriticalcomponentoftheframeworkanditisusedtodifferentiatebetweenthecorrectandincorrectsystemexecution.Atestoracleisusedtoidentifyfailure-revealingtestinputs,whileexploringthemodeloftheSUT.Theefficacyofthetest-generationtechniquelargelydependsonthelevelofabstractionofthemodelandtheefficiencyoftheexplorationalgorithm.Acoarse-grainedmodelisrelativelyeasytocreateandexplore,butitmaymisssomeoftheimportant(failure-revealing)scenarios.Onthecontrary,averydetailedandfine-grainedmodelisdifficulttocreateandexplore.However,suchafine-grainedmodelislikelytodiscovermorefailurerevealingtestinputs.Consideringtheaccuracyandprecisionofabstraction,wefurtherclassifythetechniquesinthiscategory,basedontherespectivemodelsusedfortesting.Inthefollowingsections,wedescribeeachsuchmodelinmoredetails.

5.1TimedStateMachinesModelingstools,suchasMarkovchainshavebeenusedtomodelandtestsystemsforalongtime.Tobemorespecific,MarkovChainUsageModels(MCUM)canbedescribedasdirectedgraphs,wherethenodesofthegraphrepresentthestatesoftheSUT.Thenodesofthesystemareconnectedbyedges,representingevents(inputs)thatmayarriveatagivenstateofthesystem.Additionally,edgesareannotatedwiththeprobabilityoftheoccurrenceofanevent,whenthesystemisinagivenstate.HoweverMCUMs,bythemselves,donotprovideasuitablewayofrepresentingthetiming-relatedpropertiesoftheSUT.Suchtiming-relatedpropertiesmayrequirecertaineventstohappenbefore,afterorwithinaspecificdeadline.Sincetiming-relatedrequirementsareoftenanintegralpart

ofreal-timeembeddedsystem(eg,inautomotiveapplications),MCUMswereextendedtocapturesuchrequirements.OneoftheearliestsuchextensionsofMCUMswasproposedin[5],wheretheextendedMCUMsarereferredtoasTimedUsageModels(TUMs).SimilartotheconventionalMCUMs,allpaths,fromthestartstatetotheendstateinaTUM,representfeasibleexecutionsoftheSUT.Figure7Aprovidesasimpleexampleofatimedusagemodel.However,therealsoexistssomekeydifferencesbetweenanMCUMandaTUMthatarelistedinthefollowing:

FIGURE7 Simpleexampleshowing(A)timedusagemodeland(B)timedautomata.

•SimilartotheconventionalMCUMs,aTUMhasasetofstatestocapturethefeasibleusageofthesystem.However,inTUM,anadditionalprobabilitydistributionfunction(pdf)isassociatedwitheachstate.Thispdfencodesthetime,forwhichtheSUTwillbeintherespectivestate.

•InTUM,eachtransitionbetweentwostatesistriggeredbyastimulus.Additionally,edgesconnectingthestatesareassociatedwithtwovariables,atransitionprobabilityandaprobabilitydistributionfunction(pdf)ofstimulustime.Asthenamesuggests,thetransitionprobabilitycapturestheprobabilityoftherespectivetransitionbetweentwostates.Therefore,thetransitionprobabilityhasasimilarroletothatofconventionalMCUMs.Thepdfofthestimulustimerepresentsthedurationofexecutionofthestimulusonthesystem,atagivenstate.

•InadeterministicMCUM,therecouldbeatmostonetransition(fromagivenstate)foragivenstimulus.However,inaTUM,thenextstatenotonlydependsonthestimulus,butalsoonthedurationoftheexecutionofthestimulus.Thisfeatureisrequiredtocapturetiming-relateddependenciesinthesystem.Additionally,tomaintainconsistency,thepdfsofstimulustime,originatingfromastate,donotoverlap.

Oncethemodelofthesystemhasbeencreated,avarietyofmodel-explorationtechniquescanbeusedtogeneratetestcases.Forinstance[5]and[6]performasimplerandomwalkoftheTUMmodeltogeneratetestcaseswhileotherworkssuchas[7]and[8],havedesignedcoveragemetricstoguidethetest-generationprocess.Inparticular,worksin[7]and[8],combinetheusageofTUMswithdependenciesbetweenthedifferentcomponentsoftheSUT.Thisallowsthemtogeneratetestcasesthatnotonlyrepresent

differenttimingscenarios,butalsocapturedependenciesbetweencriticalsystemcomponents.

Anotherlineofwork[9]proposetoextendfinitestatemachinesmodel(FSM)toincorporatetiming-relatedconstraints.Suchamodelismostcommonlyknownastimedautomata(TA).Intimedautomata,anFSMisaugmentedwithafinitenumberofclocks.TheseclocksareusedtogeneratebooleanconstraintsandsuchconstraintsarelabeledontheedgesoftheTA.Additionally,clockvaluescanbemanipulatedandresetbydifferenttransitionsoftheautomata.Thebooleanconstraintssuccinctlycapturesthecriteriafortherespectivetransitionbeingtriggered.Timedautomataalsohasthefeaturetolabeltime-criticalstates.Forinstance,statesmarkedasUrgentorCommittedimplythatnotimecanbespentinthesestates.Besides,whileexploringthemodel,certainstates(suchasstatesmarkedasCommitted),havepriorityoverotherstates.Theseadditionalfeaturesmaketheprocessofmodelingintuitiveandalsomakethemodeleasiertoread.Figure7Bprovidesasimpleexampleoftimedautomata.Amajordifferencebetweentheworks(eg,worksin[5–8])thatuseTUMasamodelingapproachascomparedtoworks(eg,workin[9])thatusetimedautomata,isinthemodelexploration.Whereastheformeruseeitherrandomorguidedwalksofthemodeltogeneratetestcases,thelateruseevolutionaryalgorithmstoexplorethemodelandgeneratetestcases.

5.2MarkovDecisionProcessOneofthekeyassumptions,whichweremadewhiledesigningTUMs(asdescribedintheprecedingsection),wasthattheprobabilitydistributionsfortransitionswereknownapriori.Thisisusuallytruefordeterministicsystems.However,asarguedbytheworkin[10],suchtransitionprobabilitiesareoftenunavailablefornon-deterministicsystems.Therefore,whentestingnon-deterministicsystemsfornon-functionalproperties,suchasreliability,TUMsdonotpresentasuitableapproach.Forsuchsystems,theworkof[10]proposesanapproachbasedonMarkov-DecisionProcess(MDP).Inparticular,thesystem-levelmodelingisperformedviaMDPs.SinceMDPscansupportnon-determinism,itisasuitableplatformforcapturingthenon-determinisminasystem.OnceanMDPmodeliscreatedforthesystem,theworkin[10]usesacombinationofhypothesistestingandprobabilisticmodelcheckingtocheckreliabilityconstraints.Hypothesistestingisastatisticalmechanism,inwhich,oneofmanycompetinghypothesisarechosenbasedontheobserveddata.Inparticular,[10]useshypothesistestingtoobtainthereliabilitydistributionofthedeterministiccomponentsinthesystem.Suchreliabilitydistributionarecomputedwithinthespecificerrorboundsthattheuserneedstoprovide.Subsequently,probabilisticmodelcheckingisusedonMDPstocomputetheoverallreliabilityofthesystem.

Theworkof[11]usesasimilartechniquetoobtainreliabilitydistributionforareallife,healthcaresystem.Thesystemtestedin[11]isanambient-assisted-living-systemforelderlypeoplewithdementia.Suchembeddedsystemsmustfunctionreliablyunderalloperatingscenarios.Failuretodosomaycauseseriousinjuriestotherespectivepatients.Forsuchsystems,thenon-determinismintheenvironmentalfactors(eg,theinputfrom

thesensorsandhumanbehavior)makesthesystemcomplexandmakeitchallengingtoproducetherequiredreliabilityassurances.However,theworkof[11]hasshownthatanMDP-basedapproachcanbeeffectivelyusedtotestcomplex,reallifesystemsinascalableandefficientmanner.

5.3UnifiedModelingLanguageAdifferentlineofwork[12–14]usesUnifiedModelingLanguage(UML)tomodelandtestreal-timesystemsfortiming-relatedandsafety-relatedproperties.UMLprovidesawellknown(andwellaccepted)standardforsoftwaremodelinganditisusedindifferentdimensionsofsoftwaretesting.InUML,thestructureofasystemundertestcaneasilyberepresentedviatheutilitiesprovidedbyUML,suchasobjectdiagramsandcomponentsdiagrams.Additionally,thebehaviorofthemodeledsystemcanberepresentedbyusecases,statechartsormessage-sequencecharts.However,formodelingembedded,real-timesystems,UMLneedstobeextendedwithadditionalpackages,suchaspackagesfornon-functionalpropertiesandschedulingormanagementofresources.ThesepackagescanbeavailedthroughModelingandAnalysisofRealtimeEmbeddedSystemsExtension(MARTE)ofUML.Inparticular,constraints(suchastiming-relatedconstraints)onreal-timesystemcanbecapturedthroughastandardlanguageknownasObjectConstraintLanguage(OCL).Oncethesystemismodeledwithappropriateconstraints,failure-inducingtestcasescanbegeneratedbyexploringthemodel.Forinstance,thesearchtechniquesin[12,13]comparestheeffectivenessoftheirtest-generationprocessforrandomtesting,adaptiverandomtestingandevolutionaryalgorithms,whileotherworks[14]experimentwiththeeffectivenessofgeneticalgorithmsasasearchstrategy.Theseworksobservethat,atleastfortheevaluatedcasestudies,noneofthesearchstrategies(fortestgeneration)weredefinitivelysuperiorthanothers.However,subsequentworks[13]haveclaimedbetterefficiency,whensearchingforfailure-inducingtestcases,throughhybridsearchstrategies.

5.4EventFlowGraphSystematictestingofevent-drivenapplicationsfornon-functionalproperties,suchasenergyconsumption,isachallengingtask.Thisisprimarilybecauseofthefactthatlikeanyothernon-functionalproperty,informationrelatedtoenergyconsumptionisseldompresentinthesourcecode.Additionally,suchinformationmaydifferacrossdifferentdevices.Therefore,generatingenergy-consumptionannotation,foreachapplicationanddevice,isdefinitelytimeconsuminganderror-prone.Areal-lifescenarioofsuchevent-drivensystemsismobileapplications.Mobileapplicationsareusuallyexecutedonbattery-constrainedsystems,suchassmartphones.Smartphones,inturn,areequippedwithenergy-hungrycomponents,suchasGPS,WiFianddisplay.Thisnecessitatesthedevelopmentofefficientandautomatedtestingtechniquetostressenergyconsumption.Onesuchtechniquehasbeenpresentedin[15].ItautomaticallygeneratestheEventFlowGraph(EFG)[16]oftheapplicationundertest.AnEFGcanbedescribedasadirectedgraph,wherethenodesofthegraphrepresenteventsandtheedgescapturethehappens-

afterrelationshipbetweenanytwoevents.Itispossible(andoftenthecase)thatEFGsofmobileapplicationshavecycles(suchastheexampleshowninFig.8).Suchcyclestypicallydonothaveanexplicititerationbounds.Therefore,althoughanEFGhasafinitenumberofevents,anunboundednumberofeventsequencescanbegeneratedfromthesame.Thisfurthercomplicatestheprocessoftestgeneration,asanyeffectivetestingtechniqueshouldnotonlybeabletogenerateallfailure-revealingtestcases,butalsodosoinareasonableamountoftime.

FIGURE8 ModernsmartphoneshaveawidevarietyofI/Oandpowermanagementutilities,improperuseofwhichintheapplicationcodecanleadtosuboptimalenergy-consumption

behavior.Smartphoneapplicationareusuallynonlinearpiecescode,systematictestingofwhichrequiresaddressinganumberofchallenges.

Theframeworkpresentedin[15]hastwokeyinnovationsthathelpsittotacklethechallengesdescribedintheprecedingparagraph.Thefirstofthosetwoinnovationsbeingthedefinitionofametricthatcapturestheenergyinefficiencyofthesystem,foragiveninput.Todesignsuchametric,itisimportanttounderstandwhatexactlyqualifiesasenergy-inefficientbehavior.Inotherwords,letusconsiderthefollowingquestion:Doeshigh-energyconsumptionalwaysimplyhigherenergy-inefficiency?Asitturnsout[15],theanswertothisquestionisnottrivial.Forinstance,considerascenariowheretwosystemshavesimilarenergy-consumptionbehaviorbutoneisdoingmorework(hasahigherutilizationofitshardwarecomponents)thantheother.Insuchascenario,itisquiteintuitivethatthesystemwithhigherutilizationisthemoreenergy-efficientone.Takinginspirationfromthisobservation,theworkin[15]definesthemetricofE/Uratio(energyconsumptionvsutilization)tomeasuretheenergyinefficiencyofasystem.Foragiveninput,theframeworkexecutestheapplicationonarealhardwaredeviceandanalysestheE/Uratioofthedeviceatruntime.AnanomalouslyhighE/Uratio,duringtheexecutionoftheapplication,indicatesthepresenceofanenergyhotspot.Additionally,aconsistentlyhighE/Uratio,aftertheapplicationhascompletedexecution,indicatesthepresenceofanenergybug.Ingeneral,energybugscancausemorewastageofbatterypowerthanenergyhotspotsandcandrasticallyreducetheoperationaltimeofthesmartphone.WiththemetricofE/Uratio,itispossibletofindenergy-inefficientbehaviorintheSUT,foragiveninput.However,anotherchallengeistogenerateinputstostressenergybehaviorof

agivenapplication,inareasonableamountoftime.Interestingly,forsmartphoneapplications,anumberofpreviousstudies(primarilybasedonAndroidoperatingsystem)haveobservedthatmostoftheenergy-hungrycomponentscanonlybeaccessedthroughapredefinedsetofsystemcalls.Theworkin[15]usesthisinformationtoprioritizethegenerationoftestinputs.Inparticular,[15]usesaheuristic-basedapproach.Thisapproachtriestoexplorealleventtracesthatmayinvokesystemcallstoenergy-hungrycomponents.Besides,theworkalsoprioritizesinputsthatmightinvokeasimilarsequenceofsystemcallscomparedtoanalreadydiscovered,energy-inefficientexecution.

6White-BoxAbstractionInthissection,weshalldiscusssoftwaretestingmethodologiesthatarecarriedoutdirectlyontheimplementationofanapplication.Suchanimplementationmaycapturethesourcecode,theintermediatecode(aftervariousstagesofcompilation)orthecompiledbinaryofanembeddedsoftware.Whereasweonlyspecializethetestingproceduresatthelevelofabstractionstheyarecarriedout,weshallobserveinthefollowingdiscussionthatseveralmethodologies(eg,evolutionarytestingandsymbolicexecution)canbeusedtotesttheimplementationofembeddedsoftware.Theideaofdirectlytestingtheimplementationispromisinginthecontextoftestingembeddedsoftware.Inparticular,ifthedesignerisinterestedinaccuratelyevaluatingthenon-functionalbehaviors(eg,energyandtiming)ofdifferentsoftwarecomponents,suchnon-functionalbehaviorsarebestobservedatthelevelofimplementation.Ontheflipside,ifaseriousbugwasdiscoveredintheimplementation,itmayleadtoacompleteredesigningoftherespectiveapplication.Ingeneral,itisimportanttofigureoutanappropriatelevelofabstractiontorunthetestingprocedure.Weshallnowdiscussseveralworkstotesttheimplementationofembeddedsoftwareandreasonabouttheirimplications.Inparticular,wediscusstestingmethodologiesfortiming-relatedpropertiesinSection6.1andforfunctionality-relatedbehaviorsinSection6.2.Finally,inSection6.3,wediscusschallengestobuildanappropriateframeworktoobserveandcontroltestexecutionsofembeddedsoftwareandwealsodescribesomerecenteffortsinthesoftwareengineeringcommunitytoaddresssuchchallenges.

6.1TestingTiming-relatedPropertiesTheworkin[17]showstheeffectivenessofevolutionarysearchfortestingembeddedsoftware.Inparticular,thisworktargetstodiscoverthemaximumdelaycausedduetointerrupts.Inembeddedsoftware,interruptsarecommonphenomenon.Forinstance,theincomingsignalsfromsensorsornetworkevents(eg,arrivalofapacket)mightbecapturedviainterrupts.Besides,embeddedsystemsoftenconsistofmultipletasks,whichshareresources(eg,CPUandmemory).Asaresult,switchingtheCPUfromataskttoataskt′willclearlyinduceadditionaldelaytothetaskt.Suchswitchingofresourcesarealsotriggeredviainterrupts.Therefore,thedelaycausedduetointerruptsmightsubstantiallyaffecttheoveralltimingbehavior.Forinstance,inthefalldetectionapplication,eachsensormightbeprocessedbyadifferenttaskandanothertaskmightbeusedtocomputeapotentialfallofthepatient.IfallthesetasksshareacommonCPU,aparticulartaskmightbedelayedduetotheswitchingofCPUbetweentasks.

Fig.9illustratesscenarioswherethetasktocomputeafallisdelayedbyinterruptsgeneratedfromtheaccelerometerandthegyroscope.Inparticular,Fig.9Ademonstratestheoccurrenceofasingleinterrupt.Onthecontrary,Fig.9Billustratesnestedinterrupts,whichprolongedtheexecutiontimeofthecomputationtask.Ingeneral,arrivalofaninterruptishighlynon-deterministicinnature.Moreover,itispotentiallyinfeasibletotestanembeddedsoftwareforallpossibleoccurrencesofinterrupts.

FIGURE9 Interruptlatency,(A)singleinterruptand(B)Nestedinterrupts.

Theworkin[17]discussesageneticalgorithmtofindthemaximuminterruptlatency.Inparticular,thisworkshowsthatatestingmethodbasedongeneticalgorithmissubstantiallymoreeffectivecomparedtorandomtesting.Thismeansthattheinterruptlatencydiscoveredviathegeneticalgorithmissubstantiallylargerthantheonediscoveredusingrandomtesting.Anearlierwork[18]alsousesgeneticalgorithmtofindtheWCETofaprogram.Incontrastto[17],theworkin[18]focusesontheuninterruptedexecutionofasingleprogram.Morespecifically,thetestingmethod,asproposedin[18],aimstosearchtheinputspaceandmoreimportantly,directthesearchtowardWCETrevealinginputs.

ItiswellknownthattheprocessingpowerofCPUshaveincreaseddramaticallyinthelastfewdecades.Incontrast,memorysubsystemsareseveralorderofmagnitudesslowerthantheCPU.SuchaperformancegapbetweentheCPUandmemorysubsystemsmightbecriticalforembeddedsoftware,whensuchsoftwarearerestrictedviatiming-relatedconstraints.Morespecifically,ifthesoftwareisspendingasubstantialamountoftimeinaccessingmemory,thentheperformanceofanapplicationmayhaveaconsiderableslowdown.Inordertoinvestigatesuchproblems,somerecenteffortsinsoftwaretesting[19,20]haveexplicitlytargetedtodiscovermemorybottlenecks.Sucheffortsdirectlytestthesoftwarebinarytoaccuratelydeterminerequeststothememorysubsystems.Inparticular,requeststothememorysubsystemsmightbereducedsubstantiallybyemployingacache.Worksin[19,20]aimtoexercisetestinputsthatleadtoapoorusageofcaches.Morespecifically,theworkin[19]aimstodiscovercachethrashingscenarios.Acachethrashingscenariooccurswhenseveralmemoryblocksreplaceeachotherfromthecache,hence,generatingasubstantialnumberofrequeststothememorysubsystems.Forinstance,thecodefragmentinFig.10mayexhibitacachethrashingwhenthecachecanholdexactlyonememoryblock.Inthecodefragment,m1andm2replaceeachotherfromthecache,leadingtoacachethrashing.Thisbehaviorismanifestedonlyfortheprograminput‘t’.

FIGURE10 Inputdependentcachethrashing.

Theworkin[19]showsthattheabsenceofsuchcachethrashingscenarioscanbeformulatedbysystematicallytransformingtheprogramwithassertions.Subsequently,asearchprocedureonthesoftwareinputspacecanbeinvokedtofindviolationofsuchassertions.Anyviolationofanassertion,thus,willproduceacachethrashingscenario.Themethodologyproposedin[19]usesacombinationofstaticanalysisandsymbolicexecutiontosearchtheinputspaceanddiscoverinputsthatviolatetheformulatedassertions.

Theworkin[20]liftsthesoftwaretestingofembeddedsoftwareformassivelyparallelapplications,withaspecificfocusongeneral-purposegraphicsprocessingunits(GPGPU).Itiswellknownthatfuturetechnologywillbedominatedbyparallelarchitectures(eg,multicoresandGPGPUs).Forsucharchitectures,softwaretestingshouldtakeintoaccounttheinputspaceoftheapplication,aswellasthenon-deterministicnatureofschedulingmultiplethreads.Theworkin[20]formallydefinesasetofscenariosthatcapturememorybottlenecksinparallelarchitectures.Subsequently,asearchprocedureisinvokedtosystematicallytraversetheinputspaceandthespaceconsistingofallpossibleschedulingdecisionsamongthreads.Liketheapproachin[19],theworkin[20]alsousesacombinationofstaticanalysisandsymbolicexecutionforthesearch.Insummary,boththeworks[19,20]revolvearounddetectingfine-grainedeventssuchasmemoryrequests.Ingeneral,suchfine-grainedeventsareappropriatetotestonlyattheimplementationlevel(eg,softwarebinary).Thisisbecausetheoccurrenceofsucheventswouldbesignificantlydifficulttopredictatintermediatestagesofthedevelopment.

6.2TestingFunctionality-relatedPropertiesIntheprecedingSection,wehavediscussedsoftwaretestingmethodologiesthatfocusonvalidatingtiming-relatedconstraintsofembeddedsoftware.Incontrasttosuchmethodologies,theworkin[21]primarilytargetsfunctionalpropertiesofembeddedsoftware.Inparticular,authorsof[21]discusssomeuniquechallengesthatmightappearonlyinthecontextoftestingembeddedsoftwareandsystems.Thekeyobservationisthatembeddedsystemsoftencontaindifferentlayersofhardwareandsoftware.Besides,an

embeddedapplicationmaycontainmultipletasks(eg,programs)andsuchtasksmightbeactivesimultaneously.Forinstance,inourfall-detectionapplication,theaccesstohardwarecomponents(eg,gyroscopeandaccelerometers)mightbecontrolledbyasupervisorysoftware,suchasoperatingsystems(OS).Similarly,samplingsignalsfromsensorsandcomputationofapotentialfallmightbeaccomplishedbydifferenttasksthatrunsimultaneouslyinthesystem.Theworkin[21]arguestheimportanceoftestinginteractionsbetweendifferenthardware/softwarelayersanddifferenttasks.Fig.11conceptuallycapturessuchinteractionsinatypicalembeddedsystem.

FIGURE11 Interactionamongdifferenttasksandhardware/softwarelayers.

Inordertoexerciseinteractionsbetweentasksanddifferentsoftwarelayers,authorsof[21]havedescribedasuitablecoveragecriteriafortestingembeddedsystems.Forinstance,theinteractionbetweenapplicationlayerandOSlayercanhappenviasystemcalls.Similarly,theapplicationmightdirectlyaccesssomehardwarecomponentsviaapredefinedsetofapplicationprogrammerinterfaces(APIs).Theworkin[21]initiallyperformsastaticanalysistoinferdatadependenciesacrossdifferentlayersoftheembeddedsystem.Besides,ifdifferenttasksofthesystemusesharedresources,suchananalysisalsotracksthedatadependenciesacrosstasks.Forinstance,considerthepieceofcodefragmentinFig.12,wheresyscallcapturesasystemcallimplementedinthekernelmode.InthecodeshowninFig.12,thereexistsadatadependencybetweenapplicationlayervariablegandthesystemcallsyscall.Asaresult,itisimportanttoexercisethisdatadependencytotesttheinteractionbetweenapplicationlayerandOSlayer.Therefore,theworkin[21]suggeststoselecttestcasesthatcanmanifestthedatadependencybetweenvariablegandsyscall.Toillustratethedependencybetweenmultipletasks,letusconsiderthecodefragmentinFig.13.

FIGURE12 Interactionbetweenapplicationlayervariableandoperatingsystem.

FIGURE13 Interactionbetweentasksviasharedresources(sharedvariables).

Thekeyword__shared__capturessharedvariables.InFig.13,thereisapotentialdatadependencybetweenTask1andTask2.However,toexercisethisdatadependency,thedesignermustbeabletoselectaninputthatsatisfiestheconditioninput==‘x’.Theworkin[21]performsstaticanalysistodiscoverthedatadependenciesacrosstasks,asshowninthisexample.Oncealldatadependenciesaredeterminedviastaticanalysis,thechosentestinputsaimtocoverthesedatadependencies.

6.3BuildingSystematicTest-executionFrameworkSofarinthissection,wehavediscussedtestinputgenerationtovalidateeitherfunctionalornon-functionalpropertiesofembeddedsoftware.However,asdiscussedin[22,23],thereexistsnumerousotherchallengesfortestingembeddedsoftware,suchasobservabilityoffaultyexecution.Testoraclesareusuallyrequiredtoobservefaulty

execution.Designingappropriateoraclesisdifficultevenfortraditionalsoftwaretesting.Inthecontextofembeddedsoftware,designingoraclesmayfaceadditionalchallenges.Inparticular,asembeddedsystemsconsistofmanytasksandexhibitinteractionsacrossdifferenthardwareandsoftwarelayers,theymayoftenhavenondeterministicoutput.Asaresult,oracles,whicharepurelybasedonoutput,areinsufficienttoobservefaultsinembeddedsystems.Moreover,itiscumbersometobuildoutput-basedoraclesforeachtestcase.Inordertoaddressthesechallenges,authorsin[22]proposetodesignproperty-basedoraclesforembeddedsystems.Property-basedoraclesaredesignedforeachexecutionplatform.Therefore,anyapplicationtargetingsuchexecutionplatformmightreusetheoraclesandthereby,itcanavoidsubstantialmanualeffortstodesignoraclesforeachtestcase.Theworkin[22]specificallytargetsconcurrencyandsynchronizationproperties.Forinstance,testoraclesaredesignedtospecifyproperusageofbinarysemaphoresandmessagequeues,whichareusedforsynchronizationandinterprocesscommunication,respectively.SuchsynchronizationandinterprocesscommunicationAPIsareprovidedbytheoperatingsystem.Oncetestoraclesaredesigned,atestcasecanbeexecuted,whileinstrumentingtheapplication,OSandhardwareinterfacessimultaneously.Eachexecutioncansubsequentlybecheckedforviolationofpropertiescapturedbyanoracle.Thusproperty-basedtestoraclescanprovideacleaninterfacetoobservefaultyexecutions.Apartfromtestoracles,authorsin[23]discusstheimportanceofgivingthedesignerappropriatetoolsthatcontroltheexecutionofembeddedsystems.Sincetheexecutionofanembeddedsystemisoftennon-deterministic,itis,ingeneraldifficulttoreproducefaultyexecutions.Forinstance,considerthefalldetectionapplicationwhereataskreadssensordatafromasinglequeue.Ifnewdataarrives,aninterruptisraisedtoupdatethequeue.Itisworthwhiletoseethepresenceofapotentialdataracebetweentheroutinethatservicestheinterruptandthetaskwhichreadsthequeue.Unfortunately,thearrivalofinterruptsishighlynon-deterministicinnature.Asaresult,evenaftermultipletestexecutions,thetestingmaynotrevealafaultyexecutionthatcaptureapotentialdatarace.Inordertosolvethis,authorsin[23]designappropriateutilitiesthatgivesdesignerthepowertoraiseinterruptsexplicitly.Forinstance,thedesignermightchooseasetoflocationswhereshesuspectsthepresenceofdataracesduetointerrupts.Subsequently,atestexecutioncanbecarriedoutthatraiseinterruptsexactlyatthelocationsspecifiedbythedesigner.

SummaryTosummarize,inthissection,wehaveseeneffortstogeneratetestinputsandtestoraclestovalidatebothfunctionalandnon-functionalaspectsofembeddedsoftware.Acommonaspectofallthesetechniquesisthatthetestingprocessiscarriedoutdirectlyontheimplementation.Thismightbeappealingincertainscenarios,forinstance,whenthedesignerisinterestedineventsthatarehighlysensitivetotheexecutionplatform.Sucheventsincludeinterrupts,memoryrequestsandcachemisses,amongothers.

7FutureDirectionsAsdiscussedinthischapter,analysisofnon-functionalpropertiesiscrucialtoensurethatembeddedsystemsbehaveasperitsspecification.However,thereexistsanorthogonaldirectionofwork,whereanalysisofnon-functionalproperties,suchaspowerconsumption,memoryaccessesandcomputationallatencies,havebeenusedforsecurity-relatedexploits.Suchexploitsarecommonlyreferredtoasside-channelattacksandaredesignedtoextractprivatekeys2fromcryptographicalgorithms,suchasalgorithmsusedinsmartcardsandsmarttokens.Theintentionoftheattackerisnottodiscoverthetheoreticalweaknessesofthealgorithm.Instead,theattackeraimstobreaktheimplementationofthealgorithmsthroughsidechannels,suchasmeasuringexecutiontimeorenergyconsumption.Inparticular,theattackertriestorelatesuchmeasurementswiththesecretkey.Forinstance,ifdifferentsecretkeysleadtodifferentexecutiontime,theattackercanperformstatisticalanalysistomapthemeasuredexecutiontimewiththerespectivekey.Ingeneral,anynon-functionalbehaviorthathasacorrelationwithcryptographiccomputation,iscapableofleakinginformation,ifnotmanagedappropriately.Forexample,thedifferentialpowerattack,asproposedin[24],usesasimple,yeteffectivestatisticalanalysistechniquetocorrelatetheobservedpower-consumptionbehaviortotheprivatekey.Sincethen,anumberofsubsequentworkshaveproposedcounter-measures(eg,[25])againstside-channelvulnerabilitiesandbypassestothosecounter-measures(eg,[26]).Similarly,researchershavealsostudiedside-channelattacks(andtheircounter-measures)basedonothernon-functionalbehaviors,suchascomputationallatency[27,28]andmemoryfootprint[29].Eventhoughworksonside-channelattackshaveaverydifferentobjectivecomparedtothoseonnon-functionaltesting,thereexistsanumberofcommonalities.Inessence,bothlinesofworkarelookingfortestinputsthatleadtoundesirablenon-functionalbehavior.Thedefinitionofthephraseundesirablenon-functionalbehaviorisbasedonthesystemundertest(SUT).Forinstance,inanembeddedsystemthathashardtiming-relatedconstraints,anundesirableinputwouldbetheviolationofsuchconstraints.Onthecontrary,foracryptographicalgorithm,suchasimplementedinasmartcard,anundesirableinputmayleadtoinformationleaksviasidechannels.Undesirablenon-functionalbehaviorinonescenariomayleadtoperformanceloss,sometimescostinghumanlives(suchasinananti-lockbrakingsystem),whereas,intheotherscenarioundesirablenon-functionalbehaviormaycauseinformationleaks,which,inturnmayoftenleadtofinanciallosses.Itisneedlesstomotivatethefactthattestingembeddedcryptographicsystemsforsuchundesirablenon-functionalbehaviorsiscrucial.Moreimportantly,testingmethodologiesfordetectingside-channelattacksneedtobeautomated.However,asofthiswriting,thislineofresearchisfarfrombeingsolved.Newworksonthistopiccoulddrawinspirationfromearlierworksonnon-functionaltesting,suchasworksdescribedinSection3.

Anothermoregenericdirectionisrelatedtothedetectionofrootcauseandautomaticrepairofnon-functionalpropertiesinembeddedsystems.Ingeneral,thepurposeofsoftwaretestingistoexposesuboptimalorunwantedbehaviorintheSUT.Suchsuboptimalbehaviors,onceidentified,shouldberectifiedbymodifyingthesystem.Morespecifically,therectificationprocesscanbesubdividedintotwoparts:fault-localization3

androot-causedetection,followedbydebuggingandrepair.Fault-localizationis,ingeneral,themoretime-consuming(andexpensive)phaseofthismodificationprocessandtherefore,thereisahugedemandforeffective,automatedtechniquesforfault-localization.Overthepastyears,severalworkshaveproposedmethodologiesforfault-localization.However,mostoftheseworkshavefocusedonthefunctionalityofsoftware.Asofthiswriting,thereexistsalackofeffortsinfault-localizationtechniquesfornon-functionalproperties.Oneplausibleexplanationcanbethatdesigningsuchaframework,fornon-functionalproperties,issignificantlymorechallenging.Thisisbecausethenon-functionalbehaviorofsystemdependsnotonlyonthesourcecode,butalsoontheunderlyingexecutionplatform.Someofthewellknowntechniques[30]forfault-localizationincludecomparingasetoffailedexecutiontoasetofpassingexecutionsandsubsequently,derivingtherootcauseforthefault.Suchworksnarrowdownthesearchspacefortherootcausebyassigningsuspiciousnessvaluestospecificregionsofsourcecode.Asisthecasewithfault-localization,considerableresearchneedstobeperformedonautomateddebuggingandrepairofnon-functionalsoftwareproperties.Asamatteroffact,automatedprogramrepair,eveninthecontextofsoftwarefunctionality,isfarfrombeingmatured,nottomentionthelackofresearchfornon-functionalsoftwareproperties.Asforembeddedsoftwareandsystems,bothfunctionalandnon-functionalbehaviorsplaycrucialrolesinvalidatingtherespectivesystem.Wehopethatfutureworksinsoftwaretestingwillresolvethesevalidationchallengesfacedbyembeddedsystemdesigners.

8ConclusionEmbeddedsystemsareubiquitousinthemodernworld.Suchsystemsareusedinawidevarietyofapplications,rangingfromcommonconsumerelectronicdevicestoautomotiveandavionicapplications.Apropertycommontoallembeddedsystemsisthattheyinteractwiththephysicalenvironment,oftenderivingtheirinputsfromthesurroundingenvironment.Duetotheapplicationdomainssuchsystemsareusedin,theirbehaviorisoftenconstrainedbyfunctional(suchastheinput–outputrelationship)aswellasnon-functionalproperties(suchasexecutiontimeorenergyconsumption).Thismakesthetestingandvalidationofsuchsystemsachallengingtask.Inthischapter,wediscussedafewchallengesandtheirsolutionsinthecontextoftestingembeddedsystems.Inparticular,wetakeacloserlookintoexistingworksontestingnon-functionalproperties,suchastiming,energyconsumption,reliability,forembeddedsoftware.Toputtheexistingworksinperspective,weclassifytheminthreedistinctcategories,basedonthelevelofsystemabstractionusedfortesting.Thesecategoriesinclude,black-box,grey-boxandwhite-boxabstractionbasedtestingapproaches.Ingeneral,black-boxabstractionbasedtestingmethodsusesamplingbasedtechniquestogeneratefailure-revealingtestcasesforthesystemundertest.Suchmethodsconsiderthesystemasablack-boxandhenceareequallyapplicabletosimpleandcomplexsystemsalike.However,sucheaseofuseusuallycomesatthecostofeffectiveness.Inparticular,thesemethodsoftencannotprovidecompletenessguarantees(ie,bythetimethetest-generationprocesscompletes,allfailurerevealingtestinputsmusthavebeenuncovered).Thegrey-boxabstractionbasedapproachesareusuallymoreeffectivethantheblack-boxabstractionbasedapproaches.Thisisbecausesuchmethodsoftenemployanabstractmodelofthesystemundertesttogeneratefailure-revealingtestcases.Effectivenessofthesetest-generationmethodologiesisoftendictatedbythelevelofsystemabstractionbeingused.White-boxabstractionbasedtestingapproachesusetheactualsystemimplementationtogeneratefailurerevealingtestcasesandhencearecapableofprovidingmaximumlevelofguaranteetodiscoverfailurerevealinginputs.Weobservethatexistingtechniquesvaryhugelyintermsofcomplexityandeffectiveness.Finally,wehavediscussedfutureresearchdirectionsrelatedtoembeddedsoftwaretesting.Oneofwhichwasautomatedfault-localizationandrepairingofbugsrelatedtonon-functionalproperties.Anotherdirectionwasrelatedtothedevelopmentofsecureembeddedsystems.Inparticular,weexploredthepossibilityoftestingtechniquestoexploitthevulnerabilitytowardside-channelattacks.Overtherecentyears,therehavebeenanumberofworks,whichanalyzenon-functionalbehaviortoperformside-channel(securityrelated)attacks.Itwouldbeappealingtoseehowexistingtestingmethodologiescanbeadaptedtotestandbuildsecureembeddedsoftware.

AcknowledgmentTheworkwaspartiallysupportedbyaSingaporeMoETier2grantMOE2013-T2-1-115entitled“Energyawareprogramming”andtheSwedishNationalGraduateSchoolonComputerScience(CUGS).

References[1]Awearableminiaturizedfalldetectionsystemfortheelderly.http://www.fallwatch-project.eu/press_release.php.

[2]NghiemT.,SankaranarayananS.,FainekosG.,IvancićF.,GuptaA.,PappasG.J.Monte-carlotechniquesforfalsificationoftemporalpropertiesofnon-linearhybridsystems.In:Proceedingsofthe13thACMInternationalConferenceonHybridSystems:ComputationandControl,HSCC’10;2010.

[3]SankaranarayananS.,FainekosG.Falsificationoftemporalpropertiesofhybridsystemsusingthecross-entropymethod.In:Proceedingsofthe15thACMInternationalConferenceonHybridSystems:ComputationandControl,HSCC’12;2012.

[4]AnnapureddyY.S.R.,FainekosG.E.Antcoloniesfortemporallogicfalsificationofhybridsystems.In:IECON2010–36thAnnualConferenceonIEEEIndustrialElectronicsSociety.2010.

[5]SieglS.,HielscherK.,GermanR.Introductionoftimedependenciesinusagemodelbasedtestingofcomplexsystems.In:SystemsConference,20104thAnnualIEEE;2010:622–627.

[6]SieglS.,HielscherK.,GermanR.,BergerC.Formalspecificationandsystematicmodel-driventestingofembeddedautomotivesystems.In:4thAnnualIEEESystemsConference,2010;2011.

[7]SieglS.,CaliebeP.Improvingmodel-basedverificationofembeddedsystemsbyanalyzingcomponentdependences.In:20116thIEEEInternationalSymposiumonIndustrialEmbeddedSystems(SIES).2011:51–54.

[8]LuchscheiderP.,SieglS.Testprofilingforusagemodelsbyderivingmetricsfromcomponent-dependency-models.In:20138thIEEEInternationalSymposiumonIndustrialEmbeddedSystems(SIES).2013:196–204.

[9]HanselJ.,RoseD.,HerberP.,GlesnerS.AnEvolutionaryalgorithmforthegenerationoftimedtesttracesforembeddedreal-timesystems.In:2011IEEEFourthInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2011.

[10]GuiL.,SunJ.,LiuY.,SiY.J.,DongJ.S.,WangX.Y.Combiningmodelcheckingandtestingwithanapplicationtoreliabilitypredictionanddistribution.In:Proceedingsofthe2013InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2013;2013.

[11]LiuY.,GuiL.,LiuY.MDP-basedreliabilityanalysisofanambientassistedlivingsystem.In:FM2014:FormalMethods.SpringerInternationalPublishing;LectureNotesinComputerScience.2014;vol.84422014.

[12]ArcuriA.,IqbalM.Z.,BriandL.Black-boxsystemtestingofreal-timeembeddedsystemsusingrandomandsearch-basedtesting.In:Proceedingsofthe22NdIFIP

WG6.1InternationalConferenceonTestingSoftwareandSystems,ICTSS’10;2010:95–110.

[13]IqbalM.Z.,ArcuriA.,BriandL.Combiningsearch-basedandadaptiverandomtestingstrategiesforenvironmentmodel-basedtestingofreal-timeembeddedsystems.In:Proceedingsofthe4thInternationalConferenceonSearchBasedSoftwareEngineering,SSBSE’12;2012.

[14]IqbalM.Z.,ArcuriA.,BriandL.Empiricalinvestigationofsearchalgorithmsforenvironmentmodel-basedtestingofreal-timeembeddedsoftware.In:Proceedingsofthe2012InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2012;2012.

[15]BanerjeeA.,ChongL.K.,ChattopadhyayS.,RoychoudhuryA.Detectingenergybugsandhotspotsinmobileapps.In:Proceedingsofthe22NdACMSIGSOFTInternationalSymposiumonFoundationsofSoftwareEngineering;2014.

[16]MemonA.M.,BanerjeeI.,NagarajanA.GUIripping:reverseengineeringofgraphicaluserinterfacesfortesting.In:WorkingConferenceonReverseEngineering.2003:260–269.

[17]WeisslederS.,SchlingloffH.Anevaluationofmodel-basedtestinginembeddedapplications.In:2014IEEESeventhInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2014:223–232.

[18]PuschnerP.P.,NossalR.Testingtheresultsofstaticworst-caseexecution-timeanalysis.In:IEEEReal-TimeSystemsSymposium.1998:134–143.

[19]BanerjeeA.,ChattopadhyayS.,RoychoudhuryA.Staticanalysisdrivencacheperformancetesting.In:Real-TimeSystemsSymposium(RTSS),2013IEEE34th.2013:319–329.

[20]ChattopadhyayS.,ElesP.,PengZ.AutomatedsoftwaretestingofmemoryperformanceinembeddedGPUs.In:2014InternationalConferenceonEmbeddedSoftware(EMSOFT);2014:1–10.

[21]YuT.,SungA.,Srisa-AnW.,RothermelG.Anapproachtotestingcommercialembeddedsystems.J.Syst.Softw.2014;88.

[22]YuT.,SungA.,Srisa-anW.,RothermelG.Usingproperty-basedoracleswhentestingembeddedsystemapplications.In:2011IEEEFourthInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2011:100–109.

[23]YuT.,Srisa-anW.,RothermelG.SimTester:acontrollableandobservabletestingframeworkforembeddedsystems.In:Proceedingsofthe8thACMSIGPLAN/SIGOPSConferenceonVirtualExecutionEnvironments,VEE’12,London,England,UK;2012:978-1-4503-1176-2.

[24]KocherP.,JaffeJ.,JunB.Differentialpoweranalysis.1998.http://www.cryptography.com/public/pdf/DPA.pdf.

[25]AkkarM.-L.,GiraudC.AnimplementationofDESandAES,secureagainstsomeattacks.In:ProceedingsoftheThirdInternationalWorkshoponCryptographicHardwareandEmbeddedSystems,CHES’01;2001.

[26]MangardS.,PramstallerN.,OswaldE.SuccessfullyattackingmaskedAEShardwareimplementations.In:CryptographicHardwareandEmbeddedSystems,CHES2005,LectureNotesinComputerScience.2005.

[27]P.Kocher,Timingattacksonimplementationsofdiffe-hellman,RSA,DSS,andothersystems.http://www.cryptography.com/public/pdf/TimingAttacks.pdf.

[28]KöpfB.,MauborgneL.,OchoaM.Automaticquantificationofcacheside-channels.In:Proceedingsofthe24thInternationalConferenceonComputerAidedVerification,CAV’12,Berkeley,CA;Berlin:Springer-Verlag;2012:978-3-642-31423-0564–580.doi:10.1007/978-3-642-31424-7_40.

[29]JanaS.,ShmatikovV.Memento:learningsecretsfromprocessfootprints.In:Proceedingsofthe2012IEEESymposiumonSecurityandPrivacy,SP’12;Washington,DC:IEEEComputerSociety;2012:978-0-7695-4681-0143–157.doi:10.1109/SP.2012.19.

[30]JonesJ.A.,HarroldM.J.Empiricalevaluationofthetarantulaautomaticfault-localizationtechnique.In:Proceedingsofthe20thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering,ASE’05,LongBeach,CA,USA;NewYork,NY:ACM;2005:1-58113-993-4273–282.doi:10.1145/1101908.1101949.

AbhijeetBanerjeeisaPh.D.scholarattheSchoolofComputing,NationalUniversityofSingapore.HereceivedhisB.E.inInformationTechnologyfromIndianInstituteofEngineeringScienceandTechnology,Shibpur,Indiain2011.Hisresearchinterestsincludeautomatedsoftwaretesting,debugging,andre-factoringwithspecificemphasisontestingandverificationofnon-functionalpropertiesofsoftware.

SudiptaChattopadhyayisaPost-doctoralResearchFellowintheCenterforIT-Security,Privacy,andAccountability(CISPA)inSaarbrücken,Germany.HereceivedhisPh.D.incomputersciencefromNationalUniversityofSingapore(NUS)in2013.Hisresearchinterestsincludesoftwareanalysisandtesting,withaspecificfocusondesigningefficientandsecuresoftwaresystems.

AbhikRoychoudhuryisaProfessorofComputerScienceatSchoolofComputing,NationalUniversityofSingapore.HereceivedhisPh.D.inComputerSciencefromtheStateUniversityofNewYorkatStonyBrookin2000.Since2001,hehasbeenemployedattheNationalUniversityofSingapore.Hisresearchhasfocusedonsoftwaretestingandanalysis,softwaresecurity,andtrust-worthysoftwareconstruction.Hisresearchhasreceivedvariousawardsandhonors,includinghisappointmentasACMDistinguishedSpeakerin2013.HeiscurrentlyleadingtheTSUNAMicenter,alarge5-yearlongtargetedresearcheffortfundedbyNationalResearchFoundationinthedomainofsoftwaresecurity.Hisresearchhasbeenfundedbyvariousagenciesandcompanies,includingtheNationalResearchFoundation(NRF),MinistryofEducation(MoE),A*STAR,DefenseResearchandTechnologyOffice(DRTech),DSONationalLaboratories,Microsoft,andIBM.Hehasauthoredabookon“EmbeddedSystemsandSoftwareValidation”publishedbyElsevier(MorganKaufmann)Systems-on-Siliconseriesin2009,whichhasalsobeenofficiallytranslatedtoChinesebyTsinghuaUniversityPress.HehasservedinvariouscapacitiesintheprogramcommitteesandorganizingcommitteesofvariousconferencesonsoftwareengineeringincludingICSE,ISSTA,FSE,andASE.HeiscurrentlyservingasanEditorialBoardmemberofIEEETransactionsonSoftwareEngineering(TSE).

1Otherwise,thetestingprocessshouldterminatewithassurancethatthesystemfunctionalityisexpectedunderallfeasiblecircumstances.2CryptographicalgorithmssuchasAESandDESareusedtoencryptamessageinamannersuchthatonlythepersonhavingtheprivatekeyiscapableofdecryptingthemessage.3Inthiscontext,theword“fault”impliesalltypeofsuboptimal,non-functionalbehavior.

CHAPTERFOUR

AdvancesinWebApplicationTesting,2010–2014SreedeviSampath*;SaraSprenkle†*UniversityofMaryland,BaltimoreCounty,Baltimore,MD,USA†WashingtonandLeeUniversity,Lexington,VA,USA

AbstractAswebapplicationsincreaseinpopularity,complexity,andsize,approachesandtoolstoautomatetestingthecorrectnessofwebapplicationsmustcontinuallyevolve.Inthischapter,weprovideabroadbackgroundonwebapplicationsandthechallengesintestingthesedistributed,dynamicapplicationsmadeupofheterogeneouscomponents.Wethenfocusontherecentadvancesinwebapplicationtestingthatwerepublishedbetween2010and2014,includingworkontest-casegeneration,oracles,testingevaluation,andregressiontesting.Throughthistargetedsurvey,weidentifytrendsinwebapplicationtestingandopenproblemsthatstillneedtobeaddressed.

Keywordswebapplications;Softwaretesting;Webtesting;Testcasegeneration;Oracles;Testeffectiveness;Regressiontesting

1IntroductionWhenyoudojustaboutanythingonthewebthroughawebbrowser,youarelikelyinteractingwithawebapplication.Webapplicationsareapplicationsaccessiblethroughthewebthatdynamicallygeneratewebpages,oftenbasedonuserinteractions,theapplication’sdata,orotherinformation(eg,currenttimeandtheuser’slocation).Webapplicationsareoneofthemostcommonwaysthatpeopleusetointeractwithotherpeople(eg,Wordpress,Facebook,Twitter)orbusinesses(eg,bankaccounts,travel,shopping).Webapplicationsareidealforsuchinteractionsbecausetheyareavailable24hoursadaytoanyonewithinternetaccessandawebbrowser.Maintainingwebapplicationsissimplerforbothbusinessesandclients:sincethewebapplicationcoderesidesonthewebapplicationserver,changestotheapplicationcanbeupdatedinonelocationandallusersseethechanges,withoutneedingspecialsoftwareinstalledoneachclient’scomputer.

Tomaintainthehighreliabilityrequiredofwebapplications,wemustdevelopeffectivetestingstrategiestoidentifyproblemsinwebapplications.Whiletherehavebeenadvancesinwebapplicationtesting,therearestillmanyopenproblemsinthisnontraditionaldomain[1].However,thedynamic,distributednatureofwebapplicationsmakestestingdifficult.

Whileprevioussurveypapersfocusedonbroadertimeperiods[2,3]oronspecificsubfields[4–7],wewillfocusonwebapplicationtestingapproachesforcorrectnesspublishedbetween2010and2014.Withthenumberofpublicationsincreasingandresearchers’abilitiestofocusoneachpublicationdecreasing[8],suchfocusedsurveysareincreasinglyimportant.

Inthischapter,wedescribewebapplicationarchitecture,technologies,andcharacteristicsinSection2.Section3presentsthechallenges,commonresearchquestions,andapproachestotestingwebapplications.InSection4,wepresentthestateoftheartinwebapplicationtesting,includingadistantreadingofthepaperswecovered.WeconcludeinSection5withtheopenquestionsinwebapplicationtesting.

2WebApplicationsWebapplicationsareanexampleofadistributedsystem—specifically,aclient/serverarchitecture,wheretheclientsarewebbrowsersandtheserversarethewebapplicationservers.Fig.1showsthesimplest,three-tieredversionofthewebapplicationarchitecture.Thewebapplicationservercouldbeimplementedasmultiple,load-balancedservershandlingrequestsfrommanyclients.Similarly,thedatastoretiercouldalsobeimplementedonmultiplemachines,thuslendingtoann-tierarchitecture.Theapplicationdatastorecouldbemaintainedindatabases,thefilesystem,andexternalservices.

FIGURE1 Webapplicationarchitecture.

ThebrowsersandserverscommunicateviatheHTTPprotocol[9],astatelessprotocol,meaningthateachrequestisindependentofotherrequests.Humanusersmakerequestsusingaclientbrowser,eg,GoogleChrome,MozillaFirefox,Microsoft’sInternetExplorer,Apple’sSafari,andOperatotheserver,eg,Apache[10],ApacheTomcat[11],IBM’sWebSphere[12],andGoogleAppEngine[13].

AsimplifiedHTTPrequestisshowninFig.2.Arequesthasarequesttype,typicallyeitherGETorPOST,aresource(the“R”inURL),andoptionalparametername/valuepairs.Theparametersarethedatainputstothewebapplication.Arequestmayalsoincludecookies[14],whichcontaindatathatispassedbetweenthebrowserandtheservertomaintainstateforthesession.

FIGURE2 Examplerequesttoawebapplication.

Thewebapplicationserverprocessestherequest,servinguptherequestedresource,basedontheinputs.Theserver’sresponse—typicallyanHTML[15]document—isrenderedbythebrowserfortheuser.

Awebapplicationistypicallyimplementedusingavarietyofprogramminglanguages.HTMListhestandardmarkuplanguageusedtocreatewebpages.TheHTMLdocumentoftenreferencesCascadingStyleSheets(CSS)[16]thatdefinethepresentation,style,and

layoutofthedocument.SomeHTMLdocumentsalsoincludescripting—mostcommonlyJavaScript—toallowdynamicuserinteractionwiththewebpage.Ajax—asynchronousJavaScriptandXML[17]—isasetoftechnologiesthatallowdeveloperstoupdatepartsofthewebpagethroughcommunicationwiththewebapplicationserverwithoutupdatingthewholepage.TheresultofusingAjaxisthattheuser’sexperienceismorelikeusingadesktopapplication.JavaScriptlibraries(eg,jQuery1)andframeworks(eg,AngularJS2,Bootstrap3)havebeendevelopedto(1)improvetheresponsivenessofwebsitesandapplicationsforthevarietyofdevicesonwhichwebsitesareviewed,(2)providecross-browsercompatibility,and(3)allowfasterdevelopmentofdynamicuserexperiencesontheweb.

Manydifferentprogramminglanguagescanbeusedtoimplementtheserversideofawebapplicationtogeneratewebresponses.AccordingtoasurveybyWebTechnologySurveys[18],PHP,ASP.NET,andJavaarethemostcommonserver-sideprogramminglanguagesforthetop10millionsiteswheretheprogramminglanguageisknown.Whiletheapplicationserveruseddependsontheprogramminglanguageusedtoimplementthewebapplication,theclientbrowserisindependentoftheserver’sprogramminglanguagechoice.MorerecentdevelopmentinwebapplicationsaretousewebservicessuchasRESTfulAPIs.

Webapplications’heterogenousenvironmentintermsoflanguages,architectures,components,andplatformsgivesrisetoanumberoftestingchallenges,whichwediscussindetailinthenextsection.

3ChallengestoWebApplicationTestingTestersfaceseveralchallengestotestingwebapplications:

•Afastdevelopmentcycle:Whenfaultsarefoundornewfeaturesareimplemented,softwareupdatesneedtobeappliedquicklywithlittledowntime.

•Distributedarchitecture:Faultscanoccurintheserver-(includingthedatatier)orclient-sidecomponentsorintheintegrationofthesecomponents.Thus,itcanbedifficultfordeveloperstodeterminethecauseofafailure.

•Multiplelanguages:Sincewebapplicationsareoftenwritteninmultipleprogramminglanguages,developersmustemploydifferenttechniquestoperformprogramanalysisoftheirapplication’scomponentsbasedonthelanguageusedtoimplementthecomponent.Furthermore,HTMLandCSSareamarkupandstylesheetlanguage,respectively,thatrequiredifferentvalidationtechniques.

•Multipleoutputs:TheresultsofexecutingwebapplicationcodecanbeseeninthebrowserintheformofthepresentationofthegeneratedHTMLdocumentorchangestothedocument,intheapplicationdatastore,inemailmessages,etc.Alloftheseoutputsmustbeverifiedforcorrectness.

•Dynamicbehavior:Sometypesofwebapplicationshavecodegeneratedonthefly,ie,dynamiccodegeneration.Duetothis,purelystaticanalysistechniquesmaynotfullybeabletotestthewebapplication.

•Cross-browser,cross-platformcompatibility:Sinceuserscanuseavarietyofbrowserstoaccesswebapplicationsonavarietyofplatforms,theyexpectasimilarexperience.However,despitestandardsandspecifications,browsersmayimplementfunctionalityslightlydifferently.

•Large,complexcodebase:Thedistributedarchitectureofwebapplicationsnecessitatesaratherlargecodebasethatisoftendifficultforprogrammerstounderstand.AsWebtechnologyevolves,thecodebecomesmorecomplexbyusingmultiplecodinganddevelopmentframeworks,makingasynchronouscalls,creatingrich,dynamicuserexperiences,etc.

Tomeetthesechallenges,practitionersandresearchershavedevelopedavarietyoftestingtechniques.Fig.3depictsabroadoverviewofthewebapplicationtestingprocess.Thetestingprocesscouldbeviewedassimilartothetestingprocessofothertypesofapplications,however,theuniquenessofwebapplicationtestingcomesfromthedefinitions,models,andformatsthatneedtobedevelopedfortheindividualartifacts,suchasforatestcase’sinput,expected/actualoutput,andoraclecomparator.

FIGURE3 Testingarchitecture.

Atestcaseismadeupofinputtothewebapplicationandtheexpectedoutputfromthewebapplication.Sometimes,stateinformationisalsoincludedinatestcasebecausewebapplicationtestcases,inparticular,maynotbeindependentofeachotherandtheunderlyingsession/databasestate.Theinputsandexpectedoutputsdependonthepartofthewebapplicationbeingtested.Forexample,ifserver-sidecodeisbeingtested,theinputislikelyanHTTPrequestandtheoutputislikelytheHTTPresponse,typicallyanHTMLdocument,aswellasotherrelevantoutputs.

Academicandindustryresearchersandcommercialtoolshaveproposedandevaluateddifferentformsoftheseartifacts.Forexample,thepopularwebtestingtoolSeleniumIDE[19]usesatestcaseformatofstepsthatauserperformsonawebsite,storedintabularform,writteninadomainspecificlanguagecalledSelenese.TheSeleniumtestcasealsocontainstheexpectedoutputandtheoraclesareintheformofassertions.Researchersinthedomainofcapture-replaytestingcalleduser-session-basedtestinghavedefinedatestcaseasasequenceofuseractionsstoredastextfileswithHTTPrequestsorusinganXMLnotation.

Toaddressthisopenareaofwebapplicationtestcasegeneration,researchershaveaskedthefollowingresearchquestion:

ResearchQuestion1:Howshouldtestersdefine,model,andgeneratetestcases(inputsandexpectedoutputs),suchthat,whenexecuted,thetestcaseswillcoveralargeportionoftheunderlyingapplicationcodeandwillexposefaultsinwebapplications?

InSection4.1,wereviewtheadvancesinresearchfortestcasegenerationforwebapplicationtestingthatbroadlyanswertheabovequestion.

Researchershavealsodefinedseveraloraclecomparatorsbasedondifferentformatsofexpectedoutput,suchastheentireHTMLpage,oronlythestructureoftheHTMLpageintheformofasequenceofHTMLtags,etc.Assertionshavealsobeenusedasoraclesintheliterature.Developingoraclesisadifficultproblem,especiallythesubareasofautomatingthefaultdetectionprocessanddefiningoraclecomparatorsthatareeffectiveatidentifyingfaultyexecutions.Tothisend,researchershavestudiedthefollowingresearch

question:

ResearchQuestion2:Givenatestcase,howcanatesterdetermineautomaticallyifanapplicationfails?

InSection4.2,wereviewadvancesinthedevelopmentoftestoracles.

Thenotionofwhentostoptestingisonethatisoftendiscussedinthesoftwaretestingindustry.Adequacycriteriaareusedtodeterminewhentostoptesting,aswellastoevaluatethethoroughnessofthetestingconductedsofar.Thisisanareaofresearchthatisstillgrowingforwebapplications,withmostresearchersusingthetraditionaladequacycriteriaofstatement,branch,andconditioncoverage.Researchersarebeginningtoaskthequestion:

ResearchQuestion3:Whattechniquesandcriteriacanbedevelopedandevaluatedtodeterminethoroughnessofawebapplicationtestsuite?

Toaddressthisquestion,inourliteraturesearch,wefoundthatresearchhasfocusedondevelopingadequacycriteria(Section4.3.1),operatorsandtechniquesformutationtesting(Section4.3.2),andfaultseverityclassification(Section4.3.3).

Anotheraspectofwebapplicationtestingreferstothetestingandmaintenanceofapplicationsastheapplicationevolvesandnewversionsofthewebapplicationarecreated.Here,additionalchallengesarise,suchascreatingnewtestcasesfortestingthenewandchangedpartsoftheapplication,repairing,andreusingtestcasesfrompreviousversionsofthesystem,aswellas,managingthesizeofaregressiontestsuitetomaintainhigheffectiveness.Inthissubdomain,researchershaveaddressedthequestion:

ResearchQuestion4:Howcanatestercreatetestcasesforchanged/newpartsofthecodeaswellasmaintainanexistinglargeregressiontestsuite?

WeelaborateonseveraldifferentwaysinwhichthisquestionisaddressedintheliteratureinSection4.4.

4WebApplicationTesting,2010–2014Todevelopthissurveyofrecentadvancesinwebapplicationtesting,wesearchedtheACMandIEEEdigitallibrariesforpaperspublishedbetween2010and2014.Weusedthefollowingsearchstringwhensearchingthelibraries

(((((((web)ORwebsite)OR“webapplication”)ORajax)ORjavascript)ORphp)ANDtest)

Fromthatstartingpoint,welookedatpapersreferencedorcitedbythepaperwefoundandlookedattheauthors’websitesforadditionalpapers.

Tokeepthesurvey’sscopemoremanageable,wedidnotincludepapersthatwerefocusedonstaticanalysis,programrepair(eg,[20,21]),andnonfunctionalpropertieslikesecurity,accessibility,performance,andcross-browsertesting.Inaddition,sinceAliMesbahrecentlypublishedasurveyofadvancesoftestingJavaScript-basedapplications[6],wedonotcoverJavaScripttestinginthiswork.

Inthissection,wepresenttheadvancesinwebapplicationtestingpublishedbetween2010and2014.Weendwithatextualanalysisofthepaperstoidentifytrendsinthearea.

4.1TestCaseGenerationTestcasegenerationistheproblemofcreatingtestcasesthatareeffectiveatcoveringunderlyingcodeanddetectingfaults.Severalapproachesareproposedfortestcasegeneration,suchasmodel-based,search-based,concolic,etc.Weelaborateoneachoftheseapproachestotestcasegenerationinthefollowingsubsections.

4.1.1Model-BasedTestCaseGenerationModel-basedtestcasegeneration,asitsnamesuggests,hasfocusedondevelopingamodeloftheapplicationandthentraversingthemodeltocreatetestsequences,andeventually,executabletestcases.Modelscanbedevelopedinseveralways,eg,bycapturingnavigationbehavioroftheapplicationorbystaticanalysis.

Tungetal.[22]buildapagenavigationgraphandgeneratetestpathsfromthegraph.Then,theyextractdataandcontroldependenciesfromthesourcecodetoassistwithtestcasegeneration.Datadependenceisobservedwhenonepagetransfersdatatoanotherpage.Whenonepagecanbereachedonlyafterfollowinglinks(linkdependence)orasaresultofaconditionalstatementinanotherpage,controldependenceisobserved.Theyperformmaximalgraphwalktogeneratetestpathswhichareaugmentedwithdependencies.Theyimplementtheirapproachinaprototypetoolandevaluatewithasmallcasestudy.

OffuttandWu[23]proposeanewapproachtomodelingwebapplicationsthatmodelstheatomicsectionsofawebapplication,specificallytheuserinterfacescreens.TheauthorsarguethattheirAtomicSectionModel(ASM)isimplementationtechnologyindependentwhilealsoallowinganalysesthatenablebettertestingbystatically

representingdynamicpages.TheASMismadeupoftwocomponents:(1)the“unit-level”model,thecomponentinteractionmodel(CIM)and(2)the“system-level”model,theapplicationtransitiongraph(ATG).ThummalaandOffutt[24]implementedtheASMinatoolcalledWASP(WebAtomicSectionProject)andevaluatedtheeffectivenessofthemodel.

Chenetal.[25]proposemodelingauser’snavigationofawebapplication,specificallythebrowserinteractionsandso-calledAdvancedNavigations,ie,requestednavigationswhoseresponsesdependontheuser’sorapplication’sstateorhistory.Theauthorsmodelthepagenavigationswithanaugmentedfinitestatemachine(FSM).Togeneratetestcases,theauthorssuggesttraversingthemodelwithmodificationstohandlecyclesinthenavigationsandgeneratingfinitesequences.

Torsel[26]modelthewebapplicationasadirectedgraph.Inaddition,theyproposetocapturevariablesofbasictypes,suchasString,intwoscopes,permanentandsession,andalsoavariabletypetoholddatafromexternalsources,likeadatabase.Fortestcasegeneration,theyconductabreadth-firstsearchtoexplorethedirectedgraphtoidentifylogicaldependenciesbetweennavigationpathsandbuildadependencygraph.Fromthedependencygraph,pathsareselectedforexecutionastestcases.Theyalsoprovidesomeannotationintheirmodelthatcanserveastestoracles.

Songetal.[27]proposetheuseofafinitestateautomatontomodeltheserver-sideinteractionsanduserinteractionsinawebapplication.Theyusethenotionofsynchronousproductofclientandserver-sideFSMmodelstobuildthemodelincrementally.Depth-firsttraversaloftheFSMisusedtogeneratetestcases.

Enderlinetal.[28]extendcontract-basedtestingusinggrammarstogeneratetestcases,testdata,andoraclesfortestingPHPapplications.Inpriorwork[29],theauthorsdevelopedthenotionofcontractswithrealisticdomains,whichareusedtorepresentallkindsofdataandtoassigndomainstotestdatafortestingPHPapplications.Inpriorwork,theydevelopedtheregularexpressiondomaintodescribesimpletextualdataandinthiswork,theydevelopthegrammardomaintodescribecomplextextualdata.Togeneratetestcases,theyfirstcomputetestdatafromcontractsusingthreealgorithmsforthegrammardomain(a)uniformrandom,(b)boundedexhaustive,and(c)rulecoveragebased,thentheyrunthetestcases,anduseruntimeassertioncheckingofthecontractasoracles.

WEBMATEisatoolbuiltontopofSelenium[19]byDallmeieretal.[30].Thetoolconsiderstheserversideofthewebapplicationasablackbox,focusingonthebrowser(HTML/CSS/JavaScript)interface.Byexploringtheapplication’sinterface,WEBMATEcreatesausagemodel,essentiallyafinitestateautomaton.Toexploreforms,thetoolappliesheuristicstofindvaluesforinputfields.Theauthorsperformedanexperimentalstudytoshowhowthetoolimprovescoverageoveratraditionalwebcrawlerandpresentcross-browsercompatibilitytestingasanapplicationofthetool.Thetoolwasanacademicprototypebuthasbeenfurtherdevelopedintoacommerciallyavailabletool[31,32].

Schuretal.[33,34]presentanapproachimplementedinatool,ProCrawl,tocrawltheuserinterfaceofawebapplicationandobservebehavioroftheapplicationtocreatea

behaviormodel,andthengenerateandexecutetestcasestocovertheunobservedbehavior.Thebehaviormodelisafinitestateautomaton(FSA)wherethenodesrepresentstatesandthetransitionsrepresentactionsthatuserperformedtochangethestate.Theirtoolcanhandlemultiplesimultaneoususersaccessingthewebsystem.Byusingagraphwalkalgorithmforpathgeneration,theygeneratetestcasesasSeleniumscriptstotestthewebsystem.

4.1.2StatisticalTestCaseGenerationAnotherapproachtogeneratingtestcasesistocreatestatisticalmodelsofuseraccessesthatrepresentactualuserbehavioratalowercostthandirectlyusingtheusersessionsastestcases[35]andwithoutlimitingtestcasestowhatusersactuallydid.Themostrecentworkinthisareaexplores(1)configuringmodelsofusers’navigationandtheresultingtestcasesfromthesemodels[36,37]and(2)creatingprivilegedrole-specificmodels[38].

Tobetterunderstandhowtomodelusers’behaviorand,thus,developtestcasesthatrepresentusage,Sprenkleetal.[36,37]proposedvariousconfigurationsforrepresentinguserbehaviorandempiricallyexaminingthetradeoffsandimplicationsofthosechoicesontheresultingtestcases.Sprenkleetal.buildonSantetal.’s[39]workoncreatingstatisticalmodelsofusersessionstogeneratetestcases,focusingonthenavigationmodel.Theauthorsfoundthatarelativelysmallnumberofuseraccessesareneededtocreatenavigationmodelsthatgeneratehighlyrepresentativeusernavigationbehaviorintestcases,whichisimportantbecauseusersdonotaccessawebapplicationrandomly.Theauthorssuggestrepresentinguserrequestsbytheresourceandparameternamestobalancescalabilityandrepresentationrequirements.Furthermore,theauthorssuggestthatatestercantunetheamountofhistoryusedtopredictauser’snavigationalnextstepdependingonthetester’sgoals:increasingtheamountofhistoryresultsinmorecloselymodelingtheuser’snavigationalbehavioratthecostoflargerspaceandtimeandofyieldingfewertestcasesthatrepresentwhatauserislikelytodobutnotobservedintheuseraccesslogs.Thejournalversionoftheirwork[37]extendedthenumberofexperiments,analyses,anduseraccesslogsusedintheoriginalpaper[36].

Webapplicationsoftenhavedifferenttypesofusers—someofwhichhaveprivilegedaccesstospecializedfunctionality.Sprenkleetal.[38]proposedrefiningtheirstatisticalmodelsbypartitioningtheusersessionsbytheiraccessprivilegeandfeedingthepartitionedusersessionsintotheirmodelandtestcasegenerators.Theirempiricalstudyshowedthattheresultingtestcasesareoftensmallerthantheaggregatemodelsandrepresentthespecificusertypewhilealsoallowingnew,likelynavigations.Sincepartitioningusersessionsbytheiruserprivilegemaynotbenefitallapplications,theauthorssuggestanalyzingtheoverlapbetweenthemodelsintermsofthecommonnodesandtherelativefrequencythatthecommonnodesareaccessedintheuseraccessestodetermineifthecommonaccessissufficientlylowtowarrantgeneratinguser-type-specifictestcases.

4.1.3ConcolicTestCaseGeneration

Artzietal.[40]expandontheirpreviouswork[41]andproposegeneratingtestsuitesusingacombinationofstaticanddynamicanalysis.Specifically,theauthorsproposecombiningconcreteandsymbolic(concolic)andconstraintsolvingtoautomaticallyanddynamicallydiscoverinputvaluesfortestcases.Theauthorsfocusontwotypesofwebapplicationfailures:(1)runtimecrashesorwarningsand(2)invalid/malformedHTMLdocuments.Thus,theirapproachincludesanHTMLvalidatortodetectfailures.Theirtechniqueinvolvesgeneratingacontrol-flowpredicatebasedonagiveninput(perhapsempty),modifyingthepredicatetoyieldadifferentcontrol-flowpath,anddeterminingtheinputthatwillyieldthisnewpath,thuscreatinganewtestcase.Inaddition,theapproachmaintainssharedsessionstate.TheauthorsimplementtheirapproachforPHPapplications—themostcommonserver-sidescriptinglanguage—inatoolcalledApollo.

Statisticalfaultlocalizationisanapproachtofindingthecauseoffaultsincodebyexecutingtestcasesandthendeterminingwhichexecutedcodeelementscorrelatewiththemostfailedtestcases.Alimitationtousingstatisticaltestingisthatalargetestsuitemustbeavailable.Artzietal.[42]addressthislimitationbyproposingconcolictechniquestogeneratetestsuitesdesignedforfaultlocalization.ThetechniquesincludesixvariationsontheTarantula[43,44]algorithm,whichisusedtopredictstatementsthataremostlikelytocausefailuresbasedonfailedtestcases,combinedwiththeauthors’proposedenhanceddomainsandoutputmapping.Theenhanceddomainforconditionalstatementsallowsmoreaccuratelocalizationoferrorscausedbymissingbranches.Theoutputmappingmapsprogramstatementstooutputfragmentstheygenerate,whichcanthenbeusedtohelplocalizethefault.TheauthorsimplementedtheirtechniquesinApollo,atoolforPHPapplications,thatautomaticallyfindsandlocalizesmalformedHTMLerrors[41].

Thesameauthors[45]explorethetradeoffsbetweengeneratedtestsuites’sizeandlocalizationeffectiveness.Theauthorsproposetechniquestodirectgenerationofnewtestcasesthataresimilartofailedtestcasesusingvarioussimilaritycriteria.Theirhypothesisisthatsimilar,failedtestcaseswillbebetterabletolocalizefailure-causingstatements.TheapproachwasimplementedinApollo[41].Theauthorsfoundthatusingpath-constraintsimilaritygeneratedasmallertestsuitesizewiththebestfaultlocalization.

Inaddition,theauthorscombinedandexpandedontheirfaultlocalizationina2012journalpaper[46].BeyondtheirpreviousvariationsonTarantula[42],theauthorsalsoenhancedthefaultlocalizationtechniquesofOchiai[47]andJaccard[48]usingtheenhanceddomainforconditionalstatementsandasourcemapping—therenamedoutputmapping.TheauthorsimplementedthenewtechniquesinApollo[41]andevaluatedthetechniquesinalargeexperimentalstudy.AnenhancedversionofOchiaiandthepath-constraintsimilarity-basedgenerationyieldedthebestresultsforfaultlocalizationeffectiveness.

4.1.4Requirements-BasedTestCaseGenerationRequirements,whetherformalornaturallanguagebased,areoftenusedduringmanualandautomatictestcasegeneration.Inthissubsection,wepresentresearchthatinvestigatesusingrequirementsforautomatictestcasegeneration.

MatosandSousa[49]proposeusingusecasesandformalrequirementstocreateSeleniumfunctionaltestcasesandwebpageswhichareuserinterfaceprototypes.Theinputstotheirtoolareusecases,systemglossaryanduserinterfacespecificationswritinginacontrollednaturallanguagethatthetoolcaninterpret.TheirtoolisimplementedasanEclipseplugin.

Thummalapentaetal.[50]presentanewtechniquetoautomaticallygeneratetestcasesbyfocusingonlyoninterestingbehaviorsasdefinedbybusinessrules,whichareaformoffunctionalspecificationusedintheindustry.Formally,abusinessruleistripleconsistingofanantecedent,aconsequentandasetofinvariantconditions.Intheirtechnique,theyfirstbuildanabstractstate-transitiondiagram(STD),wherethenodesrepresentequivalentstates,astheycrawltheapplication’sGUI.Inthenextstepoftheirtechnique,foreachbusinessrule,theyidentifyabstractpathsrelevanttothebusinessruleandrefinethepathsusingastricternotionofstateequivalenceuntilatraversablepathisfound.Thisfinalsetofpathsarethetestcasesthatareexecutedtotesttheapplication,whichalsocoveralltheinitiallyidentifiedbusinessrules.Assertioncheckingontheconsequentconditionofthebusinessruleservesasanoracleaswell.TheyimplementedtheirapproachinatoolcalledWATEG,WebApplicationTestCaseGenerator.

4.1.5Search-BasedTestCaseGenerationAlshahwanandHarman[51]explorethenotionofusingsearch-basedalgorithmsfortestcasegeneration.Thesearch-basedapproachtotestgenerationtheyproposehastheunderlyinggoalofmaximizingbranchcoverageintheresultanttestsuite.Theystartwithperformingstaticanalysisofthesourcecodeandthenusetheinformationgatheredduringthesearch-basedphase.Forthesearch-basedphase,theyuseanalgorithmderivedfromKorel’sAlternatingVariableMethod[52],intowhichtheyincorporateconstantseedingandtheideaofusingvaluesmineddynamicallyastheapplicationexecutesintothesearchspaceforspecificbranchingstatements.TheyimplementedtheirapproachinatoolcalledSWAT,Search-basedWebApplicationTesterandevaluateditseffectiveness.

Incontrasttootherresearchinthisarea,TappendenandMiller[53]takeadifferentapproachinthattheyleveragecookiesfortesting.Theauthorsarguethatcookiesarethecommonstaterepresentativesforallwebapplicationsandthusshouldbeleveragedintesting.TheyproposeusingthegeneticsearchalgorithmEvolutionaryAdaptiveRandom(EAP)fortestgenerationtocreatetestswherecookiesareeitherpresentorabsentinrequests.Theauthorsalsoproposeseveraloracles,discussedinSection4.2.

4.1.6RegeneratingTestCasesAlshahwanandHarman[54]suggestthatregeneratingtestsuitesfromexistingtestsuites—forexample,byreorderingtestsequences—canincreasethetestingvalueofthetestsuite.Theauthorsproposeusinganovel,value-awaredataflowanalysisontheserver-sidecodetodeterminewhichreorderingswillbemosteffectiveinexercisingtheserver-sidestate.SincetheHTTPprotocolisstateless,leveragingthedataflowofsessionvariables,cookies,andtheapplicationdatastoreintestsuiteregenerationresultsinastate-awaretest

suite.TheauthorsimplementedtheirapproachforPHPapplicationsinatoolcalledSART(StateAwareRegenerationTool).

4.1.7TestInputGenerationWhilenotspecifictowebapplications,theapproachthatShahbazetal.[55]proposetogeneratevalidandinvalidteststringsasinputstostringvalidationmethods—acommontypeofmethodinwebapplications—canbeappliedtowebapplicationtesting.Forexample,theirsubjectstringvalidationmethodsusedintheexperimentsincludemethodsfromwebapplications.Theauthorsproposeusingtextanalysisofidentifiers,websearches,regularexpressionqueries,andmutationtogeneratetheteststrings.Thewebqueryapproach—definedusingtextanalysisofidentifiers—wasfirstproposedbythesameauthorsintheir2012paper[56].

Fujiwaraetal.[57]proposegeneratingtestdatausingaspecificationoftheapplication’sbasicdesignthatiswritteninUMLandObjectConstraintLanguage(OCL).Giventhedatabasetablesstructures,userinterfaceelements,triggerableevents,theapproachfocusesonthebehaviors.Ratherthanmanuallygeneratingtestdata,theauthorsproposesolvingconstraintsusingaSatisfiabilityModuloTheoriessolver,whichcanhandlehighlystructureddatatypes,likestringandtable.Whiletheapproachseemspromising,itrequireswell-trainedpeopletospecifythemodel.

4.2OraclesDeterminingwhetheratestcasepassesorfailsisadifficultproblem,especiallyforwebapplicationsthathaveavarietyofoutputs(eg,webpages,datastores,emailmessages)thataresometimesdynamicallygeneratedornondeterministic.Recentadvanceshavefocusedonthewebpageoutputs—notsimplymalformedHTMLbutmorenuancedfailuresmanifestedinthepages.

Dobloyietal.[58]presenttechniquestoautomaticallycomparetestcaseoutputs,ie,XML/HTMLdocuments,duringregressiontesting.Theirapproach—implementedinthetoolSmart—isbasedonamodelthatexploitssimilaritiesinhowwebapplicationsfail.TheauthorsfirstinspectasmallportionofregressiontestingoutputmanuallyandidentifystructuralandsyntacticfeaturesinthetreestructuresofXMLdocumentsthatindicatedifferencesthathumansshouldinvestigateforfailure.Thefeaturesarethenusedtotrainacomparatortoapplyathresholdtoidentifywhichoutputandtestcasesrequirehumaninspection.Sincetheapproachfocusesonthetree-structuredoutput,theapproachmissesfaultsinvolvingimagesandthepresentationofHTMLelements.

deCastroetal.[59]presentanextensiontoSeleniumRC,calledSeleniumDB,thatallowsfortestingwebapplicationsthatinteractwithdatabases,suchasMySQLandPostGreSQL.Specifically,theirtoolallowsestablishingdatabaseconnectionsandcomparingtestdatawithdatastoredinthedatabase.TheyaugmentedSeleniumRC’scorewithsixnewassertfunctionsthatallowforcomparingdataoutputtedduringtestingwithdatathatexistsinthedatabaseoftheapplication.Forexample,anassertstatementthat

wasaddedtotheSeleniumRCcodechecksforthelastrecordinsertedinthedatabase.

MahajanandHalfonddevelopedtechniquesforfindingHTMLpresentationfailuresusingimagecomparisontechniques[60]combinedwithsearchingtechniquestoidentifythefailure’srootcause[61].MahajanandHalfond’sfirsttakeonfindingHTMLpresentationfailures[60]leveragesimagecomparisontechniques—comparinganimageoftheexpectedwebpageandascreenshotoftheactualwebpage,findingthepixel-leveldifferencesbetweenthetwoimages,andmappingthepixel-leveldifferencestowhichHTMLelementsarelikelytobethecauseofthefailure.MahajanandHalfond’sfollowupwork[60]improvedfindingafailure’srootcauseautomaticallyusingasearch-basedtechnique,wherepossiblefaultyelementsarepermutedtoapossiblycorrectvalueandtheresultingpageiscomparedwiththeexpectedimage.Iftheresultingpagematchestheexpectedimage,thenthepermutedfaultyelementisthelikelyrootcause.Bothpaperscontainanexperimentalevaluationoftheirapproachonwebpagesinwideuseandindicatepromisingresults.

Theauthorssuggestthatfutureworkincludescustomizingthesearchspacebasedontherootcause,handlingmultiplefailureswithinonepage,andhandlingwhenthefaultdoesnotvisuallyappearwithinthefaultyHTMLelement.Theauthorshavetwofollow-uppaperstoappearin2015[62,63],outofthescopeofthispaperandafteroursubmissiondeadline.

TappendenandMiller[53]alsoproposedoraclesthatdonotdependontheircookie-basedtestingmethodology.TheauthorsassertthattheirproposedstructuralDOMsimilaritymetrics(similartoDobolyiandWeimer’s[58]),HTMLcontentsimilaritymetrics,andahybridofthetwometricswascriticaltocompletetheircookie-basedautomatedtesting.

4.3EvaluatingTestingEffectivenessAfterdevelopingatestingtechnique,aresearchermustthenevaluatethetechnique’seffectiveness.

4.3.1TestAdequacyCriteriaTestadequacycriteriahavemultipleuses,suchastoassessthoroughnessofatestsuiteandalsotodeterminewhentostoptesting.

Alafietal.[64]proposenewcriteriaforwebapplicationtestingbasedonpageaccess,serverenvironmentvariableuse,anddatabaseinteractions.Thecriteriaaredefinedascoveringallserverpagesatleastonce,coveringallserverenvironmentvariablesatleastonce,andcoveringallSQLstatements(includingdynamicallyconstructedstatements)atleastonce.Theyfurtherdescribehowtoinstrumentanapplicationtocapturetheinformationrelevantforthesecriteria,suchasservervariables,databaseinteractionsandpageaccessinformationandhowtorecordthecoverage.

AlshahwanandHarman[65]suggestanewadequacycriterion,outputuniqueness,andproposefourdefinitionsforthiscriterionwithvaryinglevelsofstrictness.Thebasicidea

ofthiscriterionistoclassifythedifferentwaysinwhichtheoutputofHTMLwebpagescanbeunique,eitheruniqueinentirecontent,oruniqueintagsstructure,oruniqueincontent,oruniqueintagsandattributes.Then,theyderivenewtestcasesbymutatingeachtestcaseinanoriginaltestsuite,executingthenewmutatedtestcase,anddeterminingifthenewtestcaseproducesauniqueoutputasperoneoftheearlierfourdefinitions.

Sakamotoetal.[66]addresstheintegrationtestingproblemforwebapplications.Theircontributionsincludeproposinganewcoveragecriterion,templatevariablecoveragecriterion,andpresentingatechniquetogenerateskeletontestcodetoimprovethetemplatevariablecoveragecriterion.Templatesrefertoatemplatingsystemusedontheclientorserver-sidethathelpsthedevelopmentofreusableHTMLelements4.TemplateenginesreplacethesetemplatevariableswithactualvaluestocreateHTMLpages.ThetemplatevariablecoveragecriterionmeasurescoverageofvariablesandexpressionsthatareembeddedinHTMLtemplates.TheirworkisimplementedinatoolcalledPOGen.Thetoolconsistsofthefollowingcomponents:HTMLtemplateanalyzer,HTMLtemplatetransformer,andatestcodegenerator.

4.3.2MutationTestingMutationtestingisanapproachtoevaluatethefault-revealingeffectivenessoftestingtechniques.Inmutationtesting,codeismutatedusingmutationoperatorstocreateafaultyversionofthecode;themutationprocesshappensmultipletimestocreatemultiplefaultyversionsofthecode.Ifthetestingtechniquerevealsthefault,thetechniqueissaidto“kill”themutant.

PraphamontripongandOffutt[67]developedweb-specificmutationoperatorsandimplementedtheseoperatorsinatoolcalledwebMuJavathatisbuiltonmuJava[68],aframeworkforautomaticallygeneratingmutantsinJavaprograms.Themutationoperatorsfocusonhowwebapplicationcomponentscommunicate,eg,throughURLsandforms.WhilemostoftheoperatorsareappliedtoHTML—whichisgeneraltoallapplications,threearespecifictoJSPbutcouldbetranslatedtosimilarfunctionalityinotherserver-sideprogramminglanguages.Theauthorsevaluatedtheirmutationoperatorsandtoolononesmallapplicationthatshowedpromisingresultsbutalsotheneedforamutationoperatorthatmodifiesavariable’sscope,eg,frompagetosessionorapplicationscope.Withthismutationtool,researcherscanevaluatetheabilityoftheirtestingapproachestodetectfaultsinwebapplications.

4.3.3FaultSeverityDobolyiandWeimer[69]proposeamodelofconsumer-perceivedfaultseverity,whichtheyargueisthemostappropriatewaytomodelfaultseveritybecausewebapplicationsareusercentric.Givenauser-centricfaultseveritymodel,developerscanprioritizethemostseverefaults.Theauthorsproposetwomodels—oneofwhichisfullyautomated—basedonfeaturesinHTMLdocumentsthatlikelyindicateafailure.Bothmodelsoutperformedhumansindeterminingconsumer-perceivedseverity.Basedonafault

severitystudyof17subjectapplications,theauthorsconcludethattraditionalfaultseeding,whichisoftenusedtoevaluatetheeffectivenessoftestingtechniques,doesnotgenerateuniformlyseverefaultsintermsofconsumerperceptionandisthereforenotnecessarilyagoodtechniqueforevaluatingtestingtechniques.

4.4RegressionTestingRegressiontestingtakesplaceduringthemaintenancephaseofthesoftwaredevelopmentlifecycle.Commonchallengesthatareencounteredduringregressiontestingare(a)managingthesizeandqualityofanevolvingtestsuiteand(b)creatingnewtestcasesorreusingoldtestcasesfortestingnew/changedparts.Toaddressthefirstchallenge,researchershaveproposedtestcaseselectionstrategies,suchasreductionandprioritization.Forthelatterchallenge,identifyingthechangedpartsofthecodefortestcasegenerationandtestcaserepairofoldertestcasesareoftenthestrategiesthatareused.Thissectiondescribestheadvancesinregressiontestingforthedomainofwebapplicationtesting.

4.4.1PrioritizationApproachesBryceetal.[70]developauniformrepresentationtomodelbothGUIandwebsoftware(bothareevent-drivensoftwarethathavetraditionallybeenstudiedindependently)anddevelopasetofprioritizationcriteriabasedonthisnewmodelthatcanbeusedtotestbothGUIandwebsoftware.First,theydevelopauniformrepresentationforthebuildingblocksoftestingevent-drivensoftware,specifically,unifiedterminologyforawindow,anaction,aparameter,avalueandatestcase.Atestcaseismodeledasasequenceofwindowsandassociatedparametervalues.Then,theypresenteightprioritizationcriteria,thatarebasedontestcaseproperties,suchasthenumberoftwo-wayinteractionsbetweenparametersthatarecoveredinatestcase(Interaction-basedcriteria),thelengthofthetestcase(Count-basedcriteria)asmeasuredbytotalnumberofactions,totalnumberofparameter-values,uniquenumberofactionsinatestcase,andthenumberoffrequentlyoccurringsequencesofactions(measuredinthreedifferentways)inatestcase(Frequency-basedcriteria).TheyconductanempiricalstudywithfourGUIandthreewebapplicationsandapplytheproposedprioritizationcriteria.Theyfindthattwocriteriaarealwaysamongthetopthreeineffectivenessforallthesubjectapplications.Oneofthetwocriteriaorderstestcasesindecreasingorderofnumberoftwo-wayinteractionsbetweenparameters,andtheothercriterionordersindecreasingorderoflengthintermsofnumberofparametervalues.Thesetwocriteriaarerecommendedbytheauthorsaseffectiveprioritizationcriteriafortestingevent-drivensoftware.Theauthorsalsoexploretheeffectivenessofhybridprioritizationcriteria,wheretheyprioritizetestcasesbasedonafrequencycriteriauntil10%to20%ofthetestsuiteiscoveredwithoutanincreaseinAPFD[71]andthenswitchtooneoftheinteraction-basedcriteria.Intheirexperimentalstudy,theyfindthathybridcriteriaimproveeffectivenesswhencomparedtotheindividualcriteriausedincreatingthehybrid.

Sampathetal.[72]presentatool,CPUT,toprioritizeandreduceuser-session-based

testcasesofwebapplications.Thetool,developedinJava,consistsoftwomaincomponents.ThefirstcomponentisaloggingmodulethattheydevelopedforApachewebserverthatcollectsallpertinentusageinformationthatisneededtocreateuser-session-basedtestcases.ThesecondcomponentisthemainCPUTtoolthatallowsimportofApachewebserverlogs,whichcanthenbeconvertedintouser-session-basedtestcasesinXMLformat.CPUTstorestheimportedlogfileandthetestcasesinaPostgreSQLdatabase.CPUTallowsforimportingnewlogfiles,appendingtoanexistinglogfilethathaspreviouslybeenimported,andanoverwritepreviouslyimportedlogfilecapability.Thetestcasescanthenbeprioritizedorreducedbyseveralexperimentallyverifiedcriteria,suchascombinatorial,length-based,andfrequency-basedcriteria[70].CPUTalsodisplayssomestatisticsabouteachuser-session-basedtestcase,suchasthenumberofrequestsinthetestcase,thenumberofparametervalues.Thetoolwritestheprioritized/reducedtestsuitetoatextfilewhichcanbeusedbytesterstoidentifytestscasestoexecuteduringtheirmaintenancetestingcycles.

Gargetal.[73]presentatwo-levelapproachtoprioritizewebapplicationtestcases.Theybuildafunctionaldependencegraph(FDG),generatedfromaUMLdiagramoftheapplication,tomodelfunctionaldependenciesbetweenmodulesinawebapplication.Theauthorsalsocreateaninterproceduralcontrolgraph(ICG)fromthesourcecodeforeachfunctionalmoduleintheFDG.Thetestsuitefortheapplicationispartitionedintodifferenttestsetsthatcanbetiedtoafunctionalmodule/nodeintheFDG.TestcaseswithinatestsetaretiedtosubmodulesmodeledintheICG.Thetwo-levelapproachtoprioritizationproposedinthispaperisbasedoncriteriathatrelyonfirstassigningprioritiestomodulesintheFDGandthentosubmodulesintheICG.ModulesintheFDGareprioritizedbasedonnewandmodifiedfunctionalitiesthatareaddedormodifiedintheFDG,eg,anewlyintroducednodeintheFDGrepresentsnewlyaddedfunctionalityandthusgetshighestpriority,modifiedFDGnodesgetnexthigherpriority,etc.ModuleswithintheICGareprioritizedbasedonthedegreeofmodificationofthenodeintheICGwhichisdeterminedbythechangeinnumberoflinesofcode,eg,anewICGnodegetsthehighestpriority,remainingnodesareassignedprioritiesindecreasingorderofdegreeofmodification.Finally,testcasesareprioritizedusingcriteriainincreasinganddecreasingorderof(a)distanceofmodifiednodesfromtherootFDGnode,(b)numberoffunctionalmodulesexecuted,and(c)numberofchanges(asidentifiedinICGnodes)executed.AsmallexperimentalstudyrevealsthatprioritizingtestsbasedontheshortestpathofmodifiednodesfromtherootFDGnodehasthehighestAPFD[71]inthefirst10%ofthetestsuite.Inanotherwork[74],thesameauthorsproposedistributingthetestsetsonmultiplemachinesandexecutingtheminparalleltoreducetestexecutiontime.Eachmachineisapproximatelyallocatedanequalnumberoffunctionalmodulesandalsoequalprioritiesoffunctionalmodules.

Gargetal.[75]alsoproposedanewautomatedtestcaseprioritizationtechniquethatautomaticallyidentifieschangesinthedatabaseandprioritizestestcasessuchthatdatabasefaultsmaybedetectedearly.Theyuseafunctionaldependencegraphandaschemadiagramwhichmodelstherelationshipbetweenthedatabasetablesandfieldsintheirprioritizationapproach.Usinglogfilesthatcapturedetailsofthedatabase,they

identifytheFDGmodulesthataremodifiedasaresultofdatabasechanges(ascapturedinthelogfiles)andassignprioritiestothemodules,eg,assigninghigherprioritytoFDGmoduleswhosemodificationsareduetonewtablesintheschemadiagram.Asmallexperimentalevaluationshowedthattheirapproachisabletodetect70%oftheseededdatabasefaultsbyexecuting10%ofthetestsuite.

Sampathetal.[76]proposeorderingreducedsuitestofurtherincreasetheeffectivenessoftestsuitereductionstrategies.Testsuitereductionisaregressiontestingstrategytoreducethesizeofthetestsuitethatisexecutedbyusingseveralcriteriathatwillallowtheselectionofasmallersetoftestcasesthatarecomparableineffectivenesstotheentireoriginalsuite.Testcaseprioritization,ontheotherhand,strivestokeepallthetestcasesintheoriginalsuite,butproposesthattheybeorderedbasedonsomecriteriasuchthattheorderedtestsuitecanfindfaultsearlyinthetestexecutioncycle.Intheirwork,Sampathetal.[76]firstreducetestsuitesusingreductioncriteria[77]thatselectasmallertestsetbasedoncharacteristicsliketheactualbaserequestscoveredinatestcase,theactualparameternamesthatarecoveredinatestcase,etc.,withthegoalofcreatingareducedsetoftestcasesthatcoverallbaserequestsofthewebapplication,andallparameternamesinthewebapplication,respectively.They,thenprioritizethereducedsuitesbythenapplyingprioritizationcriteria(specifically,count-,interaction-,andfrequency-basedcriteria)thatareshowntobeeffectiveinrelatedwork[70].Thisapproachledtothecreationof40criteriathatorderreducedsuiteswhichareempiricallyevaluatedbySampathetal.usingthreewebapplications,seededfaultsanduser-session-basedtestcases.Anothercontributioninthisworkisthedevelopmentofanewmetricthatcanbeusedtocomparetestsuitesofunequallengths.Thecommonmetricsusedtoevaluatetheeffectivenessofprioritizedtestsuites,APFD[71]andAPFD_C[78]requirethatthecomparedtestsuitesbeofthesamesize.However,thereducedsuitescomparedinSampathetal.’sworkwereofvaryingsizesandthuscouldnotbecomparedusingthetraditionaleffectivenessmetrics.Therefore,theydevelopedaneweffectivenessmetricMod_APFD_Cthatallowstheeffectivenessevaluationofprioritizationeffectivenessoftestsuitesofunequallengths.Thenewmetrictakesintoaccountthenumberofuniquefaultsdetected,thetimetogeneratetheorderedreducedsuite,andthetimetoexecutetheorderedreducedsuite.Throughtheirempiricalstudy,theyfindthatinseveralcases,theorderedreducedsuitesaremoreeffectivethanapurereducedsuiteandapureprioritizedsuite,thuslendingevidencetothecreationofapromisingnewapproachtoregressiontesting.

Dobuneh[79]etal.proposeandevaluateahybridprioritizationcriterionfortestingwebapplications.TheirhybridcriterionfirstorderstestcasesbynumberofcommonHTTPrequestsinthetestcase,then,orderstestcasesbasedonthelengthofHTTPrequestchains,andfinally,bythedependencyofHTTPrequests.Intheirexperimentalstudywithonesubjectapplication,theyfindthatthehybridcriterionfindsalltheseededfaultssoonerthanthefirstandsecondcriteriausedinthehybrid,butiscomparabletothethirdcriterionondependencyofHTTPrequests.Similarresultsareobservedwhenevaluatingthetimetakenbythehybridandtheindividualcriteriatogeneratetheprioritizedtestsuites.

4.4.2TestSuiteReductionApproachesTestsuitereductionisaregressiontestingstrategywherethegoalistocreateasmallertestsuitethatisaseffectiveastheoriginaltestsuite.

HuangandLu[80]proposeatestsuitereductionapproachthatusesusersessionsandserviceprofilesoftheapplicationthatdescribefunctionsoftheapplication,andapplyconceptanalysis[81],amathematicalclusteringtechnique,toperformthereduction.TheserviceprofileisessentiallyasequenceofURLsthatisderivedfromeitherfunctionalspecificationsorbyreverseengineeringfromthesourcecode.Inaserviceprofile,datadependencebetweenURLsandcontroldependencebetweenURLsiscaptured.Amatrixthatmapsusersessionstotheserviceprofilestheycoveriscreatedandfedasinputtoconceptanalysiswhichthenclusterstheusersessionssuchthatalltheusersessionsinaclustercoverthesetofserviceprofilesinthecluster.Theheuristictoselectusersessionsfromtheclusterscreatedbyconceptanalysisselectsusersessionsthatcoverallserviceprofilesoftheapplication.Theauthorsalsoillustratehowconceptanalysismaybeusedinthecaseofevolvingservices,whereservicesareadded/modifiedordropped.AsmallexperimentalstudywithaPetShopwebapplicationispresentedandtheirapproachisevaluated.

Liuetal.[82]alsoproposeaclusteringapproachtoreducethesizeofauser-session-basedtestset.Clusteringrequiresasimilaritymeasurebasedonwhichsimilarusersessionsaregroupedtogether.Liuetal.measuresimilaritybetweenusersessionsbasedonthenumberofsameURLsandthenumberofsameparametertypescontain.Aftercomputingthesimilarityinthismanner,theyapplyagglutinatehierarchyclusteringalgorithmtoclustertheusersessions.Theyproposeanalgorithmtoselectfromtheseclusterstoavoidselectionofredundantuser-session-basedtestcases.Theyfindthatthereducedsuitegeneratedinthismannerisaseffectiveastheoriginaltestsuitewithinamarginof1–2%inblock,functioncoverageandseededfaultdetection.

4.4.3CreatingTestCasesforChangedPartsofCodeMarbacketal.[83]presentanapproachthatfocusesonidentifyingareasofcodethathavechangedusingimpactanalysisandgeneratingnewtestcasesfortheareasimpactedbythechangeusingprogramslicing.TheyfirstconstructProgramDependenceGraphsfortwoconsecutiveversionsoftheapplicationusingtheAbstractSyntaxTreesgeneratedbyaPHPCompiler.Then,theyuseaprogramdifferencingtooltoidentifyareasofcodethathavechangedandaprogramslicertocalculateareasofcodethatareaffectedbythechange.Finally,theyusetheslicestogeneratetestpaths(ortestcases),butfirsttheygatherconstraintsonstringandnumericvaluesintheslicesandresolvethemusingconstraintsolvers.Atestexecutionenginetakesthetestpathsandtheresolvedinputvaluesandusesthemasinputtothewebapplication.Theybuilduponthisworkin[84]withmoresubjectapplicationsandexperiments.

Oneoftheshortcomingsobservedintheearlierworkwasthatresolvingconstraintsoninputsrequiredalotofeffort.Hossainetal.[85]presentatechniquetoidentifyconstraintvaluesfrompreviousversionsoftheprogramsthatcanbereusedwhenexecuting

regressiontestpathsinthenewversion.Throughanexperimentalstudy,theyfindthatalargenumberofvariableconstraintscanbereusedfrompreviousversions.Thecentralideaintheirapproachistocomparethedefinitionsandusesofvariablesbetweenpreviousandcurrentversionsoftheapplicationtodetermineifthesameconstraintsonvariablescanbeused.

4.4.4Maintaining/RepairingaRegressionTestSuiteChoudharyetal.[86]proposeatechniquetoautomaticallyrepairwebapplicationtestscripts.Specifically,theyareabletorepairtestcasesthatfailduetoachangeinvaluesofpageelementsthatcausesamismatchbetweenexpectedandactualresultsandtestcasesthatfaultduetomovedormodifiedwebpageelements.Intheirapproach,theytakeasinputtheexecutionofthetestcaseontheoldversionoftheapplication(wherethetestcaseworks)andthenewversionoftheapplication(wherethetestcasefails)andcompareseveralDOMpropertiesinthetwoexecutionstolocatefailingpointsandrepairthetestcases.

KumarandGoel[87]exploretheuseofmodelingtheoriginalandmodifiedwebapplicationasevent-dependencygraphsthatarethenconvertedintoeventtreestoremovecyclicdependencies,andfinallycomparethetreestoidentifychangednodesandnodesthatareaffectedbythechangednodes.Identifyingtheaffectednodesintheapplicationisthenusedtoguideregressiontestselection.Threetypesofdependenciesaremodeledintheevent-dependencygraph,specifically,linkdependencies,invisibleeffectandvisibleeffectdependencies.ChopraandMadan[88]explorethepossibilityofusingtestpathsobtainedfromapageflowdiagramofawebapplicationtotesttheapplication.

Leottaetal.[89]usetheconceptofpageobjectpatternwhichisanabstractionofawebapplication’spagetoimprovemaintainabilityofSeleniumWebDrivertestcasesandconductacasestudyinanindustryenvironmenttoevaluatetheeffectivenessoftheapproach.TheyfoundthatwhenSeleniumWebDrivertestcasesarewrittenwithoutusinganypatterns,thecodeinthetestmethodsishighlycoupledwiththeimplementationinthewebpage.Thisleadstoproblemsduringmaintenance,ie,whenthewebpagecodeisupdated,allthetestcasesneedtoundergoupdatingaswell.Thecentralideainusingpageobjectpatternsisthateachpageisabstractedintoapageobjectandthefunctionalityofferedbythewebpagebecomesmethodsofferedbythepageobject,whichcanthenbecalledinatestcase.Thisallowsforcreatingtestcasesthatcanbechangedrelativelyeasilywhenthewebpageundergoeschanges.

Leottaetal.[90]alsoaddresstheproblemconstantlyfixingbrokenwebpageelementlocators(ie,XPATHlocators)whentheapplicationisundergoingchanges.TheydevelopanalgorithmthatautomaticallygeneratesrobustXPATH-basedlocatorsthatcouldworkinanewerversionoftheapplicationaswell,thusreducingtheagingoftestcases.TheydefinefourtransformationsthataredesignedtomaketheXPATHexpressionsinthewebapplicationmorespecificanditerativelyapplyoneofthesetransformationsstartingfromthemostgeneralXPATHexpression.

AndrewsandDo[91]useaFSM-basedmodelofthewebapplication,FSMWeb,

developedinpriorworktocreatetestcases.Astheapplicationundergoeschangesandthemodelchanges,theyclassifytestcasesasreusable,obsolete,andre-testabletestcases.Then,theyevaluatethecost-benefittradeoffsofapplyingbruteforceregressiontestingandselectiveregressiontestingbyquantifyingthecostsinvolvedineachcase.Further,theyproposetwoassumptions,first,thatthecostofexecutingandvalidatingatestcaseisproportionaltothelengthofthetestcaseorthenumberofinputsoneachedge,andsecond,thecostofclassification(asobsolete,reusable,orre-testable)isproportionaltothesizeofthetestsuite.Theyconductacasestudycomparingthetworegressiontestingapproachesanddiscussadecisionmakingprocessforpractitioners.

Hirzeletal.[92]adaptanexistingselectiveregressiontestingtechniquedevelopedforJavaapplicationstoworkinthecontextofGoogleWebKitcompiler,whichconvertsJavacodeintoJavaScript.Theideaistoselecttestcasesthatexecutechangedpartsofthecode.WebtestcasesexecuteJavaScriptcodebutthecodethatischangedistheJavacode.Therefore,whentheJavacodeundergoeschanges,thechangesneedtobetracedbacktoJavaScriptwhichisdifficulttoaccomplishbecauseofcodeobfuscation.Also,additionalcodeoriginatingfromlibrariesanddynamictypingmakethetrackbacktoJavadifficult.TheyfirstbuildcontrolflowgraphsofthetwoJavaapplicationversionsandcomparethem.SincethetestcasesareexecutedontheJavaScriptcodeandnottheJavacode,theyneedamappingbetweenthetestcasesandtheJavacode.Toestablishamapping,theyinstrumentthetestcasesbyintroducingacodeidentifier(CID)formethods,statementsorexpressionswhichcanbetracedfromtheJavaScripttotheJavacodeforthatcodeentity.Aftercomparingtheoldandnewversionsoftheapplication,testcasesareselectedforreexecutioniftheytouchatleastonechangedJavacodeentityasidentifiedbytheCID.TheirtechniqueisimplementedasanEclipsepluginandtheyempiricallyevaluatetheirapproach.

4.4.5EmpiricalStudiesChristopheetal.[93]conductanempiricalstudytodeterminehowwidelySelenium’stestcasesareusedinopensourcesoftwareandhowlongthesetestcasesaremaintainedovertimeastheapplicationevolves.Asaresultoftheseinvestigations,theyareabletoidentifythepartsofafunctionaltestcasethataremostpronetochange.Theystudied287GitHubrepositoriesandclassifiedtheapplicationsaswebserviceproviders,frameworksforbuildingwebapplications,Seleniumextensions,webapplicationexamplesforlearning,andamiscellaneouscategory.Theyfoundthat25%oftheapplicationsfromthewebservicescategoryuseSeleniumtoalargeextentfortesting.TheyalsofoundthatSeleniumtestcasesaremodifiedastheprojectevolves.Andfinally,theyfoundthattheconstantsandassertstatementsarethemostfrequentlychangedstatementsastheSeleniumtestcasesevolve.

4.5DistantReadingofSurveyedPapersWeusedthePaperMachines5extensionforZotero6toperformtextualanalysesofthe60paperswesurveyinthischapter.InadditiontothestandardEnglishstopwords,weadded

wordsspecifictothiscorpus,alphabetically:acm,application,applications,case,cases,conference,fig,figure,googletag,ieee,international,paper,proceedings,pubad,public,section,software,standard,table,target,test,testing,tests,andweb.ThesetextanalysesarenotperfectinthattheyrelyonextractingthetextfromthePDFfiles.Ourspotcheckingofthetextextractionshowedthatmosttextextractionwasperformedcorrectly;however,therearesomeissueswithspecialcharacters,suchasdashes,quotes,andligatures,like“ff.”Theanalysesalsodonotweightwordsdifferently,forexample,wordsfoundintitlesandfootnotesarecountedthesame.Despitetheselimitations,webelievethesedistantreadingsallowustoseetrendsintheresearch.

ThewordcloudinFig.4showstheproportionalprevalenceofwordsinthecorpus(inourcase,thesurveyedpapers),wherethemostcommonwordsareinthelargestfont.Asexpected,themostcommonwordsarerelatedtowebapplications,eg,user,HTML,pages,request/s,sessions,server,database,browser,form,http,parameter;code,eg,code,program,source,expressions,statements,line/s,variable/s;andtesting,eg,values,suite/s,fault/s,failure/s,coverage.

FIGURE4 Awordcloudofthemostcommonwordsinthepapersstudied.

Distinctcharacteristicofwebapplicationsarereflectedinthewordcloud,suchasnavigation(sequence/s,transition/s,control)anduserinteraction(users,GUI,interactions).Inaddition,weseewordslikePHPandJava,whicharethelanguagesusedtoimplementthewebapplication.Inoursurvey,wedidnotfindworkthatdevelopedtechniquesspecifictootherlanguage/frameworks,suchasASPorRubyonRails.Ofthewordsthatimplyacommonwebapplicationfunctionality(eg,searchandorder),loginisdistinctbecauseitisunlikelytohavemeaningsotherthanthewebapplicationfunctionality(asopposedtosearch-basedtestingor“inorderto”statements).Login’sprominenceseemstoimplythatitisimportantfunctionalitythatresearchersmustaddress.

Wordslikemodel/s,technique/s,approach/es,andalgorithmaswellasresult/sand,toalesserextent,evaluationwords(eg,study,percentage,andaverage)areallprominentinthecloud,implyingthatresearchersdevelopednovelapproachesandevaluatedthoseapproaches.Theprominenceoftool/sandautomateimpliesthatresearchersimplemented

theirapproachesinautomatedtools.Oursurveysupportsbothoftheseobservations.

Withrespecttowordsrelatedtothetestingresearchfocus,generation,regression,prioritization,reduction,andlocalizationareallprominentinthewordcloud.Ontheotherhand,thewordoracleismuchsmaller.Again,oursurveyreflectstherelativeweightsofthesetopics.

Whilethewordcloudpresentsindividualwords,Fig.5presentstheproportionalprevalenceof40topicsinthepapers(y-axis)bytheirpublicationdate(x-axis).7Despiteimperfectionsintheanalysis(eg,twopaperspublishedin2010arerepresentedin2009),thevisualizationhelpstoshowtheresearchtrends.Thefigurehighlightseightofthemostcommontopics.Wechosenottohighlighttopicsthatrepresentthedocuments’metadataortopicsthatrepresentcomponentsofwebapplications(eg,{user,server,request}or{function,document,http}).Despitebeingmoreprevalent,thesetopicsarelistedafterthehighlightedtopics.

FIGURE5 Atopicmodelofthemostcommongroupingsofwordsinthepapersstudied.

Ingeneral,thevisualizationsupportsourclaimthatthepapersemphasizeapproachesandtechniques(eg,{fault,sever,model},{user,model,session},{user,requir,tool},{page,navig,model})overempiricalstudies.Usertools(fourthfrombottom)andusermodeling(thirdbarfromthebottom)haveremainedfairlyconstantinprevalenceoverthetimeperiodperhapsanindicationofhowimportantusersare,withrespecttotestinganduseofwebapplications.

Whilefaultlocalization(bottombar)hasremainedrelativelyconstantinprevalencethroughoutthetimeperiod,otherfault-relatedtopicsareonthedecline.Workonfaultseveritymodels(secondbarfromtop)issharplydeclininginprevalence,yetisstillanopenproblemandcouldbeadirectionforfuturework.Thefourthbarfromthetop{generat,fault,execut}isalsosurprisinglyonthedeclineinprevalence;perhapstherecentincreaseinprevalenceofstaticanalysistechniques(eg,theslightincreasein{model,state,behavior},topbar,despitenotincludingmanypapersonstaticanalysistechniquesinoursurvey)explainsthedecline.

Thetopicoftestsuiteselection(representedby{suit,priorit,reduc},thirdfrombottom)isonanupwardstrend.Asapplicationsizes—andthereforetheirtestsuites—continuetogrow,itiscrucialthattesterscantestinmorecost-effectiveways,andthusitisimportantthatresearchaddressesthisimportantproblem.

Whilemostofthetopics(eg,faultlocalization)seemtohaveconstantprevalencethroughoutthetimeperiod,{user,session,model}and{fault,sever,model}arelessprevalentmorerecently.Researchontraditionalusersession-basedtesting—{user,model,session)inthebottombar—seemstobeonthedecline,while{session,user,request}isnot.Inoursearchforrelevantpapers,wefoundworkonJavaScript-basedcapture-replay,whichisalsousersessionbased.Wedonotincludethisworkinoursurveyorinthisanalysisbecauseitwasoutofscopeand,therefore,isnotreflectedinthefigure.Workonseveritymodelsislackingandcouldbeadirectionforfuturework.

5ConclusionAswebapplicationscontinuetogrowincomplexityandinpopularity,testingtheircorrectnesswillcontinuetodeservetheattentionofresearchers.Inthischapter,westudiedandreportedontheadvancesinwebapplicationtestingliteraturebetweentheyearsof2010and2014.Wefoundthatresearchinthisperiodbroadlycouldbecategorizedintheareasoftestcasegeneration,oracledevelopment,criteria,andapproachestoevaluateeffectivenessofatestsuite,andregressiontestingapproaches.

Inthissection,wesummarizesomeopenareasofresearchthatweseetrendinginthenextfewyearsbasedonouranalysisofthepast4yearsofresearch.

•WefoundthatseveraltoolsandtechniquesdevelopedintheliteraturearebuiltontopofSelenium[19],eitherasanextensiontoSeleniumRC,orbuiltontopofWebDriver.TheendgoalofincorporatingwithSeleniumistocreatetestcasesthatSeleniumcanexecute.TheresearchcommunityrecognizestheprevalentuseofSeleniumintheindustryandthushasfocusedondevelopingadd-onstoattractthetargetaudiencethatalreadyusesSeleniumintheirtestingprocess.ThisleadsustobelievethatSeleniumisawebtestingtoolthatwillbecomeincreasinglypopularinacademiaandindustryinthefuture.

•Anothertrend,orlackthereof,thatweobservedisthelimitedresearchintheareaofdevelopingadequacycriteriaforwebapplications.Severalresearchersfocusedonfaultdetectionasanevaluationmeasureoronusingtraditionalcoveragecriteriaofstatement,method,andbranchcoverage.Withthedifferenttypesofweblanguagesusedtoday,thedevelopmentanduseofframeworkssuchasRailsandthepopularityofJavaScriptinallaspectsofwebdevelopment,theremightbescopefordevelopmentofnewcoveragecriteriathattargetuniquecharacteristicsofwebapplications.Incompanieswheresystematictestingisfollowed,adequacycriteriatendtobethemostcommonmethodtodeterminewhentostoptesting.Thus,advancementsinthisdomainandpracticalapplicabilityofproposedcriteriacouldfindwidespreadacceptance.

•Intermsofevaluatingeffectivenessoftestingtechniques,wefoundverylimitedresearchintheareaofdevelopingfaultseverityclassifications.Faultseverityisacommonlyusedmetricintheindustrytoprioritizetestinganddevelopmentefforts.Advancesindevelopingsolid,empiricallyevaluatedfaultseveritycriteriaforwebapplicationtesting,thataddressesfaultseverityfromthepointofviewofmultiplestakeholderscouldbeofsignificantimportanceinbothacademicandindustrycircles.

•Inthedomainoftestcasegenerationfortraditionalapplications,wefindanincreasinguseofsearch-basedalgorithms.Applyingsearch-basedalgorithmstotestcasegenerationwebdomainisonlyrecentlygainingimportance.Thiscouldbeanareaofresearchthatcouldseegrowthinthenearfuture,asscalabletestcasegenerationcontinuestobeachallengeinthewebapplicationtestingdomain.

•Anothertrendthatweobservedfromouranalysisisthatresearcherstendtofocusondevelopingnewapproachesandtechniquestotestwebapplications,whetherintheareaoftestgenerationororacledevelopment,etc.Thoughmostresearchersincludeastrong

empiricalcomponentthatevaluatestheeffectivenessoftheirapproach,thereisalackofempiricalstudiesthatcompareandcontrastthevariousapproachestowebtesting,lackofstudiesthatreplicateexistingstudiesandresults,andalackofsurveysandqualitativestudiesdesignedtounderstandtestingpracticesinsmall/largecompanies,etc.Empiricalresearchcanserveasastrongfoundationforidentifyingnewresearchproblemsthatthecommunitycanaddressinthefuture.Webelievemoreempiricalresearchisanareathewebtestingcommunitycanfocusonandbenefitfrominthenearfuture.

•Muchoftheworksurveyedisfocusedonindividualcomponentsontheserver-sideortheclient-sidetogreatresults.Severalauthorsmentionthatfutureworkinvolvescombiningtheirworkwithothers’work.Wecanenvisionmanypossiblefruitfulcollaborations.Forexample,theconcolictestcasegenerationworkinSection4.1.3maybeabletobecombinedwiththeoracleworkthatfindsdifferencesinHTMLfromSection4.2toyieldevenbetterfaultlocalization.

•Finally,asawholegenreofapplicationsandservicesaremovingtothecloud,webelievethattestingtechniquesthatcanscaletothecloud[94]andthenotionofofferingwebtestingasaserviceinthecloudareareasofresearchthatcouldgainprominenceinthecomingyears.

References[1]OrsoA.,RothermelG.Softwaretesting:aresearchtravelogue(2000-2014).In:ProceedingsoftheFutureofSoftwareEngineering,FOSE2014,NewYork,NY,USA;ACM;2014:978-1-4503-2865-4117–132.doi:10.1145/2593882.259388500005.

[2]GarousiV.,MesbahA.,Betin-CanA.,MirshokraieS.Asystematicmappingstudyofwebapplicationtesting.Inf.Softw.Technol.0950-58492013.;55(8):1374–1396.doi:10.1016/j.infsof.2013.02.006.http://www.sciencedirect.com/science/article/pii/S095058491300039600015.

[3]LiY.-F.,DasP.K.,DoweD.L.Twodecadesofwebapplicationtesting–asurveyofrecentadvances.Inf.Syst.0306-43792014.;43:20–54.doi:10.1016/j.is.2014.02.001.http://www.sciencedirect.com/science/article/pii/S030643791400027100003.

[4]AlalfiM.H.,CordyJ.R.,DeanT.R.Modellingmethodsforwebapplicationverificationandtesting:stateoftheart.Softw.Test.Verif.Reliab.1099-16892009.;19(4):265–296.doi:10.1002/stvr.401.http://onlinelibrary.wiley.com/doi/10.1002/stvr.401/abstract,00064,

[5]LiX.,XueY.Asurveyonserver-sideapproachestosecuringwebapplications.ACMComput.Surv.0360-03002014;46(4):54:1–54:29.doi:10.1145/254131500006.

[6]MesbahA.Chapterfive–advancesintestingJavaScript-basedwebapplications.In:MemonA.M.,ed.AdvancesinComputers.Elsevier;201–235.2015.;vol.97.http://www.sciencedirect.com/science/article/pii/S006524581400011400000.

[7]SampathS.Chapter3–advancesinuser-session-basedtestingofwebapplications.In:HursonA.,MemonA.,eds.AdvancesinComputers.Elsevier;87–108.2012.;vol.86.http://www.sciencedirect.com/science/article/pii/B978012396535600003X00000.

[8]ParoloP.D.B.,PanR.K.,GhoshR.,HubermanB.A.,KaskiK.,FortunatoS.Attentiondecayinscience.arXiv:1503.01881[physics].2015.http://arxiv.org/abs/1503.01881arXiv:1503.01881.

[9]W3C.HTTP–HyperTextTransferProtocolOverview.2015.http://www.w3.org/Protocols/.

[10]Apache.ApacheHTTPserverproject.2015.http://httpd.apache.org/.

[11]Apache.ApacheTomcat.2015.http://tomcat.apache.org/.

[12]IBM.WebSphereapplicationserver.2015.http://www-03.ibm.com/software/products/en/appserv-was.

[13]Google.Appengine–runyourapplicationsonafully-managedPlatform-as-a-

Service(PaaS)usingbuilt-inservices–GoogleCloudPlatform.2015.https://cloud.google.com/appengine/.

[14]W3C.HTTPstatemanagementmechanism.2015.http://www.w3.org/Protocols/rfc2109/rfc2109.

[15]W3C.HTML,theweb’scorelanguage.2015.http://www.w3.org/html/.

[16]W3C.Cascadingstylesheets.2015.http://www.w3.org/Style/CSS/Overview.en.html.

[17]W3C.JavaScriptWebAPIs.2015.http://www.w3.org/standards/webdesign/script.html.

[18]WebTechnologySurveys.Usagestatisticsandmarketshareofserver-sideprogramminglanguagesforwebsites,April2015.2015.http://w3techs.com/technologies/overview/programming_language/all.

[19]Selenium.SeleniumHQbrowserautomation.2015.http://www.seleniumhq.org/.

[20]NguyenH.V.,NguyenH.A.,NguyenT.T.,NguyenT.N.Auto-locatingandfix-propagatingforHTMLvalidationerrorstoPHPserver-sidecode.In:201126thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering(ASE);2011:13–22.doi:10.1109/ASE.2011.610004700016.

[21]SamimiH.,SchferM.,ArtziS.,MillsteinT.,TipF.,HendrenL.AutomatedrepairofHTMLgenerationerrorsinPHPapplicationsusingstringconstraintsolving.In:Proceedingsofthe34thInternationalConferenceonSoftwareEngineering,ICSE’12,Piscataway,NJ,USA;IEEEPress;2012:978-1-4673-1067-3277–287.http://dl.acm.org/citation.cfm?id=2337223.233725700039.

[22]TungY.-H.,TsengS.-S.,LeeT.-J.,WengJ.-F.Anovelapproachtoautomatictestcasegenerationforwebapplications.In:201010thInternationalConferenceonQualitySoftware(QSIC);IEEE;2010:978-1-4244-8078-4399–404.doi:10.1109/QSIC.2010.33.http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=556299300015.

[23]OffuttJ.,WuY.Modelingpresentationlayersofwebapplicationsfortesting.Softw.Syst.Model.2010;9(2):257–280.doi:10.1007/s10270-009-0125-4ISSN1619-1366,1619-1374.

[24]ThummalaS.,OffuttJ.Anevaluationoftheeffectivenessoftheatomicsectionmodel.In:DingelJ.,SchulteW.,RamosI.,AbrahoS.,InsfranE.,eds.Model-DrivenEngineeringLanguagesandSystems,no.8767inLectureNotesinComputerScience.SpringerInternationalPublishing;2014:978-3-319-11652-535–49.978-3-319-11653-2http://link.springer.com/chapter/10.1007/978-3-319-11653-2_3.

[25]ChenS.,MiaoH.,SongB.,ChenY.Towardspracticalmodelingofwebapplicationsandgeneratingtests.In:20104thIEEEInternationalSymposiumonTheoreticalAspectsofSoftwareEngineering(TASE).2010:209–217.

doi:10.1109/TASE.2010.2500003.

[26]TorselA.Automatedtestcasegenerationforwebapplicationsfromadomainspecificmodel.In:2011IEEE35thAnnualComputerSoftwareandApplicationsConferenceWorkshops(COMPSACW);2011:137–142.doi:10.1109/COMPSACW.2011.32.

[27]SongB.,GongS.,ChenS.Modelcompositionandgeneratingtestsforwebapplications.In:2011SeventhInternationalConferenceonComputationalIntelligenceandSecurity(CIS);2011:568–572.doi:10.1109/CIS.2011.13100001.

[28]EnderlinI.,DadeauF.,GiorgettiA.,BouquetF.Grammar-basedtestingusingrealisticdomainsinPHP.In:2012IEEEFifthInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2012:509–518.doi:10.1109/ICST.2012.13600003.

[29]EnderlinI.,DadeauF.,GiorgettiA.,OthmanA.B.Praspel:aspecificationlanguageforcontract-basedtestinginPHP.In:WolffB.,ZadiF.,eds.TestingSoftwareandSystems,no.7019inLectureNotesinComputerScience.Berlin:Springer;2011:978-3-642-24579-464–79.978-3-642-24580-0http://link.springer.com/chapter/10.1007/978-3-642-24580-0_6.

[30]DallmeierV.,BurgerM.,OrthT.,ZellerA.WebMate:generatingtestcasesforWeb2.0.In:WinklerD.,BifflS.,BergsmannJ.,eds.SoftwareQuality.IncreasingValueinSoftwareandSystemsDevelopment,no.133inLectureNotesinBusinessInformationProcessing.Berlin:Springer;2013:978-3-642-35701-555–69.978-3-642-35702-2http://link.springer.com/chapter/10.1007/978-3-642-35702-2_5.

[31]ZellerA.Wearecreatingastart-upinwebtesting.2013.http://andreas-zeller.blogspot.com/2013/03/we-are-creating-start-up-in-web-testing.html.

[32]TestfabrikConsulting+SolutionsAG.webmate.2015.https://app.webmate.io/.

[33]SchurM.,RothA.,ZellerA.ProCrawl:miningtestmodelsfrommulti-userwebapplications.In:Proceedingsofthe2014InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2014,NewYork,NY,USA;ACM;2014:978-1-4503-2645-2413–416.doi:10.1145/2610384.262805100000.

[34]SchurM.,RothA.,ZellerA.Miningbehaviormodelsfromenterprisewebapplications.In:Proceedingsofthe20139thJointMeetingonFoundationsofSoftwareEngineering,ESEC/FSE2013,NewYork,NY,USA;ACM;2013:978-1-4503-2237-9422–432.doi:10.1145/2491411.2491426.

[35]ElbaumS.,RothermelG.,KarreS.,FisherM.Leveraginguser-sessiondatatosupportWebapplicationtesting.IEEETrans.Softw.Eng.0098-55892005;31(3):187–202.doi:10.1109/TSE.2005.36.

[36]SprenkleS.,PollockL.,SimkoL.Astudyofusage-basednavigationmodelsandgeneratedabstracttestcasesforwebapplications.In:Proceedingsofthe2011FourthIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation,ICST’11,Washington,DC,USA;IEEEComputerSociety;2011:978-

0-7695-4342-0230–239.doi:10.1109/ICST.2011.34.

[37]SprenkleS.E.,PollockL.L.,SimkoL.M.Configuringeffectivenavigationmodelsandabstracttestcasesforwebapplicationsbyanalysinguserbehaviour.Softw.Test.Verif.Reliab.1099-16892013.;23(6):439–464.doi:10.1002/stvr.1496.http://onlinelibrary.wiley.com/doi/10.1002/stvr.1496/abstract00003.

[38]SprenkleS.,CobbC.,PollockL.Leveraginguser-privilegeclassificationtocustomizeusage-basedstatisticalmodelsofwebapplications.In:Proceedingsofthe2012IEEEFifthInternationalConferenceonSoftwareTesting,VerificationandValidation,ICST’12,Washington,DC,USA;IEEEComputerSociety;2012:978-0-7695-4670-4161–170.doi:10.1109/ICST.2012.9600005.

[39]SantJ.,SouterA.,GreenwaldL.Anexplorationofstatisticalmodelsforautomatedtestcasegeneration.In:ACMSIGSOFTSoftwareEngineeringNotes.ACM;1–7.2005.;vol.30.http://dl.acm.org/citation.cfm?id=1083256.

[40]ArtziS.,KiezunA.,DolbyJ.,TipF.,DigD.,ParadkarA.,ErnstM.D.Findingbugsinwebapplicationsusingdynamictestgenerationandexplicit-statemodelchecking.IEEETrans.Softw.Eng.0098-55892010;36(4):474–494.doi:10.1109/TSE.2010.3100075.

[41]ArtziS.,KiezunA.,DolbyJ.,TipF.,DigD.,ParadkarA.,ErnstM.D.Findingbugsindynamicwebapplications.In:Proceedingsofthe2008InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA’08,NewYork,NY,USA;ACM;2008:978-1-60558-050-0261–272.doi:10.1145/1390630.139066200158.

[42]ArtziS.,DolbyJ.,TipF.,PistoiaM.Practicalfaultlocalizationfordynamicwebapplications.In:2010ACM/IEEE32ndInternationalConferenceonSoftwareEngineering;265–274.doi:10.1145/1806799.1806840.2010;vol.1.

[43]JonesJ.A.,HarroldM.J.Empiricalevaluationofthetarantulaautomaticfault-localizationtechnique.In:Proceedingsofthe20thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering,ASE’05,NewYork,NY,USA;ACM;2005:1-58113-993-4273–282.doi:10.1145/1101908.110194900601.

[44]JonesJ.A.,HarroldM.J.,StaskoJ.Visualizationoftestinformationtoassistfaultlocalization.In:Proceedingsofthe24thInternationalConferenceonSoftwareEngineering,ICSE’02,NewYork,NY,USA;ACM;2002:1-58113-472-X467–477.doi:10.1145/581339.58139700680.

[45]ArtziS.,DolbyJ.,TipF.,PistoiaM.Directedtestgenerationforeffectivefaultlocalization.In:Proceedingsofthe19thInternationalSymposiumonSoftwareTestingandAnalysis,ISSTA’10,NewYork,NY,USA;ACM;2010:978-1-60558-823-049–60.doi:10.1145/1831708.1831715.

[46]ArtziS.,DolbyJ.,TipF.,PistoiaM.Faultlocalizationfordynamicwebapplications.IEEETrans.Softw.Eng.0098-55892012;38(2):314–335.doi:10.1109/TSE.2011.7600018.

[47]AbreuR.,ZoeteweijP.,vanGemundA.J.C.Anevaluationofsimilarity

coefficientsforsoftwarefaultlocalization.In:Proceedingsofthe12thPacificRimInternationalSymposiumonDependableComputing,PRDC’06,Washington,DC,USA;IEEEComputerSociety;2006:0-7695-2724-839–46.doi:10.1109/PRDC.2006.18.

[48]ChenM.Y.,KicimanE.,FratkinE.,FoxA.,BrewerE.Pinpoint:problemdeterminationinlarge,dynamicInternetservices.In:ProceedingsoftheInternationalConferenceonDependableSystemsandNetworks,2002,DSN2002;2002:595–604.doi:10.1109/DSN.2002.1029005.

[49]deMatosE.C.B.,SousaT.C.Fromformalrequirementstoautomatedwebtestingandprototyping.Innov.Syst.Softw.Eng.2010;6(1-2):163–169.doi:10.1007/s11334-009-0112-5ISSN1614-5046,1614-5054.

[50]ThummalapentaS.,LakshmiK.V.,SinhaS.,SinhaN.,ChandraS.Guidedtestgenerationforwebapplications.In:Proceedingsofthe2013InternationalConferenceonSoftwareEngineering,ICSE’13,Piscataway,NJ,USA;IEEEPress;2013:978-1-4673-3076-3162–171.http://dl.acm.org/citation.cfm?id=2486788.2486810.

[51]AlshahwanN.,HarmanM.Automatedwebapplicationtestingusingsearchbasedsoftwareengineering.In:201126thIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering(ASE);2011:3–12.doi:10.1109/ASE.2011.610008200049.

[52]KorelB.Automatedsoftwaretestdatageneration.IEEETrans.Softw.Eng.0098-55891990;16(8):870–879.doi:10.1109/32.57624.

[53]TappendenA.F.,MillerJ.Automatedcookiecollectiontesting.ACMTrans.Softw.Eng.Methodol.1049-331X2014;23(1):3:1–3:40.doi:10.1145/255993600003.

[54]AlshahwanN.,HarmanM.Stateawaretestcaseregenerationforimprovingwebapplicationtestsuitecoverageandfaultdetection.In:Proceedingsofthe2012InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2012,NewYork,NY,USA;ACM;2012:978-1-4503-1454-145–55.doi:10.1145/2338965.2336759.

[55]ShahbazM.,McMinnP.,StevensonM.Automaticgenerationofvalidandinvalidtestdataforstringvalidationroutinesusingwebsearchesandregularexpressions.Sci.Comput.Program.0167-64232015.;97(Part4):405–425.doi:10.1016/j.scico.2014.04.008.http://www.sciencedirect.com/science/article/pii/S016764231400172500002.

[56]McMinnP.,ShahbazM.,StevensonM.Search-basedtestinputgenerationforstringdatatypesusingtheresultsofwebqueries.In:2012IEEEFifthInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2012:141–150.doi:10.1109/ICST.2012.9400027.

[57]FujiwaraS.,MunakataK.,MaedaY.,KatayamaA.,UeharaT.Testdata

generationforwebapplicationusingaUMLclassdiagramwithOCLconstraints.Innov.Syst.Softw.Eng.1614-50462011;7(4):275–282.doi:10.1007/s11334-011-0162-31614-5054.

[58]DobolyiK.,SoechtingE.,WeimerW.Automatingregressiontestingusingweb-basedapplicationsimilarities.1433-2787Int.J.Softw.ToolsTechnol.Transfer.1433-27792010;13(2):111–129.doi:10.1007/s10009-010-0170-x00000.

[59]deCastroA.,MacedoG.A.,CollinsE.F.,Dias-NetoA.C.ExtensionofSeleniumRCtooltoperformautomatedtestingwithdatabasesinwebapplications.In:20138thInternationalWorkshoponAutomationofSoftwareTest(AST).2013:125–131.doi:10.1109/IWAST.2013.6595803.

[60]MahajanS.,HalfondW.G.J.FindingHTMLpresentationfailuresusingimagecomparisontechniques.In:Proceedingsofthe29thACM/IEEEInternationalConferenceonAutomatedSoftwareEngineering,ASE’14,NewYork,NY,USA;ACM;2014:978-1-4503-3013-891–96.doi:10.1145/2642937.264296600003.

[61]MahajanS.,LiB.,HalfondW.G.J.RootcauseanalysisforHTMLpresentationfailuresusingsearch-basedtechniques.In:Proceedingsofthe7thInternationalWorkshoponSearch-BasedSoftwareTesting,SBST2014,NewYork,NY,USA;ACM;2014:978-1-4503-2852-415–18.doi:10.1145/2593833.259383600000.

[62]MahajanS.,HalfondW.G.J.DetectionandLocalizationofHTMLPresentationFailuresUsingComputerVision-BasedTechniques.In:Proceedingsofthe8thIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);IEEE;2015.

[63]MahajanS.,HalfondW.G.J.WebSee:AToolforDebuggingHTMLPresentationFailures.In:Proceedingsofthe8thIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST)-ToolTrack;IEEE;2015.

[64]AlalfiM.H.,CordyJ.R.,DeanT.R.Automatingcoveragemetricsfordynamicwebapplications.In:201014thEuropeanConferenceonSoftwareMaintenanceandReengineering(CSMR);2010:51–60.doi:10.1109/CSMR.2010.2100010.

[65]AlshahwanN.,HarmanM.Augmentingtestsuiteseffectivenessbyincreasingoutputdiversity.In:201234thInternationalConferenceonSoftwareEngineering(ICSE);2012:1345–1348.doi:10.1109/ICSE.2012.622708300003.

[66]SakamotoK.,TomohiroK.,HamuraD.,WashizakiH.,FukazawaY.POGen:atestcodegeneratorbasedontemplatevariablecoverageingray-boxintegrationtestingforwebapplications.In:CortellessaV.,VarrD.,eds.FundamentalApproachestoSoftwareEngineering,no.7793inLectureNotesinComputerScience.Berlin:Springer;2013:978-3-642-37056-4343–358.978-3-642-37057-1http://link.springer.com.ezproxy.wlu.edu/chapter/10.1007/978-3-642-37057-1_25.

[67]PraphamontripongU.,OffuttJ.Applyingmutationtestingtowebapplications.In:2010ThirdInternationalConferenceonSoftwareTesting,Verification,andValidationWorkshops(ICSTW);2010:132–141.doi:10.1109/ICSTW.2010.38

00021.

[68]MaY.-S.,OffuttJ.,KwonY.R.MuJava:anautomatedclassmutationsystem.Softw.Test.Verif.Reliab.1099-16892005.;15(2):97–133.doi:10.1002/stvr.308.http://onlinelibrary.wiley.com/doi/10.1002/stvr.308/abstract.

[69]DobolyiK.,WeimerW.Modelingconsumer-perceivedwebapplicationfaultseveritiesfortesting.In:Proceedingsofthe19thInternationalSymposiumonSoftwareTestingandAnalysis,ISSTA’10,NewYork,NY,USA;ACM;2010:978-1-60558-823-097–106.doi:10.1145/1831708.183172000007.

[70]BryceR.C.,SampathS.,MemonA.M.Developingasinglemodelandtestprioritizationstrategiesforevent-drivensoftware.IEEETrans.Softw.Eng.0098-55892011;37(1):48–64.doi:10.1109/TSE.2010.12.

[71]RothermelG.,UntchR.H.,ChuC.,HarroldM.J.Prioritizingtestcasesforregressiontesting.IEEETrans.Softw.Eng.0098-55892001;27(10):929–948.doi:10.1109/32.962562.

[72]SampathS.,BryceR.C.,JainS.,ManchesterS.Atoolforcombination-basedprioritizationandreductionofuser-session-basedtestsuites.In:Proceedingsofthe201127thIEEEInternationalConferenceonSoftwareMaintenance,ICSM’11,Washington,DC,USA;IEEEComputerSociety;2011:978-1-4577-0663-9574–577.doi:10.1109/ICSM.2011.608083300007.

[73]GargD.,DattaA.,FrenchT.Atwo-levelprioritizationapproachforregressiontestingofwebapplications.In:201219thAsia-PacificSoftwareEngineeringConference(APSEC);150–153.doi:10.1109/APSEC.2012.34.2012;vol.200001.

[74]GargD.,DattaA.Parallelexecutionofprioritizedtestcasesforregressiontestingofwebapplications.In:ProceedingsoftheThirty-SixthAustralasianComputerScienceConference–Volume135,ACSC’13,Darlinghurst,Australia,Australia;AustralianComputerSociety,Inc.;2013:978-1-921770-20-361–68.http://dl.acm.org/citation.cfm?id=2525401.252540800001.

[75]GargD.,DattaA.Testcaseprioritizationduetodatabasechangesinwebapplications.In:2012IEEEFifthInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2012:726–730.doi:10.1109/ICST.2012.16300006.

[76]SampathS.,BryceR.C.Improvingtheeffectivenessoftestsuitereductionforuser-session-basedtestingofwebapplications.Inf.Softw.Technol.0950-58492012;54(7):724–738.doi:10.1016/j.infsof.2012.01.00700010.

[77]SampathS.,SprenkleS.,GibsonE.,PollockL.,GreenwaldA.S.Applyingconceptanalysistouser-session-basedtestingofwebapplications.IEEETrans.Softw.Eng.0098-55892007;33(10):643–658.doi:10.1109/TSE.2007.70723.

[78]ElbaumS.,MalishevskyA.,RothermelG.Incorporatingvaryingtestcostsandfaultseveritiesintotestcaseprioritization.In:Proceedingsofthe23rd

InternationalConferenceonSoftwareEngineering,2001,ICSE2001;2001:329–338.doi:10.1109/ICSE.2001.919106.

[79]NejadDobunehM.R.,JawawiD.N.A.,GhazaliM.,MalakootiM.V.Developmenttestcaseprioritizationtechniqueinregressiontestingbasedonhybridcriteria.In:20148thMalaysianSoftwareEngineeringConference(MySEC);2014:301–305.doi:10.1109/MySec.2014.698603300000.

[80]HuangY.,LuL.Amethodologyfortestsuitreductioninuser-session-basedtesting.In:2010IEEEFifthInternationalConferenceonBio-InspiredComputing:TheoriesandApplications(BIC-TA);2010:864–868.doi:10.1109/BICTA.2010.564523900005.

[81]BirkhoffG.LatticeTheory,Volume5.AmericanMathematicalSoc.ColloquiumPublications;1940.

[82]LiuY.,WangK.,WeiW.,ZhangB.,ZhongH.User-session-basedtestcasesoptimizationmethodbasedonagglutinatehierarchyclustering.In:InternetofThings(iThings/CPSCom),2011InternationalConferenceonand4thInternationalConferenceonCyber,PhysicalandSocialComputing;2011:413–418.doi:10.1109/iThings/CPSCom.2011.135.

[83]MarbackA.,DoH.,EhresmannN.AneffectiveregressiontestingapproachforPHPwebapplications.In:2012IEEEFifthInternationalConferenceonSoftwareTesting,VerificationandValidation(ICST);2012:221–230.doi:10.1109/ICST.2012.10200006.

[84]DoH.,HossainM.AnefficientregressiontestingapproachforPHPwebapplications:acontrolledexperiment.Softw.Test.Verif.Reliab.1099-16892014.;24(5):367–385.doi:10.1002/stvr.1540.00000http://onlinelibrary.wiley.com/doi/10.1002/stvr.1540/abstract00000.

[85]HossainM.,DoH.,EdaR.Regressiontestingforwebapplicationsusingreusableconstraintvalues.In:Proceedingsofthe2014IEEEInternationalConferenceonSoftwareTesting,Verification,andValidationWorkshops,ICSTW’14,Washington,DC,USA;IEEEComputerSociety;2014:978-1-4799-5790-3312–321.doi:10.1109/ICSTW.2014.35.

[86]ChoudharyS.R.,ZhaoD.,VerseeH.,OrsoA.WATER:WebApplicationTEstRepair.In:ProceedingsoftheFirstInternationalWorkshoponEnd-to-EndTestScriptEngineering,ETSE’11,NewYork,NY,USA;ACM;2011:978-1-4503-0808-324–29.doi:10.1145/2002931.2002935.

[87]KumarA.,GoelR.Eventdriventestcaseselectionforregressiontestingwebapplications.In:2012InternationalConferenceonAdvancesinEngineering,ScienceandManagement(ICAESM);2012:121–12700004.

[88]ChopraR.,MadanS.Reusingblackboxtestpathsforwhiteboxtestingofwebsites.In:2013IEEE3rdInternationalAdvanceComputingConference(IACC);2013:1345–1350.doi:10.1109/IAdCC.2013.651442400000.

[89]LeottaM.,ClerissiD.,RiccaF.,SpadaroC.Improvingtestsuitesmaintainabilitywiththepageobjectpattern:anindustrialcasestudy.In:2013IEEESixthInternationalConferenceonSoftwareTesting,VerificationandValidationWorkshops(ICSTW);2013:108–113.doi:10.1109/ICSTW.2013.19.

[90]LeottaM.,StoccoA.,RiccaF.,TonellaP.ReducingwebtestcasesagingbymeansofrobustXPathlocators.In:2014IEEEInternationalSymposiumonSoftwareReliabilityEngineeringWorkshops(ISSREW).2014:449–454.doi:10.1109/ISSREW.2014.1700000.

[91]AndrewsA.,DoH.Trade-offanalysisforselectiveversusbrute-forceregressiontestinginFSMWeb.In:Proceedingsofthe2014IEEE15thInternationalSymposiumonHigh-AssuranceSystemsEngineering,HASE’14,Washington,DC,USA;IEEEComputerSociety;2014:978-1-4799-3466-9184–192.doi:10.1109/HASE.2014.3300002.

[92]HirzelM.SelectiveregressiontestingforwebapplicationscreatedwithGooglewebtoolkit.In:Proceedingsofthe2014InternationalConferenceonPrinciplesandPracticesofProgrammingontheJavaPlatform:VirtualMachines,Languages,andTools,PPPJ’14,NewYork,NY,USA;ACM;2014:978-1-4503-2926-2110–121.doi:10.1145/2647508.2647527.

[93]ChristopheL.,StevensR.,DeRooverC.,DeMeuterW.Prevalenceandmaintenanceofautomatedfunctionaltestsforwebapplications.In:2014IEEEInternationalConferenceonSoftwareMaintenanceandEvolution(ICSME);2014:141–150.doi:10.1109/ICSME.2014.36.

[94]CaiJ.,HuQ.Analysisforcloudtestingofwebapplication.In:20142ndInternationalConferenceonSystemsandInformatics(ICSAI);2014:293–297.doi:10.1109/ICSAI.2014.7009302.

SreedeviSampathisanAssociateProfessorintheDepartmentofInformationSystemsattheUniversityofMaryland,BaltimoreCounty.SheearnedherPh.D.andM.S.inComputerandInformationSciencesfromtheUniversityofDelawarein2006and2002,respectively,andherB.E.degreefromOsmaniaUniversityinComputerScienceandEngineeringin2000.Herresearchinterestsareintheareasofsoftwaretestingandqualityassurance,webapplications,softwaremaintenanceandsoftwaresecurity.Shehasservedontheprogramcommitteesofinternationalconferences,suchastheInternationalConferenceonSoftwareTestingVerificationandValidation(ICST),InternationalSymposiumonSoftwareReliabilityEngineering(ISSRE),andtheInternationalConferenceonEmpiricalSoftwareEngineeringandMeasurement(ESEM).SheisamemberoftheIEEEComputerSociety.

SaraSprenkleisanAssociateProfessorofComputerScienceatWashingtonandLeeUniversity.ShereceivedherPh.D.inComputerandInformationSciencesfromtheUniversityofDelawarein2007.SheearnedherM.S.inComputerSciencefromDukeUniversityin2004,andherB.S.fromGettysburgCollegein1999.Hercurrentresearchfocusesonautomaticallytestingwebapplicationsandwebservices,includingcost-effectiveapproachestogeneratingtestcasesanddeterminingthattheapplicationisbehavingappropriately.Sheisalsoexploringthechallengesindevelopingandtestingwebapplicationsfordigitalhumanitiesprojects.Shehasservedontheprogramcommitteesofinternationalconferences,suchastheInternationalConferenceonSoftwareTestingVerificationandValidation(ICST)andInternationalSymposiumonSoftwareReliabilityEngineering(ISSRE).SheisamemberoftheACMandtheIEEEComputerSociety.

1https://jquery.com/2https://angularjs.org/3http://getbootstrap.com/4https://developers.google.com/closure/templates/5http://papermachines.org/

6https://www.zotero.org/7Wecannotgetamorereadablefigure,eg,alargerfontforthelegendorgrayscalepatterns,fromthePaperMachinestool.Also,theorderofthebarsisnotthesameastheorderofthelegend.

CHAPTERF IVE

ApproachesandToolsforAutomatedEnd-to-EndWebTestingMaurizioLeotta*;DiegoClerissi*;FilippoRicca*;PaoloTonella†*DIBRIS,UniversitàdiGenova,Genova,Italy†FondazioneBrunoKessler,Trento,Italy

AbstractTheimportanceoftestautomationinwebengineeringcomesfromthewidespreaduseofwebapplicationsandtheassociateddemandforcodequality.Testautomationisconsideredcrucialfordeliveringthequalitylevelsexpectedbyusers,sinceitcansavealotoftimeintestingandithelpsdeveloperstoreleasewebapplicationswithfewerdefects.Themainadvantageoftestautomationcomesfromfast,unattendedexecutionofasetoftestsaftersomechangeshavebeenmadetoawebapplication.Moreover,modernwebapplicationsadoptamultitierarchitecturewheretheimplementationisscatteredacrossdifferentlayersandrunondifferentmachines.Forthisreason,end-to-endtestingtechniquesarerequiredtotesttheoverallbehaviorofwebapplications.

Inthelastyears,severalapproacheshavebeenproposedforautomatedend-to-endwebtestingandthechoiceamongthemdependsonanumberoffactors,includingthetoolsusedforwebtestingandthecostsassociatedwiththeiradoption.Theycanbeclassifiedusingtwomaincriteria:thefirstconcernshowtestcasesaredeveloped(ie,Capture-ReplayandProgrammableapproaches),while,thesecondconcernshowtestcaseslocalizethewebelementstointeractwith(ie,Coordinates-based,DOM-based,andVisualapproaches),thatiswhatkindoflocatorsareusedforselectingthetargetGUIcomponents.

Fordevelopersandprojectmanagersitisnoteasytoselectthemostsuitableautomatedend-to-endwebtestingapproachfortheirneedsamongtheexistingones.Thischapterprovidesacomprehensiveoverviewoftheautomatedend-to-endwebtestingapproachesandsummarizesthefindingsofalongtermresearchprojectaimedatempiricallyinvestigatingtheirstrengthsandweaknesses.

KeywordsWebtesting;Testautomation;Capture-replaywebtesting;Programmablewebtesting;DOM-basedwebtesting;Visualwebtesting;Pageobjectpattern;Robustlocators;Selenium;Sikuli

1IntroductionWebapplicationsarekeyassetsofoursociety.Aconsiderablesliceofmodernsoftwareconsistsofwebapplicationsexecutedintheuser’swebbrowser,runningoncomputersorsmartphones.Thewebhasasignificantimpactonallaspectsofoursocietyandinthelastyearshaschangedthelifeofbillionsofpeople.Associations,enterprizes,governmentalorganizations,companies,scientificgroupsusethewebasapowerfulandconvenientwaytopromoteactivities/productsandcarryouttheircorebusiness.Peopledailyuseonlineservicesassourceofinformation,meansofcommunication,sourceofentertainment,andvenueforcommerce.Inasentence,webapplicationspervadeourlife,beingcrucialforamultitudeofeconomic,socialandeducationalactivities.

Theimportanceofthewebinourlivesstressesthequalitywithwhichtheseapplicationsaredevelopedandmaintained[1].End-to-endwebtestingisoneofthemainapproachesforassuringthequalityofwebapplication[2].Thegoalofend-to-endwebtestingisexercisingthewebapplicationundertestasawholetodetectasmanyfailuresaspossible,whereafailurecanbeconsideredasadeviationfromtheexpectedbehavior.Inmanysoftwareprojects,end-to-endwebtestingisneglectedbecauseoftimeorcostconstraints.However,theimpactoffailuresinawebapplicationmaybeveryserious,rangingfromsimpleinconvenience(eg,malfunctionandsousers’dissatisfaction),economicproblems(eg,interruptionofbusiness),uptocatastrophicimpacts.

Thesimplestsolutionistomanuallyinteractwiththewebapplicationunderdevelopmenttoseeifitbehavesasexpected.Unfortunately,thispracticeiserrorprone,timeconsuming,andultimatelynotveryeffective.Forthisreason,mostteamsautomatemanualwebtestingbymeansofautomatedtestingtools.Theprocesscontainsafirstmanualstep:producingthetestcodeabletoinstrumentthewebapplication.Testcodeprovidesinputdata,operatesonGUIcomponents,andretrievesinformationtobecomparedwithoracles(eg,usingassertions).Themainbenefitoftestautomationcomesfromthefastandunattendedexecutionofatestsuiteaftersomechangeshavebeenmadetothewebapplicationundertest(ie,forregressionpurposes).

1.1ApproachestoAutomatedEnd-to-EndWebTestingEnd-to-endtestingofwebapplicationsisatypeofblackboxtestingbasedontheconceptoftestscenario,thatisasequenceofsteps/actionsperformedonthewebapplication(eg,insertusername,insertpassword,clicktheloginbutton).Oneormoretestcasescanbederivedfromasingletestscenariobyspecifyingtheactualdatatouseineachstep(eg,username=John.Doe)andtheexpectedresults(ie,definingtheassertions).Theexecutionofeachtestcasecanbeautomatedbyimplementingatestscriptfollowinganyoftheexistingapproaches(eg,ProgrammableandDOM-basedlocalization).Thechoiceamongthevariousapproachesdependsonanumberoffactors,includingthetechnologyusedbythewebapplicationsandthetools(ifany)usedforwebtesting.

Broadlyspeaking,therearetwomainorthogonalcriteria[3]toclassifytheapproachestoend-to-endwebtestingthatarerelatedto:(1)testscriptsimplementationand(2)web

pageelementslocalization.Fig.1showsaclassificationgridbasedonthesetwocriteriathatcanbeappliedtoexistingtools.Forwhatconcernsthefirstcriterion,wecanfindtwomainapproaches[4]:

FIGURE1 Orthogonalclassificationofend-to-endwebtestingapproachesandtools.

Capture-Replay(C&R)WebTestingconsistsofrecordingtheactionsperformedbythetesteronthewebapplicationGUIandgeneratingatestscriptthatrepeatssuchactionsforautomated,unattendedreexecution.

ProgrammableWebTestingaimsatunifyingwebtestingwithtraditionaltesting,wheretestscriptsarethemselvessoftwareartifactsthatdeveloperswrite,withthehelpofspecifictestingframeworks.Suchframeworksallowdeveloperstoprogramtheinteractionswithawebpageanditselements,sothattestscriptscan,forinstance,automaticallyfill-inandsubmitformsorclickonhyper-links.

Anautomatedend-to-endtestcaseinteractswithseveralwebpageelementssuchaslinks,buttons,andinputfields,anddifferentmethodscanbeusedtolocatethem.Thus,concerningthesecondcriterion,wecanfindthreedifferentcases[3]:

Coordinate-basedlocalization:thetoolsimplementingthisapproachjustrecordthescreencoordinatesofthewebpageelementsandthenusethisinformationtolocatetheelementsduringtestcasereplay.Thisapproachisnowadaysconsideredobsolete,becauseitproducestestscriptsthatareextremelyfragile.Hence,itisnotconsideredanyfurtherinthiswork.

DOM-basedlocalization:thetoolsimplementingthisapproach(eg,SeleniumIDE1andSeleniumWebDriver2)locatethewebpageelementsusingtheinformationcontainedintheDocumentObjectModel(DOM)and,usually,provideseveralwaystolocatewebpageelements.Forinstance,SeleniumWebDriverisabletolocateawebpageelementusing:(1)thevaluesofattributesid,name,andclass;(2)thetagnameoftheelement;(3)thetextstringshowninthehyperlink,foranchorelements;(4)CSSand(5)XPathexpressions.Notalltheselocatorsareapplicabletoanyarbitrarywebelement;eg,locator(1)canbeusedonlyifthetargetelementhasauniquevalueofattributeid,name,orclassintheentirewebpage;locator(2)canbeusedifthereisonlyoneelementwiththechosentagnameinthewholepage;and,locator(3)canbeusedonlyforlinksuniquelyidentifiedbytheirtextstring.Ontheotherhand,XPath/CSSexpressionscanalwaysbeused.Infact,asabaseline,theuniquepathfromroottotargetelementintheDOMtreecanalwaysbeturnedintoanXPath/CSSlocatorthatuniquelyidentifiestheelement.

Visuallocalization:thetoolsimplementingthisapproachhaveemergedrecently.TheymakeuseofimagerecognitiontechniquestoidentifyandcontrolGUIcomponents.The

toolSikuliIDE3andSikuliAPI4belongtothiscategory.

1.2ChapterContributionsFortestdevelopersandprojectmanagersitisnoteasytoselectthemostsuitableautomatedend-to-endwebtestingapproachfortheirneedsamongtheexistingones.Thischaptergivesacomprehensiveoverviewoftheautomatedend-to-endwebtestingapproachesandsummarizesthefindingsofalongtermresearchproject[3–6]aimedatempiricallyinvestigatingtheirstrengthsandweaknesses.

Themaincontributionofthischapteristwofold:

•Providingthereaderwithacompleteoverviewofthemostrelevantapproachesinthecontextoftheautomatedend-to-endwebtesting.

•Analyzingsuchapproachesinordertopointouttheirstrengthsandweaknesses.

Theconsideredapproacheshavebeenanalyzedbymeansofaseriesofempiricalstudies[3–4]inwhichwedevelopedandcomparedseveraltestsuitesforsixopensourcewebapplications.Inparticular:

(1)Concerningtestcasedevelopmentandmaintenance,weempiricallyinvestigatedthetrade-offbetweencapture-replay(C&R)andprogrammablewebtesting[4].Inadditiontovalidatingourinitialhypothesis,thatC&Rtestcasesarecheapertowritefromscratchthanprogrammabletestcases,butarealsomoreexpensivetomaintainduringsoftwareevolution,wehavedeterminedthenumberofsoftwarereleasesafterwhichthesavingsofprogrammabletestcasesovercomethecostsinitiallypaidfortheirdevelopment(seeFig.2).Wehaveinstantiatedourempiricalanalysisfortwostate-of-the-practicetools,SeleniumIDE(ie,C&R),andSeleniumWebDriver(ie,programmable).

FIGURE2 EvolutioncostsforC&Randprogrammabletestcases.

(2)Concerninghowtestcaseslocalizethewebelementtointeractwith,wehaveevaluatedandcomparedthevisualandDOM-basedapproaches[3]considering:therobustnessoflocators,theinitialtestsuitedevelopmenteffort,thetestsuiteevolutioncost,andthetestsuiteexecutiontime.Ourempiricalassessmentoftherobustnessoflocatorsisquitegeneralandtoolindependent,whilethedevelopers’effortforinitialtestsuitedevelopment

andtheeffortfortestsuiteevolutionweremeasuredwithreferencetospecificimplementationsofthetwoapproaches.Wehaveinstantiatedsuchanalysisfortwotools,SikuliAPIandSeleniumWebDriver,bothadoptingtheprogrammableapproachbutdifferinginthewaytheylocalizethewebelementstointeractwithduringtheexecutionofthetestcases.Indeed,SikuliAPIadoptsthevisualapproach,thususingimagesrepresentingportionsofthewebpages,whileSeleniumWebDriveremploystheDOM-basedapproach,thusrelyingontheHTMLstructure.Sincevisualtoolsareknowntobecomputationaldemanding,wealsomeasuredandcomparedthetestsuiteexecutiontime.

Thefindingsreportedinthischapterprovidepracticalguidelinesfordeveloperswhowanttomakeaninformeddecisionamongtheavailableapproachesandwhowanttounderstandwhichofthemcouldfitmoreorlesswellforaspecificwebdevelopmentcontext.

Thechapterisorganizedasfollows:Sections2and3provideanoverviewonthemainclassicalapproachestoautomatedend-to-endwebtestingandreportseveralexamplesoftoolsinstantiatingsuchapproaches.Specifically,thesesectionsdescribeshowthetestcasesdevelopmentapproaches(ie,capture-replayandprogrammable)canbecombinedwiththeDOM-basedandthevisuallocalizationapproaches.Section4describeshowtheevolutionofthewebapplicationimpactsonthetestcasescreatedbyfollowingeachapproach.Section5summarizesanddiscussestheresultsoftheempiricalstudiesweconductedtoanalyzethestrengthsandtheweaknessesofvariousapproachesforautomatedend-to-endwebtesting.Section6analysessometoolsandtechniquesthathavebeenrecentlyproposed,overcomingthelimitationsoftheclassicalapproachestoautomatedend-to-endwebtesting.InparticularSection6.1providessomeexampleoftools/techniquesthatgobeyondthesimpleadoptionofoneapproach,ie,solutionsthatareabletocombinemoreapproachesatthesametime(eg,Visual+DOMbasedorC&R+Programmable).Then,Section6.2analysesasetoftechniquesthathavebeenproposedintheliteratureinordertosolvespecificproblemsinthecontextofautomatedend-to-endwebtesting(robustness,testcaserepairuponsoftwareevolution,pageobjectcreation,andmigrationbetweenapproaches).Section7concludesthechapter.

2Capture-ReplayWebTestingCapture-Replaywebtestingisbasedontheusageofcapture/replayautomatedtools.Thesetoolshavebeendevelopedtovalidateinteractiveapplications(GUIorwebapplications).Usingacapture/replaytool,asoftwaretestercanrunawebapplicationandrecordtheentireinteractionsession.Thetool(semi-)automaticallyrecordsalltheuser’seventsinthenavigatedwebpages,suchasthekeypressesandmouseclicks,inascript,allowingasessiontobererunautomaticallywithoutfurtherhumaninteraction.Finally,thetestscriptiscompletedbyaddingoneormoreassertionstotherecordedactions.Capture/replaytoolssupportautomaticregressiontestingbyreplayingagiventestscriptonanewversionoftheWebApplicationUnderTest(WAUT).

Implementingtestcasesusingthisapproachisarelativelysimpletask.Evenapersonwithoutprogrammingexpertisecaneasilybuildacompletetestsuiteforacomplexwebapplication.However,thisapproachhasafewdrawbacks[4]:(1)testscriptsgeneratedusingtheC&Rapproachoftencontainhard-codedvalues(eg,testinputsandassertionvalues),thathavetobechangedifanythingvariesduringtheevolutionoftheWAUTorifthesametestcasehastobeexecutedusingdifferentinputvalues(eg,toexercisevariouscornercases);(2)testscriptsarestronglycoupledwithwebpages,withtheconsequencethatevenasmallchangeintheWAUT(eg,inthelayoutofthepage)leadstooneormorebrokentestcases(eg,testscriptsfailtolocatealink,aninputfieldorasubmissionbuttonbecauseofthelayoutchange)and,hence,(3)oftendifferenttestscriptscomposingatestsuitecontainalotofduplicatedcode.Indeed,whenthesamefunctionalityisinvokedwithinmultipletestcases(eg,login),thesamecodefragmentsarerepeatedacrossallsuchtestscripts.Wewillelaboratemoreontheseaspectsinthesecondpartofthechapter.

2.1DOM-BasedCapture-ReplayWebTestingAsseenbefore,DOM-basedtoolslocatethewebpageelementsusingtheinformationcontainedintheDocumentObjectModel.SeleniumIDEisthetoolforwebapplicationtestingadoptingtheDOM-basedCapture-Replayapproachthatweconsideredinourempiricalevaluation.SeleniumIDEisacompleteIDE(IntegratedDevelopmentEnvironment),assuggestedbyitsname,forthedevelopmentandexecutionofend-to-endwebtests.ItisimplementedasaFirefoxextension.Itprovidesseveralsmartfeaturessuchas:(1)testerscanrecord,edit,anddebugtestcaseswrittenintheSeleneselanguage5;(2)testerscanusesmartfieldselection(eg,usingIDs,names,orXPathlocators,asneeded)andcaninvokealocatorassistancefunction;(3)testerscansavetestcasesasHTML,orexportthemasJavacode(useful,asfirststep,forthemigrationtowardtheprogrammableapproach);and(4)assertionsonthevariouswebpageelementsareautomaticallysuggestedbythetooltotesters.

Letusassumethatwehavetotestaportionofawebapplicationusedtoauthenticateusers.Inaverysimplifiedcase,wehaveaportionofthehomepage(eg,calledhome.asp)thatallowsuserstoentertheircredentials,ie,usernameandpassword(seeFig.3).Whentheuserhasinsertedthecredentialsandhasclickedonthe“Login”button,

theapplicationevaluatesthecredentialscorrectness.Ifcredentialsarecorrect,theusername(eg,John.Doe),containedinaHTMLtagwiththeattributeID=“LoggedUser,”andthelogoutbuttonarereportedintheupperrightcornerofthehomepage.Otherwise,theloginformisstillshowninthehome.asppage.

FIGURE3 Fragmentofhome.asp—Pageandsource.

Asanexample,wereportatestcaseforthissimplefunctionalityimplementedusingthecapture/replayfacilityofSeleniumIDE(seeFig.4).ThetestscriptproducedbySeleniumIDEperformsavalidlogin,usingcorrectcredentials(ie,username=John.Doeandpassword=123456)andverifiesthatinthehomepagetheuserresultstobecorrectlyauthenticated(assertText,id=LoggedUser,John.Doe).ItcanbenoticedthatallwebelementsarelocatedusingthevaluesoftheidattributesthatcanbefoundintheDOM.Specifically,duringthetestscriptrecordingphase,SeleniumIDEisabletodetecttheactionsperformedonthewebpageelementsandtoautomaticallygeneratethelocatorsforsuchwebelements.SeleniumIDEcontainsalocatorgenerationalgorithmthatproduceslocatorsusingdifferentstrategies(implementedbytheso-calledlocatorbuilders)anditranksthemdependingonaninternalrobustnessheuristic.

FIGURE4 TestLogintestcaseinSeleniumIDE.

2.2VisualCapture-ReplayWebTestingSeveralCapture-Replaytoolsareavailabletoimplementtestcasesadoptingvisuallocators.AnexampleisSikuliIDE,astand-aloneopensourcetoolthatallowstoautomateeveryactionthatcanbeperformedonacomputerGUI(ie,itisnotdedicatedonlytowebapplications).SikuliIDEassiststhetesterintestcaseimplementationandallowsthetestertoautomaticallyreexecutethevarioustestcasesteps.Fig.5reportsaSikuliIDEtestscriptfortheloginfunctionalitydescribedbefore.TheSikuliIDEtestscriptperformsavalidlogin,usingcorrectcredentials(ie,insertstheusername,John.Doe,andthepassword,123456,intheinputfieldsmatchingtherespectivevisuallocators,andthenclickstheloginbutton)andverifiesthatinthehomepagetheuserresultstobecorrectlyauthenticated(ie,itverifiesthatthenameoftheuser,John.Doe,isshowninthehomepage).InSikuliIDE,testcaserecordingissemiautomatic:thetesterhastospecifywhatactionshewantstoperformontheuserinterface;creationofthevisuallocatorsisassistedbythetool(testersmustindicatetheportionofpageonwhichtheactionhastobeperformed).

FIGURE5 TestLogintestcaseinSikuliIDE.

3ProgrammableWebTestingProgrammablewebtestingisbasedonmanualcreationofatestscript.Webtestscriptscanbewrittenusingad-hoclanguagesandframeworksorgeneralpurposeprogramminglanguages(suchasJavaandRuby)withtheaidofspecificlibrariesabletoplaytheroleofthebrowser.Usually,theselibrariesextendtheprogramminglanguagewithuserfriendlyAPIs,providingcommandsto,eg,clickabutton,fillafield,andsubmitaform.Testscriptsarecompletedwithassertions(eg,JUnitassertionsifthelanguagechosenisJava).

Testscriptsbuiltinthiswayaremoreflexiblethantheonesbuiltusingcapture/replaytools[4].Forexample,programmabletestingallowsdeveloperstotakeadvantageofconditionalstatements,usedwhenmultipletestcaseshavedifferent,condition-dependentbehaviors.Theyalsosupportthecreationofdata-driventestcases,ie,testcasesthatareexecutedmultipletimes,eachtimewithdifferentarguments(ie,differentinputsandexpectedvalues).Thus,ingeneral,programmablewebtestingtechniquescanhandlethecomplexityofwebsoftwarebetterthanC&Rwebtestingtechniques.Ontheotherhand,theyrequirenontrivialprogrammingskillsandtheydemandforsubstantialinitialdevelopmenttime.

3.1ThePageObjectandFactoryPatternsThePageObject6,7patternisaquitepopularwebtestdesignpattern,whichaimsatimprovingthetestcasemaintainabilityandatreducingtheduplicationofcode.APageObjectisaclassthatrepresentsthewebpageelementsasaseriesofobjectsandthatencapsulatesthefeaturesofthewebpageintomethods.AdoptingthepageobjectpatternintestscriptimplementationallowstesterstofollowtheSeparationofConcernsdesignprinciple,sincethetestscenarioisdecoupledfromtheimplementation.Indeed,alltheimplementationdetailsaremovedintothepageobjects,abridgebetweenwebpagesandtestcases,withthelatteronlycontainingthetestlogics.Thus,allthefunctionalitiestointeractwithortomakeassertionsaboutawebpageareofferedinasingleplace,thePageObject,andcanbeeasilycalledandreusedwithinanytestcase.UsuallypageobjectsareinitializedbyaPageFactory,8afactoryclassthatchecksthecorrectmappingandinitializationofthewebelements.UsageofPageObjectandPageFactorypatternsreducesthecouplingbetweenwebpagesandtestcases,promotingreusability,readability,andmaintainabilityofthetestsuites[4,5].

3.2DOM-BasedProgrammableWebTestingInthiswork,weconsiderSeleniumWebDriverasarepresentativetoolforimplementingDOM-basedprogrammablewebtestsuites(forshort,WebDriver).WebDriverisastate-of-the-arttool,widelyusedforwebapplicationtesting[7].Itprovidesacomprehensiveprogramminginterfaceusedtocontrolthebrowser.WebDrivertestcasesarewrittenintheJavaprogramminglanguage,byintegratingWebDrivercommandswithJUnitorTestNGassertions.WechoseWebDriverasrepresentativeofDOM-basedprogrammabletools,because:(1)itisaquitematuretool,(2)itisopen-source,(3)itisoneofthemostwidely-

usedopen-sourcesolutionsforwebtestautomation,(4)duringourpreviousindustrialcollaborations,wegainedaconsiderableexperienceinitsusage[5,6].

InFig.6,weshowanexampleofasimpleWebDrivertestcaseforourrunningexampleapplication,correspondingtoasuccessfulauthentication.Thisautomatedtestcasesubmitsavalidlogin,usingcorrectcredentials(ie,username=John.Doeandpassword=123456)andverifiesthatinthehomepagetheuserappearsascorrectlyauthenticated(thestring“John.Doe”isdisplayedinthetop-rightcornerofthehomepage,asverifiedbymethodcheckLoggedUser).

FIGURE6 TestLogintestcaseinSeleniumWebDriver.

ThefirststepforbuildingthistestcaseiscreatingtheHomePage.javapageobject(seeFig.7),correspondingtothehome.aspwebpage.ThepageobjectHomePage.javaoffersamethodtologintotheapplication.Ittakesininputusernameandpassword,insertstheminthecorrespondinginputfieldsandclickstheLoginbutton.Moreover,HomePage.javacontainsalsoamethodthatverifiestheauthenticatedusernameintheapplication.AsshowninFig.7,webpageelementscanbelocatedusingdifferentkindsofDOM-basedlocators(eg,ID,LinkText,XPath).

FIGURE7 HomePagepageobjectinSeleniumWebDriver.

Thesecondsteprequirestodevelopthetestcasemakinguseofthepageobjectmethods(seeFig.6).Inthetestcase,first,aWebDriverobjectoftypeFirefoxDriveriscreatedtocontroltheFirefoxbrowserasarealuserdoes;second,WebDriver(ie,thebrowser)opensthespecifiedURLandcreatesapageobjectthatinstantiatesHomePage.java;third,usingmethodlogin(…),thetesttriestologinintheapplication;finally,thetestcaseassertionischecked.

3.3VisualProgrammableWebTestingThetestingtoolbelongingtotheVisualProgrammablecategorythatweconsideredinthisworkisSikuliAPI.SikuliAPIisanopen-sourcevisualtoolabletoautomateandtestgraphicaluserinterfacesusingscreenshotimages.Itprovidesimage-basedGUIautomationfunctionalitiestoJavaprogrammers.

Asanexample,theSikuliAPIversionofthetestLogintestcaseisshowninFig.8,whiletherelatedpageobjectisgiveninFig.9.ThetestcasedevelopedinSikuliAPIperformsthesameconceptualstepsastheWebDrivertestcase.Thefirstoperation,CommonPage.open(…),aimsatopeningthebrowserataspecifiedURL.Inapurelyvisualtestcase,thisinvolveslocatingandclickingontheFirefoxicononthedesktop,insertingtheURLintotheaddressbar,andthenclickingonthe“go”arrow(theseoperationsareencapsulatedintheclassCommonPage).

FIGURE8 TestLogintestcaseinSikuliAPI.

FIGURE9 HomePagepageobjectinSikuliAPI.

ThefollowingstepsarebasicallythesameinSikuliAPIandSeleniumWebDriver,theonlydifferencesbeingthatinSikuliAPIdriverisnotaparameteroftheHomePageconstructorandtheassertioncheckingmethoddoesnotneedanystringparameter.Onthecontrary,SikuliAPI’spageobjectisquitedifferentfromSeleniumWebDriver’s.AsshowninFig.9,thecommandlocateisinvokedtosearchfortheportionofawebpagethatlooksliketheimagerepresentingtherenderingofthewebelementtobelocated.Theimagemusthavebeenpreviouslysavedinthefilesystemasafileormustbeavailable

online.Oncethewebelementhasbeenlocated,aScreenRegionisreturnedbymethodlocate,whichcanbeusedtoperformoperationssuchasclickingandtypingintoit(see,eg,methodtypeinFig.9).

Thus,inSikuliAPIlocatorsareimages.WhileusingDOM-basedtoolsitispossibletoverifywhetheranHTMLelementcontainstextualinformation(seethelastlineinFig.7),withvisualtoolsitisnecessarytocheckthatthepagecontainsanimagedisplayingsuchtext(seeFig.9,methodcheckLoggedUser).Moreover,someusefulandquitegeneralSeleniumWebDrivermethodsarenotnativelyavailableinSikuliAPI(eg,click()andsendKeys()).Thus,whenusingSikuliAPI,theymustbeimplementedexplicitlyinthepageobjectclassasauxiliarymethods(eg,methodsclick()andtype()).

4TestCaseEvolutionThemainbenefitsofadoptingtestautomationarethepossibilityof:(1)executingthetestcasesmoreoften,(2)findingbugsintheearlystageofdevelopment,and(3)reusingtestcodeacrosssuccessivereleasesofthewebapplicationundertest(ie,forregressiontesting).Themaincostassociatedwithtestautomationisrelatedtothefragilityoftestcases:afragiletestisatestthatisbrokenwhenthewebapplicationundertestisslightlymodified.Specifically,whenawebapplicationevolvestoaccommodaterequirementchanges,bugfixes,orfunctionalityextensions,testcasesmaybecomebroken(eg,testcasesmaybeunabletolocatesomelinks,inputfields,andsubmissionbuttons),andsoftwaretestershavetorepairthem.Thisisatediousandexpensivetasksinceithastobeperformedmanuallybysoftwaretesters(automaticevolutionoftestsuitesisaresearchtopicunderactiveinvestigation[8]).

Dependingonthekindofmaintenancetaskthathasbeenperformedonthetargetwebapplication,asoftwaretesterhastoexecuteaseriesoftestcaserepairactivitiesthatcanbecategorizedintotwotypes:logicalandstructural.

LogicalChangesinvolvethemodificationofthewebapplicationfunctionality.Torepairthetestcases,thetesterhastomodifyoneormorestepsofthebrokentestcasesand,whenadoptingtheprogrammableapproach,thecorrespondingpageobjectsmayalsoneedtobemodifiedornewonesneedtobecreatedaccordingly.Anexampleofachangerequest(CR1)thatneedsalogicalrepairactivityisenforcingthesecuritybymeansofstrongerauthenticationandthusaddinganewwebpage,containinganadditionalquestionthatisdisplayedtotheuserwhensheclicksontheloginbuttonofpagehome.asp,showninFig.3.

StructuralChangesinvolvethemodificationofthewebpagelayout/structureonly.Forinstance,inthewebpageofFig.3thestringoftheloginbuttonmaybechangedtoSubmit(CR2)ortheid=“UID”maybechangedtoid=“UserID”(CR3).Usually,theimpactofastructuralchangeissmallerthanalogicalchange.Torepairthetestcasesafterastructuralchange,itisoftenenoughtomodifyoneormorelocalizationlines,ie,linescontaininglocators.

Thestrategyusedbyasoftwaretestertorepairatestcasedependsmainlyontwofactors:(1)thetoolusedtobuildthetestcases(C&R,likeSeleniumIDE,orprogrammable,likeSeleniumWebDriver)and(2)thekindofchange(logicalorstructural).

C&RApproach+logicalchange.Thetesterkeepstheportionofscriptuptothecommandthatprecedesthebrokenactioncommand,deletestherest,andcapturesthenewexecutionscenariobystartingfromthelastworkingcommand.Forinstance,inthecaseofaSeleniumIDEtestscript,if(CR1)isimplemented,theassertionshowninFig.4willfailandthetesterwillhavetodeleteit.Then,thetesterhastocompletethetestscriptstartingfromthecommandclickAndWait,id=loginandcapturingthenewscenario,whichincludesthenewwebpageprovidingtheadditionalauthenticationquestion.SimilarchangeshavetobeimplementedinthecaseoftheSikuliIDEtest

scriptshowninFig.5,inparticularbyremovingline6andrecordingthenewadditionalsteps.

C&RApproach+structuralchange.Thetestermodifiesthelocatorsortheassertionvaluesusedinthetestscript.InthecaseofSeleniumIDE,sherunsthetestscriptandfindsthefirstbrokencommand(ie,theSelenesecommandthatishighlightedinredaftertestcaseexecution),whichcanbeanactioncommand(eg,typeorclick)oranassertion.Atthispoint,thetesterrepairsthebrokencommandandthenreexecutesthetestscript,possiblyfindingthenextbrokencommand(ifany).Forexample,if(CR3)isimplementedthenthetestscriptshowninFig.4needstoberepaired.ThetesterhastoreplaceUIDwithUserIDinthecommandusedtoinserttheusernameintheinputfield.TherepairprocessissimilarinthecaseofSikuliIDE.ItisinterestingtonotethatastructuralchangecanaffectdifferentlyDOM-basedandvisualtestscripts.Indeed,incase(CR3)isimplemented,nomodificationsarerequiredtotheSikuliIDEtestscriptshowninFig.5,while(CR2)requirestomodifybothSeleniumIDEandSikuliIDEtestscripts.

ProgrammableApproach+logicalchange.Dependingonthemagnitudeoftheexecutedmaintenancetask,thetesterhastomodifythebrokentestcasesand/orthecorrespondingpageobjects.Insomecases,newpageobjectshavetobecreated.Forexample,if(CR1)isimplementedthenthetesterhastocreateanewpageobjectforthewebpageprovidingtheadditionalauthenticationquestion.Moreover,shehastorepairthetestLogintestcaseinFig.6(andsimilarlytheoneshowninFig.8),addinganewJavastatementthatcallsthemethodofferedbythenewpageobject.

ProgrammableApproach+structuralchange.Thetestermodifiesoneormorepageobjectsthatthebrokentestcaselinksto.Forexample,inthecaseofSeleniumWebDriver,if(CR2)isimplemented,thetesterhastorepairtheline:@FindBy(linkText=“Login”)intheHomePage.javapageobject(seeFig.7).Similarly,inthecaseofSikuliAPI,thetesterhastoupdatetheimagelogin.pngintheHomePage.javapageobject(seeFig.9).

5AnalysisoftheApproachesThissectionsummarizesaseriesofempiricalstudies[3–5]weconductedtoanalyzethestrengthsandtheweaknessesofvariousapproachestoautomatedend-to-endwebtesting.Weanalyzedthefourmainapproachescurrentlyadoptedinthecontextofautomatedfunctionalwebtesting.Inparticular,forwhatconcernsthetestscriptsimplementation,wecomparedC&RandProgrammableapproaches,whileconcerningthewebpageelementslocalizationwecomparedVisualandDOM-basedapproaches.

Theresultsofthesestudiesareinterpretedaccordingtotwoperspectives:(1)projectmanagers,interestedinunderstandingwhichapproachcouldleadtopotentiallylowercostsandcouldmaximizethereturnoftheinvestment;(2)researchers,interestedinempiricaldataabouttheimpactofdifferentapproachesonwebtesting.

Theexperimentshavebeenconductedonasampleofsixopen-sourcewebapplicationsfromSourceForge.net(seeTable1).MoredetailsontheemployedwebapplicationscanbefoundinourICWEpaper[3].Weconsideredonlyapplicationsthat:(1)arequiterecent,sothattheycanworkwithoutproblemsonthelatestreleasesofApache,PHP,andMySQL,technologieswearefamiliarwith(sincetheanalyzed/usedtechniquesandtoolsoperateontheHTMLcodeprocessedbytheclientbrowser,theserversidetechnologiesdonotaffecttheresultsofthevariousstudies);(2)arewellknownandused(someofthemhavebeendownloadedmorethan100,000timeslastyear);(3)haveatleasttwomajorreleases(wehaveexcludedminorreleasesbecausewithsmalldifferencesbetweenreleasesthemajorityofthelocators—and,thus,ofthecorrespondingtestcases—areexpectedtoworkwithoutproblems);(4)belongtodifferentapplicationdomains.

Table1WebApplicationsfromSourceForge.net

Description webSiteMantisBT Bugtrackingsystem sourceforge.net/projects/mantisbt/PPMA Passwordmanager sourceforge.net/projects/ppma/Claroline Collaborativelearningenvironment sourceforge.net/projects/claroline/AddressBook Address/phonebook,contactmanager sourceforge.net/projects/php-addressbook/MRBS Meetingroomsmultisitebookingsystem sourceforge.net/projects/mrbs/Collabtive Collaborationsoftware sourceforge.net/projects/collabtive/

5.1ExperimentalProcedureOverall,theempiricalevaluationhasbeenperformedasfollows(furtherdetailsoneachexperimentcanbefoundinourpreviousworks[3,4]):

(1)Sixopen-sourcewebapplicationshavebeenselectedfromSourceForge.netasexplainedbefore.

(2)Foreachselectedapplication,fourequivalenttestsuiteshavebeenbuilt.Weimplementedthemchoosingatoolinstantiatingeachcombinationoftheconsideredapproaches:SikuliAPI(ie,Programmable+Visual),SeleniumWebDriver(ie,Programmable+DOMbased),SeleniumIDE(C&R+DOMbased)andSikuliIDE(C&R

+Visual).TheDOM-basedtestsuitesweredevelopedforourfirstwork[4],theSikuliAPItestsuitesforthefollowingwork[3],whiletheSikuliIDEtestsuiteshavebeenspecificallybuiltforthiswork.

Allthetestsuiteshavebeendevelopedfollowingwell-knownbestpractices.Forinstance,regardingtheSeleniumWebDriverandSikuliAPItestsuites(programmableapproach)thepageobjectpatternwasusedand,concerningtheSeleniumIDEandWebDrivertestsuites(DOM-basedlocalization)IDlocatorswerepreferredwheneverpossible(ie,whenHTMLtagsareprovidedwithIDs),otherwiseName,LinkText,CSS,andXPathlocatorswereused.

Foreachtestsuite,wemeasuredthenumberoflocatorsproducedandthedevelopmenteffortfortheimplementationasclocktime.Eachtestsuiteisequivalenttotheothersbecausetheincludedtestcasestestexactlythesamefunctionalities,usingthesamesequencesofactions(eg,locatingthesamewebpageelements)withthesameinputdataandoracle.

(3)Eachtestsuitehasbeenexecutedagainstthesecondreleaseofthewebapplication.First,werecordedthefailedtestcases.Wealsocheckedthatnorealregressionbugswerefoundandthatallthefailureswereduetobrokenlocatorsortomodificationstothetestcaselogics.Then,inasecondphase,werepairedthebrokentestcases.Wemeasuredthenumberofbrokenlocatorsandtherepaireffortasclocktime.Finally,forcomparingtheefficiencyofthevariouslocalizationtechniques,weexecuted10times(toaverageoveranyrandomfluctuationoftheexecutiontime)eachSikuliAPIandSeleniumWebDrivertestsuiteandrecordedtheexecutiontimes.

5.2TestSuiteDevelopmentCostsWehaveanalyzedthedevelopmenteffortassociatedwiththevariousapproaches.RegardinghowtestcasesaredevelopedwefoundthatC&Rtestsuites(ie,SeleniumIDEandSikuliIDE)requiredconsistentlylessdevelopmenttimethanprogrammabletestsuites(ie,SeleniumWebDriverandSikuliAPI),seeFig.10.FocusingforinstanceonthetwoSeleniumtools,wenoticedthatinallsixcases,thedevelopmentoftheSeleniumWebDrivertestsuitesrequiredmoretimethantheSeleniumIDEtestsuites.

FIGURE10 Overalltestsuitesdevelopmenttime(minutes).

Varyingthelocalizationapproachalsoinfluencesthetestsuitedevelopmenttime.Fig.10clearlyshowsthatDOM-basedtestsuitesrequirelesstimefortheirdevelopment.Focusingonthetwotoolsadoptingtheprogrammableapproach(ie,SeleniumWebDriverandSikuliAPI),wefoundthatinallsixcases,developmentoftheWebDrivertestsuitesrequiredlesstimethantheSikulitestsuites(withareductionbetween22%and57%).

Summary:EmployingC&RtoolsandadoptingDOM-basedlocatorscontributetoreducetheoveralldevelopmenttime.

5.3TestSuitesEvolutionCostsThen,weanalyzedtheevolutioneffortassociatedwiththevariousapproaches.Forwhatconcernsthetoolclassificationcriterion“howtestcasesaredeveloped”,wefoundthatC&Rtestsuites(ie,SeleniumIDEandSikuliIDE)requiredconsistentlymoreevolutiontimethanprogrammabletestsuites,asshowninFig.11.FocusingforinstanceonthetwotoolspartoftheSeleniumframework,wenoticedthatforalltheapplications:(1)therepairtimeoftheSeleniumIDEtestsuiteswaslongerthantherepairtimeoftheSeleniumWebDrivertestsuites;and,(2)thenumberofrepairedSeleniumIDEtestcasesisgreaterorequaltothenumberofrepairedSeleniumWebDrivertestcases.TheevolutionoftheSeleniumIDEtestsuitesrequiredfrom19%moretimeto104%moretime.

FIGURE11 Overalltestsuitesevolutiontime(minutes).

Adoptingadifferentlocalizationapproachalsoinfluencesthetestsuiteevolution.Fig.11showsthatDOM-basedtestsuitesrequirelesstimefortheirevolution.Focusingonthetwotoolsimplementingtheprogrammableapproachwefoundthatresultsdependontherespectiverobustnessofthetwokindsoflocators(DOMbasedvsVisual)employedbythetwotoolsandthusfollowthesametrend:infourcasesoutofsix,repairingtheSeleniumWebDrivertestsuitesrequiredlesstime(from33%to57%less)thanrepairingSikuliAPItestsuites,inonecaseslightlymore.Injustonecase(ie,Collabtive)SeleniumWebDriverrequiredsubstantially(10×)moreeffortthanSikuliAPI.

Summary:Employingaprogrammabletoolcontributestoreducetheevolutioncostsforrepairingautomatedend-to-endtestsuites.Concerningthewebelementlocalizationapproach,theDOM-basedapproachcontributedtoreducetheevolutioncostsinmostcases.

5.4OverallDevelopmentandEvolutionCostsIntheprevioussections,wehaveanalyzedthecostsfor(1)developingand(2)maintainingautomatedend-to-endtestsuitesseparately.Concerningthewebelementlocalizationapproach,theDOM-basedapproachguarantees,inmostofthecases,lowercostsduringbothtestsuitesdevelopmentandevolution.Ontheotherhand,concerninghowtestsuitesaredevelopedandevolved,wefoundthatadoptingtheC&Rapproachispreferableduringinitialtestsuitesdevelopment,whiletheprogrammableoneispreferableduringtestsuitesevolution.

Thusaninterestingquestionforthepractitioneris:“WhendoprogrammabletestsuitesbecomeconvenientwithrespecttotheC&Rones?”.

BycomparingthetwotoolspartoftheSeleniumframework,wecannoticethatinmost

ofthecasesthecumulativecostofinitialdevelopmentandevolutionofprogrammabletestcases(ie,SeleniumWebDrivertestsuites)islowerthanthatofC&Rtestcases(ie,SeleniumIDEtestsuites)afterasmallnumberofreleases(moreprecisely,between1and3releases).

WeestimatedthatprogrammabletestcasesaremoreexpensivetowritefromscratchthanC&Rtestcases,withamedianratiobetweenthetwocostsequalto1.58.Duringsoftwareevolution,testsuiterepairissubstantiallycheaperforprogrammabletestcasesthanforC&Rtestcases,withamedianratioequalto0.65.Suchcost/benefittrade-offbecomesfavorabletotheprogrammabletestsuitesafterasmallnumberofreleasesthemedianofwhichis1.94.Themostimportantpracticalimplicationoftheseresultsisthatforanysoftwareprojectwhichisexpectedtodelivertwoormorereleasesovertime,programmabletestcasesofferanadvantageousreturnoftheinitialinvestment.Infact,aftertwoormorereleases,theevolutionofthetestsuiteswillbeeasierandwillrequirelesseffortifaprogrammableapproach(suchasWebDriver)isadopted.However,specificfeaturesofagivenwebapplicationmightmakethetrade-offmoreorlessfavorabletoprogrammabletests.Inparticular,thepossibilitytocapturereusableabstractionsinpageobjectsplaysamajorroleinreducingthetestevolutioneffortforprogrammabletestcases.Inthefollowing,weanalyzeeachfactorthatmightaffectthetrade-offbetweenC&Randprogrammabletestcases.

Summary:Accordingtoourestimate,aftertwomajorreleases,programmabletestcasesbecomemoreconvenientthanC&Rones.Ofcourse,theactualbenefitsmaydependonspecificfeaturesofthewebapplicationundertest.

5.5Capture&ReplayvsProgrammableApproachesInthefollowing,wereportthereasonsandimplicationsoftheresultsreportedintheprevioussections,inparticularfocusingonthecomparisonbetweenC&Randprogrammableapproaches,consideringrespectivelythetwotoolsthatarepartoftheSeleniumframework(SeleniumIDEandSeleniumWebDriver).

5.5.1NumberofPageObjectsPerTestCaseWeobservedthatthenumberofpageobjectspertestcasevariesconsiderablyamongtheconsideredapplications(from0.20to0.73pageobjectspertestcase).Thisnumbergivesanindicationofthedegreeofreusethatpageobjectshaveacrosstestcases.Ahigherreuseisassociatedwithalowermaintenanceeffort,sincereusedpageobjectswillbemaintainedonlyonceforalltheirclients(seetheexample—ontheright—inFig.12).Thevariabilityobservedfortheobjectsusedinourstudyisduetodifferentcharacteristicsoftheseapplications.

FIGURE12 Relationbetweentestcasesandpageobjects.

Thedegreeofreusemayamplifyorreducethebenefitsofadoptingthepageobjectpatternandthentheprogrammableapproach.

Forinstance,inthecaseofMantisBT(numberofpageobjectspertestcasesequalto0.73)wehavethat14pageobjects(among17)areusedbyonlyoneortwotestcasesthathavebeenrepaired.Inthiscase,wehavefewadvantagesintermsofmaintenanceeffortreductionfromadoptingthepageobjectpattern,sinceeachrepairactivityonapageobject,donejustonce,affectsonlyoneoratmosttwotestcases.Ontheotherhand,inourstudywefoundthreeapplicationsoutofsixthathaveanumberofpageobjectspertestcasesabout0.25orlower.Thus,inthesecases,thepotentialadvantagesofadoptingthePOpatternarehigher.

Itisinterestingtonotethat,atthebeginningofthetestsuitedevelopment,alotofpageobjectshavetobecreated.Forinstance,thefirsttestcasecouldrequiretocreateevenfourorfivepageobjects.Butusually,asnewtestcasesareaddedtotheexistingtestsuite,thenumberofnewpageobjectsthatthetesterhastocreatedecreases.Indeed,thetesterhastocreateapageobjectforeachlogicalpageofthewebapplication(eg,login,home,userdetails),whilehecouldpotentiallydevelopatestforeachpaththatcouldbefollowedtoreachaspecificpage.Thus,probably,comprehensivetestsuites(ie,testingthewebapplicationindepth)benefitmoreofthepageobjectpatternsincethelevelofpageobjectsreuseishigher.

Summary:Thewebpagemodularityofthewebapplicationundertestaffectsthebenefitsofprogrammabletestcases.Webapplicationswithwellmodularizedfunctionalities,implementedthroughreusablewebpages,areassociatedwithreusablepagesobjectsthataremaintainedjustonceduringsoftwareevolution.

5.5.2NumberofTestCasesRepairedInfivecasesoutofsix,thenumberofrepairedtestcasesislowerwhenSeleniumWebDriverisusedinsteadofSeleniumIDE.Atfirstsight,thisresultcouldappearstrangesinceeachpairoftestsuites(IDEandWebDriver)hasbeendevelopedequivalently,using

thesamelocators.Actually,thenumberofbrokentestcasesisthesameforeachpairoftestsuites,butthenumberofrepairedtestcasesislowerwithSeleniumWebDriverbecauseoftheadoptionofthepageobjectpattern.Withthepageobjectpatterneachofferedmethodcanbereusedmoretimesinatestsuite.Thus,achangeatthelevelofthepageobjectcanrepairmorethanonetestcaseatonce(seetheexampleinFig.13).Clearly,thereductionofthenumberofrepairedtestcasesisrelatedwiththenumberoftimesamethodinapageobjectis(re-)used.

FIGURE13 Pageobjectadoptionandrepaireffort.

Letusconsideraspecificexample.InClaroline,betweenthetwoconsideredreleasesamodificationofthepartoftheapplicationmanagingtheloginprocessoccurred.Sincethismodificationinvolvedalsotheattributeusedtolocatetheusercredentialssubmissionbutton,allthetestcaseswereimpacted(sinceallofthemstartwiththeauthentication).IntheSeleniumWebDrivertestsuitewerepairedonlythepageobjectofferingmethodDesktopPagelogin(Stringuser,Stringpass).Inthisway,weautomaticallyresolvedtheproblemfortheentiretestsuite.Onthecontrary,intheSeleniumIDEtestsuite,wehadtomodifyallthetestcases(ie,40testcases).

Summary:Pageobjectsreusereducesdramaticallythetestcasesrepaireffort.

5.5.3NumberofLocatorsGlobally,inthesixtestsuites,wehaveapproximatelythesamenumberofmodificationsmadetoaddresslogicalchanges(ie,108and122,respectively,inSeleniumIDEandWebDriver),butwecanobserveahugedifferenceintermsofmodifiedlocatorstorepair

thebrokentestcasesduetostructuralchanges(respectively727outof2735locatorschangedwithIDEvs162outof487locatorschangedwithWebDriver).

Summary:Adoptingthepageobjectpatternavoidstheduplicationoflocatorsaswellastheneedfortheirrepeated,consistentevolution.

Notethatforeachtargetwebelement,thelocatorsusedbySeleniumIDEandWebDriverareexactlythesameaswellastheirrobustness.ThustheproblemoftheC&Rapproachisonlyduetothelocatorsduplication.

5.5.4OtherAdvantagesoftheProgrammableApproachAsalreadymentioned,aprogrammabletoollikeSeleniumWebDriverusuallyoffersacomprehensiveprogramminginterfaceandsoahigherflexibilityascomparedtoastandardC&RasSeleniumIDE.Forexample,SeleniumWebDriverallowsdeveloperstocreatetestcasesenrichedwithfunctionalitiesthatnativelySeleniumIDEdoesnotprovide,suchas:conditionalstatements,loops,logging,exceptionhandling,andparameterizedtestcases.Inourtestsuiteswehavenotusedthesefeaturestokeepasmuchaspossiblefairthecomparisonbetweenthetwoapproachesandtoobtainequivalenttestsuites,butaprojectmanagershoulddefinitelyconsideralsothesefeatureswhenselectingbetweenthetwotestingapproaches.Inapreviouswork[5],wefoundthatthesefeaturesareindeedquiteusefulinindustrialprojects.

Summary:Additionalbenefitsofprogrammabletestcases(eg,parametrictestcases)shouldbetakenintoaccountwhenchoosingbetweenprogrammableandC&Rwebtesting.

5.6VisualvsDOM-BasedApproachesInthefollowingwereportthereasonsandimplicationsoftheresultsreportedintheprevioussections,inparticularfocusingonthecomparisonbetweenvisualandDOM-basedapproaches,consideringrespectivelythetwotoolsSikuliAPIandSeleniumWebDriver.

5.6.1NumberofLocatorsTheadoptionofdifferentlocalizationapproachescanpotentiallyinfluencethenumberoflocatorsthathavetobecreated.Inourexperiments,thenumberoflocatorsrequiredtobuildthesixtestsuitesvariesfrom81to158whenadoptingthevisualapproachandfrom42to126whenadoptingtheDOM-basedone.Forallsixapplicationsthevisualapproachhasalwaysrequiredahighernumberoflocators.Consideringthedatainaggregateform,wecreated45%moreVisuallocatorsthanDOMlocators(706visualvs487DOM-based

locators).Thevisualapproachrequiresmorelocatorsinthefollowingsituations:(1)webelementschangingtheirstate,(2)elementswithcomplexinteraction,and(3)data-driventestcases.

WebElementsChangingtheirState.Whenawebelementchangesitsstate(eg,acheckboxischeckedorunchecked,seetheexampleinFig.14),avisuallocatormustbecreatedforeachstate,whilewiththeDOM-basedapproachonlyonelocatorisrequired.Thisoccurredduringthedevelopmentofallthevisualtestsuites(eg,SikuliAPI)anditisoneofthereasonswhy,inallofthem,wehavemorelocatorsthanintheDOM-basedtestsuites(eg,SeleniumWebDriver).Asaconsequence,moreeffortbothduringthedevelopmentandmaintenanceisrequiredtocreatethevisualtestsuites(quiteoftenmorethanonelocatorhadtobecreatedandlaterrepairedforeachwebelement).

FIGURE14 Statechanges:VisualvsDOM-basedlocalization.

WebElementswithComplexInteraction.Complexwebelements,suchasdrop-downlistsandmultileveldrop-downmenus,arequitecommoninmodernwebapplications.Forinstance,letusconsideraformthataskstoselectthemanufacturerofthecar(seeFig.15).Typically,thisisimplementedusingadrop-downlistcontainingalistofmanufacturers.ADOM-basedtoollikeSeleniumWebDrivercanprovideacommandtoselectdirectlyanelementfromthedrop-downlist(intheexampleonlyoneID-basedlocatorisrequired).Onthecontrary,whenadoptingthevisualapproachthetaskismuchmorecomplex.Onecould,forinstance:(1)locatethedrop-downlist(morepreciselythearrowthatshowsthemenu)usinganimagelocator;(2)clickonit;(3)iftherequiredlistelementisnotshown,locateandmovethescrollbar(eg,byclickingthearrow);(4)locatetherequiredelementusinganotherimagelocator;and,finally,(5)clickonit.AllthesestepstogetherrequiremoreLOCsinthepageobjectsandlocators.Actually,inthiscasethevisualapproachperformsexactlythesamestepsthatahumantesterwoulddo.

FIGURE15 Complexinteraction:VisualvsDOM-basedlocalization.

Data-drivenTestCases.Oftenintheindustrialpractice[5],toimprovethecoveragereachedbyatestsuite,testcasesarereexecutedmultipletimesusingdifferentvalues.Thisisverywellsupportedbyaprogrammabletestingapproach.However,benefitsdependonthespecificprogrammableapproachthatisadopted(eg,visualvsDOM-based).Forinstance,inSeleniumWebDriveritispossibletousedatafromvarioussources,suchasCSVfilesordatabases,oreventogeneratethematruntime.InSikuliitisnecessarytohaveimagesofthetargetwebelements,soevenifwecanusevariousdatasources(eg,tofillinputfields),whenassertionsareevaluated,imagesarestillneededtorepresenttheexpecteddata.Forthisreason,inthevisualapproachitisnotpossibletocreatecompletedata-driventestcases(ie,includingbothdatadriveninputsandassertions).ThishappensbecauseintheDOM-basedapproachthereisaclearseparationbetweenthelocatorforawebelement(eg,anIDvalue)andthecontentofthatwebelement(eg,thedisplayedstring),sothatwecanreusethesamelocatorwithdifferentcontents(eg,testassertionvalues).Onthecontrary,usingavisualtool,thelocatorforawebelementandthedisplayedcontentarethesamething,thusifthecontentchanges,thelocatormustbealsomodified.Moreover,itisimportanttohighlightthat,ifnecessary,parameterizingthecreationofDOM-basedlocatorsisusuallyaneasytask(eg,.//*[@id=‘list’]/tr[X]/td[1]withX=1..n),whilethisisnotthecasewithvisuallocators.Inourcasestudy,weexperiencedthislimitationofthevisualapproach,sincewehad,

ineachtestsuite,atleastonetestcasethatperformsmultiple,repeatedoperationsthatdifferonlyinthedatavaluesbeingmanipulated,suchas:insert/removemultipledifferentusers,projects,addresses,orgroups(dependingontheconsideredapplication).Insuchcasesweused:(1)asingleparameterizedlocatorinSeleniumWebDriver,and(2)severaldifferentimagelocatorsinSikuliAPI(eg,forevaluatingtheassertions),withtheeffectthat,inthesecondcase,thenumberoflocatorsrequiredissubstantiallyhigher.

Summary:AdoptingtheVisualapproachrequirestogeneratemorelocatorsthanwiththeDOM-basedapproach.Thishappensinthefollowingsituations:(1)webelementschangingtheirstate,(2)webelementswithcomplexinteraction,and(3)data-driventestcases.

5.6.2LocatorsRobustnessThelocalizationmethodaffectstherobustnessofthelocators.Inourexperiments,weobservedthatinthemajorityofthecases(fouroutofsix),DOM-basedlocatorsaremorerobustthanvisuallocators.However,theresultisnotclear-cut.Generally,DOM-basedlocatorsaremorerobustbutincertaincases,dependingonthekindofmodificationsbetweenconsecutivereleases,visuallocatorsprovedtobemorerobust.Whentherenderingoftheuserinterfacechangesacrossreleases,visuallocatorsaremorelikelytobebroken,whileinothercasestheoppositemayhappen,thatis,visualrenderingremainsstablewhiletheunderlyingDOMchanges.Inthefollowing,weanalyzethelattercase,whichwename“ChangesbehindtheScene.”

Summary:DOM-basedlocatorsprovedtobeingeneralslightlymorerobustthanVisuallocators.However,muchdependsonthespecificcharacteristicsofthewebapplication.

ChangesbehindtheScene.SometimestheHTMLcodeismodifiedwithoutanyperceivableimpactonhowthewebapplicationappears.Anextremeexampleischangingthelayoutofawebapplicationfromthe“deprecated”table-basedstructuretoadiv-basedstructure,withoutaffectingitsvisualaspectinanyrespect.Inthiscase,thevastmajorityoftheDOM-basedlocators(inparticularthenavigationalones,eg,XPath)usedbyDOM-basedtoolsmaybebroken.Onthecontrary,thischangeisalmostinsignificantforvisualtesttools.AsimilarproblemoccurswhenautogeneratedIDlocators(eg,id1,id2,id3,…,idN)areusedinDOM-basedlocators.Infact,thesetendtochangeacrossdifferentreleases,whileleavingcompletelyunaffectedthevisualappearanceofthewebpage(hence,nomaintenanceisrequiredonthevisualtestsuites).Forexample,theadditionofanewlinkinawebpagemightresultinachangeofallIDsoftheelementsfollowingthenewlink.Such“changesbehindthescene”occurredinourempiricalstudyandexplainwhy,inthecaseofCollabtive,theSikulitestsuitehasrequiredbyfaralowermaintenanceeffort.Indetail,acrossthetwoconsideredreleases,aminorchangehasbeenappliedtoalmostalltheHTMLpagesofCollabtive:anunuseddivtaghasbeenremoved.ThislittlechangeimpactedquitestronglyseveraloftheXPathlocators(XPathlocatorswereusedbecauseIDswerenotpresent)intheWebDrivertestsuite.Themajorityofthe36locators(allofthemareXPaths)wasbrokenandhadtoberepaired(anexampleofrepairisfrom…/div[2]/…to…/div[1]/…).NochangewasnecessaryontheSikulivisualtestsuiteforthisstructuralchange.Overall,inSikuli,wehadonlyfewlocatorsbroken.Forthisreason,

thereisalargedifferenceinthemaintenanceeffortbetweenthetwotestsuites.AsimilarchangeacrossreleasesoccurredalsoinMantisBT,althoughithadalowerimpactinthisapplication.

5.6.3ExecutionTimeTheexecutiontimerequiredwhenadoptingtheVisualapproachwasalwayshigherthanthetimerequiredwhenadoptingtheDOM-basedapproach.Thisisexpected,sinceexecutinganimagerecognitionalgorithmrequiresmorecomputationalresources(andthus,generally,moretime)thannavigatingtheDOM.However,surprisingly,thedifferenceinpercentagebetweenthetwoapproachesisnothigh,beingonly30–48%.Itisnotsomuch,consideringthat:(1)Sikuliisaquiteexperimentaltool,(2)itisnotfocusedonwebapplicationtestingand,(3)manualmanagementofthepageloadingdelay(achievedthroughsleepcommands)isnotoptimal.Thereasonforsuchadelayisthatabrowserneedstimetoopenawebpage.Thus,beforestartingtoperformanyactiononthepagethetestautomationtoolhastowait.SeleniumWebDriverprovidesspecificcommandstodealwiththisproblem(ie,waitingforthewebpageloading).InSikuliAPIthisisnotavailableandtestershavetoinsertanexplicitdelay(eg,Thread.sleep(200)).However,accordingtoourestimates,theoverheadduetothewebpageloadingdelayisnotamajorpenaltyforSikuliAPI(only20–40spertestsuite)ascomparedtothetotalprocessingtime.Indeed,wecarefullytunedthedelaysinordertofindthesmallestrequired.

Summary:DOM-basedtestsuitesrequirelesstimetocompletetheirexecutionw.r.t.visualtestsuites.Howeverthedifferenceisnotsohigh.

5.6.4RepeatedWebElementsWheninawebpagetherearemultipleinstancesofthesamekindofwebelement(eg,aninputbox),creatingavisuallocatorrequiresmoretimethancreatingaDOM-basedlocator.Letusconsideracommonsituation,consistingofaformwithmultiple,repeatedinputfieldstofill(eg,multiplelines,eachwithName,Surname),allofwhichhavethesamesize,thusappearingidentical.Insuchcases,itisnotpossibletocreateavisuallocatorusingonlyanimageofthewebelementofinterest(eg,therepeatedNameinputfield),butwehaveto:(i)includealsosomecontextaroundit(eg,alabel)inordertocreateanunambiguouslocator,(ie,animagethatmatchesonlyonespecificportionofthewebpage),asintheexampleinFig.16,or,whenthisisnoteasilyfeasible,(ii)locatedirectlyauniquewebelementclosetotheinputfieldofinterestandthenmovethemouseofacertainamountofpixels,inordertoreachtheinputfield.Bothsolutionslocatethetargetwebelementbymeansofanother,easiertolocate,element(eg,alabel).Thisisnotstraightforwardandnaturalforthetestdeveloper(ie,itrequiresmoreeffortandtime).Actually,bothsolutionsarenotquiteconvenient.Solution(i)requirestocreatelargeimagelocators,includingmorethanonewebelement(eg,thelabelandthecorrespondinginputfield).Ontheotherhand,evenifitallowstocreateasmalllocatorimageforonly

onewebelement(eg,thelabel),Solution(ii)requirestocalculateadistanceinpixels(similarlytofirstgenerationtools),whichisnotsosimpletodetermine.Bothsolutionshaveproblemsincaseofvariationsoftherelativepositionsoftheelementsinthenextreleasesoftheapplication.Thus,thisfactorhasanegativeeffectonboththedevelopmentandmaintenanceofthevisualtestsuites.

FIGURE16 Visuallocatorcreationforrepeatedwebelement.

Summary:CreatingVisuallocatorscouldbedifficultwhenawebpagecontainsmultipleinstancesofthesamekindofwebelement(eg,alistofinputboxesinaform).

5.6.5InvisibleElementsSeleniumWebDriverlocatesthewebpageelementsbyusingtheDOMsoitdoesnotcareofwhatisactuallydisplayedbythebrowserGUI.Onthecontrary,SikulicanlocateonlyelementsthatarevisibleinthebrowserGUI.Letusconsideralongformtobefilled.UsingSikuli,itisnecessarytoscrolltheforminordertomaketheneededpageportionvisible.ThisrequirestoaddsomeLOCstotheSikulitestsuites.Thus,thisfactorcouldhaveanegativeimpactonthedevelopment(toinserttherightscroll)andmaintenance(eg,iftheelementchangespositionsandthepreviouslyinsertedscrollleavesitinvisible)oftheSikulitestsuites.Inourcasestudy,allthetestsuiteshaveatleastatestcaseusingmousescroll.Forinstance,inClaroline,duringthecreationofanewcourseitisnecessarytoscrollthepagetoreachsomeofthewebelementsintheform.ThesamehappensinMRBSwhenfillingtheroomreservationform.

Summary:DOM-basedtestsuitescaninteractwithalltheDOM-elementsregardlessofwhethertheyaredisplayedonthescreenornot.

5.6.6WebPageLoadingUsually,whenmovingfromonewebpagetoanother(eg,afterhavingsubmittedaformorclickedonalink),alittleamountoftimeisrequiredforloadingthenewwebpage.Ifthetestcasegoesaheadwithouttakingintoaccountthis,itmaynotfindthenexttargetwebelementandthusitmayreturnanerror.SeleniumWebDriverprovidesspecificcommandstodealwiththisproblem,9ie,explicitandimplicitwaits.Inourtestsuitesweusedimplicitwaits,allowingWebDrivertopolltheDOMforacertainamountoftime(eg,upto5s)whentryingtofindawebelementifitisnotimmediatelyavailable.InSikulithisisnotavailableandtestershavetoinsertanexplicitdelay(eg,Thread.sleep(1000)).Thisfactorcouldhaveanegativeimpactonthedevelopment(whentherightwaitingtimeshavetobeinserted)andmaintenance(eg,incasetherequiredwaitingtimeschange)oftheSikulitestsuites.

Summary:Choosingatoolthatprovidesspecificcommandsformanagingwebpageloadingisusefulforcreatingtestcaseseasily.Ifequippedwithsuchcommands,testcasesarealsomorerobustandfastertoexecute.

5.6.7TestCaseComprehensibilityThelocatorsusedbythetwoapproacheshaveoftenadifferentdegreeofcomprehensibility.Forinstance,bycomparingavisualandaDOM-basedlocator(eg,anXPathorCSSexpression),itisclearthatthevisuallocatorismucheasiertounderstandthanthecorrespondingDOM-basedlocator(seetheexamplesinFig.17).Infact,thevisualapproachworksinamannerthatisclosertohumansthantheDOM-basedapproach.Inourcasestudy,weexperiencedthisseveraltimes.Forinstance,duringtestsuitemaintenance,understandingwhyalocatorisbrokenisgenerallyeasierandfasterwithSikulithanwithWebDriver.

FIGURE17 ExamplesofvisualandDOM-basedlocators.

Summary:Forthetester,VisuallocatorsareusuallysimplertomatchwiththeactualwebpageelementsthanDOM-basedlocators.

5.6.8TestSuitesPortabilityvsRenderingValidationWhenaVisualtestsuiteisexecutedonadifferentmachine,wherethescreenresolutionorthefontpropertiesaredifferent,itmayhappenthattestcasesdonotworkproperly.Weexperiencedthisproblem2timeswhileexecutingtheSikuliAPItestsuitesontwodifferentcomputers:inonecasebecausethedefaultfontsizewasdifferent,resultinginbrokenimagelocators,andinanothercasebecausethescreenresolutionwaslowerthanexpected,thusmoremousescrolloperationswererequired.Ontheotherhand,Visualtestsuitesthatbehavedifferentlyindifferentplatformsmaypointtowebpagerenderingissues,asforinstancetheexistenceofwebelementsthatbecomeunreachableifthescreenresolutionistoolow.So,thetwoapproacheshavecomplementarystrengths.

Summary:TheDOM-basedandtheVisualapproacheshavedifferentstrengths:theformercreatesportabletestsuites,whilethelattercantestthecorrectrenderingofwebpagesacrossplatforms.

6OvercomingtheLimitationsoftheExistingApproachesIntheprevioussectionsweprovidedanoverviewoftheexisting,consolidatedapproachestoautomatedend-to-endwebtestingandsomeexamplesoftoolsimplementingthem.Inthissectionwediscussrecentresearchworksandexistingcommercialtoolsthattrytoovercometheirmainlimitations.

6.1HybridWebTestingInthelastfewyears,researchersandcompanieshaveproposedhybridsolutionsemployingmoreapproachesatthesametimethatthereforedonotfitourclassification.Inthefollowingweprovidethreeexamplesofthem.

6.1.1ContextualCluesYandrapallyetal.[9]proposedacapture-replayapproachandacorrespondingtoolcalledata-qvthatlocatesthewebelementsofinterestwithoutrelyingsolelyontheunderlyingDOMoronimagerecognition.Theideaisidentifyingawebelementbymeansofso-calledcontextualclues,ie,labelsorimagesintheproximityofthetargetwebelementthattogetheruniquelyidentifyitonthewebpage.Inthiswayata-qvgeneratesscriptsthatarerelativelyplatformandtoolagnostic,therefore,highlyportableandresilienttochangesininternalpropertiesorvisualrenderings.Specifically,theapproachfirsttriestocreatealocatorbyusingacombinationoflabels(ie,strings)thatcanbefoundinthepage(ie,intheDOM)andcheckswhethertheyareabletodefineauniquesubtreeoftheDOMcontainingthetargetwebelement.

Letusconsiderawebpagereportingasmartphonecatalogue.Thetargetwebelement,eg,the“ViewDetails”link,couldbelocatedusingthefollowinglocator:“ViewDetails”near“IPhone6”.Ifinthiswayitisnotpossibletofindalocatorforthewebelementofinterest,theapproachtriestocomputethreeothertypesoflocators:negation,proximity-based,andordinal-basedlocators.Negationlocatorsaddnegationpredicatesinordertoobtainexpressionsabletouniquelyidentifythewebelement.LetusassumethatwehavetwoidenticalportionsofthecataloguepagedescribingIPhone6andIPhone6Plus,andthattheword“Plus”isreportedinanestedtag(eg,underthesmartphonename).WewanttoseethedetailsofthestandardIPhone6(ie,thetestcaseshouldclickonits“ViewDetails”link).Insuchcaseanexampleofnegationlocatoris“ViewDetails”NOTnear“Plus”near“IPhone6”.

Proximity-basedlocatorsarebasedonlabelsthatarevisuallyclosetothetargetwebelementintheleft,bottom,right,andtopdirections(eg,CheckBoxunder“IPhone6”).Finally,ifthetechniquedoesnotfindanyproximity-basedlocators,itattemptstogenerateordinal-basedonesthataremoreexpressiveandincludebothdirectionalandpositionalclues(eg,secondCheckBoxunder“IPhone6”).Finally,ifallprevioustechniquesfailtofindalocator,ata-qvresortstoanXPathexpression.

Theadvantagesofthisapproacharetwofold:(1)locatorsbasedoncontextualcluesprovedtoberobust[9],and(2)testcasesareverysimpletoreadevenforanonexpert,seeforinstancetheexampleinFig.18,showinganata-qvtestcaseforourrunningexample.

FIGURE18 TestLogintestcaseinata-qv.

6.1.2RanorexRanorexisaGUItestautomationframeworkfortestingdesktop,web-basedandmobileapplications.Concerningautomatedend-to-endwebtesting,Ranorexisabletorecordthestepsperformedbythetesteronthewebapplicationandtocreateanexecutabletestcasefromthem.Thecreationoftheassertionsisaidedandthetestercanchooseamongasetofpossibleproposals.ThusRanorexbehavesasaC&Rtool,but,ontheotherhand,itprovidesalsosomefunctionalitiestypicalofprogrammabletool,suchas:(1)CodeModularization:oncethetestcaseshavebeenrecorded,itispossibletogroupsequencesofstepsinordertocreatereusableprocedures(asdonewiththePOpattern);(2)DataDriven:itispossibletoreexecutethesametestcaseusingdifferentvaluesstoredininternal(simpledatatables)orexternal(ExcelorCSVfiles,SQLDatabases)datasets;(3)ModuleDevelopment:itispossibletodeveloptestcodemodulesusingforinstancetheC#andVB.NETlanguagesandthentointegratethemwiththerecordedtestcases.

Concerningthelocalizationmethods,thetoolsupportsboththeDOM-basedandtheVisualapproaches.Indeed,RanorexemploystheRanoreXPathlanguage,anexpressionlanguagesimilartoXPath,providingasearchmechanismforfindingsingleormultiplewebelementswithinawebpage.Atthesametime,itprovidesalsothecapabilityofdefiningvisuallocatorsbothforlocalizingwebelementstointeractwithandfordefiningassertions.

6.1.3JAutomateJAutomateisacommercialtoolabletocreatetestscriptssimilarlytohowtheycanbeproducedusingSeleniumIDE.Indeed,thetesterclickstherecordbutton,performsthetestcasestepsandfinallycompletesthetestscriptbyinsertingassertions.Testscriptrecordingisfullyautomatic,sincethetoolisableto(1)detecttheactionsexecutedontheuserinterface(eg,clickonabutton,writeinaninputformorscrollthewebpageusingthemouse)and(2)generatethelocators.JAutomateisbasedonthevisuallocalizationofthetargetwebelementsbutitisalsoabletoprovidefunctionalitiesthatgobeyondthetypicalvisualapproach[10],likeverifyingthatatextisdisplayedinawebpagebymeansof:(a)runtimegenerationofavisuallocatorrepresentingsuchtext,and(b)anOCR(OpticalCharacterRecognition)algorithm,bothofwhichareveryusefulforcreatingdata-driventestcases.Incaseofseveralidenticalimagesonthescreen,itispossibletospecifywhich

hastobeselectedbyusinganindexposition,similarlytohowthisisdoneinanXPathexpression(eg,//input[3]).Moreover,JAutomateemploystwocomplementaryimagerecognitionalgorithms[11],which,oncecombined,canidentifyimageswithinvertedcolorsorimageswithtransparentbackgrounds.JAutomatetriestoovercomesomelimitationstypicalofexistingC&Rtoolsbyintegrating/combiningfeaturesoftheprogrammableapproach[10],forinstance,byprovidingconstructsto(1)implementloops,(2)createparametrictestcases(eg,byloadingdatavaluesfromCSVfiles)and(3)call/includeothertestscripts.Moreover,itprovidesanAPIfordevelopingtestscriptsdirectlyinJava.AJAutomatetestcasethathasabehaviorclosetotheonesshownintheprevioussectionsisshowninFig.19.

FIGURE19 TestLogintestcaseinJAutomate.

6.2ReducingtheDevelopmentandMaintenanceEffortInthelastyears,severalresearchersinvestigatedthefollowingproblemsassociatedwithautomatedend-to-endtestingofwebapplication:(1)improvingtherobustnessofthelocatorsusedinthetestcases,(2)automatingtestcasesrepairduringsoftwareevolution,

(3)automatingthegenerationofpageobjects,and(4)automatingthemigrationacrossdifferentapproaches.Theprovidedsolutions,stronglybasedonstatic,dynamicandcodetransformations[12,13],canbeveryusefulforreducingthetestcasedevelopmentandmaintenanceeffort.

6.2.1ImprovingLocatorsRobustnessTestscriptsheavilyrelyonlocatorsforinteractingwiththeelementsonawebpage—forinstancetoidentifyandfilltheinputportionsofawebpage(eg,theformfields),toexecutesomecomputations(eg,bylocatingandclickingonbuttons),andtoverifythecorrectnessoftheoutput(bylocatingthewebpageelementsshowingtheresults).Locatorsneedtobecheckedforcorrectnessandpossiblyupdatedateverynewreleaseofthesoftware.Sometimesevenaslightmodificationoftheapplicationundertesthasamassiveimpactonlocators.Thisproblemmakesthemaintenanceofwebtestsuitesextremelyexpensive.Severalapproachesandtechniqueshavebeenproposedtoincreaselocatorsrobustnesssuchforexample,ContextualClues[9](seeSection6.1),Robula[14,15],Montotoetal.[16],andMultiLocator[17].

Robula[14,15]isanalgorithmabletoautomaticallygeneraterobustDOM-basedwebelementlocators.ThealgorithmstartswithagenericXPathlocatorthatreturnsallnodes(“//*”).Ittheniterativelyrefinesthelocatoruntilonlytheelementofinterestisselected.Insuchiterativerefinement,Robulaappliesasetofrefinementtransformations,accordingtoasetofheuristicXPathspecializationsteps.TheempiricalevaluationofthealgorithmshowsthatRobulageneratesconsistentlylocatorsthataremuchmorerobustthanthoseproducedbyexistingstateofthearttoolsandalgorithms.

Montotoetal.[16]proposedanalgorithmforidentifyingthetargetelementsduringthenavigationofAJAXwebsites.ThealgorithmstartsfromasimpleXPathexpressionthatisprogressivelyaugmentedwithtextualandattributeinformation.Moreprecisely,thealgorithmfirsttriestoidentifythetargetelementaccordingtoitsassociatedtext(iftheelementisaleafnode)andthenitconjuncts,oneaftertheother,predicatesbasedontheattributevalues(withoutprescribinganyspecificorderofinsertion).Ifthisisnotsufficientforgeneratingauniquelocatorforthetargetelement,eachancestor(andthevaluesofitsattributes)issubjectedtothesameprocedure,untiltherootoftheDOMisreached.

WhilealgorithmsexistthatproducerobustDOM-basedlocatorstobeusedinwebtestscripts,noalgorithmisperfectanddifferentalgorithmsareexposedtodifferentfragilitieswhenthesoftwareevolves.Basedonsuchobservation,anewtypeoflocatorhasbeenproposed,namedMultiLocator[17],whichselectsthebestlocatoramongacandidatesetoflocatorsproducedbydifferentalgorithms.Suchselectionisbasedonavotingprocedurethatassignsdifferentvotingweightstodifferentlocatorgenerationalgorithms.Experimentalresultsshowthatthemulti-locatorismorerobustthaneachsinglelocator(about−30%ofbrokenlocatorsw.r.t.themostrobustsinglelocator)andthattheexecutionoverheadrequiredbythemultiplequeriesdonewithdifferentlocatorsisnegligible.

6.2.2AutomaticTestCaseRepairOtherapproachesfocusonhowtoreducetheeffortrelatedtotherepairofbrokentestcases.Thegeneralproblemofrepairingbrokentestcaseshasbeenaddressedbyseveralresearchers(eg,TestCareAssistant[18]andReAssert[19]),butonlyafewtechniquesarefocussedonrepairingautomatedend-to-endwebtestcases:

water[20]isatoolthatsuggestschangesthatcanbeappliedtorepairtestscriptsforwebapplications.Thistechniqueisbasedondifferentialtesting:bycomparingthebehaviorofatestcaseontwosuccessiveversionsofthewebapplicationandanalyzingthedifferencebetweenthetwoexecutions,watersuggestsrepairsthatcanbeappliedtoupdatethescripts.

Testcasesmakeuseofinputdatawhileexercisingthewebapplicationundertest.Onefactorlimitingtheusageofwebapplicationtestautomationtechniquesisthecostoffindingappropriateinputvalues.Tomitigatethisproblem,Elbaumetal.[21]proposedafamilyoftechniquesbasedonuser-sessiondata.Usersessiondataarecollectedinpreviousreleasesandthenreusedasinputdatainthecurrentrelease.However,usersessiondataalsosuffertheevolutionproblem:sessiondatamaybecomeinvalidduetochangesintheapplication.Techniquesrepairingtheinputdatahavebeenproposedtoalleviatethisproblem.ForinstanceHarmanandAlshahwan[22]investigatedanalgorithmbasedontheconceptof“sessionrepair.”Inparticular,whenasessionbecomesinvalidduetochangesinthewebapplication,thealgorithmattemptstorepairitbyconstructinganewsessionthatcloselyresemblestheoriginal.

6.2.3AutomaticPageObjectsGenerationAswehaveseen,theadoptionofthepageobjectpatternimprovesthetestsuitemaintainabilitybyreducingtheduplicationofcodeacrosstestcases.Theimplementationofpageobjectsisusuallydonemanually.Someopensourceframeworkshavebeenproposedtoassistthetesterduringthecreationofpageobjects.ThesetoolsmostlywraptheHTMLcontentofthepageandofferanaidedcreationofthesourcecode.Themostimportantonesare:OHMAP,10SWDPageRecorder,11WTFPageObjectUtilityChromeExtension.12Despitethesetoolsprovideusefulfeatures,mostoftheeffortisstillputontesters.Moreover,theysufferseverallimitations[23],inparticular:(i)onlyonepageatatimeistakenintoaccount,withoutconsideringanynotionofdynamismorwebapplicationstructure,(ii)onlyasubsetofwebelementsthatcanbeusedinatestistakenintoaccount,(iii)thegeneratedcodeconsistsofabasicclassskeleton,whilethekeycharacteristicsofthepageobjects(ie,exposingthewebapplicationfunctionalitiesthroughmethods)arelacking.Especiallythislastimportantfeatureiscompletelymissinginallthesetools.Recently,astepaheadhasbeenmadewiththeproposalofApoGen.

ApoGen[23]isatoolfortheautomaticgenerationofpageobjectsforwebapplications.ThetoolautomaticallyderivesatestingmodelbyreverseengineeringthetargetwebapplicationandusesacombinationofdynamicandstaticanalysistogenerateJavapageobjectsfortheSeleniumWebDriverframework.ApoGenconsistsofthreemainmodules:aCrawler,aStaticAnalyzer,andaCodeGenerator.TheinputofApoGenisanywebapplication,togetherwiththelogincredentialsifnecessary,whiletheoutputisasetof

Javafiles,representingacodeabstractionofthewebapplication,organizedusingthePageObjectandPageFactorydesignpatterns,assupportedbytheSeleniumWebDriverframework.Apreliminarystudycomparingthegeneratedpageobjectswiththeonescreatedmanuallybyahumantestershowspromisingresults.

6.2.4AutomaticMigrationbetweenApproachesAnimportantpointofconcernishowtoautomatethemigrationofanexistingtestsuitetodifferentapproaches.MigratingfromtheC&Rtotheprogrammableapproachisrelativelyasimpletaskandisalreadysupportedbysometools.Forinstance,SeleniumIDEallowstomigratethetestcasesrecordedinSelenesetoSeleniumWebDrivertestcasesimplementedinJava.Inthisway,thetestercanthenrefactortheJavacodeinordertotakefulladvantageoftheprogrammableapproach.Ontheotherhand,itismuchmorecomplexmigratingthetestcasestodifferentlocalizationapproaches(eg,tothevisualapproach).AproposalinthisdirectionisPesto.

Pesto[24]isatoolproposedforautomatingthemigrationofaDOM-basedwebtestsuite,createdusingSeleniumWebDriverandadoptingthepageobject(PO)pattern,intoavisualwebtestsuitebasedontheSikuliimagerecognitioncapabilities,whilestilladoptingthePOpattern.EvenifPestohasbeendevelopedtotransformSeleniumWebDrivertestsuitestoSikuliAPI,thetechniquesandarchitecturalsolutionsadoptedforitsimplementationarequitegeneralandcanbeeasilyusedwithinanywebtesttransformationactivityinvolvingabstractionssimilartotheonesprovidedbythePO.PestodeterminesautomaticallythescreenpositionofawebelementlocatedintheDOMbyaDOM-basedtestcase.Itthendeterminesarectangleimagecenteredaroundthewebelementsoastoensureuniquevisualmatching.Basedonsuchautomaticallyextractedimages,theoriginaltestsuiteisrewrittenintoavisualtestsuite.Experimentalresults[24]showthatthisapproachisaccurate,hencepotentiallysavingsubstantialhumaneffortinthecreationofvisualwebtestsfromDOM-basedones.

7ConclusionsInthischapterwehaveprovidedanoverviewofthemostrelevantapproachesandtoolstoautomatedend-to-endwebtesting.First,foreachapproachwehavegivenadetaileddescriptionbasedonexistingimplementations;second,wehavedeeplyanalyzedtheirstrengthsandweaknessesbydiscussingtheresultsofaseriesofempiricalstudies.Third,wehavedescribedsomerecenttechniquesandtoolsthattrytoovercomethelimitationsoftheexistingapproachesbycombiningthemintohybridmethods.Finally,wehaveanalyzedasetoftechniquesthathavebeenproposedintheliteratureinordertosolvespecificproblemsinthecontextoftheautomatedend-to-endwebtesting.

Concerningthemethodsusedfordevelopingandmaintainingwebtestcases,wefoundthatprogrammabletestsinvolvehigherdevelopmenteffort(between32%and112%)butlowermaintenanceeffort(withasavingbetween16%and51%)thanC&Rtests.Wehaveestimatedthat,onaverage,aftertwomajorreleases,programmabletestcasesbecomemoreconvenientthanC&Rones.However,theactualbenefitsdependonspecificfeaturesofthewebapplication,includingitsdegreeofmodularity,whichmapstoreusablepageobjectsthatneedtobeevolvedonlyonce,whenprogrammabletestcasesareused.Moreover,thereareusefulfeaturesofprogrammabletestcases,suchasthepossibilitytodefineparametricandrepeatedtestscenarios,whichmightfurtheramplifytheiradvantages.

Concerningtheapproachusedforlocalizingthewebelementstointeractwith,wefoundthatDOM-basedlocatorsaregenerallymorerobustthanvisuallocators,andthatDOM-basedtestcasescanbedevelopedfromscratchatlowercost.Mostofthetimestheyarealsoevolvedatlowercost.However,onspecificwebapplicationsvisuallocatorswereeasiertorepair,becausethevisualappearanceofsuchapplicationsremainedstableacrossreleases,whiletheirstructurechangedalot.DOM-basedtestcasesrequiredalowerexecutiontimethanvisualtestcases,duetothecomputationaldemandsofimagerecognitionalgorithmsusedbythevisualapproach,althoughthedifferenceisnotdramatic.Overall,thechoicebetweenDOM-basedandvisuallocatorsisapplication-specificanddependsquitestronglyontheexpectedstructuralandvisualevolutionoftheapplication.Otherfactorsmayalsoaffectthetesters’decision,suchastheavailability/unavailabilityofvisuallocatorsforwebelementsthatareimportantduringtestingandthepresenceofadvanced,RIAfunctionalitieswhichcannotbeeasilytestedusingDOM-basedlocators.Moreover,visualtestcasesaredefinitelyeasiertounderstand,which,dependingontheskillsoftheinvolvedtesters,mightalsoplayaroleinthedecision.

References[1]RiccaF.,TonellaP.Detectinganomalyandfailureinwebapplications.IEEEMultimed.1070-986X2006;13(2):44–51.doi:10.1109/MMUL.2006.26.

[2]RiccaF.,TonellaP.Analysisandtestingofwebapplications.In:Proceedingsofthe23rdInternationalConferenceonSoftwareEngineering,ICSE2001;IEEE;2001:25–34.

[3]LeottaM.,ClerissiD.,RiccaF.,TonellaP.Visualvs.DOM-basedweblocators:anempiricalstudy.In:Proceedingsofthe14thInternationalConferenceonWebEngineering(ICWE2014),LNCS,Toulouse,France;Springer;322–340.doi:10.1007/978-3-319-08245-5_19.2014;vol.8541.

[4]LeottaM.,ClerissiD.,RiccaF.,TonellaP.Capture-replayvs.programmablewebtesting:anempiricalassessmentduringtestcaseevolution.In:Proceedingsofthe20thWorkingConferenceonReverseEngineering,WCRE2013,Koblenz,Germany;IEEE;2013:272–281.doi:10.1109/WCRE.2013.6671302.

[5]LeottaM.,ClerissiD.,RiccaF.,SpadaroC.ImprovingtestsuitesmaintainabilitywiththePageObjectpattern:anindustrialcasestudy.In:Proceedingsofthe6thIEEEInternationalConferenceonSoftwareTesting,VerificationandValidationWorkshops,ICSTW2013;IEEE;2013:108–113.doi:10.1109/ICSTW.2013.19.

[6]LeottaM.,ClerissiD.,RiccaF.,SpadaroC.ComparingthemaintainabilityofSeleniumWebDrivertestsuitesemployingdifferentlocators:acasestudy.In:Proceedingsofthe1stInternationalWorkshoponJoiningAcadeMiAandIndustryContributionstotestingAutomation,JAMAICA2013;ACM;2013:53–58.doi:10.1145/2489280.2489284.

[7]vanDeursenA.Beyondpageobjects:testingwebapplicationswithstateobjects.ACMQueue.1542-77302015;13(6):20:20–20:37.doi:10.1145/2791301.2793039.

[8]MirzaaghaeiM.Automatictestsuiteevolution.In:Proceedingsofthe19thACMSIGSOFTSymposiumandthe13thEuropeanconferenceonFoundationsofSoftwareEngineering,ESEC/FSE2011,Szeged,Hungary;ACM;2011:978-1-4503-0443-6396–399.

[9]YandrapallyR.,ThummalapentaS.,SinhaS.,ChandraS.Robusttestautomationusingcontextualclues.In:Proceedingsofthe2014InternationalSymposiumonSoftwareTestingandAnalysis,ISSTA2014,SanJose,CA,USA;ACM;2014:978-1-4503-2645-2304–314.doi:10.1145/2610384.2610390.

[10]SwiftingAB.JAutomateManual.2014.

[11]AlegrothE.,NassM.,OlssonH.H.JAutomate:atoolforsystem-andacceptance-testautomation.In:Proceedingsofthe6thIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation,ICST2013;IEEE;2013:978-0-7695-4968-2439–446.doi:10.1109/ICST.2013.61.

[12]TonellaP.,RiccaF.,MarchettoA.Recentadvancesinwebtesting.Adv.Comput.2014;93:1–51.

[13]RiccaF.,TonellaP.,BaxterI.D.Webapplicationtransformationsbasedonrewriterules.Inf.Softw.Technol.2002.;44(13):811–825.URLhttp://dblp.uni-trier.de/db/journals/infsof/infsof44.html#RiccaTB02.

[14]LeottaM.,StoccoA.,RiccaF.,TonellaP.ReducingwebtestcasesagingbymeansofrobustXPathlocators.In:Proceedingsofthe25thIEEEInternationalSymposiumonSoftwareReliabilityEngineeringWorkshops,ISSREW2014;IEEE;2014:449–454.doi:10.1109/ISSREW.2014.17.

[15]M.Leotta,A.Stocco,F.Ricca,P.Tonella,ROBULA+:analgorithmforgeneratingrobustXPathlocatorsforwebtesting,J.Softw.Evol.Process(underreview)

[16]MontotoP.,PanA.,RaposoJ.,BellasF.,LopezJ.AutomatedbrowsinginAJAXwebsites.DataKnowl.Eng.0169-023X2011.;70(3):269–283.doi:10.1016/j.datak.2010.12.001.URLhttp://www.sciencedirect.com/science/article/pii/S0169023X10001503.

[17]LeottaM.,StoccoA.,RiccaF.,TonellaP.Usingmulti-locatorstoincreasetherobustnessofwebtestcases.In:Proceedingsof8thIEEEInternationalConferenceonSoftwareTesting,VerificationandValidation,ICST2015;IEEE;2015:1–10.doi:10.1109/ICST.2015.7102611.

[18]MirzaaghaeiM.,PastoreF.,Pezze’M.Automatictestcaseevolution.Softw.Test.Verif.Reliab.1099-16892014;24(5):386–411.doi:10.1002/stvr.1527.

[19]DanielB.,DigD.,GveroT.,JagannathV.,JiaaJ.,MitchellD.,NogiecJ.,TanS.H.,MarinovD.ReAssert:atoolforrepairingbrokenunittests.In:Proceedingsofthe33rdInternationalConferenceonSoftwareEngineering,ICSE2011;IEEE;2011:1010–1012.doi:10.1145/1985793.1985978.0270-5257.

[20]ChoudharyS.R.,ZhaoD.,VerseeH.,OrsoA.WATER:webapplicationtestrepair.In:Proceedingsofthe1stInternationalWorkshoponEnd-to-EndTestScriptEngineering,ETSE2011,Toronto,Ontario,Canada;ACM;2011:978-1-4503-0808-324–29.

[21]ElbaumS.,RothermelG.,KarreS.,FisherIIM.Leveraginguser-sessiondatatosupportwebapplicationtesting.IEEETrans.Softw.Eng.0098-55892005;31(3):187–202.doi:10.1109/TSE.2005.36.

[22]HarmanM.,AlshahwanN.Automatedsessiondatarepairforwebapplicationregressiontesting.In:Proceedingsofthe1stInternationalConferenceonSoftwareTesting,Verification,andValidation,ICST2008;2008:298–307.doi:10.1109/ICST.2008.56.

[23]StoccoA.,LeottaM.,RiccaF.,TonellaP.Whycreatingwebpageobjectsmanuallyifitcanbedoneautomatically?In:Proceedingsof10thIEEE/ACMInternationalWorkshoponAutomationofSoftwareTest,AST2015,Florence,

Italy;IEEE;2015:70–74.doi:10.1109/AST.2015.26.

[24]LeottaM.,StoccoA.,RiccaF.,TonellaP.AutomatedgenerationofvisualwebtestsfromDOM-basedwebtests.In:Proceedingsof30thACM/SIGAPPSymposiumonAppliedComputing,SAC2015,Salamanca,Spain;ACM;2015:775–782.doi:10.1145/2695664.2695847.

MaurizioLeottaisaresearchfellowattheUniversityofGenova,Italy.HereceivedhisPhDdegreeinComputerSciencefromthesameUniversity,in2015,withthethesis“AutomatedWebTesting:AnalysisandMaintenanceEffortReduction.”Heisauthororcoauthorofmorethan30researchpaperspublishedininternationalconferencesandworkshops.HiscurrentresearchinterestsareinSoftwareEngineering,withaparticularfocusonthefollowingthemes:WebApplicationTesting,FunctionalTestingAutomation,BusinessProcessModelling,EmpiricalSoftwareEngineering,Model-DrivenSoftwareEngineering.

DiegoClerissiisaPhDstudentinComputerScienceattheUniversityofGenova,Italy.In2015hereceivedhismasterdegreefromthesameUniversity,withthethesis:“TestCasesGenerationforWebApplicationsfromRequirementsSpecification:PreliminaryResults.”Atthetimeofwritingheiscoauthorof10researchpaperspublishedininternationalconferencesandworkshops.HisresearchinterestsareinSoftwareEngineering,Model-BasedTesting,SoftwareTesting,WebApplications,SystemModeling.

FilippoRiccaisanassociateprofessorattheUniversityofGenova,Italy.HereceivedhisPhDdegreeinComputerSciencefromthesameUniversity,in2003,withthethesis:“Analysis,TestingandRe-structuringofWebApplications.”In2011hewasawardedtheICSE2001MIP(MostInfluentialPaper)award,forhispaper:“AnalysisandTestingofWebApplications.”Heisauthororcoauthorofmorethan100researchpaperspublishedininternationaljournalsandconferences/workshops.HewasProgramChairofCSMR/WCRE2014,CSMR2013,ICPC2011,andWSE2008.Amongtheothers,heservedintheprogramcommitteesofthefollowingconferences:ICSM,ICST,SCAM,CSMR,WCRE,andESEM.From1999to2006,heworkedwiththeSoftwareEngineeringgroupatITC-irst(nowFBK-irst),Trento,Italy.DuringthistimehewaspartoftheteamthatworkedonReverseengineering,Re-engineering,andSoftwareTesting.Hiscurrentresearchinterestsinclude:Softwaremodeling,Reverseengineering,EmpiricalstudiesinSoftwareEngineering,Webapplications,andSoftwareTesting.Theresearchismainlyconductedthroughempiricalmethodssuchascasestudies,controlledexperiments,andsurveys.

PaoloTonellaisheadoftheSoftwareEngineeringResearchUnitatFondazioneBrunoKessler(FBK),inTrento,Italy.HereceivedhisPhDdegreeinSoftwareEngineeringfromtheUniversityofPadova,in1999,withthethesis:“CodeAnalysisinSupporttoSoftwareMaintenance.”In2011hewasawardedtheICSE2001MIP(MostInfluentialPaper)award,forhispaper:“AnalysisandTestingofWebApplications.”Heistheauthorof“ReverseEngineeringofObjectOrientedCode,”Springer,2005.HeparticipatedinseveralindustrialandEUprojectsonsoftwareanalysisandtesting.Hiscurrentresearchinterestsincludecodeanalysis,webandobjectorientedtesting,search-basedtestcasegeneration.

1http://seleniumhq.org/projects/ide/2http://seleniumhq.org/projects/webdriver/3http://www.sikuli.org/4https://code.google.com/p/sikuli-api/5EachSeleneselineisatriple:(command,target,value).See:http://release.seleniumhq.org/selenium-core/1.0.1/reference.html6http://martinfowler.com/bliki/PageObject.html

7https://code.google.com/p/selenium/wiki/PageObjects8https://code.google.com/p/selenium/wiki/PageFactory9http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp10http://ohmap.virtuetech.de/11https://github.com/dzharii/swd-recorder12https://github.com/wiredrive/wtframework/wiki/WTF-PageObject-Utility-Chrome-Extension

AuthorIndex

SubjectIndex

ContentsofVolumesinthisSeries