fundamental principles of testing - final · 2018-05-30 · testing is critical to enforcement of...

TheFundamentalPrinciplesofTesting

Copyright©2016Anti-MalwareTestingStandardsOrganization,Inc.Allrightsreserved.Nopartofthisdocumentmaybereproducedinanyform,inanelectronicretrievalsystemorotherwise,withouttheprior

writtenconsentofthepublisher.

2

NoticeandDisclaimerofLiabilityConcerningtheUseofAMTSODocuments

ThisdocumentispublishedwiththeunderstandingthatAMTSOmembersaresupplyingthisinformationforgeneraleducationalpurposesonly.Noprofessionalengineeringoranyotherprofessionalservicesoradvice isbeingofferedhereby. Therefore,youmustuseyourownskillandjudgmentwhenreviewingthisdocumentandnotsolelyrelyontheinformationprovidedherein.

AMTSObelievesthattheinformationinthisdocumentisaccurateasofthedateofpublicationalthoughithasnotverifieditsaccuracyordeterminedifthereareanyerrors.Further,suchinformationissubjecttochangewithoutnoticeandAMTSOisundernoobligationtoprovideanyupdatesorcorrections.

Youunderstandandagreethat thisdocument isprovidedtoyouexclusivelyonanas-isbasiswithoutanyrepresentationsorwarrantiesofanykindwhetherexpress, impliedorstatutory. Without limitingthe foregoing, AMTSO expressly disclaims all warranties of merchantability, non-infringement,continuousoperation,completeness,quality,accuracyandfitnessforaparticularpurpose.

InnoeventshallAMTSObeliableforanydamagesorlossesofanykind(including,withoutlimitation,any lost profits, lost data or business interruption) arising directly or indirectly out of any use of thisdocument including, without limitation, any direct, indirect, special, incidental, consequential,exemplary and punitive damages regardless of whether any person or entity was advised of thepossibilityofsuchdamages.

Thisdocument isprotectedbyAMTSO’s intellectualpropertyrightsandmaybeadditionallyprotectedbytheintellectualpropertyrightsofothers.



3

TheFundamentalPrinciplesofTestingThefollowingrepresentasummaryoftheprinciplesapplicabletoanti-malwaretestingthatshouldbefollowedbytesters,publicationsandvendors.Theseprinciplesarebasedonourbeliefthateverybodyinvolved in such testingmust behave ethically, test properly and communicate in a fair and accurateway.Foradditionalinformation,pleasereviewguidelinesforeachitemincludedbelow.

1. Testingmustnotendangerthepublic.

2. Testingmustbeunbiased.

3. Testingshouldbereasonablyopenandtransparent.

4. Theeffectivenessandperformanceofanti-malwareproductsmustbemeasured inabalancedway.

5. Testers must take reasonable care to validate whether test samples or test cases have beenaccuratelyclassifiedasmalicious,innocentorinvalid.

6. Testingmethodologymustbeconsistentwiththetestingpurpose.

7. Theconclusionsofatestmustbebasedonthetestresults.

8. Testresultsshouldbestatisticallyvalid.

9. Vendors, testers and publishers must have an active contact point for testing relatedcorrespondence.



4

GuidelinestotheFundamentalPrinciplesofTesting

Principle1:Testingmustnotendangerthepublic.

ThisprincipleisfundamentaltothecharterandpurposeofAMTSOandeachofitsmembers.Thepublichas the right to expect that the development and sale of anti-malware products, the review of suchproducts and publication of those reviews are all done, fundamentally, to protect them. Thus, theforemostprincipleoftestinganti-malwareproductsisthatneithertheproductsnortherelatedtestingshouldendangerthepublic.Infurtheranceofthisprinciple,testersmustfollowappropriateprocedurestoavoidaccidental releaseof samplesatall times. Inaddition,newmalwaremustnotbecreated fortestingpurposes.

Q.Whatareconsideredtobe“appropriateprocedures”?

A. Itisexpectedthatanytestingenvironmentwillutilizeindustry-standardbestpracticestoensurethatmalwaresamplesarenotaccidentallyreleasedandthatriskstothepublicareavoided.

Q.Whatismeantby“creationofnewmalware”?

A. Thisreferencehashistoricallyreferredtothecreationofnewvirusesorstrainsofmalware,oneobjectionbeingbasedontheprinciplethattherearemorethanenoughsamplesavailableinthewild for everyone. Thismandatehas been complicatedby introductionof packers and virtualmachines, inviting the question as to whether utilizing these vehicles could be deemed tochangethecharacteristicsofpre-existingmalwaretothepointthatitcouldbedeemed“new.”Thereare legitimatereasonstochangeexistingmalwarecharacteristics for testingpurposes–this principle is not included in order to preclude such testing. To be clear, however, thisprinciple is included to demonstrate unanimous disapproval by AMTSO of the idea of thecreation of new viruses or other malware and the related risk to the public. If you wish tocontactAMTSOaboutthesematterspleasesendan [email protected] formoreinformation.

Principle2:Testingmustbeunbiased.

Webelievethatanti-malwaretesting,byitsnature,shouldbeunbiased–eachproductmustbetreatedequally.Whetherthetest iscommissionedbyavendortosupportamarketingmessageorbyamajormagazinetorunastoryonproductefficacy, it is theobligationofthetestertoconductthetest inanethicalmanner,andtopresenttruthfulandunbiasedresults.

Therearemanycircumstanceswherevendorsmayprovidefinancialincentivestoapublicationortester.Theseincentivesareneitherunusualnorbydefinitionunethical,andmaybeobtainedthroughtestingcommissions or advertising revenue, for instance. Although generally innocuous, to avoid theappearanceof impropriety,webelieve that these relationships,whensignificant, shouldbedisclosed.



5

Thus,tomeetthisPrinciple2,weencouragetestersandpublisherstopubliclydisclosetheexistenceofanysuchsignificantfinancialrelationshipswithareviewedpartyoraffiliate.

Q.Whatwouldconstituteasignificantfinancialincentive?

A. The intent of this principle is to avoid bias and conflicts of interest in product testing andreporting.Thus, thisdisclosureshould includeanyrelationshipthatcouldpotentially influencethetester,including:(i)whetherthepublicationortesterhasreceivedrevenuefromavendororaffiliatewithregardtoanyparticulartest,and(ii)whetherthepublicationortesterreceivesasignificant portion of its overall revenue from a particular vendor.While testers are asked todisclosethesourceoftheirsamplesinthetestingdetails,provisionofsamplesingeneralisnotconsideredafinancialincentive.

Q.Howshouldthisdisclosurebemade?

A. Ideally,eachtesterandpublicationwillprovidethisdisclosureasafootnotetoeachpublishedreport,orwillprovidealinkorotherreferencetowheresuchinformationcanbefound.

Principle3:Testingshouldbereasonablyopenandtransparent.

AMTSO recognizes that some publicationsmay not be always comfortablewith the disclosure of themethodology of published tests. However, AMTSO feels strongly that having open and transparenttesting is critical to enforcement of these fundamental principles, and to ensuring reliability andconsistencyinanti-malwaretesting.Asaresult,webelievethatanytestreleasedtothepublicmustbeaccompaniedby,orreferencethelocationof,detailsregardingthetestandtestingmethodology.

Detailsregardingthespecifictestshouldincludethefollowinginformation:

1. Whichsolutionsweretested?

2. Howwerethesolutionsobtainedandupdated?

3. Howwerethesamplesortestcasesobtainedandvalidated?(Seealsoprinciple5.)

4. Whatversionsoftheproductswereused?

5. Whatproductsettings/configurationswereused?

6. Whenandunderwhatconditionswasthetestconducted?

7. What environment was the test conducted in? (for example, the operatingsystem/environmentversion,servicepacksapplied,andotherprogramsthatwererunningatthetime)?

Detailsregardingthespecifictestingmethodologyshouldincludethefollowinginformation:

1. Howwerethetestsamplesortestcasesselected?

2. Whatwerethesourcesofmaliciousandinnocentsamplesortestcases?



6

3. Howwerethemaliciousandinnocentsamplesortestcasesapplied?

4. Howwastheresponseofthesolutionsmeasured?

5. Was the test “apples to apples” (comparing products of similar type andfunctionality),or“applestooranges”(comparingproductsofsignificantlydifferenttypeand/orfunctionality)?

6. If“applestooranges”,howwerethevarioussolutionscompared?

7. Howweretheresultscalculatedandinterpreted?

Q.Whereshouldthetestandtestingmethodologybedisclosed?

A. Ideally,thisinformationwillbeincludedinthepublishedreport,eitherinthebodyofthereportorbya link to the relevant information. Ifpublicationsareunableorunwilling to include thisinformation, testers can themselves make this information available on their website with areferencetoaspecificorgeneraltest.

Q.Musttestersprovidefeedbackand/orsamplestovendors?

A. No.However, AMTSOencourages testers to provide vendorswith constructive and adequatefeedback inatimely fashionaboutspecific faultsanddeficiencies (e.g.crashes, falsepositives,falsenegatives,etc.)This feedbackcanbe in the formof technicaldetails, reproductionsteps,logfiles,memorydumps,samples,etc.

Principle4:Theeffectivenessandperformanceofanti-malwareproductsmustbemeasuredinabalancedway.

It is difficult – and can be misleading – to summarize product efficacy with a single measurement.Testersareencouragedtopresentmultiplemeasurementsofproductperformanceindifferentareasinordertoallowuserstomakeaninformeddecision.

Forinstance,testersshouldappropriatelybalancefalsenegativeandfalsepositivetestcases.Aproductthatissuccessfulatdetectingahighpercentageofmalwarebutsuffersfromahighfalsepositiverate,maynotbe“better”thanasolutionwhichcatcheslessmalwarebutwhichgenerateslessfalsepositives.

Principle5: Testersmusttakereasonablecaretovalidatewhethertestsamplesortestcaseshavebeenaccuratelyclassifiedasmalicious,innocentorinvalid.

It has often been the case that seemingly reliable testing results are, in fact, not valid, because thesamplesused in the testsweremisclassified.Forexample, ifa testerdetermines thataproducthasahigh rate of false positives, that result could be wrong if some samples were wrongly classified asinnocent.Thus,itisourpositionthatreasonablecaremustbetakentoproperlycategorizetestsamplesortestcases,andweespeciallyencouragetesterstorevalidatetestsamplesortestcasesthatappeartohavecausedfalsenegativeorfalsepositiveresults.



7

Similarly,careshouldbe takento identifysamples thatarecorrupted,non-viableor thatmayonlybemaliciousincertainenvironmentsandconditions.

Principle6:Testingmethodologymustbeconsistentwiththetestingpurpose.

Tests must address the intended or stated purpose of the publisher’s related review or article. Webelievethatpublishersshouldstatetheobjectiveoftheirtestsclearly,andthattestmethodologyshouldbeconsistentwiththestatedtestobjective.(Forexample,publishingtestresultsinaconsumer-targetedmagazinewithoutmakingitclearthatthetestwasconductedoncorporateproductsbecausethisdoesnotsimulatetargetuserexperience.)

Foradditionalreference,DavidHarleyhaspublishedapaperthatanalyses indetailtheproblemswithone test that displayed a certain inconsistency between the test objective described and themethodologyused.

Seehttp://geekpeninsula.files.wordpress.com/2013/09/av_comparative_guide.pdf

Principle7:Theconclusionsofatestmustbebasedonthetestresults.

Thisprincipleaddressesacommonhighlevelproblemwithpublishingconclusionsalongsidetestingdatathatarenotsupportedbythosedata.(Forexample,drawingbroadand/orinaccurateconclusionsfromnarrowtestdata.)

Principle8:Testresultsshouldbestatisticallyvalid.

Testers should use a sufficient quantity of test samples, test cases or scenarios for results to bestatisticallysound. Inaddition,thetester’sanalysisofmeasurementerrors is importantandshouldbepublished.Ingeneral,AMTSOrecommendsusingasmanytestscenariosaspossible.

For additional reference, Igor Muttik has published a paper that analyses in detail how insufficientquantityofsamplesortestcasescanproducerandomtestresults.

Seehttp://publications.muttik.net/avar2001-cookingacomparative.pdf

Principle9:Vendors,testersandpublishersmusthaveanactivecontactpointfortestingrelatedcorrespondence.

An “active contact point” is a current and monitored point of accessibility (via phone, fax or email)providedbyvendors,PRdepartments, testersandpublishers.Relevant correspondence regarding thesubject product, test or testingmethodology should be answered by the vendor, tester or publisherwithinareasonabletimeframe.

______________________________________________________________________________

ThisdocumentwasadoptedbyAMTSOonOctober31,2008