practices in jasss, 2001-2012 anarchy reigns: a ...'mega-journal' plos-one, exemplifies...

©CopyrightJASSS

SimonDAngusandBehroozHassani-Mahmooei(2015)

MonashUniversity,Australia

"Anarchy"Reigns:AQuantitativeAnalysisofAgent-BasedModellingPublicationPracticesinJASSS,2001-2012

JournalofArtificialSocietiesandSocialSimulation 18(4)16<http://jasss.soc.surrey.ac.uk/18/4/16.html>DOI:10.18564/jasss.2952

Received:03-Feb-2015Accepted:17-Sep-2015Published:31-Oct-2015

AbstractAgentBasedModelling(ABM),apromisingscientifictoolset,hasreceivedcriticismfromsome,inpart,duetoaclaimedlackofscientificrigour,especiallyinthecommunicationofitsmethodsandresults.Totesttheveracityoftheseclaims,weconductastructuredanalysisofover900scientificobjects(figures,tables,orequations)thatarosefrom128ABMpaperspublishedintheJournalofArtificialSocietiesandSocialSimulation(JASSS),duringtheperiod2001to2012inclusive.Regrettably,wefindconsiderableevidenceinsupportofthedetractorsofABMasascientificenterprise:elementaryplottingattributesareleftoffmoreoftenthannot;basicinformationsuchasthenumberofreplicatesorthebasisbehindaparticularstatisticarenotincluded;andfew,ifany,establishedmethodologicalcommunicationstandardsareapparent.Inshort,'anarchyreigns'.WhilstthestudywasconfinedonlytoABMpapersofJASSS,weconcludethatiftheABMcommunitywishesitsapproachtobeacceptedfurtherafield,authors,reviewers,andeditorsshouldtaketheresultsofourworkasawake-upcall.

Keywords:AgentBasedModelling,SocialSciences,Simulation,Publishing

Introduction

"Whilethetheoreticalandexperimentalfoundationsofagent-basedsystemsarebecomingincreasinglywellunderstood,comparativelylittleefforthasbeendevotedtounderstandingthepragmaticsof(multi-)agentsystemsdevelopment—theeverydayrealityofcarryingoutanagent-baseddevelopmentproject.Asaresult,agentsystemdevelopersareneedlesslyrepeatingthesamemistakes,withtheresultthat,atbest,resourcesarewasted—atworst,projectsfail."

"Itiswidelyacknowledgedthat…agent-basedmodels,canplayanimportantroleinfosteringunderstandingofthedynamicsofcomplexsystems.…However,current[agent-based]modellingpracticehastwosubstantialshortcomings:(1)Thereasoningbehindthechoiceofacertainhumandecisionmodelisoftennotwelldocumented;insufficientempiricalortheoreticalfoundationsaregiven;orthedecisionmodelisonlyassumedonanad-hocbasis…(2)Oftenthemodelisnotdescribedinatransparentmanner(clearandcomplete)thatwouldallowforreproducibilityandfacilitatethecommunicationofthemodelanditsresults."

1.1 ThetwoquotationsaboveconciselydescribeatragedyinthestoriedhistoryofAgent-BasedModelling(ABM).Thetragedybeingthat,whilstdescribingessentiallythesameparadox–thepromiseofABMapproachesinthesocialsciencesjuxtaposedagainstthelackofwelldevelopedpracticesinABMscience–thefirstquote(Wooldridge&Jennings1998)predatesthesecond(Mülleretal.2013)by15years.[1]

1.2 Ofcourse,duringthisperiod–onewhichhasseenanewglobalfinancialcrisis(GFC),theemergenceofseveralpotentiallypandemicinfectiousdiseases,andtheriseandriseofnetwork-basedsocialmediaplatforms–thechampionsofABMmethodologiesfromallrelevantfieldshavere-emphasised,inprominentpublishingplatforms,theneedfor,andrenewedrelevanceof,ABMmethods:DoyneFarmerandDuncanFoley(2009),MarkBuchanan(2009),andJean-PhilippeBouchard(2008)allleveragedthe'meltdown'ofthe2007–

http://jasss.soc.surrey.ac.uk/18/4/16.html 1 25/01/2016

2009GFCtocallforABMsinEconomicsatNature;whilstJoshuaEpstein,alsowritinginNature,pointedtothenetworkcomplexityinheritintheemergingH1N1outbreakof2009topointouthowsuitedABMsaretomodellinginfectiousdiseases(Epstein2009).Inshort,thepromiseofthemethodologyseems,duringthisperiod,aliveandwell.

1.3 Meanwhiletimelyandcompellingarticlesofthegenericform"FieldX,meetABM…,"werewrittenfor,amongstothers,the'humansystems'modellingcommunitybyEricBonabeauintheUSflagship,ProceedingsoftheNationalAcademyofSciences(Bonabeau2002),themind—actionsocio-economiccommunitybyNigelGilbertandPietroTerna(2000)inthe(thenbrandnew)MindandSocietyjournal,andforthesociologycommunitybyMichaelMacyandRobertWillerintheAnnualReviewofSociology(2002),eacharrivingatthestartofthiscrucialperiod.

1.4 Intermsofresults,afewABMpractitionershavefoundareceptiveaudiencefortheirinsightfulworks.Creative,unashamedlyABM,papershavefoundtheirwayintotopfield(e.g.intheEconomicsciences:AmericanEconomicReview(albeit,Papers&ProceedingsoftheAEA),JournalofEconomicDynamicsandControl,JournalofEconomicBehaviourandOrganisation)andgeneralist(e.g.Science)journalsduringthisperiod(Geanakoplosetal.2012;Howitt&Clower2000;Dosietal.2010;Limetal.2007).

1.5 However,outsideofthesefewhigh-pointsoftheABMsocialsciences,ABMstudiesingeneralhavefaredratherlesswell.Leombrunietal.'s((2005)surveyofthetop20Economicand10Socialsciencesjournalsfoundonlyahandfulofpublishedstudies(7intheformer,11inthelatter)uptothatpointintimewhichusedanABMmethodology.Recentpublishingpracticesseemlittledifferent.

1.6 VariousexplanationsaregivenforABM'sdifficultreceptioninfieldjournals.Chiefamongsttheseseemtobeaperceivedlackofwhatmightbecalled'intuitivetransparency'relativetotheclosed-form,deductive,toolsetsavailable.JoshuaMEpstein(2006),inhisclassiccontributiontotheHandbookofComputationalEconomics(vol2),writes(ch34),

"Therealreasonsomemathematicalsocialscientistsdon'tlikecomputationalagent-basedmodellingisnotthattheapproachisempiricallyweak(innotableareas,it'sempiricallystrongerthantheneoclassicalapproach).It'sthatitisn'tbeautiful."

1.7 Inasimilardirection,LeombruniandRichiardi(2005)considerthetwokeyperceivedproblemsof'Economists'withABMstudies(lackofgeneralisability,identificationorestimationproblems)standbehindtheheadlinecommentthatABMs,'don'tproveanything'.(Caseinpoint,LeombruniandRichiardi'spaperwaspublishednotbyaprogressive,forward-looking,Economicsfieldjournal,butratherbyPhysicaA:statisticalmechanicsanditsapplications–not,oneassumes,acommonlyreadjournalbymosteconomists.)

1.8 Whilstthethe'proof'(or'beauty')problemisratherover-statedbyfieldeditorsandreviewers(andishandledverywellbytheauthorsjustmentioned),alingeringproblemremains:thecommunicationofABMmethodsandresults–thisisanABMsciencepracticeissue.

1.9 ThesimplefactisthatABMstudiesoftendrawtheirheritageapartfromthedeductivemathematicalsciences,instead,theybuildonthesciencesofsoftwaredesignandnumericalsimulation.Andhere,ABMpractitionersinthesocialsciencesseemtohavesufferedheavilyfromalackofwell-establishedcommunicationtoolsandstandards.Richiardietal.(2006)capturethispointwell(paragraph1.5),

"Agent-basedmodelshavesolidmethodologicalfoundations.However,thegreaterfreedomtheyhavegrantedtoresearchers(intermsofmodeldesign)hasoftendegeneratedinasortofanarchy(intermsofdesign,analysisandpresentation)."[emphasisadded]

1.10 Theygoontoelaboratethisanarchyasfollows(paragraph1.5),

"a)Thereisnoclearclassificationofthedifferentwaysinwhichagentscanexchangeandcommunicate:everymodelproposesitsowninteractionstructure.

"b)Thereisnotastandardwaytotreattheartificialdatastemmingfromthesimulationruns,inordertoprovideadescriptionofthedynamicsofthesystem,andmanyarticlesseemtoignorethebasicsofexperimentaldesign.Often,thecomparisonbetweenartificialandrealdataisoverlynaïf,andtheparameters'valuesarechosenwithoutproperdiscussion;and

"c)Toooften,itisnotpossibletounderstandthedetailsoftheimplementationofanagent-basedsimulation.Thismakesreplicationadifficult,sometimesimpossibletask,thusviolatingthebasicprincipleofscientificpracticeandconfiningtheknowledgegeneratedbyagent-basedsimulationstonomorethananecdotalevidence."[enumerationadded]

1.11 …Anarchyindeed.Sadly,andbringingusbacktotheopeningquotationfromWooldridgeandJenning's(1998)article,thisexactproblemwasidentified(understandablythen)forthe'newfield'ofABMeightyearsprior(section8.2),

"Inafieldasnewasagentsystems,therearefewestablishedstandardsthatadevelopercanmakeuseofwhenbuildingtheagentspecificcomponentsofanapplication."

1.12 Sincethistime,therehave,ofcourse,beenrealattemptstoorderthe'anarchy'ofABMdevelopmentandcommunication.Asearlyas2000thepowerfulUnifiedModelingLanguage(UML)wasarticulatedtoincludeagents(Odelletal.2000)andsoofferedapotentialstandardisationcandidate(atleastofoneaspectofABMdevelopment),whilsttheOverview,Designconcepts,andDetails(ODD)(Grimmetal.2006)andODD+D(ODD+'Decision',Mülleratal.2013)protocolsof2006and2013respectivelyarosefromaninitiativewhichsoughttoresponddirectlytothe'anarchy'problemofABMdevelopment.Meanwhile,otherauthorshavefocussedtheircontributionstospecificareasofABMpracticesuchasmodelvalidation(Law2005;Windrumetal.2007;Moss2008)orvalidationandreplication(Heathetal.2009)and2-Dimensional,spatialvisualisation(Kornhauseretal.2009).Whilsttheseapproachesareallhighly

http://jasss.soc.surrey.ac.uk/18/4/16.html 2 25/01/2016

relevantandimportant(andinsomefocussedareasoftheliteratureappeartobehavinganimpactonthequalityofABMpractice,Grimmetal.2010),wewishtoprovidethecommunitywithaslightlymorequantitativeperspectiveonthestateofaffairs.

1.13 Specifically,inthisstudy,wefocusonthethird(c)aspectofRichiardietal.'s(2006)'anarchy'enumeration–thatofthebasiccommunicationofmethodologiesandresultsviaastudy'svisualshort-hand–thetables,figuresandequationsthatitemploystodescribeitsscience(whatweshallcallapaper's'objects').Weconsiderthatthissimpleaspecthasreceivedlittleattentioninthestandardisationeffortstodate.WherepreviousauthorshavespenttheirtimeonthedevelopmentanddesignaspectsofABMs,itwouldappearthatthecommunityhaslargelyassumedthatthequalityofbasiccomponentsofABMcommunicationwasingoodorder.Wecontendthatevenifamodelisdescribedaccuratelyandhelpfully,perhapsusingtheODDorODD+Dprotocol,andevenifthemodelhasbeenwellvalidated,itwillstillfailasapieceofscienceifitskeymethodologicalandresultsobjectsareofpoorquality.Thisdistinctionmattersperhapsmoresoinourvisually-orientated,short-attention-span,eraofscientificpublishing.Indeed,theso-called'mega-journal'PLOS-One,exemplifiesthistrend,inaskingauthorstonominatea'strikingimage'(forourstudy:'object')duringsubmission.Theypresumablyknowwellthevalueofsuchanimagefortheirsocialmediaandmarketingplatformintegrations.

1.14 Weconductastructuredanalysisofover900scientificobjects(figures,tables,orequations)thatarosefrom128ABMpaperspublishedintheJASSS,duringtheperiod2001to2012inclusive.Byfocussingonasingleoutlet,wereduceanyinter-journalvarianceonpublishingandeditorialstandardsandallowauthors'ownpracticestocometothefore.

1.15 ThejournalJASSSwaschosenforthestudyfortwoprinciplereasons:first,itiswell-regardedasageneralsocialsciencesABMstudy'clearing-house'withanactiveandengagedreadership,sostandswellforcross-disciplinarysocialsciencesABMpublishingtrends;andsecond,JASSS,beinganopen-access,online,HTML-based,journallendsitselftofacilemulti-yearstudyofthatwhichwepropose–paperscouldbeidentified,harvestedandanalysedwithease.WenotealsothatJASSSactsasacrucial'reverberationboard'forthetransmissionofrelevantideasbetweenthesocialandphysicalsciences(Squazzoni&Casnici2013).

1.16 Finally,thestudyperiod2001-2012coversthiscrucial'earlydecade'ofsocialsciencesABMscience.Asmentionedearlier,notonlyweremultiple,keycontributionsmadetointroduceABMmethodstovarioussocialsciencesfieldsatthestartofourstudyperiod,butalsotheperiodcoversalmostallofthemajorcontributionstothestandardisationprogramofABMsocialsciencemodeldevelopmentandcommunicationwhichhasbeenongoingsinceatleast2000.IftheseeffortshavehadanyearlyimpactsonABMcommunicationpractices,theseshouldbeevidentinourstudy.

TheABMPaperSample

EnrollingJASSSPublicationsintothePrimaryDatabase

2.1 World-wide-web(WWW)linkstoallscientificarticles(notreviews,forumetc.)publishedinJASSSfromvolume1to15(1998-2012)wereobtainedfromtheindexpage(e.g.http://jasss.soc.surrey.ac.uk/1/1/1.html),producing487links.

2.2 Ascriptwaswrittentodownloadthe.htmlfileofeacharticlefromtheJASSSsiteandthenprocessthemeta-dataoftheHTMLfilestoobtainthepaper'sfields(title,authors,keywords,abstract,volume,issue,dateofpublication).Insevencases,themeta-dataformatdeviatedfromtheJASSSstandard,causingthearticlestobedroppedfromthesample(e.g.http://jasss.soc.surrey.ac.uk/11/4/3.html).Thisproduced480articleswithsuccessfullyharvestedmeta-data.

2.3 Next,acorpusofuniquekeywords(the'Subject'meta-datacontentintheJASSStemplate,foranexample,seeAppendixA)(note,keywordscanbephrases)wasbuilt,takingcareofnon-materialkeywordvariationssuchasthepresenceorabsenceofahyphen.Inall,1501keywordsweregatheredinthisstep.

2.4 Finally,anarticlewasenrolledintotheprimarydatabaseifitcontainedoneofthefollowing'ABM'keywordschosenbytheauthorsafterreviewingtheuniquekeywordset,eitherexactly,orasakeywordroot(e.g.'multiagents',or'multiagentsystems'wouldmatchwith'multiagent')(numberofmatchesinparentheses):'agentbased'(151),'multiagent'(39),'socialsimulation'(31),'individualbased'(9),'multiagent'(8),'agentsimulation'(2),and'artificialagents'(1).Ofthe480papers,220uniquepaperscontainedamatchingkeywordwereenrolledintheprimarydatabase.Themajorityofpapers(202)hadasingleABMkeywordmatch,16matchedtwoABMkeywords,andtwomatchedthree.

2.5 Toassistwithreplication,afulllistingoftheresulting937objects,theirhomepaperID,andbasicdescriptorsisgiveninacomma-separatedvaluefileonlinewiththiswork.

Methods

Defining&ValidatingtheObjectTaxonomy

3.1 Sincenocleartaxonomyofobjectsexiststoourknowledgeintheliterature,theauthorssetaboutbuildingausefulandfaciletaxonomytodescribethepublicationpracticesineachABMpaper.Ourmethodologyhasbeenguidedbyexperiencearisingfromrelatedsocialsciencestaxonomy/encodingexercises,albeitofatextualnature(Haraetal.2000;Rourke&Anderson2004).Buildingthetaxonomyorganicallyproceededin6cyclicalsteps:

1. Twoauthorsreviewedthemethodsandresultsobjectscontainedwithinarandomlyselected20papersub-sampleofthe

http://jasss.soc.surrey.ac.uk/18/4/16.html 3 25/01/2016

primarydatabase(SampleA).2. Adrafttaxonomywascompiledcollaboratively,includinghierarchicaldescriptions.3. Thedrafttaxonomywasthenapplied,independentlybytwoauthors(SDA,BH-M)toSampleA.4. Thesameauthorsthenmettodiscussdisagreementsandimperfectionsinthedrafttaxonomyleadingtoarefinedtaxonomy.5. Therefinedtaxonomywasthenappliedindependentlybytwoauthors(SDA,BH-M)toafurther20paperrandomsub-sample

fromtheprimarydatabase(SampleB).6. Asecondmeetingwasthenconvenedbetweenthetwoauthorstovalidateandfurtherrefinethetaxonomyleadingtothefinal

taxonomy,andclarifyitsapplicationtothepooled(SamplesA+B)40papersampleusedinprevioussteps.

3.2 Note,sinceourfocusofanalysisisonthedecisionsofauthorsastohowtheypresentthemethodsandresultsofABMstudies,weskipduplicateobjecttypesfoundinanyarticle.Thatis,thefirstinstanceofagivenobjectisstudied,withsecond,third,andsubsequentobjectshavingthesamegeneralattributesasthefirst,notincludedintheanalysis.Typically,subsequentobjectsofthesamekindwerepresentedwithidenticalfeatures(orlackoffeatures)asthefirstobject,presumablysinceauthorscreatefigures,ortables,via'templates'.

SummaryoftheObjectTaxonomy

Table1:SummaryoftheObjectTaxonomy.Countsofobjectsateachlevelofthehierarchyaregiveninthecells.

http://jasss.soc.surrey.ac.uk/18/4/16.html 4 25/01/2016

3.3 Table1providesasummarypresentationoftheObjectTaxonomy.Ascanbeseen,alltheobjectsarefirstcategorisedbasedonwhethertheyareusedintheMethodsorResultssections.AgivenobjectisthenlabelledasFigure,TableorEquationbasedonwhattheauthorsoftheoriginalpapershavenamedthem.Finally,foreachobject,dependingonwhereithasappearedanditstype,wecollectedarangeoffurtherinformationrelevanttotheobjectaspresentedinthetable.

ApproachtoanObject'sSurroundingContext

3.4 Ifspecificcharacteristicsoftheobject'snature(e.g.the'Target'ofaMethodsFigureorTable,ortheGranularityofaResultsFigureorTable,referTable1)werenotobviousfromthecontentsoftheobject,oritscaption,theinformationwassoughtfromthesurroundingtext.However,inthespecificcasewherethesimulationresultsfigureswerestudiedfortheirquality(seefinalResultssection,'QualityofABMresultsplottingovertime'below)amorestringenttestwasappliedrequiringthekeyinformationwewerelookingfortobepresentinthefigureitselforfigurecaptiononly.Weareoftheviewthataresultsfigureshouldbe,asmuchaspossible,self-contained.Here,weareinfluencedbythewidelyused(e.g.bytheeditorsoftheProceedingsoftheNationalAcademyofSciencesoftheUSA)"ScientificStyleandFormat:TheCSEManualforAuthors,Editors,andPublishers"(8thedition)(CouncilofScienceEditors2014,section30.3)(seeespeciallysection30.2.2ofthisguide,'FigureCaption').

ExamplesofTaxonomicObjects

3.5 Note:inallcases,exceptwherestatedotherwise,exampleobjectsfrompapersareprovidedwithoutexplicitattribution.Ourintentionisnottosingleoutparticularauthors,butrathertopointtogeneraltrendsinthediscipline.SourceJASSSarticlecitationinformationforanyexampleprovidedcanbesoughtfromthecorrespondingauthor.

3.6 InFigures1to9weprovideexamplesofthetopthree(byincidenceinthedatabase)objectsusedtocommunicatemethodologiesinthedatabase.

http://jasss.soc.surrey.ac.uk/18/4/16.html 5 25/01/2016

Figure1.ExampleMethodsobject'Look-uptable'.Inthiscase,eachrowofthetabledescribesfeaturesofa'resource'usedinthemodel.

Figure2.ExampleMethodsobject'Schematicdiagram'.Thefigureindicatesrelationshipsbetweenexampleagentsinthemodel('Seller001','Buyer001')andfunctions,withoutadheringtoUMLformalism,ora'Flow-chart'presentation.

http://jasss.soc.surrey.ac.uk/18/4/16.html 6 25/01/2016

Figure3.ExampleMethodsobject'Screen-shot'.Atypicalscreen-shot,inthiscase,annotatedandtakenfromtheNetLogosoftwareplatform.

3.7 InFigures1,2and3,examplesofthe'Look-upTable','SchematicDiagram',and'Screen-shot'objectsarepresented.Fortheformer,weweredirectedbythenomenclatureoftheauthors,where'Figure'wasused,the'Figure'taxonomyobjectclasseswereemployed,whilst'Table'inducedthe'Table'taxonomyclasses.'SchematicDiagram',asintheexample(Figure2),wasusedtocodeanyfigurewhichconveyedtherelationshipbetweenmultipleaspectsofthemodel(agents,procedures)butdidnotconformtoeitheraflow-diagramorUML-formalism.Wehavenotsoughttoclassifythesefiguresfurther,owingtotherichvarietyofsymbolicandrelationalelementsemployedbyauthors.

Figure4.ExampleMethodsobject'Parameterinitialisation'.Eachrowofthetableprovidesparametervaluesforthreesettingsinthemodel.

http://jasss.soc.surrey.ac.uk/18/4/16.html 7 25/01/2016

Figure5.ExampleMethodsobject'Processes(behaviours,rules)'.Eachrowofthetableprovidesdefinitionsandthresholdsforvariousrulesandquantitiesinthemodel.

Figure6.ExampleMethodsobject'Experimentalsetup'.Eachrowofthetablepresentsthechoicesofdecisionrulesandparameterstestedineachexperiment.

http://jasss.soc.surrey.ac.uk/18/4/16.html 8 25/01/2016

Figure7.ExampleEquationobjects.a:'A-temporal',b:'Discrete-time',andc:'If-else'.

3.8 Figures4,5and6provideexamplesofhighlyused'Table'classobjects,being,unsurprisingly,focussedonparameterinitialisation,ruledefinitions,and(numericalsimulation)experimentalconditions.Figure7(a,bandc)provideexamplesoftheprominentequationtypesusedbyauthorsinthemethodssection.

Training&EncodingtheentirePrimaryDatabase

3.9 Next,aresearchassistant,familiarwithABMscience,wastrainedinapplyingthefinaltaxonomy,beforeapplyingittoSampleA.Areviewandtrainingmeetingwasheldwithoneoftheauthors(BH-M)toprovideguidanceandclarificationtotheresearchassistant,beforetheyappliedthefinaltaxonomytoSampleB.Afinaltrainingmeetingwasheldwithoneoftheauthors(BH-M)tocompletethetrainingandensurestrongcoherencewiththeencodingapproachoftheauthors.

3.10 Finally,theresearchassistantwentontoencodetheentirePrimaryDatabase,recordingdecisionswithanonlineformtofacilitatedataentry.Validationoftheresearchassistant'sapplicationofthefinaltaxonomytothePrimaryDatabaseincluded:on-going,ad-hoc,conferringwithoneoftheauthors(BH-M);randomcheckingbythesameauthoroftheencoding(around10%ofthecodedobjectsinall);andthenauthorcheckingofanyresidual'flagged'objects.A'flag'wasusedwherevertheresearchassistantwashesitantforanyreasoninthecorrectencodingtouse.Finally,adifferentresearchassistantreviewedall'flagged'objectstoconfirmthatthecorrecttaxonomyhadbeenapplied.

3.11 Duringcoding,anumberofpaperswerefoundnottobeABMinnatureandweredroppedfromthedatabasefollowingasimplerule:ifthepaperdidnotconveythemethodsandresultsofascientificenquiryusinganABMmodel,itwasdropped.Forexample,somepaperswerefoundtodiscussABMtheoryorpractice(suchasDeichselandPyka's(2009)'APragmaticReadingofFriedman'sMethodologicalEssayandWhatItTellsUsfortheDiscussionofABMS')orwereintendedas'positionpapers'foralternativedisciplines(suchasAhrweiler's(2011),'ModellingTheoryCommunitiesinScience').Additionally,paperspublishedpriorto2001(1998–2000)weredroppedduetothesmallnumberofresultantABMpapersintheseyears.

3.12 Intheend,theStudyDatabasecontained937objects(139equations,599figures,199tables)drawnfrom128papers,spanningtheyears2001–2012.

Quantitativeanalysis

3.13 QuantitativeanalysiswascarriedoutwithacombinationoftoolsincludingMicrosoftExcel(pivottables)andMATLAB(R2014b)(visualisations).

Results

Summaryofthedata

Table2:Uniquepapers,byyear,intheStudyDatabase.

Year Count2001 42002 92003 132004 72005 112006 112007 142008 132009 112010 182011 82012 9Total 128

http://jasss.soc.surrey.ac.uk/18/4/16.html 9 25/01/2016

Table3:Objectcounts,byMethodorResults,andbyObjectType,byyear,intheStudyDatabase.

Year Methods Results TotalEqtn. Figure Table Eqtn. Figure Table Total

2001 10 2 20 2 342002 6 21 6 22 5 602003 16 32 20 29 8 1052004 8 19 14 20 2 632005 10 33 6 23 7 792006 16 15 11 29 9 802007 17 20 14 37 10 982008 21 29 10 37 6 1032009 9 26 9 27 8 792010 15 36 16 1 39 9 1162011 5 22 9 12 3 512012 15 20 7 21 6 69Total 138 283 124 1 316 75 937

4.1 Tables2and3presentsummarystatisticsforpapersandobjects,respectively,intheStudyDatabase.Duetotherelativelysmallnumberofpapersenrolledeachyearovertheperiod,communicationpracticesovertimewillbestudiedatamulti-yearaggregatelevelbelow.Perhapsunsurprisingly,Table3demonstratesthetendencytoconveymethodologicalaspectsofastudypredominantlywiththeuseofabalanceofequations,figuresandtables,whilstresultsareseldomcommunicated(oranalysed)inequationform(apointwereturntolater),andlargelyfindexpressioningraphical(figure)presentation.

4.2 Toanalysepublishingpracticesovertime,westudytheincidenceofpaperswhichhaveatleastoneobjectofagiventype.AspresentedinTables2and3,itisobviousthatsomeyearshaveasmall(<5)numberofpapersorgivenobjecttype.Thus,todrawmeaningfulconclusions,weaggregateoverfour,threeyear,periods:2001–2003,2004–2006,2007–2009,and2010–2012(inclusive).

Temporalpatternsinmethodologicalpresentation

Table4:ABMmethodsobjectuseovertime.Percentofpapersinthedatabasepublishedwithinthegivenperiod,havingatleastonemethodsobjectofagiventype,sortedbydescendingincidenceinfinalperiod.

Type '01-'03 '04-'06 '07-'09 '10-'12Look-up-table 50 59 58 66Schematicdiagram 58 45 58 57Screen-shot 35 31 32 37Pseudo-code 0 17 16 26XYplotofvarioustype 15 3 11 14Raw-code 15 7 11 11Flow-chart 19 28 18 9Map 12 3 5 9Matrix 0 10 11 6UML 8 10 11 3Example-agent 12 17 0 0Table(other) 12 0 0 0Misc(allothers) 23 21 11 20Relevantpapersinperiod(count) 26 29 38 35

http://jasss.soc.surrey.ac.uk/18/4/16.html 10 25/01/2016

Figure8.PatternsinABMmethodsobjectuseovertime.Eachbubblerepresentsthepercentageofpapersinthedatasetpublishedduringthegiventhreeyearbandwhichincludedatleastoneoftheobjectsindicated.Note:sizingofthebubblesisnon-linear.

4.3 InTable4,andvisualisedinFigure8,trendsinpublishingpracticesformethodologicalpresentationarestudied.Weusetheincidence-rateofanobject,definedasthefractionofrelevantpapers(i.e.thoseincludingmethodologicalobjects)inthegivenperiodintheStudyDatabasewhichutilisedtheobjectatleastonce,asthesummarymeasurethepracticesundertakenbyauthors.Wenotethatpercentagesdonothavetosumwithinacolumn,asonepapermayexhibitmorethanoneobjecttype.

4.4 Forthesakeoftheanalysis,wepoolFigureandTableobjecttypeswhichtogethergives25uniquemethodsobjects.Further,wefocusononlythoseobjectswhichobtainanincidence-rateofatleast10%inoneofthefourperiods.Wefind12suchobjecttypes(Table4)whichfitthiscriterion,coveringbetween80%and89%ofallrelevantpapersinthatperiod.Prominentobjecttypesincludedin'Misc(allothers)'include'Algorithm','3Dmodelofenvironment','Agenttypehistogram',and'Video'(recall,JASSSiswhollypublishedonlineandmulti-mediaisencouraged,thoughapparentlyusedsparingly).

4.5 AscanbeseeninFigure8,thetopobjectchoices:'Look-up-table','Schematicdiagram'and'Screen-shot'areverystableoverthestudyperiod.Whatisperhapssurprisingisthatthemoreformal,structured,modeldescriptiontoolofUMLappearstohaveallbutdisappearedasaformofcommunicationovertime.Alternatively,andperhapsmoreencouragingly,raw-code,andpseudo-code(arguablythemostinformativeanddetaileddescriptionofthemechanicsofamodel)haveretained,orslightlyincreased,theirincidence-rateovertime.

Temporalpatternsinresultspresentation

Table5:ABMresultsobjectsovertime.Percentofpapersinthedatabasepublishedwithinthegivenperiod,havingatleastoneresultsobjectofagiventype,sortedbydescendingincidenceinfinalperiod.

Type '01-'03 '04-'06 '07-'09 '10-'12Simulation-only 96 100 100 97Empirical-only 0 8 8 10Mixed:Empirical–simulation 0 0 8 13Theory-only 8 12 3 3Mixed:Theory–simulation 8 4 3 3Notreported 4 0 0 0Relevantpapersinperiod(count) 24 26 38 31

http://jasss.soc.surrey.ac.uk/18/4/16.html 11 25/01/2016

Figure9.PatternsinABMresultsobjectuseovertime.Eachbubblerepresentsthepercentageofpapersinthedatasetpublishedduringthegiventhreeyearbandwhichincludedatleastoneoftheobjectsindicated.Note:sizingofthebubblesisnon-linear.

4.6 InTable5,andvisualisedinFigure9,wepresentasimilaranalysisforresultsobjects,again,poolingFigureandTableobjecttypes.Inthisanalysis,wefocusonthebasisoftheresultspresentation–whethertheresultsaredrawnfromsimulationresultsonly(i.e.quantitiesdrawnfromartificialdatasets),empiricalresultsonly(I.e.,quantitiesdrawnfromsurvey,ormeasureddatasets),theoreticaloutcomes(i.e.numericalcalculationsofparametricsystems,typicallywithoutstochasticsources),orsomecombinationoftheabove.

4.7 Whatthedatainthetableandthefigureshowisthatthepredominantpractice,byapproximatelyninetooneistheuseofsimulationdataonlyfromwhichtodrawresults.Thatis,authorsofidentifiedABMstudies,inJASSS,almostexclusivelysupporttheirkeyresults(eitherastablesorfigures)withtheuseofsimulationdataonly.Authorsarenot,therefore,oftenfoundtobepresenting(forverification,orcomparison)otherquantitytypesalongwiththeirsimulationdata.Thereisperhapssomeevidenceofatrendtowardscomparisonofsimulationquantitiestoempiricalquantities('Mixed:empirical–simulation')coupledwithatendencyawayfromtheoreticalcomparison,butthesampleistoosmalltoasserttheseassignificanttrends.

QualityofABMresultsplottingovertime

4.8 WefurtheranalysethepresentationofresultsintheStudyDatabasebyassessingwhetheragivenresults(figures-only)objectexhibitsoneoffivequalityattributes.Thefiveattributeswere:

1. (XY1)XandYaxeslabelsincluded(textualorpro-numeralinnature);2. (XY2)XandYscaleincluded(requiresnumericmax/minorticks,plusunits);3. (TYP)Thebasisoftheplottedresultsindicatedclearlyonthefigure,inthecaption,orinthesurroundingtext(e.g.simulation-

only,theory-only,mixed…etc.);4. (PAR)Theparametersusedtogeneratedthedataincluded(eitheronthefigure,inthecaption,orinreferencetoatable);5. (NUM)Forsimulationresults,thecountofsimulations(N)usedtocreatethesummarydataincluded.

4.9 Eachfigurewasgivenonepointifitdisplayedanyoftheabovefiveattributes,leadingtoamaximumscoreoffive.Itshouldbestressedthatweconsidertheabovefivefeaturestobetheabsolutebareminimumforeffectivecommunicationofscientificresults,followingtheprincipleslaidoutintheUniversityofChicagoScientificStyleandFormatmanual(8thedition)(CouncilofScienceEditors2014,section30.3).

Figure10.Examplesimulationsresultsfigurefroma2012JASSSarticlescoringpoorly(score1/5)underthequalityscoringtool.Inthiscase,thefigurereceivedonepointforincludingtheXandYscale.Allotherfeatures(axeslabelling,resultsbasis,parameters

used,simulationN)weremissing.

http://jasss.soc.surrey.ac.uk/18/4/16.html 12 25/01/2016

Figure11.Examplesimulationsresultsfigurefroma2012JASSSarticlereceivinganaveragequalityscore(score3/5).Inthiscase,thefigurereceivedonepointeachforincludingtheXandYscale,axeslabelsandresultsbasis.Otherfeatures(parametersused,

simulationN)weremissing.

Figure12.Examplesimulationsresultsfigurefroma2012JASSSarticlereceivingahighqualityscore(score4/5).Inthiscase,thefigurereceivedonepointeachforincludingtheXandYscale,axeslabels,resultsbasisandparametersused.However,norecordof

thenumberofsimulationsusedwasprovided.

4.10 InFigures10,11and12,weprovide(anonymous)examples,takenfromtheStudyDatabase,ofalow(score:1/5),medium(score:3/5)andhigh(score:4/5)qualityresultsfigureasassessedbyourattributemethod.

http://jasss.soc.surrey.ac.uk/18/4/16.html 13 25/01/2016

Table6:Thequalityofsimulationresultsobjects.Datafor229resultsfigureobjectswhichwerelabelledas,orappearedtobe,simulation-backedinnature.

Attribute CountofObjectswithAttributes Exists%

Missing%

Note

XY2 210 91.7 8.3 XandYscaleincludedXY1 177 77.3 22.7 XandYaxeslabelledTYP 126 55.0 45.0 SimulationbasisclearlyindicatedPAR 92 40.2 59.8 ParametersincludedNUM 75 32.8 67.2 CountofsimulationrunsOfTotal 229

4.11 Inall,229resultplots,drawnfrom106uniquepapers,wereanalysedusingthequalitymetric.Theseobjectsallclaimed,orappeared,tobesimulation-basedinnature–thepredominantformofresultspresentationchoice.Byaclearmargin(Table6),themostprevalentattributesdisplayedweretheXY2andXY1attributeswithover91%and77%ofobjectsexhibitingXandYscales,andXandYaxeslabellingrespectively.Toaidthereader'sunderstanding,weprovideinFigure13arare'negative'exampleinwhichasimulationresultsobjectfailedthe'XY2'(XandYscale)attribute.However,perhapsalarmingly,over40%ofobjectsfailedtodemonstrateoneormoreoftheotherthreeattributes.

Figure13.Examplesimulationsresultsfigurefroma2007JASSSarticleinwhichtheXY2(XandYscale)attributewasmissing.Casesofthisspecificattributebeingmissingwererare,occurringinlessthan10%ofrelevantobjects.

Figure14.PatternsinABMresultsqualityovertime.Barsrepresentthepercentileofofthedistributionofaverageobjectscores,bypaper,forthegivenperiod.Thetotalcountofrelevantpapersineachperiodwas('01-'03')18,('04-'06)27,('07-'09)34,and('10-'12)

27respectively.Objectqualityscoringexplainedinthetext.

4.12 Ifoneassumesthatauthorsareultimatelyresponsibleforthepresentationofresultfigures,thenonecantakeanaveragescoreacrossallqualifyingresultsobjectsbypaper(author).InFigure14,wepresentthepercentiles,byaggregatetimeperiod,ofaveragepaper

http://jasss.soc.surrey.ac.uk/18/4/16.html 14 25/01/2016

scores,whereeachpaperisrepresentedbytheaverageofthescoresoftheresultsobjectswithinit,codedinourstudy.

4.13 Again,stressingthatweconsiderthefiveresultsfigureattributesasbasicscientificrequirements,theresultsarealarming,bothintermsoftheaveragequality,andthetendovertime.Forinstance,themedianpaperscorein2001–2003of2.50isonlymarginallyimprovedupon(3.00)by2004–2006,andthennotagaininthesubsequentyears.Thisindicatesthataround50%ofpaperspresentresultsfigureswhichmisstwoormoreofthebasicfiveattributesweidentify.Atthetopend,ahighpointpaperscoreof4.62isobtainedforthe95thpercentilein2004–2006,butthisslidesto4.00by2010-2012.

4.14 Thesedataindicatethatforwhateverreason,thequalityofsimulationresultspresentedbyABMauthorsinJASSSoverthelastdecadeisgenerallypoor,andthestandardsarenotimproving.Wediscusspotentialreasonsfortheseproblemsbelow.

Discussion&Conclusions

5.1 Thisstudysetouttostudyanunder-reportedaspectofABMscience:thecommunicationpracticesandqualityofmethodologicalandresultsobjectsinABMstudiesoverthelastdecade.Asnotedintheintroduction,whilstotheraspectsofABMdevelopmentandcommunicationhavereceivedacademicattentionandproposals,wearenotawareofanysuchreviewofthekindwecarryouthere.

5.2 Againstabackgroundandhistoryofearly'anarchy'ontheonehand,andattemptsatformalisationontheother,wewereinterestedtoseeiftherewereanytrendstowardsorderordisorderinABMpublishingpractices,andinthegeneralqualityofthesepracticesforscientificends.

5.3 Below,wedrawtogetherthreekeyconclusionsfromourwork,andprovidesometentativereflectionsontheirpossiblecauses.

Conclusion1ABMscience'smethodological'anarchy'showsnosignsofsubmissiontoformalism

5.4 WeareindebtedtoRichiardietal.'s(2006)'anarchy'descriptor.Lookingatthepracticesrevealedinthisstudy,weseenoevidencethatthestateofaffairsischanging.Threedominantmodesofvisualcommunicationwereintheascendencyin2001–2003(Look-up-tables,'Schematic'diagrams',andScreen-shots),andthesamethreesatinthesamepositionadecadelater(Figure8).Whilstlook-up-tablesandscreen-shotscouldperhapsbeleftasidefromthisanalysis,astheyservetheirown,specific,communicativepurpose,theuseofrelationaldiagramsnotfittinganyparticularformalism(whichwecall'schematics')isinteresting.Indeed,whenonenotesthattheincidence-rateofUMLandFlow-chartdiagramshasdeclinedduringthedecade,wecanonlyconcludethatthesocialsciencescommunityhasturneditsbackon,orperhaps,hasneverproperlyengagedwith,theuseofmoreformalvisuallanguagesforconveyingtothereaderthecorerelationshipsamongstmodelcomponentsandagents.

5.5 ApotentialexplanationforthislackofengagementisofferedbyHeathetal.'s(2009)'manyfieldsofstudy'conclusiontotheirsurveyof297ABMpapers,

"ABMisconnectingdiversefields.Thefieldsofbiology,business,ecology,economics,themilitary,publicpolicy,socialscienceandtraffic,amongothers,alluseABM.ThesediversefieldsaretryingtounderstandcomplexsystemsandareusingABMasaonecommontool.…afterreviewingthesurveyedarticlesitisclearthateachfieldhasdevelopedtheirownABMterminologytodescribetechniques,applicationsandresults,havetheirownABMstandardsandtheirownABMphilosophies."(paragraph4.4)

5.6 Whilstthepaperssurveyedherearealldrawnfromthe'socialsciences'communityastheywerepublishedbyJASSS,asimilardrivercouldbeconjectured.OneofthepowerfulhallmarksofJASSSisitswideembraceofagent-basedsimulationpapersfromallacrossthesocialsciences.However,thisdiversitywillbringwithitadiversityofphilosophiesofknowledge,andadiversityofexpertisewithquantitativeandcomputationalmethods.Inparticular,WooldridgeandJennings'(1998)'Agent-orientedDevelopment'pitfall#4.3comestomind,"Youforgetyouaredevelopingsoftware"[emphasisretained].WooldridgeandJenningswrite,

"Unfortunately,becausetheprocess[ofdevelopinganyagentsystem]isexperimental,itencouragesthedevelopertoforgetthattheyareactuallydevelopingsoftware.Projectplanstendtobepre-occupiedwithinvestigatingagentarchitectures,developingcooperationprotocols,andimprovingcoordinationandcoherenceofmulti-agentactivity.Mundanesoftwareengineeringprocesses—requirementsanalysis,specification,design,verification,andtesting—becomeforgotten.Theresultofthisneglectisaforegoneconclusion:theprojectflounders,notbecauseofagent-specificproblems,butbecausebasicsoftwareengineeringgoodpracticewasignored."[emphasisretained]

5.7 Doestheanthropologist,writingtheirfirsteverABMin(say)NetLogo,grippedbythenotionthattheirtheoreticalhunchcan,forthefirsttime,bemodelledandvisualisedinreal-timebeforetheireyes,thinkthattheyarenowa'softwareengineer'?(Dotheyevenknowwhatthatis?)Theanswerisobviously'no'.OuranthropologistgetsonwithusingtheABMtooltosupporttheirscientificconclusion.Alongtheway,however,thedevelopment,validation,andcommunicationoftheirmethodologyandresultswillverylikelybe'home-spun'.Andso,anotherbespokedaughterofthe'schematic'classisborn.

Conclusion2ThescientificpresentationofABMresultsneedsimmediate,remedial,attention

5.8 Thesecondconclusionrestsonthealarmingpatternsuncoveredbyoursimple'5attributes'qualitysurveyofsimulation-basedresultsplots(Table6,Figure14).That60%ofthese'results'figuresdidn'tclearlyindicatetheparametersusedtogeneratethem,northenumberofsimulationsbehindthem,strikessomewhatofamortalblowtoanyhopeofsuccessfulreplication.Or,atthepaper-levelofobservation,thatthe75thpercentileaverageresultsfigurequalityscoreislessthan3.5,duringthemostrecentperiod(2010–2012)intheStudyDatabasesuggeststheproblemofadequateresultspresentationpracticeiswide-spreadandcontinuestoday.

http://jasss.soc.surrey.ac.uk/18/4/16.html 15 25/01/2016

5.9 Atfacevalue,itwouldseemthateithertheinterpretationorenforcementofJASSS'ownauthorguidelinesisatfaultastheyappearclearonthematterofreplication,

"Authorsarestronglyencouragedtoincludesufficientinformationtoenablereaderstoreplicatereportedsimulationexperiments."

5.10 However,theongoingtextoftheseguidelinesfocussesmoreontherequestthatfullmodelcodebemadeavailablethroughathird-partysite,ratherthanthedetailsneededtoactuallyusethemodeltoreplicatetheresults.Indeed,point(4)ofJASSS'sstated'Refereeguidelines'continuesinthis'algorithmic-centric'view,askingrefereestocommentexplicitlyon,

"Ifthearticledescribesasimulationmodel,isthereenoughdetailprovidedfortherelevantoutputfromthemodeltobereplicatedbyareader(thedescriptionmightbeintheformofanalgorithm,pseudo-code,oraccesstothesimulationprogramitself)?"

5.11 Whilstthequestionis'replication',theapparentsufficientanswer,accordingtoJASSS'guidelines,isthatauthorsprovideanindicationofthealgorithmusedtogeneratetheresults.Ofcourse,ifthe'simulationprogramitself'isprovided,thenitispossible,withasuitablywrittencode-base(andrunfiles)toidentifytheexactparametersusedforeveryresultinthepaper.However,anythingshortofthiswillnotsuffice:'analgorithm',or'pseudo-code',onitsown,willnotleapintoactionandproducetheresultsinthepaper.

5.12 Here,thereseemsaclearandsimpleopportunityforJASSS(atleast)totightenitsguidelinesaroundresults,broadeningits'algorithmic-centric'viewofreplicationtoincludealltheinformationrequiredtoreplicatetheresults,includingforexample,parameters,initialisationsettings,random-numberstreamdefinitions,andperhapsevenhardwarearchitecture.Again,suchconsiderationsnodoubtariseimmediatelytothemindofthesoftwareengineer,butnot,ouranthropologist(oreconomist,orsociologist,or…)colleague(weincludeourselveshere).

Conclusion3ParametricinterpretationbyestimationofABMsimulationdataappearstobemissinginaction

5.13 CloseinspectionofTable3highlightsonefurtherpatterninthepracticesrelatedtoanalysingABMresults:thelackofestimationofABMsimulationdata.The'smoking-gun'forthisconclusionisthealmostcompletelackofequationobjectswithintheresultssectionsofthe128papersreviewed,indeed,justone'results'equationwasidentifiedbyourmethods.WhilstwereadilyacknowledgethattherearemanycaseswhereparametricestimationofABMoutputsisunnecessarytojustifyscientificconclusionsduetotheinherentunpredictabilityoremergentcomplexityofclassesofABMmodels,therearemanyothercases,(forexample,whereanABMmodelistobecomparedtoempiricaleconomicdata)whereestimationisnecessary.Thatjustonepaperappearedtogodownthispath,wouldseemthatestimation,asananalysistechnique,orasaparametertuningtool,iseffectivelyunknowninthesocialsciencesABMcommunity.

5.14 Tobefair,thereislikelyculturalandtechnicalreasonsatplaybehindConclusion3.Culturally,werefertothe'manyfieldsofstudy'argumentproposedaboveinreflectiononConclusion1.ABMsocialsciencedemands,attimes,avariedandreasonablytechnicalskill-set:notonlymustthesocialscientisthavetheirownfield-specificknowledgeandnoveltyofcontribution,aswehaveseen,theyshouldalsoideallybeawareofsomebasicsoftwareengineeringprinciples,andhere,theywouldnowappeartoneedfurthertobefamiliarwithtime-seriesanalysisandestimationtechniques.SupposingagainthatJASSSwelcomesahigherproportionof'manyfields'authorswhoareexploringABMasatooltoilluminatetheirdisciplinethanotherjournals,thatalargeproportionofpapersdonotgodownthepathofestimationisunderstandable.

5.15 However,thereisasecond,technicalreasonthatcouldbeadvanced.Supposethatwearewrong,andthatJASSSauthorspredominantlywouldliketouseestimationtechniquestoassesstheartificialdatatheirmodelsproduceandtheyarewellversedin'standard'approaches,thentheywouldquicklydiscoverthatestimationofABMscanbepresentsomeuniquechallenges.TwofeaturesofalargenumberofABMscausetrouble–first,thenormallylargenumberofparameters,andsecond,theexistenceofnon-lineardynamics(oftenthereasonforchoosingABMtechniquesinthefirstplace).Together,theseattributescauseenormousdifficultyforestimation.Thereissome'hope'onthehorizon,however:veryrecentlyGrazziniandRichiardi(2014,2015)havebeguntoprovidesomecredibleoptionsfortheestimationofergodicandnon-ergodicABMtime-seriesinthepresenceoftheseattributes.WhilstGrazzini'searlier,butstillrelativelyrecent(2012)studyinJASSSshouldbehighlightedagaintotheABMsocialscientistscommunityasitenablesanauthortoidentifywhethertheirABMdataexhibitsstationarityandergodicityinthefirstplace(crucialquestionsforchoiceofestimationtechnique).

5.16 Hence,Conclusion3shouldnotbeseenasallthatsurprising,andshouldbereadmoreasaconfirmationofthestateofABMsocialscienceingeneral.However,withtheeffortsnowbeingexpendedonthisproblem(seeespeciallythecompanionworkofLeeetal.2015),ifasimilarpaucityofequation-basedestimationanalysisofABMoutputswasdiscoveredoverthenextdecade,differentconclusions(andprescriptions)wouldneedtobemade.

5.17 Beforedrawingfinalconclusionswereturntothemostobviouslimitationofourstudy:theuseonlyofJASSSABMpapersforouranalysis.Weacknowledgeseveralproblemshere.First,selectionbias:byconfiningourselvestoJASSSwehavenovisionofthequalityofABMstudiespublishedelsewhere:paperssubmittedtoJASSScouldbeofhigherorlowerqualitythanthe'fieldmedian';orpaperscouldexpressanunusualmixofsocialscience,artificiallife,andABMmethodologies,impactingthewaythattheyarepresentedtowardsabespoke'JASSS'style.Whilstsuchbiases(andotherslikethem)areimportant,theydonot,inouropinion,diminishtheresponsibilityoftheJASSStowardsbuildingupitspublicationstandards,JASSScan(anddoes)playacriticaleducationalroleinthepreparation,presentationandprosecutionofABMscience,apointwereturntobelow.

5.18 Second,editorialflux:itispossiblethatduringourstudyperiod,editorialpoliciesweakenedortightened,orfollowedsomeotherfunctionalform,perhapstopursueother(reasonable)motivationssuchasexpandingthe'reach'or'inclusiveness'oftheJASSS

http://jasss.soc.surrey.ac.uk/18/4/16.html 16 25/01/2016

community.Indeed,JASSSitselfpublishedcurrentJASSSEditorSquazzoniandCasnici's(2013)studywhichadvocatedspecificeditorialpolicyprescriptionsforJASSStobetterserveitsaimssuchaskeepingacloserwatchonJASSS'squantifiableinter-disciplinaryimpactandpotentiallytargetingspecific,un-tapped,domainsthroughspecialissues.Whilstitishardtoobtainameasureofsuchpolicydynamics,ourstudy'smainconclusionsencourageadoptionofminimumpractices,generictoallABMpapers,regardlessoffieldorspecificeditorialfocus,hence,weagainwouldsubmitthatanynuancededitorialmovementsapplyatalayerabovetheonetowhichwearestudying.Inanycase,theEditorialoversightofJASSSduringthisperiodwasasstableasonecouldhopeforanyjournal:asingle(foundational)editoroversawthefirst17yearsofthejournal'slife,generouslyoverlappingwithourstudyperiod.Theeditor,asfounder,builtupthejournalfromtheground,andhasrightlyreceivedenormousgratitudefromthesocialsimulationcommunityfortheircontributionstoJASSS'ssuccess(Elsenbroich&Badham2015).

5.19 Onefurtherlimitationisworthmakingclear.Weacknowledgethattheoverallclarityofapaper'sscientificcontributionmayormaynotrestontheformalismofitspresentationstyle.Inthiswork,wehavemerelytabulatedthetrendsintheABMpublishingcommunity,expressedthroughJASSS.Whatwecannotconcludeiswhetherthepaperswhichadoptoneorotherformalisminthecommunicationofmethodsandresultsareactuallyanyclearerthanthosewhichdonot.Theanswertosuchaquestionwouldrequireanaltogetherdifferentmethodology,presumablyinvolvingtrainedhumanreadersand/orreplicationattemptsofthesciencethatagivenpaperpresents.Weseesuchconsiderationsasanaturalextensionofthisworkandwouldencourageotherstoimaginecreativeexperimentaldesignswhichcouldidentifythe'best'communicationformalismofmethodsandresultsforABMworks.

5.20 Inonesense,ourstudyconcludeswithoutaparticularlynovelcontribution:weappearmerelytohavefoundnewwaystocataloguetheanarchyofABMsocialsciencespublicationpractices,oralternatively,thelackofuptakeofthevariousproposalsto'order'therealm.Onthisquestion,itwouldseemthattheABMsocialsciencescommunityhasachoice:itcangoonpermittingthe'anarchy',choosingtoapplylaissez-fairepublicationpracticestandardstothecommunicationofmethodologiesandresults,or,itcanattempttoapplyorderbyenforcingoneormorestandards.

5.21 Affinityordissatisfactionwith'anarchy'willlikelyturnonone'sstancetowardsthemeritsofdiversity.Ontheonehand,proponentsofanarchicpublishingpracticescouldpointtothebenefitsofenhancedacademicfreedom,creativeexpression,andthenotionoffittingtheauthors'personalsenseofthe'right'methodologicalorresultsobjectpresentationtothescientificclaimathand,withlittlereferencetonormsorstandards.AsmentionedaboveinConclusion1,giventhediversityofgeneratingfields(eachwiththeirownfield-norms)whoseektopublishtheirABMworksinJASSS,thereisnoneedtoencouragesuchdiversity,itwillariseorganicallyduetoeacharticle'sprovence.Furthermore,whoistosaythatoneauthor'sexperimentationorinnovationinpresentingtheirABMmethodorresultswillnotbeself-evidentlybrilliant,andsoreceivewideradoptionamongstABMauthorsviaimitationforthebenefitofall?Thereareindeedpotentialmeritstoanarchy.Thatsaid,letusadvanceanalternativeperspectiveonthemeritsofanarchy.IfourreadingofJASSSisright,thatitisaplatformofchoiceformanyfirst-timeABMauthorsfromthesocialsciences,thenJASSSmustbeseenasfarmorethansimplya'socialsciencessimulationpublication',JASSSisalsoapowerfuleducationaltool:thearticlesinJASSSwillimplicitlyformacorpusofideas,methodsandpracticestolearnfrom,extend,andultimatelyimitate.ThismattersbecauseiftheABMmethodologyistomakeprogressoutsideofsimulation-specificjournalslikeJASSSandintothe'main-stream'topfieldjournalsofourrespectivedisciplines,thenJASSScanplayanimportantroleindeveloping(enforcing)thebeststandardsoftheABMdiscipline'inhouse'beforeauthorstaketheirideastoeditorsandrefereesforwhomABMwill(still)beanentirelynewmethodologicalapproach.If,overtime,socialscientistsaremadetodevelopstandardapproaches,thenthatstandardwillbelearnedandunderstoodbyfieldeditorsandreferees,annullinganeasy(rejection)chargethatABMpaperslackintelligibilityortransparencybasedonthecommunicationofthemethodologyorresultsalone.Insummary,therecouldbestronglong-runbenefitstofinding(andcontinuouslyrefining)socialscienceABM'sformalvoiceonthepresentationofmethodsandresults.

5.22 Butourstudyhasdonemorethanconfirmthisanarchy,ithasidentifiedabasicproblemwithavastnumberofABMresultspublishedinJASSSoveradecade:thequalityofpresentedresults.Thissituationcannotcontinue.Itisamatteroffundamentalscientificpracticeandreproducibility–asupposedlycherishedfeatureofABMscience.Westressthatwearenotsuggestingthatworksstudiedinoursurveydonothavegoodscientificpointstomake,theseworkshaveallpassedthepeer-reviewprocessandassuchmustconveyimportantscientificfindings.However,theABMcommunitycannotontheonehandgrumbleabouttheslowuptakeofABMscienceintopfieldjournals,whilstontheother,failatpracticingbasicscientifichygienewhenitcomestopresentingitsresults.Again,viewedthroughaneducationallens,JASSShasarealopportunity,byitsinstructionstoreferees,anditssubmissionrequirements,tosetminimumstandardsforreplicationandresultspresentation.

5.23 Welookforwardtocontributingtothefurtherdevelopmentandenforcementofbest-practicestandards,andcallonthesocialsciencesABMcommunitytodothesame.

Acknowledgements

WethankTayaAnnable,andPenelopeMealyfortheirgenerousresearchassistanceandcontributionsinpreparingthisstudy,andJu-SungLeeandMatteoRichiardiwhoprovidedhelpfulcommentsonanearlierversionofthismanuscripts.Wealsothankthefouranonymousrefereeswhoprovidedextensive,interesting,andhelpfulperspectivesonourworkwhichservedtoimprovetheclarityofitsmessage.Allerrorsaretheauthors.

Notes

1Onecouldgofurtherbackstill.Forexample,Starbuck'sfascinating(1983)paperreviewingtheprospectsforsimulationinthesocial

http://jasss.soc.surrey.ac.uk/18/4/16.html 17 25/01/2016

sciencesoffersthemodernABMpractitionermuchmaterialforsoberreflection.

AppendixA-ExampleJASSSHTMLheaderusedfortermextraction,keywordsfromthe'Subject'identifierhighlighted

Paper:Huet,S.,Edwards,M.andDeffuant,G.(2007),"TakingintoAccounttheVariationsofNeighbourhoodSizesintheMean-FieldApproximationoftheThresholdModelonaRandomNetwork",JournalofArtificialSocietiesandSocialSimulation,10(1)10http://jasss.soc.surrey.ac.uk/10/1/10.html.

HTML:

References

ARHWEILER,P.(2011).ModellingTheoryCommunitiesinScience.JournalofArtificialSocietiesandSocialSimulation,14(4),8<http://jasss.soc.surrey.ac.uk/14/4/8.html>.

BONABEAU,E.(2002).Agent-basedmodeling:Methodsandtechniquesforsimulatinghumansystems.ProceedingsoftheNational

http://jasss.soc.surrey.ac.uk/18/4/16.html 18 25/01/2016

AcademyofSciencesoftheUnitedStatesofAmerica,99(suppl3),7280–7287.[doi:10.1073/pnas.082080899]

BOUCHAUD,J.-P.(2008).Economicsneedsascientificrevolution.NatureMaterials,455(7217),1181.[doi:10.1038/4551181a]

BUCHANAN,M.(2009).Economics:Meltdownmodelling.NatureNews,460(7256),680.[doi:10.1038/460680a]

COUNCILOFSCIENCEEDITORS(2014),ScientificStyleandFormat,8thEdition,ScientificStyleandFormatonline(URL:http://www.shttp://www.scientificstyleandformat.org/Home.html),UniversityofChicagoPress.

DEICHSEL,S.,&Pyka,A.(2009).APragmaticReadingofFriedman'sMethodologicalEssayandWhatItTellsUsfortheDiscussionofABMs.JournalofArtificialSocietiesandSocialSimulation,12(4),6<http://jasss.soc.surrey.ac.uk/12/4/6.html>

DOSI,G.,Fagiolo,G.,&Roventini,A.(2010).SchumpetermeetingKeynesApolicy-friendlymodelofendogenousgrowthandbusinesscycles.JournalofEconomicDynamics&Control,34(9),1748-1767.[doi:10.1016/j.jedc.2010.06.018]

ELSENBROICH,C.&Badham,J.(2015).AStandingOvationforNigel:AnInformalStudy.JournalofArtificialSocietiesandSocialSimulation,18(1),16<http://jasss.soc.surrey.ac.uk/18/1/16.html>.

EPSTEIN,J.M.(2006)."RemarksontheFoundationsofAgent-BasedGenerativeSocialScience."Chapter34inHandbookofComputationalEconomicsII:Agent-basedComputationalEconomics,editedbyLeighTesfatsionandKJudd,North-holland.

EPSTEIN,J.M.(2009).Amodelapproach.NatureMaterials,460(7256),667.

FARMER,J.D.,&Foley,D.(2009).Theeconomyneedsagent-basedmodelling.NatureMaterials,460(7256),685.[doi:10.1038/460685a]

GEANAKOPLOS,J.,Axtell,R.,Farmer,D.J.,Howitt,P.,Conlee,B.,Goldstein,J.,etal.(2012).GettingatSystemicRiskviaanAgent-BasedModel.AmericanEconomicReview,PapersandProceedings,102(3),53–58.[doi:10.1257/aer.102.3.53]

GILBERT,N.,&Terna,P.(2000).Howtobuildanduseagent-basedmodelsinsocialscience.Mind&Society,1,57–72.[doi:10.1007/BF02512229]

GRAZZINI,J.(2012).AnalysisoftheEmergentProperties:StationarityandErgodicity.JournalofArtificialSocietiesandSocialSimulation15(2),7<http://jasss.soc.surrey.ac.uk/15/2/7.html>.

GRAZZINI,J.,&Richiardi,M.(2014).PartialIdentificationinNon-ergodicModels.TechnicalReport,iNET,OxfordMartinSchool,UniversityofOxford.

GRAZZINI,J.,&Richiardi,M.(2015).Estimationofergodicagent-basedmodelsbysimulatedminimumdistance.JournalofEconomicDynamics&Control,51,148–165.[doi:10.1016/j.jedc.2014.10.006]

GRIMM,V.,Berger,U.,Bastiansen,F.,Eliassen,S.,Ginot,V.,Giske,J.,etal.(2006).Astandardprotocolfordescribingindividual-basedandagent-basedmodels.EcologicalModelling,198(1–2),115–126.[doi:10.1016/j.ecolmodel.2006.04.023]

GRIMM,V.,Berger,U.,Deangelis,D.L.,Polhill,J.G.,Giske,J.,&Railsback,S.F.(2010).TheODDprotocol:Areviewandfirstupdate.EcologicalModelling,221(23),2760–2768.[doi:10.1016/j.ecolmodel.2010.08.019]

HARA,N.,Bonk,C.J.,&Angeli,C.(2000).Contentanalysisofonlinediscussioninanappliededucationalpsychologycourse.InstructionalScience,28(2),115–152.[doi:10.1023/A:1003764722829]

HEATH,B.,Hill,R.andCiarallo,F.(2009).ASurveyofAgent-BasedModelingPractices(January1998toJuly2008).JournalofArtificialSocietiesandSocialSimulation12(4),9<http://jasss.soc.surrey.ac.uk/12/4/9.html>.

HOWITT,P&Clower,R,(2000).Theemergenceofeconomicorganization,JournalofEconomicBehaviour&Organization41,55–84.

KORNHAUSER,D.,Wilensky,R.&Rand,W.(2009).Designguidelinesforagentbasedmodelvisualisation.JournalofArtificialSocietiesandSocialSimulation12(2),1<http://jasss.soc.surrey.ac.uk/12/2/1.html>.

LAW,A.(2005).Howtobuildvalidandcrediblesimulationmodels.Proceedingsofthe37thConferenceonWinterSimulation,24–32.[doi:10.1109/wsc.2005.1574236]

LEE,J.,Filatova,T.,Ligmann-Zielinska,A.,Hassani-Mahmooei,B.,Stonedahl,F.,Lorscheid,I.,Voinov,a.,Polhill,J.G.,Sun,Z.&Parker,D.C.(2015).TheComplexitiesofAgent-BasedModelingOutputAnalysis.JournalofArtificialSocietiesandSocialSimulation18(4),4<http://jasss.soc.surrey.ac.uk/18/4/4.html>.

LEOMBRUNI,R.,&Richiardi,M.(2005).Whyareeconomistsscepticalaboutagent-basedsimulations?PhysicaA:StatisticalMechanicsandItsApplications,335,103–109.

LIM,M.,Metzler,R.,&Bar-Yam,Y.(2007).GlobalPatternFormationandEthnic/CulturalViolence.Science,317(5844),1540–1544.

MACY,M.W.,&Willer,R.(2002).FromFactorstoActors:ComputationalSociologyandAgent-BasedModeling.AnnualReviewofSociology,28(1),143–166.[doi:10.1146/annurev.soc.28.110601.141117]

MOSS,S.(2008).Alternativeapproachestotheempiricalvalidationofagent-basedmodels.JournalofArtificialSocietiesandSocialSimulation,11(1),5<http://jasss.soc.surrey.ac.uk/11/1/5.html>.

http://jasss.soc.surrey.ac.uk/18/4/16.html 19 25/01/2016

MÜLLER,B.,Bohn,F.,Dreßler,G.,Groeneveld,J.,Klassert,C.,Martin,R.,etal.(2013).Describinghumandecisionsinagent-basedmodels–ODD+D,anextensionoftheODDprotocol.EnvironmentalModelling&Software,48,37–48.[doi:10.1016/j.envsoft.2013.06.003]

ODELL,J.,Parunak,H.,&Bauer,B.(2000).ExtendingUMLforagents.InG.Wagner,Y.Lesperance,&E.Yu.PresentedattheProc.oftheAgent-OrientedInformationSystemsWorkshopatthethNationalconferenceonArtificialIntelligence,Austin,TX:AnnArbor.

RICHIARDI,M.,R.Leombruni,N.Saam,andM.Sonnessa.(2006).ACommonProtocolforAgent-BasedSocialSimulation.JournalofArtificialSocietiesandSocialSimulation9(1),15<http://jasss.soc.surrey.ac.uk/9/1/15.html>.

ROURKE,L.,&Anderson,T.(2004).Validityinquantitativecontentanalysis.EducationalTechnologyResearchandDevelopment,52(1),5–18.[doi:10.1007/BF02504769]

SQUAZZONI,F.&Casnici,N.(2013).IsSocialSimulationaSocialScienceOutstation?ABibliometricAnalysisoftheImpactofJASSS.JournalofArtificialSocietiesandSocialSimulation16(1),10<http://jasss.soc.surrey.ac.uk/16/1/10.html>.

STARBUCK,W.H.(1983).Computersimulationofhumanbehavior.BehavioralScience,28(2),154–165.[doi:10.1002/bs.3830280207]

WINDRUM,P.,Fagiolo,G.,&Moneta,A.(2007).Empiricalvalidationofagent-basedmodels:Alternativesandprospects.JournalofArtificialSocietiesandSocialSimulation,10(2),8<http://jasss.soc.surrey.ac.uk/10/2/8.html>.

WOOLDRIDGE,M.,&Jennings,N.(1998).PitfallsofAgent-OrientedDevelopment(pp.385–391).PresentedattheProceedingsofthesecondinternationalconferenceonAutonomousagents.[doi:10.1145/280765.280867]

http://jasss.soc.surrey.ac.uk/18/4/16.html 20 25/01/2016