1

RESEARCHARTICLE

HowtoRankJournals

CoreyJ.A.Bradshaw1*,BarryW.Brook2

1SchoolofBiologicalSciences,TheUniversityofAdelaide,Adelaide,South

Australia5005,Australia,2SchoolofBiologicalSciences,PrivateBag55,

UniversityofTasmania,Hobart7001,Australia

*Correspondingauthor:CoreyJ.A.Bradshaw,SchoolofBiologicalSciences,The

UniversityofAdelaide,Adelaide,SouthAustralia5005,Australia;Tel:+6188313

5842;Fax:+6183134347.E-mail:corey.bradshaw@adelaide.edu.au

Keywords:Ranking;Peer-reviewedJournals;ImpactFactor;h-index;Citations;

Bibliometrics

2

Abstract1

Therearenowmanymethodsavailabletoassesstherelativecitation2

performanceofpeer-reviewedjournals.Regardlessoftheirindividualfaultsand3

advantages,citation-basedmetricsareusedbyresearcherstomaximizethe4

citationpotentialoftheirarticles,andbyemployerstorankacademictrack5

records.Theabsolutevalueofanyparticularindexisarguablymeaningless6

unlesscomparedtootherjournals,anddifferentmetricsresultindivergent7

rankings.Toprovideasimpleyetmoreobjectivewaytorankjournalswithin8

andamongdisciplines,wedevelopedaκ-resampledcompositejournalrank9

incorporatingfivepopularcitationindices:ImpactFactor,ImmediacyIndex,10

Source-NormalizedImpactPerPaper,SCImagoJournalRankandGoogle5-year11

h-index;thisapproachprovidesanindexofrelativerankuncertainty.Weapplied12

theapproachtosixsamplesetsofscientificjournalsfromEcology(n=10013

journals),Medicine(n=100),Multidisciplinary(n=50);Ecology+14

Multidisciplinary(n=25),Obstetrics&Gynaecology(n=25)andMarineBiology&15

Fisheries(n=25).Wethencross-comparedtheκ-resampledrankingforthe16

Ecology+Multidisciplinaryjournalsettotheresultsofasurveyof188publishing17

ecologistswhowereaskedtorankthesamejournals,andfounda0.68-0.8418

Spearman’sρcorrelationbetweenthetworankingsdatasets.Ourcomposite19

indexapproachthereforeapproximatesrelativejournalreputation,atleastfor20

thatdiscipline.Agglomerativeanddivisiveclusteringandmulti-dimensional21

scalingtechniquesappliedtotheEcology+Multidisciplinaryjournalset22

identifiedspecificclustersofsimilarlyrankedjournals,withonlyNature&23

Scienceseparatingoutfromtheothers.Whencomparingaselectionofjournals24

withinoramongdisciplines,werecommendcollectingmultiplecitation-based25

metricsforasampleofrelevantandrealisticjournalstocalculatethecomposite26

rankingsandtheirrelativeuncertaintywindows.27

28

Introduction29

Lovethemorloathethem,‘objective’metricsdesignedtomeasureapeer-30

reviewedjournal’sperformancerelativetoothersareheretostay.Journal31

rankingsandscoresare,rightlyorwrongly[1,2],usedubiquitouslynowby32

3

academicselectionpanelstoassessapplicanttrackrecords,byscholarschoosing33

journalstowhichtheywillsubmittheirresearchfindings,andbypublishing34

companiesseekingtomarkettheirjournals[3,4].TheISI®ImpactFactor—35

calculatedastheaveragenumberoftimesthearticlesfromajournalpublished36

withinthepasttwoyearshavebeencitedintheJournalCitationReportsyear—37

has,todate,receivedthemostattention[5]andhence,themostcriticism[6-8].38

CriticshaveshownthattheImpactFactordoesnotcomparewellamong39

disciplines[9,10],ittendstoincreaseovertimeregardlessofjournal40

performance[10,11],andthemethodsbehinditscalculationarenottransparent41

(particularly,whattypesofarticlesarecounted).Thishasledtogaming,andasa42

result,therehavebeenmanysuggestedmodificationstothealgorithm[1,2,12-43

14].Nonetheless,theImpactFactorisnowentrenchedinthepsycheof44

researchersandhasarguablychangedthedynamicofjournalassessmentand45

bibliometricsmorethananyothersinglemethod[15].46

Despiteitsestablisheddominance,theImpactFactortodayhasmany47

competitorsthatareall,tosomedegree,basedoncitationdata.Theseinclude48

interaliaISI’sothermetricssuchasthefive-yearaverageImpactFactor,the49

ImmediacyIndex,CitedHalf-life,EigenfactorScoreandArticleInfluenceScore,50

plusElsevier’sSource-NormalizedImpactPerPaper,ImpactPerPublicationand51

SCImagoJournalRank,andnowGoogle’s5-yearHirsch-type[16]index[17]and52

itsmedian[18].Ithasbeenshownthatthesedifferentmetricscandelivervastly53

differentrankingsforindividualjournals[18,19](andseeSupporting54

Informationofthispaper).Nosinglemetriccanbeviewedasidealbecausesome55

tendtooverestimatecitations(e.g.,GoogleScholar)[20],whileothers56

underestimatethem(e.g.,WebofScience-andScopus-basedmetrics)[8,21].57

Anotherissueisthesubjectiveinterpretationofjournal‘quality’,forreasons58

thatgowellbeyondmethodologicalquestionsofhowtocombineorrelativize59

citationperformance.Thisisbecausejournalreputationamongstpeers,its60

impactonpolicyandpractice,andthequalityofthearticlesthemselves,donot61

necessarilyreflectthenumberofcitationsanarticleorajournalwillultimately62

receive[22].Itisforthesereasons,andtheimpossibletaskofchoosinga‘best’63

metric,thatjournalmetricsshouldonlyeverbeconsideredasindicesofrelative,64

average-citationperformancefromwithinadiscipline-specificorpersonally65

4

selectedsampleofjournals[9].Byitself,thevalueofaparticularjournalcitation66

metricislargelymeaningless.67

Herewedescribeasimplemethodtocalculatemeanmetricranks(and68

resampleduncertaintybounds)fromspecificsamplesofjournals,andwe69

provideasimplecomputercode(RProgrammingLanguagescript)todothe70

calculations.Ourresultsprovideamoreintegratedandtransparentwayfor71

researchers,publishers,andemployerstojudgeajournalofinterestrelativeto72

anyother.Wealsovalidateourapproachwithajournal-rankingsurveyof73

perceptionsof‘quality’from188publishingecologists,bothtodemonstratethe74

approach’sutilityandtoidentifyitsbiases.75

76

MaterialsandMethods77

Metrics78

Therearemanyexistingalgorithmsusedtorankjournals.Themostwell-79

knownaretheISI®WebofKnowledge(webofknowledge.com)metrics,including80

ImpactFactor(IF):theaveragenumberoftimesarticlesfromthejournal81

publishedinthepasttwoyearshavebeencitedintheJournalCitationReports82

year;5-yearImpactFactor(IF5):theaverageIFoverlastfiveyears;Immediacy83

Index(IM):theaveragenumberoftimesanarticleiscitedintheyearitis84

published;CitationHalf-Life(HL):themedianageofarticlescitedbythejournal85

intheJournalCitationReportsyear;EigenfactorScore(EFS):isbasedonthe86

numberoftimesarticlesfromthejournalpublishedinthepastfiveyearshave87

beencitedintheJournalCitationReportsyear,butitalsoconsiderswhich88

journalshavecontributedthesecitationssothathighlycitedjournalswill89

influencethenetworkmorethanlesser-citedjournals.Referencesfromone90

articleinajournaltoanotherarticlefromthesamejournalareremoved,sothat91

EFSisnotinfluencedbyjournalself-citation[23];andArticleInfluenceScore92

(AIS):calculatedbydividingajournal’sEFSbythenumberofarticlesinthe93

journal,normalizedasafractionofallarticlesinallpublications[23].94

ElsevieralsoproducesthreejournalmetricsbasedontheScopus®citation95

database(www.journalmetrics.com),namelytheSource-NormalizedImpactPer96

Paper(SNIP):theratioofajournal'scitationcountperpaperandthecitation97

5

potentialinitssubjectfield(averagelengthofreferencelistsinafieldto98

determinetheprobabilityofbeingcitedtocorrectfordifferencesbetween99

subjectfields)[24];ImpactPerPublication(IPP):theratioofcitationsinayear100

topaperspublishedthethreepreviousyears,dividedbythenumberofpapers101

publishedinthosesameyears[24];andSCImagoJournalRank(SJR):ameasure102

ofscientificinfluenceofscholarlyjournalsthataccountsforboththenumberof103

citationsreceivedbyajournalandtheimportanceorprestigeofthejournals104

fromwheresuchcitationscome.Itisavariantoftheeigenvectorcentrality105

measureusedinnetworktheory[25,26].106

Finally,Google(scholar.google.com)hasenteredthejournalrankingfraywith107

its5-yearHirsch’s(h)indexforjournals[16](h5):thelargestnumberhsuch108

thatharticlespublishedinthepastfiveyearshaveatleasthcitationseach;and109

median5-yearh-index(h5m):themediannumberofcitationsforthearticles110

thatmakeupitsh5.Here,ajournal’sh-index(oranyothercitation-basedmetric)111

isentirelyunrelatedtotheoriginalandbetterknownh-indexofindividual112

researchers[16].113

Usingmetricsupto2013,wedecidedtoculloursampleofmetricstofiveof114

themostdisparate,toreducecross-correlationsandredundancy.IFandIF5115

werestronglycorrelated,aswereh5andh5m(S1Table).OfthethreeElsevier116

metrics(SNIP,IPPandSJR),IPPwasthemostredundant(S1Table).Wealso117

excludedEFSandAISgiventheirredundancyrelativetoIF118

(webofknowledge.com),aswellasHLbecausethatmetricdidnotprovideexact119

values>10.AlthoughEFSandAISarearguablymorecarefullyweightedmetrics,120

weusedtherawIFinthisanalysisbecauseofitsubiquity;thethreemetricsare121

alsohighlycorrelated(Spearman’sρ=0.83-0.93basedon85‘biology’journals122

listedinISI®WebofScience–datanotshown).123

OurfinallistofmetricsthereforeincludedIF,IM,SNIP,SJRandh5.Forh5,we124

foundanon-random,positiverelationshipbetweenh5andthenumberof125

articlespublishedthatyear(Spearman’sρ=0.52to0.75;S1Table),sowe126

dividedh5bylog10(n)tostandardizeitperjournal.WithineachsampleofJ127

journals,werankedeachjournalandeachmetricfrom1(highest)toJ(lowest)128

journalsusingasimplerankfunctionintheRpackage[27],withtiestreatedas129

6

meanranks.Weprovideallrawmetricdataforeachofthesamplejournals(see130

below)intheSupportingInformation.131

132

Samplejournals133

Weselectedthe‘top’100Ecology,100Medicineand50Multidisciplinary134

journalsinwhichbiologicallythemedpapersareregularlypublished(ourchoice135

ofthejournalpoolwasnecessarilysomewhatarbitrary,butbasedonreputation136

ofhigher-rankedjournals,andhistoricalmetricslikeIF5).Eachjournalineach137

samplewasrequiredtohavethefullsetofmetricsweexamined,includingtotal138

numberofarticlespublishedin2013derivedfromISI(n),totalnumberof139

citationsin2013derivedfromISI(c),IF(ISI),IM(ISI),SNIP(Elsevier),SJR140

(Elsevier),andh5(Google).141

Wealsogeneratedafourthexampleset,consistingof25journalsthatwere142

ecology-specific,butwhichalsoincludedseveralhigh-rankingmultidisciplinary143

journals(i.e.,amixofjournalsfromtheEcologyandMultidisciplinaryjournalsets144

outlinedabove).Ourrationalewasthatanecologistwouldconsidersucharange145

ofjournalstowhichshe/hemightsubmitahigh-qualitymanuscript.Thesame146

ideacouldapplyequallytoanyotherdiscipline,andsoisnotdependentonthe147

disciplineofecologyperse.Givenourparticularexpertiseinecology,weare148

confidentthatthislistincludesarepresentativeandrealisticselectionof149

relevantjournalsinourfield(althoughthefinalchoiceofjournalsisirrelevantto150

demonstrateourmethod’sutility).Finally,weincludedtwomorediscipline-151

specificsetsof25journalsfromObstetrics&Gynaecology,andMarineBiology&152

Fisheries.Ouraimherewastoexaminerankswithinaspecialistdiscipline153

(withoutmultidisciplinaryjournals)toinvestigatewithin-subdisciplinepatterns.154

155

Rankinguncertainty156

Wedidnotapplyanaprioriweightingtoanyofthefivemetricsincluded;157

insteadwecalculatedthemeanandstandarddeviationofeachmetric’srankper158

journal.Wecalculatedaresampleduncertaintyinterval(i.e.,notatrue159

confidenceintervalbecauseofthefinitesampleofjournalsconsidered)ofthat160

meanrankbyresampling(functionsampleintheRProgrammingLanguage)[27]161

7

withreplacementarandomselectionofjournalsforeachof10,000iterations162

[28].Foreachjournal,wetookthe0.025thandthe0.975thquantilesofthe163

resampledranksastheuncertaintybounds.Wealsoappliedakappa(κ)164

limitationtotheresampledselections,wherebyweonlyretainedtheresampled165

meanrankswithinκσoftheoverallaveragemean(herewesetκ=2),thereafter166

recalculatingtheaverageandstandarddeviationofthemeanrank,and167

repeatingtheprocessfivetimes.Weusedthisiterativeκσ‘clipping’approach—168

whichisoftenusedinimageprocessingtoremoveartifactswhenstackingsub-169

frames[29]—tolimittheinfluenceofoutliersinestimatingtherangeofmean170

rankacrossall10,000iterations.Itisimportanttounderstandthattheresulting171

rankuncertaintiesdonotrepresentanestimateofatruestatisticalparameter172

becauseweareonlyconcernedwithhowmuchtherelativerankofeachjournal173

intheselectionperformsasjournalsareincludedorexcludedintherandomly174

resampledselections.A‘true’combinedrankinguncertaintyistherefore175

nonsensicalbecausepresumablyonewouldneverbeinterestedinknowinga176

journal’sspecificrankrelativetoallotherexistingjournals.Inotherwords,itisa177

random-samplingprocedureonlyforthatsample.Wealsoverifiedthe178

resampledrankingbycalculatingajackknifedestimateoftherankuncertainty179

usingthejackknifefunctioninthebootstraplibrary[30]inR.Weprovideall180

theRscriptnecessarytorepeattheanalysisintheSupportingInformation.181

182

Clustering183

Todeterminewhetherparticularjournalswithinasamplefellintodistinct184

groups,weappliedbothagglomerative(function:agnes)anddivisive(function:185

diana)hierarchicalclustering(Euclideanmetric,completelinkageclusteringof186

standardizedmetricvalues)fromtheclusterlibraryinR[31].Clusteringwas187

basedonthesamefivemetricsusedintheκ-resampledapproachdescribed188

above.Toassessthestatisticalevidenceforanygroupidentified,wefurther189

appliedfunctionpvclustfromthepvclustlibrary[32]toestimatemulti-scale190

bootstrapresamplingprobabilitiesforputativeclusters[33].Forfurther191

visualizationofputativeclusters,wealsoappliedprincipalcomponentsanalysis192

tothestandardizedmetricsusingtherdafunctionintheveganlibrary[34].193

8

194

Surveyofecologistsforqualitativevalidation195

WedesignedanonlinesurveyusingGoogleForms196

(http://goo.gl/forms/5Kqz8OMtBb)aimedspecificallyatpublishingecologists197

ofanystageofcareer.Weusedtargetedemaillistsandsocialmedia(Twitter,198

Facebook,WordPress)toencourageparticipation.Weremovedanyentries199

providingane-mailaddressnotdirectlyassociatedwithatertiaryacademic200

institution,NGO,governmentagencyorprivate-sectorcorporationwithresearch201

capacity,butretainednopersonallyidentifyinginformation.Wealsoincludedan202

ecology-specific‘validation’questiontoidentifyandremoveanswersfrombots203

andnon-ecologists–wealsodeletedanysuspectentriesbasedontheanswerto204

thatquestion.Inadditiontothejournalrankingquestions(seebelow),weasked205

eachparticipanttoprovidethestageoftheircareer(Undergraduate;206

Postgraduate;TechnicalOfficer;PostdoctoralFellow;JuniorAcademic;Senior207

Academic;Professor;Other),numberofpeer-reviewedpublicationspublishedto208

date(0;2-5;6-10;11-25;26-50;51-100;>100);institutiontype(University;209

NGO;Government;PrivateSector);gender,andcurrentcountryofresidence.All210

participantswereawaretheirresponseswouldbeusedforresearchpurposes211

andpublished.212

Foreachofthe25‘ecologysample’journals,weaskedrespondentstoclassify213

thejournalintooneofthefollowingcategories:1-Elite;2-Prestigious;3-214

Reputable;4-Respectable;5-Other.Althoughourmeaningwasanordinalscale,215

weweredeliberatelyvagueabouthowarespondentshouldclassifyeachjournal216

andinterpretthecategorydescriptions,askingthemtoclassifybasedoninstinct217

andwithoutconsultinganyspecificjournalmetric.Wealsodidnotshowthem218

anyofourresultsonthemetric-basedcalculations.Ourintentionwastohavea219

journal’sreputation—intheiropinion—guidetheirselection.Toprovideamean220

rankofthe25journals,wecalculatedthemeanandstandarddeviationofthe221

categoryvaluesacrossall(188)vettedrespondents.Toexaminetheeffectof222

publicationexperienceonthesesurveyresultsandtheircorrelationtothe223

compoundranks,wesubsettedthesurveyrespondentstothosewith>50224

publishedarticlesandrepeatedtheaboveanalyses.225

226

9

Results227

Resampledranksandclusters228

Theκ-resamplingofthemeanranksacrossmetricsrevealedanoverlapping229

seriesofjournalranksperdisciplinarysample(Fig1).Usingameanormedian230

providedsimilarrankings,butwithafewsmalljournal-specificdifferences(S1231

Fig).Rankswerealsoaxiomaticallysimilarbasedonthejackknifeapproach(S2232

Fig),althoughtheestimateduncertaintywasnarrower(S3Fig)giventhelow233

numberofjournalmetrics(5)fromwhichtojackknife.FortheMedicinesample,234

thegreateroverall(sample-specific)uncertaintyamongranksmeansthatthe235

top-rankedjournalsinparticulararesimilar,withnoclearlydominantjournal236

withinthesevenorsotop-rankedjournalswithinthatsample.Giventhefinite237

sampleofjournals,therankuncertaintywindowswerenecessarilywiderinthe238

middleoftherange(S4Fig).239

Theecology-specificsampleof25journals(Ecology+Multidisciplinary)240

yieldedanotheroverlappingranking(Fig2A)thatdivergedforsomejournals241

fromtheresampledmeanranksderivedfromthesurveyofpublishingecologists242

(Fig2B).Wehadatotalof188verified-ecologistrespondents(49female,139243

male)from29countries(butmainly(69%)fromAustralia,USAandUK),withan244

approximatelyuniformdistributionofcareerstage(postgraduatetoprofessor;245

S5Fig).Respondentshadawiderangeofpublicationexperience(0to>100246

papers),andweremostly(82%)basedinuniversities(S5Fig).Inparticular,the247

journalsProceedingsoftheRoyalSocietyofLondonB-BiologicalSciences,Ecology,248

ConservationBiologyandTheAmericanNaturalisthadhighermeanreputation249

scores(andcorrespondingrankposition)thanexpectedfromtheresampled250

metric-basedmeanranks,andCurrentBiology,GlobalEcologyBiogeographyand251

BioSciencehadlower-than-expectedreputationranks(Fig2B).Allotherjournals252

fellneartotheirexpectedmeanresampledranks(Fig2B),withaSpearman’s253

rankcorrelationof0.68-0.84(median=0.77;basedon1000randomuniform254

resamplesoftherankuncertaintyinterval)betweenthecompositemetric-based255

andreputation-basedrankings(Fig2C).Recalculatingtherankcorrelationfor256

onlythosesurveyrespondentswhohadpublished≥50articles(n=58),the257

resultswerenearlyidentical(Spearman’srankcorrelation=0.67-0.83;median=258

10

0.76),buttheresampledmeanrankuncertaintyintervalswereslightlywider(S6259

Fig).260

Applyingtheclusteringmethodstotheecologysamplerevealedonlytwo261

statisticallysupportedgroupingsaccordingtofunctionpvclust:(i)Natureand262

Scienceand(ii)allremaining23journals(Fig3A).Theprincipalcomponents263

analysisrevealedthat95.3%ofthevariancewasexplainedbythefirstprincipal264

componentaxis(Fig3B),withonlyanadditional2.1%explainedbythesecond265

principalcomponentaxis(Fig3B),thusconfirmingtheScience/Natureoutliers266

groupedtogetherusingagglomerativeanddivisiveclustering.267

Thetwospecialistdisciplinesamples(Obstetrics&GynaecologyandMarine268

Biology&Fisheries)of25journalseachrevealeddifferentpatternsofranking.269

Fortheformer,thereweretwoclustersofsimilarlyrankedjournalsfromranks270

2-8andfromranks10-19,whichtendedtotruncatethespreadofranksacross271

the25journals(Fig4).Theranksofthe25MarineBiology&Fisheriesjournals272

weremoreevenlyspreadacrossthespectrumsuchthattherewerefewer273

obviousclustersofsimilarlyrankedjournals(Fig4).274

275

Discussion276

Wecontendthathavedesignedamoreobjectiveandintuitivewaythanhas277

beenpreviouslyavailabletoreflectacompositeofrelativecitation-basedranks278

fromwithinandamongspecificresearchdisciplines,byusingalogical279

combinationofmetricsthatspeaktodifferentaspectsofjournalcaliber.While280

manyotherdiscipline-specificrankingsystemshavebeenproposed,suchas281

thosebasedontotaldownloads[35],authororeditorprestige/publishing282

behavior[36-38],libraryholdings[37],databasecoverage[37],andeconometric283

analysesofcitationsandreferencingpatterns[39,40],mostsufferfroman284

inabilitytocomparejournalsacrossdisciplines,relyonoverlycomplex285

approaches,ormakequestionableassumptionsandsubjectivechoicesofthe286

componentmetrics.Further,whilesomehavealsoproposedusingseveral287

citation-basedmetricstorankjournals(e.g.,[41,42]),oursimpleapproachisthe288

onlyexistingmethodthatexplicitlyandquantitativelycombinesthemost289

11

relevantcitation-basedmetricsintoacompositerankingwithassociated,290

selection-specificuncertainties.291

Itwasnotourintention,however,todiscusstherelativemerits,shortfallsor292

qualityofthecomponentmetricsandthedatabasesfromwhichtheyare293

calculated(see[4,8,20,21]fordetaileddiscussionofmetricanddatabase294

issues).Instead,wehaveprovidedamethodtocombinecommonlyavailable295

metricsthatmostresearchersandacademicadministratorscaneasilyaccess.296

Ourparticularexamplelistsofrankedjournalsinfivebiologydisciplines297

(Ecology,Medicine,Multidisciplinary,MarineBiology&FisheriesandObstetrics&298

Gynaecology)representnovelandusefulguidesforscientistsworkinginthese299

areas(Fig1andFig4),andwehaveprovidedthecomputerscriptthatanyone300

canusetoconstructapersonallistofjournalstorankinthismanner.301

Further,byusinganindependentsurveyofecologistsfromdifferentcareer302

stages,institutiontypes,gendersandcountriesofresidence,wedemonstrated303

thattheperceptionofrelativejournalreputationislargelycapturedbythe304

combinedresampledjournalrankings.Oursampleofrespondentecologists305

providedarankingthatagreedwellwiththecombinedresampledcitation-based306

rankingswithaSpearman’scoefficientof0.68-0.84.Thiscomparesfavorablytoa307

previousvalidationsurveyofjournalreputationforphysicians,whereR2were308

between0.62(practitioners)and0.83(researchers)[22]comparedtoImpact309

Factors.Itislikelythatoursurveyresultsarenotentirelyindependentof310

citationmetricsbecauseresearchersmightbesubconsciouslyinfluencedby311

themwhenresponding.Regardlessofsomeinevitablecircularity,itisinstructive312

thatcitation-basedandreputationalsurveyrankingslargelyagreed,despite313

somediscipline-specificoutliers.Outlierjournalsthemselvesmightbeofinterest314

toresearcherstoidentifyanaspectofjournal‘quality’thatislessdependenton315

citationsthanwhatexistingmetricscurrentlyprovide.316

Othershaveattemptedtocomparethecitationperformanceofindividuals,317

fieldsandinstitutionsacrossmultipledisciplines,suchasdividingcitation-based318

performancebythetotalnumberofcitationswithinaspecificdiscipline[43],319

comparingindividualperformancetodiscipline-specifich-indexconfidence320

intervals[44],standardizingbasedoncitationdistributionfunctionsper321

discipline[45],orcomparingmultidisciplinarygroupsusingclustering-based322

12

bibliometricmaps[46].Whilesomeofthesetechniquescouldpotentiallybe323

appliedtojournalmetrics,weareconfidentthatournonparametricrankingof324

journalsfromwithinanyselectionrepresentsoneofthesimplestandmost325

intuitivewaystocomparejournalsacrossdisciplines.326

Thereareofcoursemanyconsiderationsauthorsmustcanvasswhen327

choosingwheretosubmittheirpapers[15].Intermsofmaximizingcitation328

impact,werecommendthatresearchersconsidercollectingmultiplecitation-329

basedmetricsforasampleofrelevantandrealisticjournals(i.e.,inwhichitis330

plausiblethemanuscriptcouldbepublished)—sayforexample,3–10journals—331

andcalculatethecombinedrankingsandtheiruncertaintyaswehave332

demonstratedhere(seeRcodeprovidedinS1FileandS2Fileforexampledata).333

Aftertakingintoaccounttargetaudience,journalscope,acceptanceprobability,334

handlingtime[15],discipline-specificreputationamongpeersandan335

appreciationofthejournals’overall‘quality’(howeverobjectivelyorsubjectively336

defined),suchrelativerankingsusingcombinedcitation-basedmetricsmight337

assistresearcherstochoosethemostpertinentjournalsformanuscript338

submission.Samplejournalscouldalsobeamixofspecialistand339

multidisciplinary/generalistperiodicalsbecauseourapproachtakesinto340

accountrelativerankandnottheabsolutevalueofthemetricsthemselves.341

342

SupportingInformation343

S1File,RProgrammingLanguagecode(text)forrepeatinganalysis.344

S2File,Raw2013journalcitationmetricdata.345

S1Table.Spearman’sρcorrelationmatrixfortheindividualmetricsused346

todevelopthecompositeranking.n=totalnumberofarticlespublishedin347

JournalCitationReports(JCR;2013)year;cites=totalnumberofcitationstothe348

journalintheJCRyear;h5=Google5-yearh-index;h5m=medianGoogle5-year349

h-index;IF=ISI®ImpactFactor;IF5=ISI®5-yearImpactFactor;IM=ISI®350

ImmediacyIndex;SNIP=Elsevier®Source-NormalizedImpactPerPaper;IPP=351

Elsevier®ImpactPerPublication;SJR=Elsevier®SCImagoJournalRank.352

S1Fig.Comparisonofresampledmean-andmedian-basedrankings(with353

uncertaintywindows)foreachofthreesamplesofjournals.Samplesinclude354

13

(top)Ecology,(middle)Medicine,and(bottom)Multidisciplinary.Solidgreylines355

indicate1:1correspondence(45°line);dashedblacklinesindicateleast-squares356

linearfitstothecentralvalues.357

S2Fig.Comparisonofmeanresampledandjackkniferankingsforeachof358

sixsamplesofjournals.Samplesinclude(A)Ecology,(B)EcologySample359

(EcologyandsomeMultidisciplinaryjournals),(C)Medicine,(D)Multidisciplinary,360

(E)Obstetrics&Gynaecology,and(F)MarineBiology&Fisheries.361

S3Fig.Comparedjournalrankings.Resampled(upperpanels)versus362

jackknife(bottompanels)foreachofthreejournalsamples(Ecology,Medicine363

andMultidisciplinary).364

S4Fig.Per-journalrankuncertaintybounds.Rankuncertaintyincreases365

nonlinearlythroughtoapproximatelyhalfwaythroughthesample,anddecrease366

thereafter(acharacteristicofanyregressionfit),duetotheimposedlimitof100367

journalsineachdisciplinecategory(50forMultidisciplinary).Notethegreater368

relativeuncertaintyforthemiddleranksoftheMedicinediscipline.369

S5Fig.Ecologysurveyrespondentcharacteristics(authorsarebasedin370

Australia).371

S6Fig.EcologySamplejournalranks.(A)Meanrank(±95%confidencelimits372

viaκ-resamplingwith10,000iterations)ofthetop25journalswithina373

combinedEcologyandMultidisciplinarytheme.Journalsareorderedbymean374

rankoffivemetrics:IF,IM,SNIP,SJRandh5/log10(n)(seemaintextfordetails).375

(B)Meanrank(±1σ)ofthesamejournalsassessedfromasurveyof58376

ecologistswhohadeachpublished≥50articles.Journalsabovethe1:1377

correspondence(45°line)areratedhigherbytheseecologiststhantheirmean378

metricwouldindicate,andviceversa.(C)Overall,therewasaSpearman’srank379

correlationof0.67-0.83(comparedto0.68-0.84forthefull188survey380

participants);median=0.76;basedon1,000randomuniformresamplesofthe381

rankinterval)betweenbothrankings.JournalabbreviationsfollowtheWebof382

Sciencestandard.383

S7Fig.Medianrank(±95%uncertaintylimits)ofthetop30journalsfor384

twodisparatebiologicaldisciplines:EcologyandMedicine,plusone385

Multidisciplinarytheme.Journalsareorderedbymedianrankoffivemetrics:IF,386

IM,SNIP,SJRandh5/log10(n);statisticswereestimatedusingκ-resamplingwith387

14

10,000iterations,fromatotalsampleof100journalsforEcologyandMedicine388

and50journalsforMultidisciplinary(seemaintextfordetails).Journal389

abbreviationsfollowtheWebofSciencestandard.390

S8Fig.Medianrank(±95%uncertaintylimits)of25journalswithintwo391

specialistdisciplines:Obstetrics&Gynaecology(toppanel)andMarineBiology392

&Fisheries(bottompanel).Journalsareorderedbymedianrankoffivemetrics:393

IF,IM,SNIP,SJRandh5/log10(n);statisticswereestimatedusingκ-resampling394

with10,000iterations(seemaintextfordetails).Journalabbreviationsfollow395

theWebofSciencestandard.396

S9Fig.Comparisonofmedianresampledandjackkniferankingsforeachof397

sixsamplesofjournals.Samplesinclude(A)Ecology,(B)EcologySample398

(EcologyandsomeMultidisciplinaryjournals),(C)Medicine,(D)Multidisciplinary,399

(E)Obstetrics&Gynaecology,and(F)MarineBiology&Fisheries.400

401

Acknowledgements402

WethankF.Courchampforadvice.SupportedbyanAustralianResearchCouncil403

(arc.gov.au)FutureFellowshipgrantFT110100306(CJAB).404

405

AuthorContributions406

Conceivedanddesignedthemodels:CJABBWB.Sourcedandanalyzedthedata:407

CJAB.Contributedanalysistools:CJABBWB.Wrotethepaper:CJABBWB.408

409

References410

1.AdamD.Citationanalysis:thecountinghouse.Nature.2002;415(6873):726-411

9.412

2.VanclayJ.Impactfactor:outdatedartefactorstepping-stonetojournal413

certification?Scientometrics.2012;92(2):211-38.doi:10.1007/s11192-414

011-0561-0.415

3.SmithR.Commentary:Thepoweroftheunrelentingimpactfactor—Isita416

forceforgoodorharm?InternationalJournalofEpidemiology.2006;417

35(5):1129-30.doi:10.1093/ije/dyl191.418

15

4.JacsóP.Theplausibilityofcomputingtheh-indexofscholarlyproductivityand419

impactusingreference-enhanceddatabases.OnlineInformationReview.420

2008;32(2):266-83.doi:10.1108/14684520810879872.421

5.FershtA.Themostinfluentialjournals:ImpactFactorandEigenfactor.422

ProceedingsoftheNationalAcademyofSciencesoftheUSA.2009;423

106(17):6883-4.doi:10.1073/pnas.0903307106.424

6.PLoSMedicineEditors.TheImpactFactorgame.PLoSMedicine.2006;425

3(6):e291.doi:10.1371/journal.pmed.0030291.426

7.SeglenPO.Whytheimpactfactorofjournalsshouldnotbeusedforevaluating427

research.BritishMedicalJournal.1997;314(7079):497.doi:428

10.1136/bmj.314.7079.497.429

8.JacsóP.Theprosandconsofcomputingtheh-indexusingWebofScience.430

OnlineInformationReview.2008;32(5):673-88.doi:431

10.1108/14684520810914043.432

9.RamírezA,GarcíaE,DelRíoJ.RenormalizedImpactFactor.Scientometrics.433

2000;47(1):3-9.doi:10.1023/A:1005600807292.434

10.AlthouseBM,WestJD,BergstromCT,BergstromT.Differencesinimpact435

factoracrossfieldsandovertime.JournaloftheAmericanSocietyfor436

InformationScienceandTechnology.2009;60(1):27-34.doi:437

10.1002/asi.20936.438

11.NeffBD,OldenJD.Notsofast:inflationinImpactFactorscontributesto439

apparentimprovementsinjournalquality.BioScience.2010;60(6):455-9.440

doi:10.1525/bio.2010.60.6.9.441

12.WealeAR,BaileyM,LearPA.Thelevelofnon-citationofarticleswithina442

journalasameasureofquality:acomparisontotheimpactfactor.BMC443

MedicalResearchMethodology.2004;4:14.doi:10.1186/1471-2288-4-14.444

13.ZittM,SmallH.Modifyingthejournalimpactfactorbyfractionalcitation445

weighting:Theaudiencefactor.JournaloftheAmericanSocietyfor446

InformationScienceandTechnology.2008;59(11):1856-60.doi:447

10.1002/asi.20880.448

14.PudovkinAI,GarfieldE.Rank-normalizedimpactfactor:Awaytocompare449

journalperformanceacrosssubjectcategories.ProceedingsoftheAmerican450

16

SocietyforInformationScienceandTechnology.2004;41(1):507-15.doi:451

10.1002/meet.1450410159.452

15.SalinasS,MunchSB.WhereshouldIsendit?Optimizingthesubmission453

decisionprocess.PLoSONE.2015;10(1):e0115451.doi:454

10.1371/journal.pone.0115451.455

16.HirschJE.Anindextoquantifyanindividual'sscientificresearchoutput.456

ProceedingsoftheNationalAcademyofSciencesoftheUSA.2005;457

102(46):16569-72.doi:10.1073/pnas.0507655102.458

17.BraunT,GlänzelW,SchubertA.AHirsch-typeindexforjournals.459

Scientometrics.2006;69(1):169-73.doi:10.1007/s11192-006-0147-4.460

18.Delgado-López-CózarE,Cabezas-ClavijoÁ.Rankingjournals:couldGoogle461

ScholarMetricsbeanalternativetoJournalCitationReportsandScimago462

JournalRank?LearnedPublishing.2013;26(2):101-13.doi:463

10.1087/20130206.464

19.FalagasME,KouranosVD,Arencibia-JorgeR,KarageorgopoulosDE.465

ComparisonofSCImagojournalrankindicatorwithjournalimpactfactor.466

TheFASEBJournal.2008;22(8):2623-8.doi:10.1096/fj.08-107938.467

20.JacsóP.Theprosandconsofcomputingtheh-indexusingGoogleScholar.468

OnlineInformationReview.2008;32(3):437-52.doi:469

10.1108/14684520810889718.470

21.JacsóP.Theprosandconsofcomputingtheh-indexusingScopus.Online471

InformationReview.2008;32(4):524-35.doi:472

10.1108/14684520810897403.473

22.SahaS,SaintS,ChristakisDA.Impactfactor:avalidmeasureofjournal474

quality?JournaloftheMedicalLibraryAssociation.2003;91(1):42-6.475

PubMedPMID:PMC141186.476

23.BergstromCT,WestJD,WisemanMA.TheEigenfactor™Metrics.TheJournal477

ofNeuroscience.2008;28(45):11433-4.doi:10.1523/jneurosci.0003-478

08.2008.479

24.MoedHF.Measuringcontextualcitationimpactofscientificjournals.Journal480

ofInformetrics.2010;4(3):265-77.doi:10.1016/j.joi.2010.01.002.481

25.González-PereiraB,Guerrero-BoteVP,Moya-AnegónF.Anewapproachto482

themetricofjournals’scientificprestige:theSJRindicator.Journalof483

17

Informetrics.2010;4(3):379-91.doi:484

http://dx.doi.org/10.1016/j.joi.2010.03.002.485

26.Guerrero-BoteVP,Moya-AnegónF.Afurtherstepforwardinmeasuring486

journals’scientificprestige:theSJR2indicator.JournalofInformetrics.2012;487

6(4):674-88.doi:http://dx.doi.org/10.1016/j.joi.2012.07.001.488

27.RCoreTeam.R:Alanguageandenvironmentforstatisticalcomputing.489

Vienna,Austria:RFoundationforStatisticalComputing,2014.490

28.ManlyBFJ.Randomization,BootstrapandMonteCarloMethodsinBiology,491

ThirdEdition.London:ChapmanandHall/CRCPress;2006.480p.492

29.LehmannG.Kappasigmaclipping.InsightJournal.2006;493

http://hdl.handle.net/1926/367.494

30.TibshiraniR,LeischF.bootstrap:FunctionsfortheBook"AnIntroductionto495

theBootstrap".Rpackageversion20152.2015:http://CRAN.R-496

project.org/package=bootstrap.497

31.MaechlerM,RousseeuwP,StruyfA,HubertM,HornikK.cluster:cluster498

analysisbasicsandextensions.RPackageversion1153.499

2014:http://CRAN.R-project.org/package=cluster.500

32.SuzukiR,ShimodairaH.pvclust:hierarchicalclusteringwithp-valuesvia501

multiscalebootstrapresampling.Rpackageversion12-2.502

2011:http://CRAN.R-project.org/package=pvclust.503

33.ShimodairaH.Approximatelyunbiasedtestsofregionsusingmultistep-504

multiscalebootstrapresampling.AnnalsofStatistics.2004;32(6):2616-41.505

doi:10.1214/009053604000000823.506

34.OksanenJ,BlanchetFG,KindtR,LegendreP,MinchinPR,O'HaraRB,etal.507

vegan:communityecologypackage.Rpackageversion20-10.508

2013:http://CRAN.R-project.org/package=vegan.509

35.BrownLD.Rankingjournalsusingsocialscienceresearchnetwork510

downloads.ReviewofQuantitativeFinanceandAccounting.2003;511

20(3):291-307.doi:10.1023/A:1023628613622.512

36.KorobkinR.Rankingjournals:somethoughtsontheoryandmethodology.513

FloridaStateUniversityLawReview.1999:851-76.514

18

37.EastJW.Rankingjournalsinthehumanities:anAustraliancasestudy.515

AustralianAcademicandResearchLibraries.2006;37(1):3-16.doi:516

10.1080/00048623.2006.10755319.517

38.PujolF.Rankingjournalsfollowingamatchingmodelapproach:an518

applicationtopubliceconomicsjournals.JournalofPublicEconomicTheory.519

2008;10(1):55-76.doi:10.1111/j.1467-9779.2008.00351.x.520

39.HudsonJ.Rankingjournals.TheEconomicJournal.2013;123(570):F202-521

F22.doi:10.1111/ecoj.12064.522

40.KodrzyckiYK,YuP.Newapproachestorankingeconomicsjournals.TheBE523

JournalofEconomicAnalysisandPolicy.2006;5(1):1935-682.524

41.BontisN,SerenkoA.Afollow‐uprankingofacademicjournals.Journalof525

KnowledgeManagement.2009;13(1):16-26.doi:526

10.1108/13673270910931134.527

42.MingersJ,HarzingA-W.Rankingjournalsinbusinessandmanagement:a528

statisticalanalysisoftheHarzingdataset.EuropeanJournalofInformation529

Systems.2007;16(4):303-16.530

43.PodlubnyI.Comparisonofscientificimpactexpressedbythenumberof531

citationsindifferentfieldsofscience.Scientometrics.64(1):95-9.doi:532

10.1007/s11192-005-0240-0.533

44.MalesiosCC,PsarakisS.Comparisonoftheh-indexfordifferentfieldsof534

researchusingbootstrapmethodology.QualityandQuantity.2012;535

48(1):521-45.doi:10.1007/s11135-012-9785-1.536

45.IglesiasJE,PecharrománC.Scalingtheh-indexfordifferentscientificISI537

fields.Scientometrics.2007;73(3):303-20.doi:10.1007/s11192-007-1805-538

x.539

46.vanRaanAFJ.Measurementofcentralaspectsofscientificresearch:540

performance,interdisciplinarity,structure.Measurement:Interdisciplinary541

ResearchandPerspectives.2005;3(1):1-19.doi:542

10.1207/s15366359mea0301_1.543

544

19

FigureLegends

Fig1.Meanrank(±95%uncertaintylimits)ofthetop30journalsfortwo

disparatebiologicaldisciplines:EcologyandMedicine,plusoneMultidisciplinary

theme.Journalsareorderedbymeanrankoffivemetrics:IF,IM,SNIP,SJRand

h5/log10(n);statisticswereestimatedusingκ-resamplingwith10,000iterations,

fromatotalsampleof100journalsforEcologyandMedicineand50journalsfor

Multidisciplinary(seemaintextfordetails).JournalabbreviationsfollowtheWeb

ofSciencestandard.

Fig2.(A)Meanrank(±95%uncertaintylimitsviaκ-resamplingwith10,000

iterations)ofthetop25journalswithinacombinedEcologyand

Multidisciplinarytheme.Journalsareorderedbymeanrankoffivemetrics:IF,

IM,SNIP,SJRandh5/log10(n)(seemaintextfordetails).(B)Meanrank(±1σ)of

thesamejournalsassessedfromasurveyof188ecologists.Journalsabovethe

1:1correspondence(45°line)areratedhigherbyecologiststhantheirmean

metricwouldindicate,andviceversa.(C)Overall,therewasaSpearman’srank

correlationof0.68-0.84(median=0.77;basedon1,000randomuniform

resamplesoftherankinterval)betweenbothrankings.Journalabbreviations

followtheWebofSciencestandard.

Fig3.Qualitygroupingsderivedfromsimilarmeanmetrics.Toppanel:

Agglomerativehierarchicalclusteringtreefor25EcologyandMultidisciplinary

journalsbasedontheranksoffivemetrics(IF,IM,SNIP,SJRandh5/log10(n));

calculatedasaEuclideanmetric,completelinkageclusteringofstandardized

metricvalues,seemaintextfordetails).Theclusteringmethodsrevealedonly

twostatisticallysupportedgroupings:(i)NatureandScienceand(ii)all

remaining23journals.Bottompanel:Principalcomponentsanalysisofthesame

sampleofjournalsbasedontheirmeanranksfromthesamefivemetrics.95.3%

ofthevariancewasexplainedbythefirstprincipalcomponentaxis,withonlyan

additional2.1%explainedbythesecondprincipalcomponentaxis.Thisconfirms

theScience/Natureoutliersgroupedtogetherusingagglomerativeanddivisive

clustering.

20

Fig4.Medianrank(±95%uncertaintylimits)of25journalswithintwo

specialistdisciplines:Obstetrics&Gynaecology(toppanel)andMarineBiology&

Fisheries(bottompanel).Journalsareorderedbymeanrankoffivemetrics:IF,

IM,SNIP,SJRandh5/log10(n);statisticswereestimatedusingκ-resamplingwith

10,000iterations(seemaintextfordetails).Journalabbreviationsfollowthe

WebofSciencestandard.

a1171765

Typewritten Text

Fig 1

a1171765

Typewritten Text

Fig 2

a1171765

Typewritten Text

Fig 3

a1171765

Typewritten Text

Fig 4