introduction to meta-analysis outline for todaybio501/lecturepdf/11.meta-analysis.pdf ·...
Post on 30-May-2018
235 Views
Preview:
TRANSCRIPT
Introductiontometa-analysis
Outlinefortoday
• Metaanalysiscomparedwithtraditionalreviewarticle
• Quantitativesummariesvs.vote-counting
• Howtocarryoutameta-analysis
• Effectsize
• Fixedandmixed-effects
• Associatingeffectsizeswithrelevantvariables
• Publicationbias
• Makeyourresultsaccessibletometa-analysis
Scientificstudiesonatopicareoftenrepeated
Newstudiesimprove/expandonpreviousstudies,orexaminethesameissueinadifferentstudysystem,orusingdifferentmethods
• Schoeneretal.(1983)found164publishedfieldexperimentsoninterspecificcompetition.
• Gardneretal.(2003)obtainedresultsfrom51separatestudiesreportingcoralcoverfrom294sitesfromacrosstheCaribbean.
• Belletal.(2009)found759publishedestimatesoftherepeatabilityofbehavior,from114studiesof98species.
• Vilàetal(2011)reviewed199articlesreporting1041fieldstudiesdescribingtheecologicalimpactsof135alienplanttaxa.
Amethodisneededtosummarizeresultsfrommultiplestudies
• Dr.BenjaminSpocksold50millioncopiesofBabyandChildCare1950s--1990s.Inithewrote“Ithinkitispreferabletoaccustomababytosleepingonhisstomachfromthebeginningifheiswilling”.Otherpediatriciansmadesimilarrecommendations.
• Fromthe1950sintothe1990s,morethan100,000babiesdiedofsuddeninfantdeathsyndrome(SIDS).
• Intheearly1990s,researchersrealizedthattheriskofSIDSdecreasedbyatleast50%whenbabieswereputtosleepontheirbacksratherthanfacedown.
• SubsequenteducationcampaignsledtoadramaticdropinthenumberofSIDSdeaths.
• However,researchwasavailablefrom1970thatsleepingonthestomachwashazardoustobabies.Anearliersynthesisofthedatacouldhavegottheanswermuchsooner.
Traditionalapproachisthereviewarticle
Anexpertinthefieldassemblesthestudiespublishedonatopic,thinksaboutthemcarefullyand(hopefully)fairly,andthenwritesareviewarticlesummarizingtheoverallconclusionsreached.Afirst-ratereviewarticleadvancesafieldfarbeyondameresummary.Itreviewsandcommentsonthecurrentstateofthoughtandknowledgeaboutaparticulartopic.Suchareviewwillproposenewhypotheses,uncoverpreviouslyunnoticedrelationships,andpointtonewpathsofresearch.
Thetraditionalreviewlacksaquantitativemethod
Thismightleadtotwoproblems• Bias.Inhis1986bookHowtoLiveLongerandFeelBetter,LinusPauling(theonlypersontobeawardedtwounsharedNobelPrizes)cited30studiessupportinghisideathatlargedailydosesofvitaminCreducestheriskofcontractingthecommoncold,butcitednostudiesopposingtheidea,eventhoughanumberhadbeenpublished.Notallreviewsaresobiased,buttherearefewrulesregardingselectionofstudiesforreview.
• Lackofaquantitativesummaryofresearchfindings.Reviewsdon’ttellusabouthowlargetheeffectis.
“Vote-counting”isastepintherightdirection
Dividestudiesintotwocategories:thosethatyieldedastatisticallysignificantresultsupportingtheresearchhypothesis,andthosethatdidnot.Theproportionsofstudies‘voting’fororagainstthehypothesisarethencounted.
Limitationsofvote-counting
• Bycountingonlythestatisticallysignificantstudiesvote-countingignoresallthequantitativeinformationaboutthemagnitudesofeffects.
• Tooconservative.“Votes”areaffectedbythepowerofindividualstudies,whichmaybeweak.
• Significancelevelbyitselfdoesn’tindicatewhethertwoormorestudiesobtainedthesameoutcome.
• Themagnitudeoftheeffectisdownplayed.
• Itisdifficulttoquantifytheeffectsofpublicationbias.
• Methodisunabletoweightheeffectofstudiesdifferinginsamplesize,andthereforepower.
Limitationsofvote-counting
• TheAntiplateletTrialists’Collaboration(1994)conductedameta-analysisof142randomizedexperimentstestingwhethertakingaspirinorotherantiplateletmedicationfollowingastrokeormyocardialinfarction(“heartattack”)reducedtheriskoffuturestroke.TotalN>70,000.
• Thevote:19of142studiesshowedastatisticallysignificantlybetterresultforpatientsonantiplatelettherapythanforthecontrolpatients.Twoofthe142studiesshowedasignificantlyworserateofvasculareventswithaspirintreatment.
• Yet14.7%(5400/36,711)ofpatientsinthecontrolgroupshadsubsequentvascularevents,comparedwith11.4%(4183/36,536)inthetreatedgroup.Smalleffectbutreal,accordingtometa-analysismethods.Thisconclusionsavedmanylives.
Meta-analysis,the“analysisofanalyses”
Meta-analysisreferstothestatisticalsynthesisofresultsfromaseriesofstudies(Borensteinetal2009).Themethodinvolvescompilingallknownscientificstudiesestimatinganeffectandquantitativelycombiningthemtogiveanoverallestimateoftheeffect.Meta-analysisallowsustogeneralize.Itletsusdeterminehowfrequent,howimportant,andhowconsistenteffectsareacrossavarietyofsystems. Meta-analysisgetspasttheoccasionalsensationalresult(theoneyoureadaboutinthenewspaper)toanobjectiveassessmentofalltheevidence.
Meta-analysis,the“analysisofanalyses”
Camefrommedicalresearch,inwhichallstudiesareallofthesamespecies(humans).Hereisa“forestplot”fromtheAntiplateletTrialists’Collaboration.
Meta-analysis,the“analysisofanalyses”
Ecologistsandevolutionarybiologistsattempttogeneralizeacrossamuchwiderrangeofspeciesandsystems.Thisismorechallengingthanstudiescarriedoutonasinglespecies(e.g.,humans).
Example1:Meta-analysisoftheTransylvaniaeffect
• Manypeoplebelievethatafullmooncanaffecthumanbehavior.ThewordlunacyisderivedfromtheLatinluna,moon.
• Legendsofstrangehappenings,suchaswerewolvesandvampires,havebeenconnectedtofullmoonsforcenturies.
• LordBlackstone,an18th-centuryEnglishjurist,wasthefirsttodefineaconditionofmadnessexacerbatedbythelunarcycle:“Alunatic,ornoncomposmentis,isproperlyonewhohathlucidintervals,sometimesenjoyinghissensesandsometimesnotandthatfrequentlydependinguponthechangesofthemoon.”
• RottonandKelly(1985)showedthat50%ofuniversitystudentsbelievedthatpeopleactstrangelyduringafullmoon.
• Vance(1995)reportedthatasmanyas81%ofmentalhealthprofessionalsbelievedthatthefullmoonaltersindividualbehaviour.
Example1:Meta-analysisoftheTransylvaniaeffect
RottonandKelly(1985)carriedoutameta-analysisofstudiescorrelatinghomiciderates,psychiatrichospitaladmissions,suiciderates,crisiscalls,etc.Theaverageeffectsizerwassmallerthan0.01.
Howtocarryoutameta-analysis
1. Definethequestionandscope.
• Anarrowquestionappliedtoahomogeneousgroup?“Doesaspirinreduceincidenceofmyocardialinfarction?”
• Oraheterogeneoussetofstudiesorvariables?“Howmuchgeneticvariationexistsinpopulationsforbehavioraltraits?”
• Onlyexperimentswithcontrolsandrandomization?Onlyreplicatedexperiments?Onlyexperimentswithblinding?
• Itmaybebesttoadoptareasonablywidescopeandinvestigatelaterwhetherdifferencesbetweenmethodsleadtodifferenteffectsoverall.
Howtocarryoutameta-analysis
2. Literaturesearch,gatherdata.
• Makeitexhaustivetoavoidbias.• Easily-foundstudiesaredifferentfromthosethatwecannotfindeasily.Studiesfindinglarge,statisticallysignificanteffectsaremorelikelytobepublished,morelikelytobein“first-rate”journals,andmorelikelytobereferencedinotherarticles.
• Statisticaltechniquesexisttoaccountpartiallyforpublicationbias(funnelplots)buttheydonotreplaceanexhaustivesurvey.
• Decidewhetherto(holdyournoseand)includestudiesofapparentlypoorquality.Failuretohavewell-definedcriteriacanleadtobias(wearemorelikelytodiscardapoorstudyifitdisagreeswithourpethypothesis).
Howtocarryoutameta-analysis
• Ideally,thedataobtainedshouldallbeindependent,butnon-independenceofvarioussortscreepsin(e.g.multiplestudiesbythesamelab).
• Asinglestudymayprovidemeasurementsonmultiplespecies,ormeasurementsofmultipleresponsesonthesamespecies.Includethemallortakeasummarymeasure?
• Oneorasmallnumberofspecies(e.g.,greattit)orsystems(e.g.,intertidalzone)maybeoverrepresentedintheliterature.Treatthemallasindependent?
• Itmaybeworsetoleavedataout,ortakesummarymeasures,thantothroweverydatapointintotheanalysis.
Example2:Testosteronevsaggression
Booketal.(2001)asked“Aretestosteronelevelsandaggressioncorrelatedinhumanmales?”Itincludedahugediversityoftypesofstudies:
• levelsoftestosteroneinprisonersconvictedofviolentcrimescomparedtothoseofprisonersconvictedofpropertycrimes
• levelsoftestosteroneinuniversitystudentscomparedwiththeiranswerstoquestionnairesthataskedthemforlevelsofagreementtostatementslike“Ifsomebodyhitsme,Ihitback.”
• levelsofaggressionin!KungSanmalesasdeterminedbycounting“theirscarsandsometimesstillopenwoundsintheheadregion.”
• drunkenFinnishspouse-abuserscomparedtodrunkenFinnsdrinkingquietlyinabar.
• membersof“rambunctious”fraternitiescomparedto“responsible”fraternities
Example2:Testosteronevsaggression
Below is the “funnel plot” of studies comparing human aggression to levels oftestosterone. The curves show the approximate boundaries of the critical regionsthatwouldrejectthenullhypothesisinanyonestudywithα=0.05.
Howtocarryoutameta-analysis
3. Calculateaneffectsizethatcanbecombinedacrossstudiestoproduceaquantitativesummaryofthefindings.
• Correlationcoefficientriscommonlyusedthoughnotalwaysideal,becauseeffectsizedependsontherangeofthedata.
• Oddsratio–usedinhighlyhomogeneousstudies(oddsratiosintestsofaspirinandmyocardialinfarctions).
• Responseratio:𝑅 = 𝑌$/𝑌& orlogofresponseratio:ln(𝑅)
• Standardizedmeandifference,Cohen’sdorHedges’g:
𝑔 =𝑌$ − 𝑌&𝑠pooled
𝐽(𝑚)
sisthepooledsamplevarianceandJ(m)isasmall-samplebiascorrection.
Example3:Effectivenessofmarinereserves
Halpern(2003)usedthelogofresponseratiotocomparemarinereservestocomparisonareas(orthesameareabeforereserveestablishment)inabundanceanddiversityoffishand/orinvertebrates
Howtocarryoutameta-analysis
4. Statisticalinferenceonaverageeffectsize.
Fixedeffectsmodels
• Mostcommonlyusedinmedicalstudies.
• Assumesthatthemultiplestudieshavethesamemean,differingonlybecauseofsamplingerror.Ifeverystudywereinfinitelylarge,everystudywouldyieldanidenticalresult.Noheterogeneityamongthestudies.
• Perhapsneverjustifiedunlessallstudiesconductedsimilarlyandonthesamespecies.Thisisrarelythecaseinecologyandevolution.
Howtocarryoutameta-analysis
4.Statisticalinferenceonaverageeffectsize.
Random(mixed)effectsmodels
• Randomvariationispresentamongmeansofstudiesinadditiontosamplingerror.
• Individualstudiesarethereforeestimatingdifferenttreatmenteffects.
• Mostinterestisfocusedonthecentralvalue,ormean,ofthedistributionofeffects.
• Buttheideaofarandomeffectsmeta-analysisistounderstandthedistributionofeffectsacrossdifferentstudies.
Howtocarryoutameta-analysis
Fixedeffectmodel
Effectsizeofeachstudyiis
where
€
Θistheone“true”effectsize.
Randomeffectmodel
Effectsizeofstudyiis
whereμisthegrandmeanand
€
ζi isthedeviationofthe“true”effectsizeofstudyifromthegrandmean.
Thedifferenceaffectshoweachstudyisweighedwhencalculatingtheaverageeffectsizeoverallstudies.We’lldothisintheworkshop.
Unfortunately,lmecan’tbeusedforrandomeffectsmeta-analysisinR,becauseitwon’tcalculatethenecessaryweights.Otherpackages(e.g.,meta)areavailable.
€
Yi =Θ + ε i
€
Yi = µ + ζi + ε i
Howtocarryoutameta-analysis
5. Lookforeffectsofstudyquality.Forexample,areeffectsizesdifferentonaveragebetweenstudiesthatincludedblindingandthosethatdidnot?
6. Lookforassociationswithvariablesthatmightexplainheterogeneityofeffect
sizesamongstudies.Forexample,doestheaverageeffectsizedifferbetweenstudiescarriedoutonwomensubjectsandthoseonmalesubjects?
Example4:Meta-analysisofcompetitioninfieldexperiments
Gurevitchetal(1992)studyofinter-andintra-specificcompetition,lookingonlyatstudiespublishedin1980’s
Example4:Meta-analysisofcompetitioninfieldexperiments
Theylookedforeffectsofstudyquality
Example4:Meta-analysisofcompetitioninfieldexperiments
Theylookedforassociationswithvariablesthatmightexplainvariationineffectsize
File-drawerproblemInmeta-analysis,thedifficultiescausedbypublicationbiasarecalledthefile-drawerproblem,inreferencetotheunknownstudiessittingunavailableinresearchers’filedrawersorhiddeninobscurejournals.Thefile-drawerproblemisthepossiblebiasinestimatesandtestscausedbypublicationbias.
FunnelplotFunnelplotscangiveanindicationofthebiasresultingfromsmallstudies.SomaandGaramszegi(2011)usedtheTrimfillalgorithmtofillinhypotheticalmissingstudiesinthefunnelplottoachievetheoreticalsymmetry.
Fail-safenumberThefail-safenumbercalculateshowmanymissingstudieswouldbeneededtochangetheoverallresultofthemeta-analysis.Vilàetal(2011)estimatedthenumberofstudiesthatwouldhavetobeaddedtochangetheresultsoftheirinvasiveplantmeta-analysisfromsignificanttonon-significantas37,689.Thiswastooimplausible,sotheyconcludedthattheirestimateswerereliable.(Fig1a:toplinereferstototalplantproduction;otherlinesareeffectsonnativeplants)
Makeyourresultsaccessibletometa-analysisManypublishedpapersdonotreportenoughinformationformeta-analyststoextractthenumbersthattheyneed.Asaresult,manyotherwiserelevantpapershavetobediscarded.Don’tletthishappentoyourwork.• Alwaysgivesizesofeffectsandtheirstandarderrors.AP-valuebyitselfisuseless.
• Giveestimatesofthemeansandstandarddeviationsoftheimportantvariables.
• Alwaysindicateyoursamplesizesordegreesoffreedom.• Makethedataaccessible.PublishtherawdatainthepaperordeposittoanonlinearchivesuchasDryad.
Considerameta-analysisforyourfirstthesischapter
Often,thefirstchapterofathesisisareviewoftheliterature.Ifyourreviewisthorough,andyoukepttrackoftheimportantquantitiesandfeatureofeachstudy,youmayhaveenoughforaquantitativereview–yourownmeta-analysis.
Bestpracticesformeta-analysis
Bestpracticesformeta-analysis
Rtoolforbestpractices
Discussionpapernextweek:
RichmanandPrice:Evolutionofecologicaldifferencesinwarblers(multivariateanalysis)
Downloadfrom“Handouts”taboncoursewebsite.
Presenters:Vince&Rafael
Moderators:Sarah&Takuji
top related