ioannidis 2005
DESCRIPTION
TRANSCRIPT
![Page 1: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/1.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Why most published research findings are falseArticle by John P. A. Ioannidis (2005)
Aurelien Madouasse
November 4, 2011
![Page 2: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/2.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
![Page 3: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/3.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The author: John P.A. Ioannidis
• C.F. Rehnborg Chair in Disease Prevention at StanfordUniversity (US)
• Professor of Medicine and Director of the StanfordPrevention Research Center (US)
• Chaired the Department of Hygiene and Epidemiology atthe University of Ioannina School of Medicine (Greece)
• Has a 51 page CV
![Page 4: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/4.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The author: John P.A. Ioannidis• C.F. Rehnborg Chair in Disease Prevention at Stanford
University (US)• Professor of Medicine and Director of the Stanford
Prevention Research Center (US)• Chaired the Department of Hygiene and Epidemiology at
the University of Ioannina School of Medicine (Greece)• Has a 51 page CV
![Page 5: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/5.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The journal: PLoS Medicine
• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900
![Page 6: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/6.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The journal: PLoS Medicine• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900
![Page 7: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/7.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The article (Checked 2011-10-22)
• Views: 410,087• Citations:
• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585
![Page 8: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/8.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The article (Checked 2011-10-22)• Views: 410,087• Citations:
• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585
![Page 9: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/9.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
![Page 10: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/10.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Introduction
• Published research findings sometimes refuted bysubsequent evidence
• Increasing concern false findings may be the majority
• This should no be surprising
• Here is why . . .
![Page 11: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/11.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
![Page 12: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/12.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis testing
• Consider a parameter measured in a population ofindividuals with a disease:
• Before treatment
• After treatment (Here assuming the treatment has an effect)
Some Parameter
Fre
quen
cy
![Page 13: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/13.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis testing
• Consider a parameter measured in a population ofindividuals with a disease:
• Before treatment• After treatment (Here assuming the treatment has an effect)
Some Parameter
Fre
quen
cy
![Page 14: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/14.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis
• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 15: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/15.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis
• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 16: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/16.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 17: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/17.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 18: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/18.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 19: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/19.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 20: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/20.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 21: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/21.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 22: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/22.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect
• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 23: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/23.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
![Page 24: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/24.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
![Page 25: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/25.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
![Page 26: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/26.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
![Page 27: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/27.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
![Page 28: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/28.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
![Page 29: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/29.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
![Page 30: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/30.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• For a given hypothesis, whether we get it wrong dependson:
• Whether the hypothesis is true• The magnitude of the effect• The values we choose for α and β
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
![Page 31: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/31.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper
• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
![Page 32: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/32.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses
• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
![Page 33: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/33.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True
• Hypothesis testing can be seen as testing for a disease inEpidemiology
• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
![Page 34: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/34.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology
• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
![Page 35: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/35.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity
• 1− α is the specificity• We can define a positive predictive value
![Page 36: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/36.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity
• We can define a positive predictive value
![Page 37: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/37.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
![Page 38: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/38.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
![Page 39: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/39.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
PPV =p(1− β)
p(1− β) + (1− p)α
• Ioannidis uses R = p1−p
![Page 40: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/40.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R
1+R × (1− β)R
1+R × (1− β) + 11+R × α
![Page 41: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/41.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− β)
R(1− β) + α
![Page 42: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/42.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
• Among the studies that should have been reported asnegative
• A proportion u are reported as positive because of bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
![Page 43: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/43.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
• Among the studies that should have been reported asnegative
• A proportion u are reported as positive because of bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
![Page 44: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/44.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
![Page 45: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/45.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
• Positive predictive value
PPV =p(1− β + uβ)
p(1− β + uβ) + (1− p)(α + u(1− α))
• Ioannidis uses R = p1−p
![Page 46: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/46.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− β) + uβR
R + α− βR + u − uα + uβR
![Page 47: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/47.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
Power = 0.8
![Page 48: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/48.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
Power = 0.5
![Page 49: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/49.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
Power = 0.2
![Page 50: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/50.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published
• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
![Page 51: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/51.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
![Page 52: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/52.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
![Page 53: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/53.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
![Page 54: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/54.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
![Page 55: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/55.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
![Page 56: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/56.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
![Page 57: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/57.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Total p 1 − p
• Positive predictive value
PPV =p(1− βn)
p(1− βn) + (1− p)(1− (1− α)n)
• Ioannidis uses R = p1−p
![Page 58: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/58.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− βn)
R + 1− ((1− α)n + Rβn)
![Page 59: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/59.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
Power = 0.8
![Page 60: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/60.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
Power = 0.5
![Page 61: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/61.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
Power = 0.2
![Page 62: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/62.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds
• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 63: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/63.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds
• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 64: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/64.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5
• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 65: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/65.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 66: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/66.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 67: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/67.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???
• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 68: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/68.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???
• Problem: Gold Standard
![Page 69: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/69.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
![Page 70: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/70.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• Link between magnitude of the effect, α, β and samplesize
• Trade off between α and β• Smaller effects require bigger samples
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
![Page 71: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/71.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• Link between magnitude of the effect, α, β and samplesize
• Trade off between α and β
• Smaller effects require bigger samples
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
![Page 72: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/72.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• Link between magnitude of the effect, α, β and samplesize
• Trade off between α and β• Smaller effects require bigger samples
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
![Page 73: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/73.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The corollaries follow from the proposed model
![Page 74: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/74.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
![Page 75: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/75.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 1
The smaller the studies conducted in a scientific field, theless likely the research findings are to be true
![Page 76: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/76.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 2
The smaller the effect sizes in a scientific field, the lesslikely the research findings are to be true
![Page 77: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/77.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 3
The greater the number and the lesser the selection oftested relationships in a scientific field, the less likely theresearch findings are to be true
![Page 78: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/78.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 4
The greater the flexibility in designs, definitions, outcomesand analytical modes in a scientific field, the less likely theresearch findings are to be true
![Page 79: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/79.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 5
The greater the financial and other interests and prejudicesin a scientific field, the less likely the research findings areto be true
![Page 80: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/80.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 6
The hotter a scientific field (with more scientific teamsinvolved), the less likely the research findings are to be true
![Page 81: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/81.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
![Page 82: Ioannidis 2005](https://reader034.vdocuments.site/reader034/viewer/2022051412/54943446b47959654d8b4a3e/html5/thumbnails/82.jpg)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
How can we improve the situation?
• Cannot draw firm conclusions based on a single positiveresult
• It is possible to test for something until we find what wewant!
• And this is more likely to receive attention
• Selecting research questions• Avoid marketing driven questions• Importance of pre study odds
• Increase power• Larger samples
• For research questions with high pre-study odds• To test major concepts rather than narrow specific
questions
• Research standards