publication bias & negative results
TRANSCRIPT
NEGATIVE RESULTS &PUBLICATION BIAS
PRISONER’S DILEMMA
SO WHO PUBLISHESNEGATIVE RESULTS
????
TITLE• Negativity towards negative results: a
discussion of the disconnect between
scientific worth and scientific culture.
AUTHORS
1 Faculty of Science, Medicine and Health, Illawarra Health and Medical research institute, University of Wollongong, NSW 522, Australia.
2Schizophrenia Research Institute, NSW 2010, Australia.
Natalie Matosin1,2,*, Elisabeth Frank1,2, Martin Engel1,2, Jeremy S.
Lum1,2, and Kelly A1,2.
JOURNAL :
VOLUME : 7
ISSUE : 2014
PAGES : 171 – 173
IMPACT FACTOR : 4.316
“What gets us into trouble is not what we don’t know, it’s what we know for sure that
just ain’t so.” – Mark Twain.
SCIENCE – Flawless system of knowledge.
REALITY – This is not always the case.
BACKGROUND
NEGATIVE / NEWBut what happens when you obtain results that support the null hypothesis, or do not fit with the current scientific thinking???
POSITIVEDissemination of results are straightforward when the findings are positive.
Issues surrounding Publication bias and difficulty in
communicating the negative results.
THIS ARTICLE IS ABOUT
PRESSUE ON SCIENTISTS
HIGH-IMPACT STUDIES.
ILLOGICAL or IMPRACTICAL
SEARCH OF SIGNIFICANCE
FOCUS ON POSITIVE
FILE DRAWER EFFECT
(SCARGLE 1999)
Fanelli - 2010,2012• “Papers are less likely to be
published and to be cited if they report ‘negative’ results”
(Fanelli, 2010)
• Consequently, the amount of non-significant data reported is progressively declining
(Fanelli, 2012)
REASON
Selectiveness of ‘high impact’ journals
Have a bold statement in the
submission form:Negative results are not
accepted.
Positive Results = High Impact Results
Why negative result is not considered equally important??
Science should tell all sides of the story.
Scientific principles are always under reconsideration
New evidence will refute old evidence and current scientific thinking
BUT THIS IS DIFFICULT TO ACHIEVE
ANDREW WAKEFIELD MMR CONTROVERSYUK based researcher Dr. Andrew Wakefield.
Child vaccination (specifically the MMR vaccine) causes
Incidence Of Autism (Wakefield et al., 1998).
Led to Panic & decade long decrease in child immunisation
13 studies with convincing negative results published 1998 - 2010
Support against Wakefield’s claims failed to gain the same level of attention as the original study
article retraction in 2010
TRANSITIONING BETWEEN PARADIGMSOnce the body of evidence for the competing paradigm overtakes the evidence in support of the dominant paradigm, then scientists will easily switch allegiance (Kuhn, 1970)Humans have an inbuilt need to support the status quo, and therefore have an innate difficulty in overriding pre-existing beliefs (Jost and Hunyady, 2003)
CORRECTING THE LITERATURE: AN UPHILL BATTLE
Extreme difficulties in correcting the literature have been experienced by many, but discussed openly by few.
Dr Peter Wilmshurst
“Amrinone increased heart contractility in a clinical trial” (Benotti et al., 1978)
“Although amrinone increased the strength of contraction of normal heart muscle, it did not affect contractility in patients with heart failure” (Wilmshurst, 2003)
Wilmshurst spent two decades to disseminating his negative findings
and remained unsuccessful.
DR.DAVID VAUXPeer-reviewed “a mechanism to overcome the rejection of transplanted tissue ’’ in Nature
(Bellgrau et al., 1995)News and Views’ piece, published in the same
issue of Nature
(Vaux, 1995)He started conducting experiments in his own laboratory.
But failed to produce results
He tried submitting his new findings and tried to refute his earlier article but ultimately
he failed.
“Little did we know that instead of providing an answer to transplant rejection, these experiments would teach us a great deal about editorial practices and the difficulty of correcting errors once they appear in the literature” (Vaux, 2013)
After 2 years he retracted his article.
Because the results were unpublished
Research groups had continued to follow the same lines of thought and the same paths of investigation, only to all fail in the same way, ultimately wasting time and resources.
Both unsuccessfully attempted to alter depression-like behaviour in the CD1 mouse strain with a variety of classical antipsychotics.
Both failed
SIGNIFICANCE BIAS
Statistically significant results have been shown to be three times more likely to be published than papers with null results. K.Dickerson et al,1987
MISUSED AND MISINTERPRETED
DOES NOT SUPPORT HYPOTHESIS
ITS ONLY A TOOL TO REJECT NULL HYPOTHESIS
BIAS IN META - ANALYSISFUNNEL PLOTS
Exploring Publication Bias
SYMMETRICAL – NORMAL DISTRIBUTION ASYMMETRICAL
Gaping hole where negative studies should be
WHY NEGATIVE RESULTS ARE NOT PURSUED???
• Time and effort to construct the paper.
• Surviving the peer review is not out weighed
by the benefits.• Not considered high impact
knowledge.• Will not result in a highly cited paper.
How are we going to reverse the anti-negative-finding culture?
“Perhaps a lab should have to correct for the total number of published results in a given year”
Researchers should be obligated to retract their previous works throughout the progression of their career as they “…[find] that [their] previous tests in old papers are no longer significant in light of their success and, ironically, [their] contribution to the field”
(Lieberman and Cunningham, 2009)
NEGATING THE NEGATIVITYRevolt against publication bias.
Publication bias is a common theme and it still remains an issue.
“…negative findings are still a low priority for publication, so we need to find ways to make publishing them more attractive” (O’Hara, 2011)
Direction of scientific research Not determined by the pressure to win
the ‘significance lottery’
But Systematic, hypothesis-driven
attempts to fill holes in our knowledge
It Is Our Duty As Scientists To
(1)Publish all data, no matter what the outcome, because a negative finding is still an important finding.
(2)Have a hypothesis to explain the finding.
CONCLUSIONIf the experiment- Planned properly- The data has not been manipulated- Or pulled out of context - Compiled evidence of a negative
result, thenIt should provide an explanation as to
why we are seeing what we are seeing???
Only by truly rethinking the current scientific culture,
Will negative results be esteemed for their entire value.
Only then can we work towards an improved scientific paradigm.
Having negative results isn’t bad, what’s bad is failing to report them.
REFERENCES1) Egger M, Davey Smith G, Schneider M,
Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ : British Medical Journal. 1997;315(7109):629-634.
2) Dickersin, Kay, et al. "Publication bias and clinical trials." Controlled clinical trials 8.4 (1987): 343-353.
