the reproducibility crises in biomedical research and its impact on pharmaceutical r&d...
DESCRIPTION
There has been a significant decline of productivity in pharmaceutical R&D • Many different reasons have been discussed, but little attention has been paid to the role of poor reproducibility of biomedical research in general • There are often fundamental doubts about the credibility of research data leading to costly and often futile repetition of biomedical research studies • One of the root causes for the reproducibility problem is the lack of a common quality standard for biomedical research • There have been recent initiatives by the WHO and the FD&C Division of the American Society for Quality (ASQ) to address this issue • The pharmaceutical industry needs to embrace these initiatives in order to reverse the negative productivity trendTRANSCRIPT
![Page 1: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/1.jpg)
The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity
Ülo Palm, MD, PhD, Ülo Palm, MD, PhD,
Senior Vice President Clinical Operations & BiometricsSenior Vice President Clinical Operations & BiometricsForest Research InstituteForest Research Institute
February 11, 2013February 11, 2013
1
![Page 2: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/2.jpg)
Moore’s Law:Moore’s Law:Transistor Count Transistor Count Doubling Every Doubling Every
Two YearsTwo Years
Source: Wikipedia 2
![Page 3: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/3.jpg)
Eroom’s Law:Eroom’s Law:Number Of New Drug Number Of New Drug
Approvals Per Billion US Approvals Per Billion US Dollars Dollars Halved Every Nine Halved Every Nine
YearsYears
Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March 2012, page 191 - 200
3
![Page 4: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/4.jpg)
““War On Cancer” Going On For 40 War On Cancer” Going On For 40 YearsYears
• About $200 billion spent on cancer research in the US since 1971*
• More than 1.5 million papers published*But despite some progress in prevention and
treatment• Nearly 1 in 2 men and more than 1 in 3
women diagnosed with cancer during their lifetime**
• Almost 1.5 million cancer cases and 560 000 cancer deaths in the United States in 2009**
• Cancer now the second leading cause of death in the US**
Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303, 4
![Page 5: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/5.jpg)
Some Causes Of Declining Some Causes Of Declining Productivity Of Biomedical Productivity Of Biomedical
ResearchResearch
• Biomedical R&D increasingly complex*
• Higher regulatory hurdles *• Pharmaceutical companies too big
to innovate
*Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March 2012, page 191 - 200
5
![Page 6: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/6.jpg)
Is There Another Major, Root Cause Of The Is There Another Major, Root Cause Of The Declining Productivity Of Biomedical Declining Productivity Of Biomedical Research?Research?
6
![Page 7: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/7.jpg)
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
“Five of the seven largest published studies addressing cancer prognosis did not classify patients better than chance”
7
![Page 8: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/8.jpg)
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
8
![Page 9: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/9.jpg)
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
November 2010
9
![Page 10: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/10.jpg)
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
“This is one of medicine's dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can't be reproduced”
December 2011
10
![Page 11: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/11.jpg)
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
“An empirical assessment of 18 published papers of microarray studies showed that independent analysts could perfectly reproduce the results of only two of the studies…”
December 2011
11
![Page 12: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/12.jpg)
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:2012
“Forensic bioinformatics is the art of using raw data and reported results to infer what must have happened to get from one to the other”
12
![Page 13: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/13.jpg)
13
![Page 14: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/14.jpg)
Lancet 2002Lancet 2002
• A new mass spectrometry method a highly sensitive and specific diagnostic tests for ovarian cancer
• NCI launched clinical Proteomics initiative• Companies formed to take assays based on this
method to the clinic• Turned out to be an experimental artifact due to
running of all of the controls before all of the cancers
Source:
Keith A . Baggerly and Kevin R . Coombes Handbook of Statistics in Clinical Oncology, Third EditionAntje Crowley and John Hoering Chapman and Hall/CRC 2012 Pages 605–618
Petricoin EF, Ardekani AM, Hitt BA, Levine PJ, Fusaro VA, Steinberg SM, Mills GB et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet 2002; 359(9306):572–577
14
![Page 15: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/15.jpg)
Nature Medicine 2006Nature Medicine 2006
• Microarray-based signatures of drug sensitivity derived from cell lines predict patient response to specific chemotherapeutics
• Discover magazine designated this paper as one of the top 100 breakthroughs of 2006
• Large clinical trials started using this methodology• In 2009, independent analysis demonstrated that the
data were wrong due to mislabeling and indexing errors
Source: Keith A . Baggerly and Kevin R . Coombes Handbook of Statistics in Clinical Oncology, Third EditionAntje Crowley and John Hoering Chapman and Hall/CRC 2012 Pages 605–618
Potti A, Dressman HK, Bild A, Riedel RF, Chan G, Sayer R, Cragun J et al. Genomic signatures to guide the use of chemotherapeutics. Nat Med 2006; 12:1294–1300. Retracted January 7,
2011. 15
![Page 16: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/16.jpg)
A few “What If Questions”A few “What If Questions”
16
![Page 17: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/17.jpg)
Impact On Industry Impact On Industry ProductivityProductivity
Amgen*• Published literature described that inhibition of STK33
destroyed cancer cells• Amgen could not replicate the results despite massive
research effortsBayer*• Bayer halted in 2011 nearly two-thirds of its early drug target
projects because in-house experiments failed to match claims made in the literature.
Pfizer***• Pfizer wrote off $750 million in 2012 after results of a study
with Dimebon for Alzheimer, published originally in the journal Lancet*, could not be reproduced
Source: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012 17
![Page 18: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/18.jpg)
What Are The Reasons For What Are The Reasons For The Lack Of Reproducibility The Lack Of Reproducibility
In Modern Biomedical In Modern Biomedical Research?Research?
18
![Page 19: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/19.jpg)
Modern Science Is Based On Modern Science Is Based On Collective IntelligenceCollective Intelligence
• Collective intelligence requires: – a shared body of knowledge, methods,
and techniques, a shared praxis* – a shared and agreed upon quality standard
how to plan, conduct, and report scientific work
*Source: Reinventing Discovery – The new era of networked science by Michael Nielsen, 2012
19
![Page 20: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/20.jpg)
No Well Defined Quality No Well Defined Quality Standard Exists In Biomedical Standard Exists In Biomedical
ResearchResearch
• “Genes were mislabeled due to an off-by-one indexing error”
• “The method didn’t actually work at all; it only appeared to work due to poor bookkeeping”
• “A disconnect between the numbers and the sample names rendered the predictions invalid”
• “Poor documentation allowed errors to go unnoticed until after things had proceeded to clinical trials “
Source: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in Clinical Oncology, Third Edition, Antje Crowley and John Hoering Chapman and Hall/CRC 2012 Pages 605–618 20
![Page 21: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/21.jpg)
WHO: Development Of A Common Quality WHO: Development Of A Common Quality Standard For Biomedical Research A Pressing Standard For Biomedical Research A Pressing Global NeedGlobal Need
• “The world’s population is facing serious health challenges…
• .. there is increased demand for new drugs and new principles for treatment …
• ..it is essential that basic scientific (biomedical) research , …is conducted in a proper fashion… using processes that minimize waste of resources
• ….to reduce the need for costly repetition of work already performed”
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006 21
![Page 22: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/22.jpg)
American Society For Quality (ASQ) : Best Quality American Society For Quality (ASQ) : Best Quality Practices For Biomedical Research In Drug Practices For Biomedical Research In Drug DevelopmentDevelopment
• The objective is to increase the overall productivity of biomedical research by promoting a common quality standard:
• “A common quality guideline for biomedical research in drug development will ensure the validity and credibility of scientific data from different research institutions and facilitate the mutual acceptance of research results”
22
Source: http://asq.org/quality-press/display-item/index.html?item=T919E
![Page 23: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/23.jpg)
A Vision For The FutureA Vision For The Future
End-to-end electronic•Integration•Analysis•Modeling•Simulation
Standardized and validated external research
Standardized and validated internal research
Electronic Submission
23
![Page 24: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/24.jpg)
Summary Of IssuesSummary Of Issues
• Declining productivity of Drug Development
• Poor reproducibility of Biomedical Research
• Lack of a common quality standard in Biomedical Research
24
![Page 25: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/25.jpg)
SolutionSolution
Establish a common quality standard for biomedical research in drug development•WHO and ASQ published quality guidelines for Biomedical Research•Pharmaceutical Industry should promote these guidelines to improve R&D productivity
25
![Page 26: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/26.jpg)
Thank You!
26
![Page 27: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/27.jpg)
Backup slides
27
![Page 28: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/28.jpg)
The Flow Of Research ActivitiesFrom Planning To Publishing
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006
![Page 29: The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity](https://reader036.vdocuments.site/reader036/viewer/2022062312/556318bbd8b42a5b498b50ec/html5/thumbnails/29.jpg)
““Best Quality Practices For Biomedical Research Best Quality Practices For Biomedical Research And Drug Development” And Drug Development” - Technical Report Content -- Technical Report Content -
• Management system• Organization• Project Management• Quality Management
System• Documentation• Document control /
Document approval and issue:
• Document changes
• Document storage• Technical Requirements• Test Equipment• Test methods / Method
Validation• Sampling and Chain of Custody• Materials• Legal and Ethical
Considerations• Vendor Selection and
Qualification