Comparative Effectiveness Research:
Background, Priorities, Methods
1
Sean Tunis MD, MScJuly 10, 2009
http://www.cbo.gov/ftpdocs/89xx/doc8972/02-15-GeogHealth.pdf
Quality of Evidence for Guideline Recommendations in CV disease
Robert Califf, IOM Meeting on Evidence-based Medicine, December 2007
Great Expectations
Promoted from CBO director to OMB director “better information about the costs and
benefits of different treatment options, combined with new incentive structures reflecting the information….is essential to putting the country on a sounder long-term fiscal path.”
4
The nature of “C” evidence
The paradox “Available evidence is limited or poor quality” 18,000 RCTs are published every year
The gaps, as seen by decision makers Patients are highly selected Research settings are not typical of community Missing or incorrect comparators Physiologic or surrogate outcomes, not function Results are not available when decisions made
6
7
Committee’s Definition of CER
The generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.
8
Recommendation 6
The CER Program should fully involve consumers, patients, and caregivers in key aspects of CER, including strategic planning, priority setting, research proposal development, peer review, and dissemination.
9Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009
Engaging stakeholders
• Recommendation 1: Conduct a systematic assessment of best practices for effective engagement of decision makers
• Recommendation 2: As a condition of receiving federal funding for any CER study, the investigators must form a stakeholder advisory committee whose function is based on findings of #1
AJRR 9/19 Meeting Participants
1
•Shami Feinglass (CMS)
•Mike Rapp (CMS)
•Penny Mohr (CMS)
•Tom Gross (FDA)
•Susan Gardner (FDA)
•Danica Marinac-Dabic (FDA)
•Mark Melkerson (FDA)
•Art Sedrakyan (AHRQ)
•Jean Slutsky (AHRQ)
•Sean Tunis (CMTP)
•Mark McClellan (Brookings)
•Tony Rankin (AAOS)
•John Callaghan (AAOS/Iowa)
•Kevin Bozic (AAOS/UCSF)
•David Lewallen (AAOS/Mayo)
•Jeannie Kennedy (AAOS)
•Kathryn Sale (AAOS)
•Nancy Foster (AHA)
•Teresa Lee (Advamed)
•Jeff Kang (Cigna)
•Bob McDonough (Aetna)
•Joshua Benner (Brookings)
•David Brogan (Brookings)
•Michael Gluck (CMTP)
•Randee Kastner (CMTP)
11Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009
Categories of CER Methods
• Systematic reviews of existing research
• Decision modeling, with or without cost information
• Retrospective analysis of existing clinical or administrative data
• Prospective non-experimental studies, including registries
• Experimental studies, including randomized clinical trials (RCTs)
12Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009
All methods have a role
• Inevitable tradeoff between internal validity and feasibility, generalizability, cost, time
• The nature of the research question, and the decision maker will influence best practices
• Experimental studies will have a crucial role in CER, and there is need for improving design and implementation
• Non-experimental methods hold great promise, particularly as methods are refined and data infrastructure is improved
13Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009
Debate on CCTA Coverage
• Payers/researchers » want RCT with death/AMI outcome» 20k patients, 2+ years of follow-up» Non-experimental options rejected
• Vendors / clinicians » existing evidence adequate for coverage
• Medicare final decision (March 2008)» No adequately designed studies show improved outcomes» “We believe large, well-designed prospective trials needed”» Broad coverage by local contractors retained
• NHLBI currently reviewing 3 large RCTs• No progress on shared evidentiary framework for CER
14
Recommendation 7
The CER Program should devote sufficient resources to research and innovation in the methods of CER, including the development of methodological guidance for CER study design such as the appropriate use of observational data and more informative, practical, and efficient clinical trials.
Baucus-Conrad PCOR Legislation
• “Establish and maintain methodological standards for comparative clinical effectiveness research on major categories of interventions to prevent, diagnose, or treat a clinical condition or improve the delivery of care”
15
16Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009
Effectiveness Guidance Documents
• Analogous to FDA-guidance
• Targeted to product developers, clinical researchers
» Recommendations for design of clinical studies to generate evidence that will provide “reasonable confidence” of improved health outcomes
• Multi-stakeholder advisory group, iterative draft and comment process
• EGDs under development by CMTP» Gene expression profiling for breast cancer» Treatments for chronic wounds» Cardiac imaging» Integrative medicine» Radiation oncology
Pragmatic Pharmaceutical Trials
• Optimize design of phase III trials to be more informative for post-FDA decision makers
• Clarify patient, clinician, payer evidence needs • Identify critical regulatory, methodological
financial, operational barriers• Develop PPCT guidance document• Industry, FDA, CMS, NICE, PBAC, CDR,
Consumer’s Union, Medco, BSBCA, others
17
Contact Info
• [email protected]• www.cmtpnet.org• 443-759-3116 (D)• 410-963-8876 (M)
19
Final Portfolio: 100 CER Priority Topics
TABLE 5-1 Recommended Research Priorities by Research Area
CategoryPrimaryResearch Area
Secondary Research Area Total
Health Care Delivery Systems* 23 27 50Racial and Ethnic Disparities 3 26 29Cardiovascular and Peripheral Vascular 8 13 21Geriatrics 2 19 21Functional Limitations and Disabilities 2 20 22Neurologic Disorders 6 11 17Psychiatric Disorders 7 10 17Pediatrics 1 15 16
* Although this category was described as “Safety and Quality of Health Care” in the web-based questionnaire, the category was re-labeled by the committee as “Health Care Delivery System” to be more accurate.
20
TABLE 5-3 Committee’s Recommended Research Priorities by Types of Intervention
Types of Interventions Number of Topics
Systems of Care 43Pharmacological Treatment 36Standard of Care 33Behavioral Treatment 29Prevention 24Procedures 23Provider Patient Relationships 20Treatment Pathways 19Testing, Monitoring, and Evaluation 17Devices 13Alternative Treatment 9Other 18Total 284NOTE: The total exceeds the total number of priority topics as respondents were allowed to select
multiple interventions to be compared for each topic.
21
Recommendation 5
The HHS Secretary should establish a mechanism—such as a coordinating advisory body— with the mandate to strategize, organize, monitor, evaluate and report on the implementation and impact of the CER Program.
22
Recommendation 6
The CER Program should fully involve consumers, patients, and caregivers in key aspects of CER, including strategic planning, priority setting, research proposal development, peer review, and dissemination.
23
Recommendation 6 cont’d
– The CER Program should develop strategies to reach out to, engage, support, educate, and, as necessary prepare consumers, patients, and caregivers for leadership roles in these activities.
– The CER Program should also encourage participation in CER in order to create a representative evidence base that could help identify health disparities and inform decisions by patients in special population groups.
24
Recommendation 8The CER Program should help to develop
large-scale, clinical and administrative data networks to facilitate better use of data and more efficient ways to collect new data to inform CER.
– The CER Program should ensure that CER researchers and institutions consistently adhere to best practices to protect privacy and maintain security.
25
Recommendation 8 cont’d
– The CER Program should support the development of methodologies for linking patient-level data from multiple sources.
– The CER Program should encourage data holders to participate in CER and provide incentives for cooperation and maintaining data quality.
AHRQ Systematic Review: Tx of Clinically Localized Prostate Cancer
Limited evidence on relative safety and effectiveness of major treatment options prostatectomy, brachytherapy, radiation, active surveillance
New technologies rapidly spreading without data robotic surgery, proton beam
Rigorous trials needed to compare treatment options, especially for side effects
CMS Efforts to Improve Evidence Category B IDE regulation (1996) Cover routine costs of clinical trials (2000) Coverage with evidence development (2003) Promote pragmatic clinical trials (2003) Priorities for Sec 1013 of MMA (2004) MCAC becomes MedCAC (2005) Ad hoc efforts to work with NIH
27
Tools and Strategies for CER
Methodological framework Pragmatic clinical trials Coverage with evidence development
28
CMTP Mission
Collaborative projects to promote generation of new evidence that it is more informative to patients, consumer clinicians, and payers. Fewer C’s, more B’s and A’s
We don’t do CER studies; we develop methods, policies, and collaborations to make CER happen
29
CMTP Basics
Started Jan 2006 within HealthTech in SF CHCF and Blue Shield California Foundation
Incorporated as 501c3 in Jan 2008 (Maryland) 5 member governing board; 14 on advisory board
Funding Founding members: Blue Shield California,
Kaiser, United, Aetna NPC, Pfizer, Amgen, JNJ Additional funds from foundations, government,
professional societies Staffing: 12 FTEs
Project Categories
Priorities for evidence development Facilitated research
Total joint replacement registry Proton beam vs IMRT in prostate cancer
Methods development / guidance Gene expression, chronic wounds, cardiac imaging “Pragmatic” Phase III pharmaceutical trials
Applied policy Commercial payer CED MedPAC project on Medicare CER and CED
31
Coverage with Evidence Development
Links payment to requirement for prospective data collection
Intent is to guide clinical research to address questions of interest to Medicare Medicare must approve study design
Goal to support evidence and rapid access Lower evidence threshold with commitment
to generate better info later
32
33
Examples of Medicare CED
Lung volume reduction surgery FDG-PET for suspected dementia Implantable defibrillators Off-label use of drugs for colorectal cancer FDG-PET for oncology Home testing for sleep apnea Daily dialysis Artificial heart
CED Lesson Learned
Timing: when coverage under review, may be too late for CED
Methods Difficult to design studies in coverage context Registries provide broader access; ?? validity RCTs viewed as equivalent of non-coverage
Large simple trials may help, but few examples
Payers view as benefit expansion; Vendors opposite Unclear how best to fund clinical and research costs
Banff Workshop on CED
Presentations from Scotland, UK, Australia, US, Ontario, British Columbia, Alberta
Everyone trying to solve the same policy problem – balancing access and evidence
Encountering similar pressures and compromises, repeating the same mistakes
Some hints about viable approaches Will begin work on CED Field Guide for
Payers and Policy Makers
35
Medicare Review of CCTA
EPC report from Duke (April 2006) Limited evidence of clinical utility in any
population MedCAC mtg (May 2006)
“Uncertain confidence about existing evidence”
Broad local coverage of CCTA Medicare draft policy in 12/07 proposed CED
for CCTA in “adequate” studies
Debate on CCTA Coverage
Payers/researchers want RCT with death/AMI outcome 20k patients, 2+ years of follow-up Non-experimental options rejected
Vendors / clinicians existing evidence adequate for coverage
Medicare final decision (March 2008) No adequately designed studies show improved outcomes “We believe large, well-designed prospective trials needed” Broad coverage by local contractors retained
NHLBI currently reviewing 3 large RCTs
Cardiac Imaging Think Tank
Discussed alternative methods for evaluating cardiac imaging Co-sponsors: ICER, ACC, ACR, SCCT, ASNC private payers, CMS, vendors, clinical
researchers, consumers, AHRQ, VA, etc. Goal: balance validity with feasibility
Current status Meeting summary completed Effectiveness Guidance Document on clinical
utility of non-invasive cardiac imaging
Effectiveness Guidance Documents
Analogous to FDA-guidance Targeted to product developers, clinical researchers
Recommendations for design of clinical studies to generate evidence that is adequate for decision making
“reasonable confidence” of improved health outcomes Started from insights from systematic reviews Multi-stakeholder advisory group, iterative draft and
comment process Ongoing work
Gene expression profiling for breast cancer Treatments for chronic wounds Cardiac imaging Radiation oncology
Tools and Strategies for CER
Coverage with evidence development Methodological guidance Pragmatic clinical trials
40
The blank pragmatic–explanatory continuum indicator summary (PRECIS) “wheel.”
Domain Pragmatic Trial Explanatory Trial
Participants
Participant Eligibility Criteria All participants who have the condition of interest are enrolled, regardless of their anticipated risk, responsiveness, co-morbidities, or past compliance.
Step-wise selection criteria are applied that: (a) restrict study individuals to just those previously shown to be at highest risk of unfavorable outcomes, (b) further restrict these high risk individuals to just those who are thought likely to be highly responsive to the experimental intervention, and (c) include just those high risk, highly responsive study individuals who demonstrate high compliance with pre-trial appointment-keeping and a mock intervention.
PRECIS Domains Illustrating the Extremes of Explanatory and Pragmatic Approaches to Each Domain
Domain Pragmatic Trial Explanatory Trial
Follow-up and Outcomes
Primary Trial Outcome The primary outcome is an objectively measured, clinically meaningful outcome to the study participants. The outcome does not rely on central adjudication and is one that can be assessed under usual conditions: for example, special tests or training are not required.
The outcome is known to be a direct and immediate consequence of the intervention. The outcome is often clinically meaningful, but may sometimes (early dose-finding trials for example) be a surrogate marker of another downstream outcome of interest. It may also require specialized training or testing not normally used to determine outcome status or central adjudication.
PRECIS Domains Illustrating the Extremes of Explanatory and Pragmatic Approaches to Each Domain
Pragmatic Clinical Trials Initiative
Optimize design of phase III trials to be more informative for post-FDA decision makers
Clarify patient, clinician, payer evidence needs Identify critical regulatory, methodological
financial, operational barriers Develop PPCT guidance document Industry, FDA, CMS, NICE, PBAC, CDR,
Consumer’s Union, Medco, BSBCA, others
46
47
Contact Info
[email protected] www.cmtpnet.org 443-759-3116 (D) 410-963-8876 (M)