Comparative Effectiveness Research:

Background, Priorities, Methods

1

Sean Tunis MD, MScJuly 10, 2009

http://www.cbo.gov/ftpdocs/89xx/doc8972/02-15-GeogHealth.pdf

Quality of Evidence for Guideline Recommendations in CV disease

Robert Califf, IOM Meeting on Evidence-based Medicine, December 2007

Great Expectations

Promoted from CBO director to OMB director “better information about the costs and

benefits of different treatment options, combined with new incentive structures reflecting the information….is essential to putting the country on a sounder long-term fiscal path.”

4

The nature of “C” evidence

The paradox “Available evidence is limited or poor quality” 18,000 RCTs are published every year

The gaps, as seen by decision makers Patients are highly selected Research settings are not typical of community Missing or incorrect comparators Physiologic or surrogate outcomes, not function Results are not available when decisions made

6

7

Committee’s Definition of CER

The generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.

8

Recommendation 6

The CER Program should fully involve consumers, patients, and caregivers in key aspects of CER, including strategic planning, priority setting, research proposal development, peer review, and dissemination.

9Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009

Engaging stakeholders

• Recommendation 1: Conduct a systematic assessment of best practices for effective engagement of decision makers

• Recommendation 2: As a condition of receiving federal funding for any CER study, the investigators must form a stakeholder advisory committee whose function is based on findings of #1

AJRR 9/19 Meeting Participants

1

•Shami Feinglass (CMS)

•Mike Rapp (CMS)

•Penny Mohr (CMS)

•Tom Gross (FDA)

•Susan Gardner (FDA)

•Danica Marinac-Dabic (FDA)

•Mark Melkerson (FDA)

•Art Sedrakyan (AHRQ)

•Jean Slutsky (AHRQ)

•Sean Tunis (CMTP)

•Mark McClellan (Brookings)

•Tony Rankin (AAOS)

•John Callaghan (AAOS/Iowa)

•Kevin Bozic (AAOS/UCSF)

•David Lewallen (AAOS/Mayo)

•Jeannie Kennedy (AAOS)

•Kathryn Sale (AAOS)

•Nancy Foster (AHA)

•Teresa Lee (Advamed)

•Jeff Kang (Cigna)

•Bob McDonough (Aetna)

•Joshua Benner (Brookings)

•David Brogan (Brookings)

•Michael Gluck (CMTP)

•Randee Kastner (CMTP)

11Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009

Categories of CER Methods

• Systematic reviews of existing research

• Decision modeling, with or without cost information

• Retrospective analysis of existing clinical or administrative data

• Prospective non-experimental studies, including registries

• Experimental studies, including randomized clinical trials (RCTs)

12Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009

All methods have a role

• Inevitable tradeoff between internal validity and feasibility, generalizability, cost, time

• The nature of the research question, and the decision maker will influence best practices

• Experimental studies will have a crucial role in CER, and there is need for improving design and implementation

• Non-experimental methods hold great promise, particularly as methods are refined and data infrastructure is improved

13Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009

Debate on CCTA Coverage

• Payers/researchers » want RCT with death/AMI outcome» 20k patients, 2+ years of follow-up» Non-experimental options rejected

• Vendors / clinicians » existing evidence adequate for coverage

• Medicare final decision (March 2008)» No adequately designed studies show improved outcomes» “We believe large, well-designed prospective trials needed”» Broad coverage by local contractors retained

• NHLBI currently reviewing 3 large RCTs• No progress on shared evidentiary framework for CER

14

Recommendation 7

The CER Program should devote sufficient resources to research and innovation in the methods of CER, including the development of methodological guidance for CER study design such as the appropriate use of observational data and more informative, practical, and efficient clinical trials.

Baucus-Conrad PCOR Legislation

• “Establish and maintain methodological standards for comparative clinical effectiveness research on major categories of interventions to prevent, diagnose, or treat a clinical condition or improve the delivery of care”

15

16Implementing Comparative Effectiveness Research: Priorities, Methods, and Impact June 9, 2009

Effectiveness Guidance Documents

• Analogous to FDA-guidance

• Targeted to product developers, clinical researchers

» Recommendations for design of clinical studies to generate evidence that will provide “reasonable confidence” of improved health outcomes

• Multi-stakeholder advisory group, iterative draft and comment process

• EGDs under development by CMTP» Gene expression profiling for breast cancer» Treatments for chronic wounds» Cardiac imaging» Integrative medicine» Radiation oncology

Pragmatic Pharmaceutical Trials

• Optimize design of phase III trials to be more informative for post-FDA decision makers

• Clarify patient, clinician, payer evidence needs • Identify critical regulatory, methodological

financial, operational barriers• Develop PPCT guidance document• Industry, FDA, CMS, NICE, PBAC, CDR,

Consumer’s Union, Medco, BSBCA, others

17

Contact Info

• sean.tunis@cmtpnet.org• www.cmtpnet.org• 443-759-3116 (D)• 410-963-8876 (M)

19

Final Portfolio: 100 CER Priority Topics

TABLE 5-1 Recommended Research Priorities by Research Area

CategoryPrimaryResearch Area

Secondary Research Area Total

Health Care Delivery Systems* 23 27 50Racial and Ethnic Disparities 3 26 29Cardiovascular and Peripheral Vascular 8 13 21Geriatrics 2 19 21Functional Limitations and Disabilities 2 20 22Neurologic Disorders 6 11 17Psychiatric Disorders 7 10 17Pediatrics 1 15 16

* Although this category was described as “Safety and Quality of Health Care” in the web-based questionnaire, the category was re-labeled by the committee as “Health Care Delivery System” to be more accurate.

20

TABLE 5-3 Committee’s Recommended Research Priorities by Types of Intervention

Types of Interventions Number of Topics

Systems of Care 43Pharmacological Treatment 36Standard of Care 33Behavioral Treatment 29Prevention 24Procedures 23Provider Patient Relationships 20Treatment Pathways 19Testing, Monitoring, and Evaluation 17Devices 13Alternative Treatment 9Other 18Total 284NOTE: The total exceeds the total number of priority topics as respondents were allowed to select

multiple interventions to be compared for each topic.

21

Recommendation 5

The HHS Secretary should establish a mechanism—such as a coordinating advisory body— with the mandate to strategize, organize, monitor, evaluate and report on the implementation and impact of the CER Program.

22

Recommendation 6

The CER Program should fully involve consumers, patients, and caregivers in key aspects of CER, including strategic planning, priority setting, research proposal development, peer review, and dissemination.

23

Recommendation 6 cont’d

– The CER Program should develop strategies to reach out to, engage, support, educate, and, as necessary prepare consumers, patients, and caregivers for leadership roles in these activities.

– The CER Program should also encourage participation in CER in order to create a representative evidence base that could help identify health disparities and inform decisions by patients in special population groups.

24

Recommendation 8The CER Program should help to develop

large-scale, clinical and administrative data networks to facilitate better use of data and more efficient ways to collect new data to inform CER.

– The CER Program should ensure that CER researchers and institutions consistently adhere to best practices to protect privacy and maintain security.

25

Recommendation 8 cont’d

– The CER Program should support the development of methodologies for linking patient-level data from multiple sources.

– The CER Program should encourage data holders to participate in CER and provide incentives for cooperation and maintaining data quality.

AHRQ Systematic Review: Tx of Clinically Localized Prostate Cancer

Limited evidence on relative safety and effectiveness of major treatment options prostatectomy, brachytherapy, radiation, active surveillance

New technologies rapidly spreading without data robotic surgery, proton beam

Rigorous trials needed to compare treatment options, especially for side effects

CMS Efforts to Improve Evidence Category B IDE regulation (1996) Cover routine costs of clinical trials (2000) Coverage with evidence development (2003) Promote pragmatic clinical trials (2003) Priorities for Sec 1013 of MMA (2004) MCAC becomes MedCAC (2005) Ad hoc efforts to work with NIH

27

Tools and Strategies for CER

Methodological framework Pragmatic clinical trials Coverage with evidence development

28

CMTP Mission

Collaborative projects to promote generation of new evidence that it is more informative to patients, consumer clinicians, and payers. Fewer C’s, more B’s and A’s

We don’t do CER studies; we develop methods, policies, and collaborations to make CER happen

29

CMTP Basics

Started Jan 2006 within HealthTech in SF CHCF and Blue Shield California Foundation

Incorporated as 501c3 in Jan 2008 (Maryland) 5 member governing board; 14 on advisory board

Funding Founding members: Blue Shield California,

Kaiser, United, Aetna NPC, Pfizer, Amgen, JNJ Additional funds from foundations, government,

professional societies Staffing: 12 FTEs

Project Categories

Priorities for evidence development Facilitated research

Total joint replacement registry Proton beam vs IMRT in prostate cancer

Methods development / guidance Gene expression, chronic wounds, cardiac imaging “Pragmatic” Phase III pharmaceutical trials

Applied policy Commercial payer CED MedPAC project on Medicare CER and CED

31

Coverage with Evidence Development

Links payment to requirement for prospective data collection

Intent is to guide clinical research to address questions of interest to Medicare Medicare must approve study design

Goal to support evidence and rapid access Lower evidence threshold with commitment

to generate better info later

32

33

Examples of Medicare CED

Lung volume reduction surgery FDG-PET for suspected dementia Implantable defibrillators Off-label use of drugs for colorectal cancer FDG-PET for oncology Home testing for sleep apnea Daily dialysis Artificial heart

CED Lesson Learned

Timing: when coverage under review, may be too late for CED

Methods Difficult to design studies in coverage context Registries provide broader access; ?? validity RCTs viewed as equivalent of non-coverage

Large simple trials may help, but few examples

Payers view as benefit expansion; Vendors opposite Unclear how best to fund clinical and research costs

Banff Workshop on CED

Presentations from Scotland, UK, Australia, US, Ontario, British Columbia, Alberta

Everyone trying to solve the same policy problem – balancing access and evidence

Encountering similar pressures and compromises, repeating the same mistakes

Some hints about viable approaches Will begin work on CED Field Guide for

Payers and Policy Makers

35

Medicare Review of CCTA

EPC report from Duke (April 2006) Limited evidence of clinical utility in any

population MedCAC mtg (May 2006)

“Uncertain confidence about existing evidence”

Broad local coverage of CCTA Medicare draft policy in 12/07 proposed CED

for CCTA in “adequate” studies

Debate on CCTA Coverage

Payers/researchers want RCT with death/AMI outcome 20k patients, 2+ years of follow-up Non-experimental options rejected

Vendors / clinicians existing evidence adequate for coverage

Medicare final decision (March 2008) No adequately designed studies show improved outcomes “We believe large, well-designed prospective trials needed” Broad coverage by local contractors retained

NHLBI currently reviewing 3 large RCTs

Cardiac Imaging Think Tank

Discussed alternative methods for evaluating cardiac imaging Co-sponsors: ICER, ACC, ACR, SCCT, ASNC private payers, CMS, vendors, clinical

researchers, consumers, AHRQ, VA, etc. Goal: balance validity with feasibility

Current status Meeting summary completed Effectiveness Guidance Document on clinical

utility of non-invasive cardiac imaging

Effectiveness Guidance Documents

Analogous to FDA-guidance Targeted to product developers, clinical researchers

Recommendations for design of clinical studies to generate evidence that is adequate for decision making

“reasonable confidence” of improved health outcomes Started from insights from systematic reviews Multi-stakeholder advisory group, iterative draft and

comment process Ongoing work

Gene expression profiling for breast cancer Treatments for chronic wounds Cardiac imaging Radiation oncology

Tools and Strategies for CER

Coverage with evidence development Methodological guidance Pragmatic clinical trials

40

The blank pragmatic–explanatory continuum indicator summary (PRECIS) “wheel.”

Domain Pragmatic Trial Explanatory Trial

Participants

Participant Eligibility Criteria All participants who have the condition of interest are enrolled, regardless of their anticipated risk, responsiveness, co-morbidities, or past compliance.

Step-wise selection criteria are applied that: (a) restrict study individuals to just those previously shown to be at highest risk of unfavorable outcomes, (b) further restrict these high risk individuals to just those who are thought likely to be highly responsive to the experimental intervention, and (c) include just those high risk, highly responsive study individuals who demonstrate high compliance with pre-trial appointment-keeping and a mock intervention.

PRECIS Domains Illustrating the Extremes of Explanatory and Pragmatic Approaches to Each Domain

Domain Pragmatic Trial Explanatory Trial

Follow-up and Outcomes

Primary Trial Outcome The primary outcome is an objectively measured, clinically meaningful outcome to the study participants. The outcome does not rely on central adjudication and is one that can be assessed under usual conditions: for example, special tests or training are not required.

The outcome is known to be a direct and immediate consequence of the intervention. The outcome is often clinically meaningful, but may sometimes (early dose-finding trials for example) be a surrogate marker of another downstream outcome of interest. It may also require specialized training or testing not normally used to determine outcome status or central adjudication.

PRECIS Domains Illustrating the Extremes of Explanatory and Pragmatic Approaches to Each Domain

Pragmatic Clinical Trials Initiative

Optimize design of phase III trials to be more informative for post-FDA decision makers

Clarify patient, clinician, payer evidence needs Identify critical regulatory, methodological

financial, operational barriers Develop PPCT guidance document Industry, FDA, CMS, NICE, PBAC, CDR,

Consumer’s Union, Medco, BSBCA, others

46

47

Contact Info

sean.tunis@cmtpnet.org www.cmtpnet.org 443-759-3116 (D) 410-963-8876 (M)