GPC is a major initiative of the Center for Medical Technology Policy

REAL WORLD EVIDENCEINITIATIVE REPORT

February 2017

GREEN PARK COLLABORATIVE

• A multi-stakeholder forum to clarify the evidence expectations of payers– Public and private payer requirements for evidence of effectiveness and

value for specific conditions and technologies• Key participant-stakeholders in initiatives

– Public and private payers, FDA, NIH, AHRQ, guideline developers, professional societies, life sciences companies

• Output/Activities (depending on initiative)– Effectiveness guidance documents (EGDs)– High priority topic-specific workshops and webinars

• Benefits of participation– Greater transparency of decision-maker expectations (payers, HTA,

guideline developers, health systems, etc.); substantive interaction with key stakeholders; input into methods guidance and recommendations

RWE INITIATIVE: PURPOSE

ProblemMany frameworks and checklists exist to assess the quality of observational and other real-world studies• Frameworks complex, lengthy,

difficult to use• Relatively little input from users• How users make decisions on

single studies and bodies of evidence remains unclear

SolutionDevelop easy-to-use tool for transparent assessment of RWE• User-vetted• “Informed judgment” and visual

summary of methodological rigor and relevance of RW studies

• Detailed assessment can follow, if merited

• Aims: consistency, improved understanding, transparency

APPROACH TO FRAMEWORK DEVELOPMENT

FOUR-PHASED APPROACH

Incorporation of Key Findings and Final RWE Framework

Vetting of Draft Framework and In-Person Meetings

Background Research

Stakeholder and Expert Engagement

TIMELINE

QUARTER 1

QUARTER 2 JUNE 2016 QUARTER

3 – 4FEBRUARY

2017

Identify target users in health plans, health systems, other contexts; assess decision needs of users through interviews

Convene expert workgroups Advisory Committee Methods WG Dissemination WG

Hold in-person workshop with decision-makers to road test tool

Incorporate feedback from workshop and additional user vetting

Produce final framework and interactive tool with user’s guide

ADVISORY COMMITTEE

MEMBERSJoseph Chin, CMS Sally Okun, PatientsLikeMeGregory Daniel, Duke University Tom Oliver, ASCONancy Dryer, Quintiles Eleanor Perfetto, UMD School of PharmacyScott Flanders, Astellas Edmund Pezalla, AetnaJohn Fox, Priority Health Catherine Piech, Janssen PharmaceuticalsJohnathan Jarow, FDA Megan Maguire Priolo, GBMCSachin Kamal-Bahl, Pfizer Alan Rosenberg, AnthemJulie Locklear, EMD Serono Lucy Savitz, Intermountain HealthcareJoan McClure, NCCN Marcus Wilson, HealthcoreElizabeth McGlynn, Kaiser Permanente Brande Ellis Yaist, Eli Lilly and CompanyPeter Neumann, Tufts Medical Center

METHODS WORKGROUP

MEMBERSKristen Bibeau, Teva Pharmaceuticals George Browman, University of British

ColumbiaScott Flanders, Astellas Jennifer Graff, NPC

Craig Henderson, UCSF David Henry, University of Toronto

Brad Hirsch, Flatiron Health Sachin Kamal-Bahl, Pfizer

Mark Levenson, CDER Office of Biostatistics Gary Lyman, ASCO

Jim Murray, Eli Lilly and Company Sally Morton, Virginia Tech

Josée Poirier, MeYou Health, LLC Beverly Shea, University of Ottowa

Mike Stoto (Chair), Georgetown University Timothy Vaughan, PatientsLikeMe

Mingliang Zang, Janssen

DISSEMINATION WORKGROUP

MEMBERSAylin Altan, Optum Labs Rabia Kahveci, HTA Consultant

Megan Klopchin, Eli Lilly and Company Karen Lencoski, Astellas

Julie Locklear, EMD Serono Joan McClure, NCCN

Troy Sarich, Janssen Marcus Wilson, HealthCore

Julie Simmons (Co-Chair), CMTP John Beilenson (Co-Chair), Strategic Communications & Planning

SPONSORS

SUMMARY OF CONTENT

MODULES

MODULE 1: Articulating Question

• What is the decision to be made?

• What information is required?

MODULE 2A: Assessing Relevance

• Population• Intervention• Comparators• Outcome(s)• Timing• Setting

MODULE 2B: Assessing

Rigor• Quality of

Research Question

• Potential for Bias

• Precision• Data Integrity

MODULE 2C:Effect Size

• What is the magnitude and direction of the effect?

BASIC QUESTIONS RATED WITH SCORING SYSTEM THAT CAN BE PLOTTED AND VISUALIZED

MODULE 2A (EXAMPLE)

DIRECTIONS1. Check each box as you consider the

domains within the Relevance dimension

2. Rate Relevance for each study along a continuum (1 – minimally relevant to 4 – maximally relevant)

3. Enter “0” only if there is a necessary piece of information the study fails to provide

DOMAINS If "0"PopulationInterventionComparatorPrimary OutcomeTimingSettingPopulationInterventionComparator ComparatorPrimary OutcomeTiming

ComparatorPrimary Outcome

ComparatorPrimary Outcome

STUDY 1

STUDY 2

Smith, 2005

Johnson, 2003

MODULE 3 (EXAMPLE OF OUTPUT)

• Each sphere is the data point for one study and represents the Likert-type assessments for Relevance (x-axis) and Rigor (y-axis) (Modules 2A and 2B).

• The size of the sphere indicates the magnitude of an effect (Module 2C).

• The color of the sphere indicates the direction of an effect (green=positive; white=negative or no difference) (Module 2C).

• Note: “0” scores for Relevance and Rigor are not plotted

NEXT STEPS

Testing and Demonstration• Current iteration: a free and downloadable Excel Tool (Version 1.0)

– Will continue to vet and test Tool for usefulness, relevance, reliability, and interpretation

• Focus groups and early adopter survey to inform the next iteration of the Excel Tool (Version 2.0), as well as the development of a desktop software solution

Additional Tools• Growing a library of use cases, video demonstrations for users,

webinars, and additional training tools based on feedback from early adopters

ACKNOWLEDGEMENTS

INITIATIVE TEAM

CENTER FOR MEDICAL TECHNOLOGY POLICYRachael Moloney, Research Manager Initiative Lead

Donna Messner, Sr. Vice President GPC Program Director

Sean Tunis, President & CEO Advisor

Jennifer Al Naber, GPC Program Manager Initiative Manager

Julie Simmons, Manager, Marketing & Communications

Co-chair, Dissemination Workgroup

EXTERNAL COLLABORATORSMichael Stoto, Georgetown University Chair, Methods Workgroup

John Beilenson, Strategic Communications & Planning (SCP)

Co-chair, Dissemination Workgroup