community forum american institutes for research engaging the public on the use of evidence: the...

Community ForumAMERICAN INSTITUTES FOR RESEARCH

Engaging the Public on the Use of Evidence: The Role of Public Deliberation

Kristin L. Carman, PhD

Maureen Maurer, MPH

AMERICAN INSTITUTES FOR RESEARCH2


• Explain public deliberation and application in health care

• Identify best practices for successful public deliberation

• Describe AHRQ’s Community Forum deliberative methods experiment

Session objectives



• 3-year, ARRA-funded initiative of Agency for Healthcare Research and Quality

• Seeks to:1) Expand the evidence base about

public deliberation

2) Obtain public input on a topic of value to AHRQ: the use of evidence in health care decisionmaking

Reason for this work

AMERICAN INSTITUTES FOR RESEARCH Community ForumAMERICAN INSTITUTES FOR RESEARCH

OVERVIEW OF PUBLIC DELIBERATION



• Public consultation approach • Involves lay members of the public• Includes an educational component and

a discussion-based (or deliberative) component

• Suited for ethical or values-based social issues

• Concerned with the “greater good” or societal interests

What is public deliberation?



Core elements of public deliberation

• Individuals from a broad range of perspectives gather through live or virtual channels

Convene

• Objective overview of background, issues, and options presented

Learn • Participants exchange reasons, perspectives, and values

Deliberate

• A summary or transcript of deliberation is prepared for decisionmakers

Report

Outcome:Impact on individual

participants

Outcome:Impact on civic decisionmaking



How deliberation differs from other public consultation methods

• Educational• Participant-based dialogue• Reason-based• Societal perspective & mutual

responsibility• Challenging



Examples of deliberative methods

• Citizens’ Jury• Citizens’ Panel / Council• Deliberative Focus Group• Deliberative Poll ®• Issues Forum• Study Circles• Town Hall• Hybrid approaches



• Group size, participant sample• Length, duration• Mode (online, in-person)• Recruitment method• Use of educational materials and

experts• Facilitation• Consensus as goal

How deliberative methods vary



What are the outcomes of public deliberation?

• Informed public input to the sponsor– Summary of important themes in

participants’ views– Can be used to inform policy,

programmatic, or other decisions



What are the outcomes of public deliberation? (cont.)

• Impact on participants– Increased knowledge of the

deliberative topic– Change in attitudes on deliberative

topic– Increased willingness to participate in

civic activities– Adoption of societal concerns / shift

from personal preferences


APPLICATION IN HEALTH CARE



Real-world applications of deliberative methods

• Examples– Developing a fair cost-sharing

structure(Ginsburg et al., 2012)

– Priority-setting social and health interventions (Pesce et al., 2011)



• Issue: Cost-sharing (deductibles, co-payments) can have varying financial impact on patients depending on their health care needs. What is the fairest way to structure cost-sharing when there is diversity of needs?

• Researchers: The California Health Benefit Exchange and the Center for Healthcare Decisions.

• Goals: Learn how future Exchange users prioritize the health care needs that should have greatest consideration for affordable cost-sharing.

• Evaluation: Assess response to participating in the deliberative process and its importance as input to the design of new health insurance programs.

Developing a fair cost-sharing structure



• Issue: How would you determine which social or health services to provide to improve health?

• Researchers: National Institutes of Health, Howard University, and D.C. Department of Health

• Goals: Learn how participants prioritize social or health services to improve health and understand their reasoning

• Evaluation: Assess deliberative process and whether deliberation affected participants’ knowledge on the determinants of health

Prioritizing interventions


BEST PRACTICES FOR SUCCESSFUL PUBLIC DELIBERATION



What conditions enable good public deliberation?

• Clear deliberative goals• Participant trust in the process and

outcomes• Accurate, unbiased information

(education, experts)• Reason-giving• Diversity



Design considerations

• Setting goals• Identifying and recruiting

participants• Selecting deliberative process or

method• Educating session participants• Facilitating sessions• Synthesizing output



Areas for further study

• Understanding what works best– Understanding impact on

decisionmakers and policy– Which deliberative methods work best

• How to address inequalities within context of deliberation– Recruitment strategies and incentives– Measuring equal participation– Language considerations



Check In

• Questions so far...• Discussion

– What issues do you see as appropriate for deliberation?


AHRQ’S COMMUNITY FORUM EXPERIMENT



Deliberative methods experiment: RCT

Eligible Participants Randomized

Deliberative Methods

N=960

Pre/Post K&A Survey

DM vs. Control Compare DMs

Deliberative Experiences

Survey

Compare DMs to each other

Qualitative Data Compare DMs to each other

Control

N=336Pre/Post K&A

Survey DM vs. control

76 deliberative groups in 4 U.S. cities in 4 months



Deliberative methods experiment: RCT

Experimental condition Characteristics

Brief Citizens’ Deliberation (BCD) 2 hours in-person

Online Deliberative Polling® (ODP)

5 hours onlineMeets 1.25 hours per week for 4 weeksQ & A with experts via teleconference

Community Deliberation (CD) 5 hours in-person, Meets for 2 in-person sessions with a week in betweenQ& A online postings with experts

Citizens’ Panel (CP) 20 hours in-personMeets for 3 consecutive daysPresentations and Q & A with experts

Control group Receives educational materials to review at home



Overarching deliberative question• Deliberate on the use of evidence in

healthcare decision-making:– Should individual patients and/or their doctors be

able to make any health decisions no matter what the evidence of medical effectiveness shows, or should society ever specify some boundaries for these decisions?

• Will explore 3 “variations” on the theme – Decisionmaking to encourage better health care– Decisionmaking when there are cost implications– Decisionmaking when there are complex societal

tradeoffs



Sub-questions by variation

Variations Example Questions

BetterHealth Care:

Should doctors and patients be able to make decision about medical care, regardless of what the research says?

Cost: Should there be limits on patient choice when less expensive options are available that work as well?

Societal Trade-off: What is the duty of society to protect patients from possible harm?



Deliberative questions across methods

Deliberative Method Case Study Variation

Brief Citizens’ Deliberation (BCD)

• Hospital quality • Better health care• Societal tradeoff

Online Deliberative Polling® (ODP)

• Hospital quality • Better health care• Cost implication• Societal tradeoff

Community Deliberation (CD)

• Hospital quality • URI

• Better health care• Societal tradeoff

Citizens’ Panel (CP) • Hospital quality• URI• Obesity management• Heart disease

• Better health care• Cost implication• Societal tradeoff



Overview of Community Forum evaluation

Research questions

Primary outcomes Measurement

Is public deliberation more or less effective than education only?

Changes in knowledge and attitudes

Quantitative (survey)

Which method(s) are most effective? In what ways? And why?

Changes in knowledge and attitudes

Quality of experience (process elements) and content

Quantitative (survey)

Combination



Overall process for evaluation

Determine measurement

approach

Develop surveys

Collect survey data & analyze

Develop codebook for

transcripts

Code & analyze

transcripts

Combine data for further analysis

Develop debrief forms

Analyze observation

data



Quantitative analysis: Mapping concepts to surveys

Survey Group Measure

Pre/Post Knowledge & Attitude Survey

Deliberative methods and control

Participants’ knowledge, attitudes, and beliefs about the deliberative issue

Post Deliberative Experience Survey

Deliberative methods only

Discourse quality (e.g., equal participation, tolerance of different perspectives, reasoned justification of ideas)

Implementation quality (e.g., quality of facilitation, materials, and presentations)



Qualitative analysis: Aims

1. Summarize values and ethical principles cited by participants

2. Assess whether values and ethical principles differ by method



Qualitative analysis: Approach

Focus of analyses Data source Approach

Thematic analysis: 1) Ethics and values2) Acceptable

boundaries3) Factors

Transcripts Coding – descriptive and interpretive coding

Summarized for all groups and then used in comparisons between specific methods

Quality of process: Aspects affecting nature of deliberation

Transcripts & debrief forms

Coding – descriptive and interpretive coding; counts of occurrences

Used in comparisons between specific methods



How the public views the application of evidence • What and how does the public

think about applying medical evidence?

• What matters most to the public?• How can public input be used to

inform CER?



Integrating quantitative and qualitative analyses Quantitative Qualitative

Relatively high or low scores on Deliberative Experience survey for certain methods – or groups

Deliberative quality/ process elements (e.g., equal opportunity to participate)

Differences in knowledge and attitudes scores on K & A survey for certain methods – or groups

Deliberative content (e.g., shifts, emphasis on educational materials or expert contributions)

Differences in societal vs. individual perspective based on attitude questions

Deliberative quality and content (e.g., emphasis on societal perspective, articulation of overall question)



• Next steps– Implement experiment: now – fall 2012– Analysis: summer 2012 – summer 2013 – Report of findings: fall 2013

• Information to look for on the EHC Program website– Webinars

• Using Deliberative Methods to Engage the Public: How to Design and Implement an Effective Deliberative Session (April 2012)

• Using Deliberative Methods to Engage Patients, Consumers, and the Public (December 2011)

– Literature Review (coming soon)http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/how-to-get-involved-in-the-effective-health-care-program/

Next steps and resources

http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/how-to-get-involved-in-the-effective-health-care-program/

http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/how-to-get-involved-in-the-effective-health-care-program/



• American Institutes for Research– Kristin L. Carman, Project Director

• [email protected]– Maureen Maurer

• [email protected]

• AHRQ– Joanna Siegel, Project Officer

• [email protected], 301-427-1969

AHRQ’s Effective Health Care Program site:

http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/

Key contacts

mailto:[email protected]



http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/


QUESTIONS?

community forum american institutes for research engaging the public on the use of evidence: the...

Documents

researchamerican institutes

informed public input

deliberative component

deliberative topicchange

use of evidence

civic decisionmaking

health interventions

evidence base