Meet the Evaluation Team: RAND Corporation

CalMHSA Statewide Program Partners

Statewide Coordination Workgroup Meeting

A9604-AB-2 12/2011

Overview

• Background on team and state-wide evaluation

• Approach to evaluation

• Current state of research in three initiative areas

• Evaluation team and program partners: Relationship building

A9604-AB-3 12/2011

PEI Statewide Evaluation Purpose

To evaluate to what extent and how the strategies of PEI Statewide Projects are effective in:

• Preventing Suicides

•  Improving Student Mental Health

• Reducing Mental Health Stigma and Discrimination

A9604-AB-4 12/2011

Objectives of the Statewide Evaluation

Key objectives of PEI Statewide Evaluation:

• Establish baselines and community indicators

• Conduct thorough program evaluations

•  Identify innovative programs for replication

• Promote continuous quality improvement efforts

A9604-AB-5 12/2011

PEI Statewide Evaluation Process

• CalMHSA developed a Request for Qualifications

• Statements of Qualifications reviewed by a team of experts that selected the RAND team

• CalMHSA and RAND team negotiated a Statement of Work

• RAND team develops an Evaluation Strategic Plan for discussion with CalMHSA and the SEE

• Statewide evaluation implementation and reporting

A9604-AB-6 12/2011

Components of Statewide Evaluation

• Goal-based, process-based, and outcomes-based evaluations

• Conducted at three levels: –  Individual programs –  Each of three initiatives –  Overall CalMHSA effort statewide

• Coordination and leveraging across PEI initiatives and programs

A9604-AB-7 12/2011

Who Is on the RAND Team?

• RAND Corporation – independent nonprofit public policy research organization based in Santa Monica, CA

• SRI International Policy Division – independent, nonprofit research institute based in Menlo Park, CA

• Field Research Corporation – marketing and public opinion research firm based in San Francisco, CA

• UCLA/RAND/USC NIMH – Partnered Research Center for Quality Care

A9604-AB-8 12/2011

Key Players at RAND

• Audrey Burnam – Project Leader (Admin and Design)

• Sandy Berry – Co-Project Leader (Data Acquisition)

• Rajeev Ramchand – Suicide Prevention

• Becky Collins – Stigma and Discrimination

• Bradley Stein – School Based Mental Health

• Patricia Ebener – Technical Assistance

• Mary Vaiana – Communications and Dissemination

A9604-AB-9 12/2011

What Is RAND?

• Chartered in 1948 as a nonprofit organization To further and promote scientific, educational, and charitable purposes, all for . . . public welfare and security . . .

• Mission To help improve policy and decision making through research and analysis

• Core values Quality and objectivity

A9604-AB-10 12/2011

Civil Justice

Education Substance Abuse

U.S. National Security

Energy and

Environment

Health and Health Care

Transportation and

Infrastructure

International Affairs

Science and

Technology

Population and

Aging

Terrorism and

Homeland Security

Children and

Adolescents

Mental Health

•  Providing practical solutions to complex problems

•  Supporting evaluation and implementation

•  Enhancing policy debates

•  Training future policy analysts

•  Improving knowledge and analytic methods

RAND’s Principal Research Areas

A9604-AB-11 12/2011

Our Research Is Typically Characterized by . . .

An analytic approach that is: •  Integrative, collaborative,

and multidisciplinary •  Empirical, with technical

depth and methodological rigor

•  Innovative, but informed by past findings

•  Buttressed by demanding standards of quality and objectivity

RAND strives to build long-term relationships with its clients

Issues that involve: •  Competing objectives

and perspectives •  Intersection of public/

private interests •  "Messy" data, major

uncertainties •  Implications for the

future

A9604-AB-12 12/2011

Toward a Culture of Consequences Performance-Based Accountability Systems for Public Services Brian M. Stecher, Frank Camm, Cheryl L. Damberg, Laura S. Hamilton, Kathleen J. Mullen, Christopher Nelson, Paul Sorensen, Martin Wachs, Allison Yoh, Gail L. Zellman

Recent RAND Research Addresses Many Critical Societal Issues

Invisible Wounds of War Psychological and Cognitive Injuries,

Their Consequences, and Services to Assist Recovery TERRI TANIELIAN AND LISA H. JAYCOX, EDITORS

Oil Shale Development In the United States

James T. Bartis, Tom LaTourrette, Lloyd Dixon, D.J. Peterson, Gary Cecchine

Prospects and Policy Issues

Early Childhood

Interventions

Lynn A. Karoly M. Rebecca Kilburn

Jill S. Cannon

Proven Results, Future Promise

A9604-AB-13 12/2011

Overview

• Background on team and state-wide evaluation

• Approach to evaluation

• Current state of research in three initiative areas

• Evaluation team and program partners: Relationship building

A9604-AB-14 12/2011

Evaluation Approach: Structure-Process-Outcomes

STRUCTURE

What PEI capacities &

resources are PPs developing and implementing?

•  Networks •  Needs assessment •  Service expansion •  Outreach •  Training & technical

assistance •  Screening •  Educational resources •  Marketing campaigns •  Cross-system

collaboration •  Policies & protocols

PROCESS

What intervention activities are delivered,

and to whom?

•  Participation in training & education

•  Exposure to outreach •  Exposure to media •  Use of services •  Integration of

services across organizations

SHORT TERM OUTCOMES

What are immediate targets of change?

•  Knowledge •  Attitudes •  Normative behavior •  Mental & emotional

well-being •  Help-seeking

KEY OUTCOMES

What negative outcomes are

reduced?

•  Suicide •  Discrimination •  Social Isolation •  School drop-out

A9604-AB-15 12/2011

Purposes of Evaluation

• Formative Evaluation –  Provide early feedback on measures of process and

proximal outcomes –  Feedback can inform program partners

implementation efforts, and suggest changes in structure of PEI initiatives

• Summative Evaluation –  Assess short-term and key outcomes to evaluate

overall impact –  Examine relationships between process and short-

term outcomes to understand what had impact –  Examine relationships between short-term and key

outcomes to test social change assumptions

A9604-AB-16 12/2011

Overview

• Background on team and state-wide evaluation

• Approach to evaluation

• Current state of research in three initiative areas –  Suicide prevention –  Stigma reduction and prevention –  Student mental health

• Evaluation team and program partners: Relationship building

A9604-AB-17 12/2011

What We Know About Suicide Prevention

• CalMHSA evaluation presents a significant opportunity to move the field forward because few suicide prevention initiatives have been evaluated

However

• Suicide is rare, and it is difficult to show universal approaches reduce deaths by suicide

• Evaluations often focus on short-term outcomes that are logically or empirically linked to suicide

A9604-AB-18 12/2011

Types of Programs Evidence base Informational Resources •  No evidence of effect

•  Generally considered a ‘good idea’, but is it? Gatekeeper Trainings •  Changes in knowledge and attitudes among certain trainees

Hotlines •  Lower levels of suicidality and hopelessness among suicidal callers after calls

Universal Screening •  Little evidence of effect Training Providers •  Training primary care providers on depression awareness may

reduce suicides Treatment Interventions •  Certain treatments lead to reductions in suicide or attempts

for: suicide attempters, persons with bi-polar disorder, and patients with major depression

Social/Policy Interventions •  Evidence that means restrictions lead to reductions in suicide

Comprehensive Approaches

•  Multi-component efforts implemented in organizations and institutions have reduced suicide rates

Evidence of Varying Effectiveness for Suicide Prevention Initiatives

A9604-AB-19 12/2011

The Rareness of Suicide and Data Lags Often Impede Timely Outcomes Evaluations

•  Two sources of national and state-level suicide data –  NCHS Vital Statistics System (3 year lag) –  NVDRS for 16 states (1.5 year lag)

•  CA is not a participating state in NVDRS

USA

CA

0

2

4

6

8

10

12

14

2000 2001 2002 2003 2004 2005 2006 2007 2008

Suicide rate per 100,000

Suicide in the US and California*

* Age-Adjusted, with 2000 as standard year

3775 suicides in 2008

A9604-AB-20 12/2011

Outcome Short Description Strengths Weaknesses Suicide Did an intervention lead to a reduction in

deaths by suicide? Key Outcome - Rare/Power

- Data Availability

Suicide Attempt

Did an intervention lead to a reduction in suicide attempts (objective reports or self-reported)?

Best predictor of suicide (5-15%)

Half of all suicides have no history of attempt

Suicide Ideation

Did an intervention lead to reductions in thinking about killing oneself?

Self-report suicidal thoughts

Relationship to suicide is unclear

Change in Functioning

Did an intervention lead to reductions in mental health functioning linked with suicide?

Self-reports of thoughts thought to relate to suicide

Relationship to suicide is unclear

Referral Did an intervention increase referrals or utilization of other, specific resources?

Often is the intended outcome

Relies on referral to prevent suicide

Knowledge and Attitudes

Did an intervention change the way people think about suicide, or how care professionals manage suicidal individuals?

May lead to increased referrals

Unclear whether changes in K&A are actionable

Other Behavior Change

Did an intervention change functioning in other domains that may indirectly impact suicide (e.g., promote responsible drinking)?

May eventually lead to reductions in suicides

Predictive link with suicide is often weak

Varying Outcomes Used to Evaluate Suicide Prevention Programs

A9604-AB-21 12/2011

Overview

• Background on team and state-wide evaluation

• Approach to evaluation

• Current state of research in three initiative areas –  Suicide prevention –  Stigma reduction and prevention –  Student mental health

• Evaluation team and program partners: Relationship building

A9604-AB-22 12/2011

What Do We Mean by Stigma?

• Defined as –  Prejudice –  Negative stereotypes –  Discrimination against an individual or group

• Two manifestations –  Internalized/private –  Social/public

A9604-AB-23 12/2011

Why Are Stigma and Discrimination Mental Health Issues?

• Reduce help-seeking, service utilization, and treatment compliance

• Can adversely affect self-perceptions/mental health

• Cause social exclusion/limit opportunity –  Limit family/peer support –  Reduce healthcare access –  Create employment/housing/school issues –  Result in negative interactions with criminal justice

A9604-AB-24 12/2011

Multi-Level Interventions Are Thought to Be Most Effective

Individual Change

Social Change

Policy/Practice Change

A9604-AB-25 12/2011

Individual Change

Social Change

Policy/Practice Change

Outcomes at Each Level of Change

Change in knowledge, attitudes, behavior

Shifts in community discussion,

norms

Change in organiza-

tional policies, practices

A9604-AB-26 12/2011

Three Kinds of Programs Are Used to Reduce Stigma and Discrimination

• Training presentations for police, health care providers, in schools, and in workplaces

–  Education only –  Video “contact” –  Direct/In-person contact

• Media Interventions –  Deliberate communication/media campaigns –  Advocacy and education to affect media coverage

• Policy and practice reviews

A9604-AB-27 12/2011

Evaluations of Existing Training Interventions

• A handful of small pre-post studies of police and workplace trainings have been conducted

• There is a more substantial literature on SDR trainings in college student and other school-based populations

• These evaluations suggest trainings involving contact, education, or both are effective, with contact a priority

• The changes demonstrated are limited to short term shifts in attitudes

A9604-AB-28 12/2011

Little Is Known About the Effectiveness of Media Campaigns

• Background noise: hard to distinguish effects of a campaign from other change-producing factors and unrelated secular trends

• Message spillover into the control group

• Media effects are gradual and cumulative (a “drip” effect) unless the information communicated is novel

• Matching study sample to target audience can be difficult

A9604-AB-29 12/2011

Considerations Specific to Evaluating Media Campaigns

• Test potential - are the messages effective ones?

• Document implementation – are multiple well-chosen channels used?

• Collect exposure metrics – is substantial portion of target audiences reached and with high frequency?

• Measure immediately targeted outcomes – is awareness, knowledge affected among those reached?

A9604-AB-30 12/2011

Overview

• Background on team and state-wide evaluation

• Approach to evaluation

• Current state of research in three initiative areas –  Suicide prevention –  Stigma reduction and prevention –  Student mental health

• Evaluation team and program partners: Relationship building

A9604-AB-31 12/2011

What Is Student Mental Health?

• Programs and interventions to address needs of students with emotional and behavioral problems or enhance students social and emotional learning

• Commonly focused on students, but may also involve parents and siblings

• May be staffed by school personnel, or by individuals from other organizations working in the schools; treatment services provided in schools or in local community

• Knowledge and evidence base greater for K-12 schools than for colleges and universities

A9604-AB-32 12/2011

Common Types of Student Mental Health Programs

•  Clinical services –  Efficacy depends on nature of clinical treatment provided--

school-based CBT programs have strongest evidence base

•  Targeted intervention/prevention programs –  CBITS efficacious for trauma symptoms –  FRIENDS/Coping Cat efficacious for anxiety

•  Primary prevention programs –  Project ALERT – efficacious for substance use –  SOS- quasi-randomized- effective for decreasing suicide

attempts

•  Holistic/school-wide approaches to student mental health –  PBIS – improved behaviors in pre-post and non randomized

comparison groups

A9604-AB-33 12/2011

What We Know About Student Mental Health Evaluation

• Relatively few student mental health programs have been rigorously evaluated

•  Intervention processes are most easily assessed; clinical and academic outcomes are more challenging

• Most rigorous mental health evaluations have focused on specific programs and subsets of outcomes; clinical programs more often evaluated than prevention or curricular programs

• School-based substance abuse programs have a history of more widespread evaluations when compared to mental health programs

A9604-AB-34 12/2011

Student Mental Health Programs Often Rely on Broader Community Resources

• Programs to enhance identification and referral of students for treatment may depend on factors beyond the program’s control

–  Are resources available in the community to provide treatment?

–  Can student or family pay for treatment if necessary?

–  Is student and/or family willing to accept referral?

•  Information needed to evaluate referral component of identification and referral program may be difficult to obtain

A9604-AB-35 12/2011

Issues Specific to College Mental Health

•  Field is generally not as developed as K-12 mental health

•  Common types of programs –  On campus clinical services –  General population programs to increase awareness and

decrease stigma –  Targeted programs for specific populations or issues

•  Suicide prevention •  Recognition of onset of schizophrenia •  Early identification and referral (recently often tied to

concerns about violence) •  Drug/alcohol related, Intimate partner violence, other

at risk groups

A9604-AB-36 12/2011

Student Performance Is Key Outcome, but Data Challenges Exist

• School data: grades, attendance, and test scores can be used to evaluate programs, but schools vary widely in the data available

–  Electronically captured (and if captured available) –  Time intervals (quarter, trimester, semester) –  Metrics (A, B, C, D, F vs. E, S, NS vs. 1, 2, 3) –  What information is kept and for how long –  Program may span grading periods –  Grades reflect cumulative performance over a

period of time

A9604-AB-37 12/2011

Implications for CalMHSA Evaluation

STRUCTURE

What PEI capacities & resources are PPs

developing and implementing?

•  Networks •  Needs assessment •  Service expansion •  Outreach •  Training & technical

assistance •  Screening •  Educational resources •  Marketing campaigns •  Cross-system

collaboration •  Policies & protocols

PROCESS

What intervention activities are delivered,

and to whom?

•  Participation in training & education

•  Exposure to outreach •  Exposure to media •  Use of services •  Integration of

services across organizations

SHORT TERM OUTCOMES

What are immediate targets of change?

•  Knowledge •  Attitudes •  Normative behavior •  Mental & emotional

well-being •  Help-seeking

KEY OUTCOMES

What negative outcomes are

reduced?

•  Suicide •  Discrimination •  Social Isolation •  School drop-out

A9604-AB-38 12/2011

Overview

• Background on team and state-wide evaluation

• Approach to evaluation

• Current state of research in three initiative areas

• Evaluation team and program partners: Relationship building

A9604-AB-39 12/2011

Next Steps in Statewide Evaluation

• Compile information from SOWs to learn about Program Partners

–  Contact information, schedules, resources –  Goals, elements of programs, target populations

• Conduct “get-to-know-you” visits or calls with each Program Partner

–  Clarify and update plans, understand logic models –  Review evaluation plans, including data collection –  Discuss technical assistance needs and options

Part 2

A9604-AB-40 12/2011

Development of the Strategic Plan

• Collaborate and coordinate with Program Partners

• Assess Program Partner interest and need for technical assistance

• Understand Program Partner plans for collecting baseline data and existing program evaluation plans

• Seek guidance from the SEE, CalMHSA, Program Partners, Counties and other stakeholders

A9604-AB-41 12/2011

Data for Evaluation

• May include use of data collected by individual programs, existing data, and data RAND collects

• RAND will: –  Work with Program Partners to develop

specifications for data to be provided –  Provide templates and secure methods for

providing data –  Provide reports on data completeness and quality

• Goal is to avoid duplication and excessive burden

A9604-AB-42 12/2011

Timeline for Statewide Evaluation

• Request for Qualifications issued August, 2011

• Proposals due and RAND selected October, 2011

• Negotiation of RAND SOW concluded November, 2011

•  “Get To Know You” meetings December, 2011 —January, 2012

• Draft SESP to be available for review April, 2011

• Final SESP to be adopted June, 2011

• Contract concludes June, 2014