Conducting Successful Program Evaluations:

Researchers Reveal Their Insights

Sue Palsbo, PhD

Thilo Kroll, PhD

NRH Center for Health & Disability Research

Why Do an Evaluation?

• Does the program work?

• Why?

• How?

• Who benefits?

• Who loses?

• Who pays?

• How does it compare to other programs?

Evaluation Process

Analyze data

Determine the Outcome

Collect data

State the

Performance Measure

Who Cares?

• Interested in different outcomes

• Different outcomes require different methods

• Different methods require different data sources

Funding Considerations

• Public or private?

• Contracts or grants?

• Single source or multiple?

• Lead time

• Scope and /or size

• Funding organization’s priorities

Diverse Stakeholders

Diverse interests for outcomes

Requires diverse set of measures

quantitative qualitative

Blended together for total picture

Evaluation Types

• Formative– Immediate feedback allows program

modifications and mid-course corrections

• Summative– Post-program perspective – Comprehensive overview of outcomes– Total impact of the intervention or program

Design Considerations

• Unit of analysis– Person, Provider, State

• Funding for the study

• Single vs. multi-method approach

• Short-term vs long-term outcomes

• Legacy impacts e.g. sustainability and duplication

How to Select the Evaluators

• Your evaluation identifies the questions and methods; these indicate the skill set you need

• In-house or external?

• Sole source contracting

• Competitive bidding

Example

Stakeholders

Evaluation Funding

Evaluation Design

• Consortium model

• Prospectively designed

• Contemporaneous (formative)

• Highly leveraged

• 3-year cumulative

MnDHO Goals

1. Create and maintain satisfaction

2. Promote overall well-being of enrollees

3. Meet cost and utilization goals.

Measurement Domain

1. Satisfaction

2. Quality of care

3. Utilization and patterns of care

4. Costs and rate setting

Measurement Selection Criteria

1. Use existing data

2. Avoid duplication of information.

3. Avoid recall bias.

4. Minimize respondent burden.

Preliminary Findings From Mixed-Methods Evaluation

• Longitudinal CAHPS®-based survey

– At MnDHO enrollment

– 1 year post-enrollment

• Focus groups

– MnDHO enrollees

– Eligible people who chose not to enroll

Survey Content

• General satisfaction• Care coordination experience• Access to a wide variety of health and long-

term-care services• Self-directed care• Quality of interactions with health care providers• Quality of interactions with MA or AXIS staff

Longitudinal Survey

• 100 MnDHO participants

– 35 enrollees have completed baseline and follow-up surveys.

• Baseline

– Health care experiences in the year before enrollment.

• Follow-up

– Health care experiences in MnDHO.

4 Focus Groups

• October, 2002

• 2 MnDHO enrollees

• 2 fee-for-service Medical Assistance program

Care Coordination. In the year before / after you enrolled in AXIS/UCare Complete, did anyone help manage the health care services you received from different doctors, nurses, therapists, PCAs,or equipment providers?

12

82

6

0102030405060708090

100

% o

f R

espo

nden

ts

Both Only in MnDHO Not in MnDHO

Focus Group Statements

• The MnDHO / AXIS Experience:– “She’ll (health coordinator) line me up with my

appointments and stuff. She’ll find my therapists and my physical equipment…, all that stuff.”

– “My stress level has been relieved somewhat. I’m able to focus on more vocational and future issues, as opposed to the day-to-day healthcare issues……”

Focus Group Statements

• The fee-for-service experience:• “I spend most of my time on the phone calling people,

setting up appointments, or trying to get services myself, and that is very tiring. I get exhausted because I do have MS.”

• “Our experience was so bad we had to hire our own coordinator. It seems like if the person is stable……, people don’t return their calls, people don’t seem to care……”

Self-direction in health care. In the year before / after you enrolled in AXIS/UCare Complete, were you involved as much as you wanted in making decisions about your health care?

5044

6

0102030405060708090

100

% o

f R

espo

nden

ts

Both Only in MnDHO Only out of MnDHO

Focus Group Statement

“My healthcare coordinator, she always involves me. It is our lives. They can give us all the information, but ultimately it comes down to us, what we want to do…... it comes to us making the final decision…... The healthcare coordinators are always there to give you the pros and cons.”

Recommendations• Its worth the time and money to do a program

evaluation

• Do a formative evaluation so you can make changes in real time

• Important lessons from a summative evaluation can tell you what worked and what didn’t, to improve the next effort

• Mixed methods provide more information together than a single method