Program and Program Outcomes

Program Monitoring and Evaluation

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

What does this organization do

that you respect?

Prove that you are making a difference!

1. Increased Funding

2. Participant Impact

3. Improved Community

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

ProgramA planned, coordinated

group of activities and/or services, often for a specific

purpose or to achieve a goal.

The systematic and continual documentation

of key aspects of a program in order to assess whether

the program is being implemented as intended.

Program Monitoring Quality

Improvement

Process Monitoring

Fidelity of Implementation

Performance Management

Process Evaluation

The systematic method for collecting, analyzing, and using information to answer questions about a program’s effectiveness.

Program Evaluation Program

OutcomesImpact

EvaluationOutcome

Evaluation

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

What are our goals?

Strategic Plan

Programmatic Goal

My Goal: Healthy Heart

What are the needs?

How do we identify them?

Needs Assessment

What do our participants need to reach the goal?

What are the barriers to reaching the goal?

SmokerDiet high in sat. fatLack of exercise

Diabetes

ObeseWaist > 35”

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

What do we do?

What results do we expect?

Theory of Change/Logic Model

IF THEN

It’s Participatory!!

Based on Needs and Goals

Inputs Activities Outputs Outcomes Impact

Inputs Activities Outputs Outcomes Impact

What do we need to implement the program?

FundingSpaceCurriculaTrainingParticipantsStaff

Inputs Activities Outputs Outcomes Impact

What activities need to be offered to help us achieve our intended results?

TutoringCurriculaFind HomesDistribute FoodProvide CounselingEducation

Inputs Activities Outputs Outcomes Impact

How will we know that our activities happened?

# of Hours# of Participants# Distributed# Housed# of Counseling # Completed# Contacted

Inputs Activities Outputs Outcomes Impact

What results do we expect?Change in KnowledgeChange in SkillsChange in AttitudeChange in Behaviors

Inputs Activities Outputs Outcomes Impact

What do we ultimately want to change?

Conditions in Life, such as:GraduationCollege EntryStable HousingEarningsObesity

Inputs Activities Outputs Outcomes Impact

LOGICTHENIF

Inputs Activities Outputs Outcomes Impact

Eat less fat

Vigorous Physical activity

Lose Weight

Lower LDL

Healthy Heart

# meals low sat. fat

# mins vig. activity/wk

Healthy Recipes

Sneakers

Gym Membership

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

How do we know if we’re on track?

How do we know if we achieved our results?

Inputs Activities Outputs Outcomes Impact

Program Monitoring

Inputs Activities Outputs Outcomes Impact

Eat less fat

Lose Weight

Lower LDL

Healthy Heart

# meals low sat. fat

# mins vig. activity/wk

Healthy Recipes

Sneakers

Gym Membership

Vigorous Physical Activity

Inputs Activities Outputs Outcomes Impact

Program Evaluation

Inputs Activities Outputs Outcomes Impact

Eat less fat

Vigorous Physical Activity

Lose Weight

Lower LDL

Healthy Heart

# meals low sat. fat

# mins vig. activity/wk

Healthy Recipes

Gym Membership

Sneakers

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

What do we measure?

What should we measure?

Inputs Activities Outputs Outcomes Impact

Complex

Simple

Inputs Activities Outputs Outcomes Impact

# of AttendeesHours of CounselingHours by Type of Counseling

Simple: Attendance

Complex: Service Records

Inputs Activities Outputs Outcomes Impact

# meals low sat. fat

Simple: Tally

Complex: Meal Log

Inputs Activities Outputs Outcomes Impact

Change in KnowledgeIncrease Positive Behaviors

Simple: Surveys

Complex: Observations

Inputs Activities Outputs Outcomes Impact

Lose WeightLower LDL

Complex: Blood Test

Simple: Weight

Inputs Activities Outputs Outcomes Impact

Increased GraduationReduced PovertyLess Teen PregnancyIncreased EarningsReduction in Obesity

Simple: Census Data

Complex: Participant Data

Inputs Activities Outputs Outcomes Impact

Healthy Heart

Simple: Checklist

Complex: Stress Test

Inputs Activities Outputs Outcomes Impact

Eat less fat

Walking

Biking

Lose Weight

Lower LDL

Healthy Heart

# meals w/out bacon

# days walked/biked

# hrs walked/ biked

Healthy Recipes

Sneakers

Bike

When should we collect and measure?

Inputs Activities Outputs Outcomes Impact

Eat less fat

Vigorous Physical Activity

Lose Weight

Lower LDL

Healthy Heart

# meals w/ low sat. fat

# mins vig.

activity/wk

Healthy Recipes

Sneakers

Gym Membership

= -1 lb

15 wks > 90 min. (29%)

= 300 (27%)

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

Who is responsible?

Leadership

Program Staff

Trained Evaluator

Data Collection Systems

Putting it Together

Inputs Activities Outputs Outcomes Impact

Eat less fat

Vigorous Physical Activity

Lose Weight

Lower LDL

Healthy Heart

Healthy Recipes

Sneakers

Gym Membership

# mins vig.

activity/wk

42 wks > 90 min. (81%)

# meals w/ low sat. fat

= -9 lbs= 820 (75%)

Inputs Activities Outputs Outcomes Impact

Daily Meds

Lose Weight

Lower LDL

Healthy Heart

# days with meds

Meds

Modify Program

I. The Why: Importance

a. Goals and Needs? III. The How: Questions to Ask

IV. Q&A and Discussion

II. The What: Definitions

b. Program plan? c. How are we doing? d. Data and measures? e. Capacity?

Questions? Thoughts?

Thank you!

Caroline Taggart, MPHSenior Evaluatorcaroline@viaeval.comwww.viaevaluation.com

https://prezi.com/qerbnvzqchw3/mah-theory-of-change/

Leveraging Evaluation

Hewlett Foundation Performing Arts Program Strategic Framework; Moss, 2008.

Output or OutcomeBaseline = number from prior to program (if available) or number from first period of programTarget = ideal, based on research and reasonable projections (what is realistic to achieve)Indicator = data source; what you are looking at

# meals low sat. fatBaseline = unknown; set in Year 1Target > 15 meals per week with low saturated fat Indicator = meals < estimated low saturated fat level

# minutes of vigorous physical activity Baseline = 0 (I never got my heartrate up!)Target > 90 minutes per week (3 30-minute sessions per week)Indicator = minutes of activity with perceived exertion > 15

Lower LDLBaseline = 185 mg/dL Target < 150 mg/dL (still high, but a realistic reduction for my one-year program)Indicator = lipid profile results of LDL

Baselines, Targets (Benchmarks), Indicators

Outcome: the change or result that is desired or expectedIndicator: the information (data) that will be used to measure whether the outcome was achieved

Easy example:Outcome = lose weight Indicator = weight in pounds

More complicated example:Outcome = reduce problem behaviors Indicator = number of suspensions*Often programs list “Reduce Suspensions” as the desired outcome. The program actually wants to reduce problem behaviors, which will be measured by a reduction (hopefully) in suspensions.

Same idea:Outcome = improve academic successIndicator = graduation rate

Outcomes vs. Indicators