The American Nonprofit Sector and Philanthropy IThe American Nonprofit Sector and Philanthropy I

Fall 2010Fall 2010 Program EvaluationProgram Evaluation

Kelly Wishart, MA, CECKelly Wishart, MA, CECAdministrator, Professional Development and QualityAdministrator, Professional Development and Quality

Children’s FriendChildren’s Friend

Providence, RIProvidence, RI

Overview of SessionsOverview of Sessions

11-23-10 Session I:11-23-10 Session I: Evaluation BasicsEvaluation Basics Current thinking about evaluationCurrent thinking about evaluation Uses of evaluationUses of evaluation When to evaluate and when not toWhen to evaluate and when not to Types of evaluationTypes of evaluation Vocabulary of evaluationVocabulary of evaluation Logic Models and Theory of Change ModelLogic Models and Theory of Change Model Creating Program Level OutcomesCreating Program Level Outcomes

11-30-10 Session II:11-30-10 Session II: Evaluation Planning and DesignEvaluation Planning and Design Creating Program Level OutcomesCreating Program Level Outcomes Data, Data, Data: So What, Now WhatData, Data, Data: So What, Now What Data Collection ToolsData Collection Tools Avoiding the DRIP: Data Rich Information Poor SyndromeAvoiding the DRIP: Data Rich Information Poor Syndrome

12-7-10 Session III:12-7-10 Session III: Student Presentations and FeedbackStudent Presentations and Feedback Identify and articulate at least three outcomes for a program with Identify and articulate at least three outcomes for a program with

which you have familiarity.which you have familiarity. Design at least one data collection instrument and describe how the Design at least one data collection instrument and describe how the

tool will be administered, how the data will be analyzed, used in any tool will be administered, how the data will be analyzed, used in any public reporting, etc.public reporting, etc.

Each student will present for not more than ten minutesEach student will present for not more than ten minutes

How do you know you know?How do you know you know?

““The quicker picker upper.”The quicker picker upper.”

BountyBounty

““We make a lot of the things We make a lot of the things you buy, better.”you buy, better.”

BASFBASF

Evaluation…the old days:Evaluation…the old days:

To ensure fiscal accountability (audits are To ensure fiscal accountability (audits are type of evaluation tool)type of evaluation tool)

Imposed by outsiders (funders say they Imposed by outsiders (funders say they want to know how their money was spent, want to know how their money was spent, did you do what you said you were going did you do what you said you were going to?)to?)

Conducted at the end of the program (did Conducted at the end of the program (did you do what you intended or set out to do?)you do what you intended or set out to do?)

Findings are usually used to answer simple Findings are usually used to answer simple questions and not for program decision-questions and not for program decision-making or program improvement.making or program improvement.

Evaluation today:Evaluation today:

Valued by those doing the work (direct Valued by those doing the work (direct line workers know if they are making an line workers know if they are making an impact).impact).

Looks at the process of the doing, as Looks at the process of the doing, as well as the end result. well as the end result.

Is part of the planning and Is part of the planning and implementation process and not an after implementation process and not an after thought; questions are asked before thought; questions are asked before implementation. implementation.

Organizations use the information to Organizations use the information to improve practice.improve practice.

What can evaluation tell you:What can evaluation tell you: Is the program worth doing?Is the program worth doing? How can you improve your program?How can you improve your program? How well does the program deliver How well does the program deliver

services?services? What effect does the program have?What effect does the program have? What do we need to know for decision-What do we need to know for decision-

making?making? How do program costs compare with How do program costs compare with

benefits?benefits? How does the program comply with How does the program comply with

particular standards?particular standards? What’s important to people about this What’s important to people about this

program?program?

In other words, types of In other words, types of evaluation:evaluation:

Process Outcome Formative Summative Cost-effectiveness Cost-benefit evaluation

Other common terms:Other common terms:

QualitativeQualitative QuantitativeQuantitative

Refer to the Refer to the types of informationtypes of information or data collected in an or data collected in an evaluation, not the type of evaluation, not the type of evaluationevaluation

Preparing for EvaluationPreparing for Evaluation

What do you want to know?What do you want to know?

Who will use, view the results?Who will use, view the results?

How will the information be How will the information be

used?used?

Who needs to be involved in Who needs to be involved in your evaluation?your evaluation?

Program ProvidersProgram Providers Current fundersCurrent funders Potential donorsPotential donors Program participantsProgram participants ColleaguesColleagues ResearchersResearchers PartnersPartners CompetitorsCompetitors Others…?Others…?

Program EvaluationProgram Evaluation

Program evaluation is a

systematic method for

collecting, analyzing, and using

information to answer basic

questions about a program. ACF Program Evaluation GuideACF Program Evaluation Guide

The basic steps:The basic steps:

Assemble an evaluation teamAssemble an evaluation team

Prepare for the evaluationPrepare for the evaluation

Develop an evaluation planDevelop an evaluation plan

Collect evaluation informationCollect evaluation information

Analyze your evaluation informationAnalyze your evaluation information

Prepare the evaluation reportPrepare the evaluation report

Program Objectives, 1:Program Objectives, 1:

Implementation objectives refer to what you plan

to do in your program, how you plan to do it, and

who you want to reach.

The services or training you plan to implement

The characteristics of the participant population

The number of people you plan to reach

The staffing arrangements and staff training

The strategies for recruiting participants.

Often referred to as implementation evaluationOften referred to as implementation evaluation

Program Objectives, 2:Program Objectives, 2:

Participant outcome objectives describe what

you expect to happen to your participants as a

result of your program.

Can refer to the agencies, communities, and

organizations as well as individuals.

Measure your expectations about how your

program will change participants' knowledge,

attitudes, behaviors, or awareness are your

participant outcome objectives.

Often referred to as an outcome evaluation. Often referred to as an outcome evaluation.

What kind of objectives are What kind of objectives are these:these:

Family safety will improve. Examples: prevention of

physical, sexual or emotional abuse of children;

child neglect; domestic violence.

Improve referral and service connections among

DCYF and community agencies.

Children will show greater engagement in learning

and early literacy activities.

True partnerships between target schools and local

communities are realized in the design,

implementation, and governance of the project.

Who evaluates?Who evaluates?

An outside evaluator (which may be an

individual, research institute, or consulting firm)

who serves as the team leader and is

supported by in-house staff.

An in-house evaluator who serves as the team

leader and is supported by program staff and

an outside consultant.

An in-house evaluator who serves as the team

leader and is supported by program staff.

How much does it cost?How much does it cost?

Most experts say expect the Most experts say expect the cost of evaluation to be about cost of evaluation to be about 15-20% of your total program 15-20% of your total program budget.budget.

Some funders will require a Some funders will require a minimum amount to be included minimum amount to be included in your budget.in your budget.

Who evaluates?Who evaluates?

Depends on your program’s Depends on your program’s

funding requirementsfunding requirements

Your resources and capabilitiesYour resources and capabilities

Decision time:Decision time:

Does your agency or program have funds designated for evaluation purposes?

Have you successfully conducted previous evaluations of similar programs, components, or services?

Are existing program practices and information collection forms useful for evaluation purposes?

Can you collect evaluation information as part of your regular program operations (at intake, termination)?

Are there agency staff who have training and experience in evaluation-related tasks?

Are there advisory board members who have training and experience in evaluation-related tasks?

Deciding what to evaluate.Deciding what to evaluate.

Evaluation Logic ModelEvaluation Logic Model

The logic model serves as a basis for identifying your program's implementation and participant outcome objectives.

Focus your evaluation on assessing whether implementation objectives and immediate participant outcome objectives were attained.

Deciding what to evaluate:Deciding what to evaluate:

Source: Measuring Program Outcomes: A Practical Approach, United Way of America (1996)

Key to good program evaluation is Key to good program evaluation is clearly articulated outcomes!clearly articulated outcomes!

Outcomes are benefits or changes for Outcomes are benefits or changes for individuals or populations during or after individuals or populations during or after participating in program activities. participating in program activities.

They are influenced by a program’s outputs.They are influenced by a program’s outputs. They may relate to behavior, skills, They may relate to behavior, skills,

knowledge, attitudes, values, condition, knowledge, attitudes, values, condition, status, etc. status, etc.

They are what participants know, think, or They are what participants know, think, or can do; or how they behave; or what their can do; or how they behave; or what their condition is, that is different following the condition is, that is different following the program.program.

Examples:Examples:

Financial management Financial management counseling programcounseling program OutputsOutputs = number of classes and = number of classes and

families seenfamilies seen OutcomesOutcomes = families living within = families living within

a budget, making additions to a budget, making additions to savings accounts, increased savings accounts, increased financial stabilityfinancial stability

Examples:Examples:

Neighborhood clean up Neighborhood clean up campaigncampaign Outputs = number of organizing Outputs = number of organizing

meetings held, number of meetings held, number of weekends dedicated to the workweekends dedicated to the work

Outcomes = reduced exposure to Outcomes = reduced exposure to safety hazards, increased feeling safety hazards, increased feeling of pride in neighborhoodof pride in neighborhood

What is not an outcome:What is not an outcome:

If it does not represent benefits to or If it does not represent benefits to or changes in participants.changes in participants.

If it tells nothing about whether or not If it tells nothing about whether or not participants benefitted from the participants benefitted from the service, e.g. number of participants service, e.g. number of participants reflects only volume.reflects only volume.

Participant satisfaction generally not Participant satisfaction generally not an outcome since it does not indicate an outcome since it does not indicate whether or not the participant's whether or not the participant's condition improved as a result of the condition improved as a result of the service.service.

ObjectivesObjectives

Describe what you plan to do in your program and how you expect the participants to change in a way will allow you to measure these objectives. What you plan to doWhat you plan to do Who will do itWho will do it Who you plan to reach and how manyWho you plan to reach and how many

S.M.A.R.T. ObjectivesS.M.A.R.T. Objectives

SpecificSpecific MeasurableMeasurable Action-orientedAction-oriented RealisticRealistic Time-boundTime-bound

Measurable Objectives:Measurable Objectives: Stating participant outcome objectives

in measurable terms requires you to be specific about the changes in knowledge, attitudes, awareness, or behavior that you expect to occur as a result of participation in your program.

Be able to answer the question: How will we know that the expected

changes occurred? To answer this question, you will have to

identify the evidence needed to demonstrate that your participants have changed.

AssignmentAssignment

In small groups review the In small groups review the program description.program description.

Using the Logic Model Using the Logic Model worksheet identify assumptions, worksheet identify assumptions, potential activities, outputs and potential activities, outputs and outcomes.outcomes.

Present to group.Present to group.