Gathering Evidence of Impact: A Continued Conversation

Jan Middendorf

Cindy Shuman

Office of Educational Innovation and Evaluation

Using a logic model to describe your program Using a logic model to describe your program

INPUTS OUTPUTS OUTCOMES

Program

investmentsActivities Participation Short Medium Long-term

What you do, with whomWhat you do, with whom What your clients do as a resultWhat your clients do as a result

ImpactImpact

Learning Behavior Change

Condition Change

Four Levels of Evaluation

Level 1: Reaction Level 2: Learning/Skill Building

– (short term outcome…ask at end of meeting, lesson, workshop, etc.)

Level 3: Transfer…Behavior Change– (medium term outcome… ask sometime after…maybe 6

months later ) Level 4: Results… Impact

– (long term outcome)

Kirkpatrick Model of Evaluation

Evaluation of Outcomes

How will you know what your clientele did as a result of your educational program?

What is the evidence that they used the information?

Why Evaluate?

4 main purposes of evaluation4 main purposes of evaluation

Improvement: to improve the program; to enhance quality; to manage more effectively and efficiently. The effort to enhance programs.

Accountability: to assess merit or worth; to assess effects; to assess costs and benefits. The effort to make judgments about the value of a policy or program.

Knowledge development: to gain new insights. The effort to add to the knowledge base about effective practice or to add to policy debate.

Oversight and compliance: to assess the extent to which a program follows rules, regulations, mandates or other formal expectations.

Source: Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding and improving policies and programs. San Francisco: Jossey-Bass.

Indicators: Evidence of Achieving Indicators: Evidence of Achieving OutcomesOutcomes

• What would it look like?• How would I know it?• If I were a visitor, what would I see, hear, read,

and/or smell that would tell me this “thing” exists?

If the outcome is achieved, how will you know it? What will it look like? What is the

evidence?

Indicators - EvidenceIndicators - Evidence

The information needed to answer your evaluation questions

Example: Did participant land owners or managers improve their land management practices?

Evidence: # acres or % of acres managed according to guidelines

# or quality of conservation plans implemented

Indicators - EvidenceIndicators - Evidence

The information needed to answer your evaluation questions

Example: Did participants increase their ability to achieve financial self-sufficiency?

Evidence: #, % who increased financial knowledge, #, & who reduced debt,

#,% who established an emergency fund

Have the pets been fed today?Have the pets been fed today?

How would you know that the animals have been fed? What is the evidence?

Let’s practice….Let’s practice….

What is the evidence of…– High blood pressure?– A clean neighborhood?– A popular movie?– A good carpenter?– Learning at the workshop?

Would the evidence be different for young people vs. seniors, high- vs. low-income neighborhoods, rural vs. urban residents, or by ethnicity?

Evidence is often expressed as numbers or percentages (number of…, percent of…, ratio of…, incidence of…, proportion of…).

However, not all evidence is numbers; qualitative evidence may be important.

Remember, "Not everything that counts can be counted."

Work on Evaluation Plans

Develop a question or two to understand what clientele did differently as a result of your educational program

Develop a plan for how you will gather the evidence…

– Get names and phone numbers and call them 6 months later– Ask a third party what they see that is different– Observe differences yourself– Other?

How good is the indicator?

Tangible – be able to “touch/know” the information in some way– See (observable) – Read (survey, records, etc.)– Hear (from individuals, others)– Tips: direct, specific, useful, practical, culturally

appropriate, adequate, clearly defined

What is an indicator?

An indicator is the specific information, or evidence, that represents the phenomenon you are asking about.

Indicator of fire = smoke Indicator of academic achievement = grades

How good are your questions?

Can the questions be answered given the program?

Are the questions truly important? Will the questions provide new insights? Can the questions be answered given your

resources and timeline? Have the concerns of key users been

included?

Identify key evaluation questions

Who wants to know what about this program?

Evaluation Questions

Clarify your evaluation questions Make them specific What do you need to know vs. what you

would like to know Prioritize Check: Will answers to these questions

provide important and useful information?

Components of a program

Situation Resources (Inputs) Outputs

– Activities – Participants

Outcomes– Chain of outcomes from short- to long- term

External Factors and Assumptions

What is your purpose for evaluating?

We are conducting an evaluation of _____ (program name) because ______ in order to __________.

Example: We are conducting an evaluation of the Money Quest Program because we want to know to what extent youth who participate learn and use recommended money management skill in order to report program outcomes to our funder.