association for the study and development of community maximizing impact: helping funders get smart...

Association for the Study and Development of Community

Maximizing Impact: Helping Funders Get Smart About Evaluation

Kien S. LeeAssociation for the Study and Development of Community

438 N. Frederick Avenue, Suite 315, Gaithersburg, MD 20877(301) 519-0722, ext. 108 [email protected]

Maine Philanthropy CenterOctober 8, 2008


I have to rewrite the evaluator’s report because it is too jargony and not useful. After reading it, I

still have no idea how we are doing.

I’m told an internal

evaluation is not objective

The evaluator told us that our program was

not ready for evaluation

We submit all this information to the funder,

I’m not sure anyone actually reads them

The funder had unrealistic expectations. There is no way we

can measure these outcomes given the constraints

I want to evaluate a public education

campaign, how much will it cost?


American Evaluation Association Guiding Principles

www.eval.org

A. Systematic InquiryB. CompetenceC. Integrity/HonestyD. Respect for PeopleE. Responsibilities for General and Public

Welfare


Useful Terms to Know About

• Evaluability Assessments

• Prospective Evaluations

• Retrospective Evaluations

• Formative, Process, Implementation Evaluations

• Summative, Outcome, Impact Evaluations

• Meta-Evaluations


Readiness• Clear and realistic goals, objectives, and

strategies

• Clear understanding of how strategies will lead to anticipated outcomes

• Capacity to collect the data needed

• Commitment to learning

• Resources to hire an evaluator


Internal or External Evaluator?

• Purpose

• Roles in relation to other parts of the organization

• Knowledge and skills

• Existing systems

• Resources


What Do You Look For in An Evaluator?

• Knowledge of Evaluation

• Philosophical and Methodological Orientations

• Commitment to Professional Development

• Cultural Competency

• Special Skills and Experiences Required for the Evaluation


Multiple Hats Worn by An Evaluator

Educator

Conflict Resolution Manager

Learning Facilitator

Evaluator

Advocate for Social Justice


What if you can’t afford a full-fledged evaluation?

• Identify and monitor only those outcomes and indicators that are critical

• Engage external evaluator to review your logic model, measures, and/or instruments

• Engage external evaluator to double-check quality of data and analyses

• Engage external evaluator to develop monitoring system and train staff


Ask the following questions over and over again:

•What do I want to know?•Who else needs to be involved?•Strengths and limitations?•How will the findings be used?•When will the findings be needed?•What is the best way to communicate the

findings?


• Merit or worth?• Ways to improve our work?• Compliance of grantees?• Contribute to the field?• Inform policies?

Patton, M. (1997). Utilization-focused Evaluation. Thousand Oaks, CA: SAGE PublicationsMark, M., Henry, G., & Julnes, G. (2001). Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Program. San Francisco, CA: Jossey-Bass.

What do I want to know from the evaluation?


Who else needs to be involved?

Internal Stakeholders• Board and Executive Leadership• Staff

External Stakeholders• Grantees• Community Leaders• Partners


What are the strengths and limitations of the evaluation?

• What are the theories and assumptions underlying the evaluation?

• What are the pros and cons of the evaluation design?

• What can I say or not say about the findings and their implications?


How do I plan to use the findings?

•What decisions do I need to make based on the results?

•What type of changes am I prepared to make? •What changes, if any, can be shared with some

people and not with others?


When do I need the findings?

• What is the timeframe for making decisions?

• When do my supervisor, board members, community leaders, policymakers, or others want to know about the evaluation findings?


Where will the evaluation findings be shared?• Staff meetings, press conference, board

meetings, public hearings, etc.

Who is the most effective messenger for the findings?

• Evaluator, you, your supervisor, someone else

What is the best way to communicate the findings?


Examples of different forms of communicating evaluation findings:

• Short, written communications

• Reports• Executive summaries• Newsletters, bulletins,

briefs, and brochures• Verbal presentations

• Video presentations• Posters• Working sessions• Photography• Cartoons• Poetry• Drama

Preskill, H., Torres, R., & Piontek, M. (2004). Evaluation Strategies for Communication and Reporting. Thousand Oaks, CA: SAGE Publications


Logic Modeling

• Is a graphic illustration of your theory of change• Makes explicit underlying assumptions about the

change process• Specifies connections between inputs,

strategies, activities, immediate outcomes, intermediate outcomes, and long-term outcomes

• Provides a process for reflection and improvement


The logic model in non-jargon terms!

• Provides the story line for your grantmaking and evaluation

• Pushes you to ask and answer the question, “If you do A, then what do you expect to happen? If your answer is B, tell me why you think B will occur.”

• A picture that lays out your investment hypothesis


Assessing the Foundation’s Effectiveness and Impact

• Start with the overall theory of change and logic model for the entire foundation

• Then, develop logic models for each program priority area

• Then, develop logic models for each program or initiative within priority area

• Check to ensure that each logic model supports the one “above” and “below” it


Inputs

Immediate Outcomes

Long-term Outcomes

Contextual Conditions

Example of A Simple Logic Model

Strategy #1

Strategy #2

Activities

Activities

Intermediate Outcomes


Indicators and Benchmarks

Road signs that you are moving in the right direction

• Shows the change or progress over a defined time period

• Directly related to your anticipated outcome • Can be in the form of a number or statistic where

you can see an increase or decrease or a first-time practice or procedure that marks a significant change


• Develop uniform indicators across grantees that demonstrates funder’s effectiveness

• Establish these uniform indicators ahead of time before grant is awarded

• Build data requirements into progress reporting template

• Provide technical assistance to grantees to track the uniform indicators and indicators specific to their effort

In An Initiative with Multiple Grantees and Sites….


Measuring Context

• Rival explanations or counterfactuals

• Qualitative description (converging evidence)

• Statistical methods (multiple regression, hierarchical linear modeling)


How can we best learn from evaluation?

• Conduct reflection meetings regularly• Be prepared for different scenarios • Set up appropriate structures and processes

for engaging all the stakeholders (be creative!)• Communicate mid-course adjustments that

resulted from learnings• Identify promising practices and learning

assets


Why do some evaluation reports sit on the shelf?• The evaluation purpose is not connected to

learning• The report is too technical and jargon-filled• The report does not address the learning

questions• The analysis is weak, confusing, or inappropriate• The evaluation report is too late to be useful• The findings are deemed too controversial to

discuss openly


What should the evaluation report look like?

• Things to consider based on audience needs: Knowledge of technical language Format Length Style Number and type of reports (e.g., interim and

final; internal and external)


Characteristics of a Useful Evaluation Report• Flows like the logic model • Jargon-free, well-written• Findings first, evidence second• Uses a variety of illustrations• Answers key questions• Provides recommendations for mid-course

adjustments or future improvements• Includes an executive summary


• Don’t forget – part of the reflection should focus on the evaluation itself! What would you have done differently

given data and resource constraints? How could the reflection and learning

process be improved? Did we give appropriate feedback to the

evaluator?


Key Take-aways• Determine the purpose of your evaluation and

the best evaluator for the job• Engage the appropriate stakeholders early on• Plan for the reporting, communication about the

findings, and learning process at the beginning of the evaluation

• Assess the context for presenting the findings• Know and apply AEA’s guiding principles• Hire an external evaluator for certain tasks• Anything else?

association for the study and development of community maximizing impact: helping funders get smart...

Documents

evaluator slide

evaluation slide

lee association

external evaluator

public welfare slide

internal evaluation

social justice slide

evaluation kien