irene f. goodman, ed.d., president, goodman research group, inc. denise callahan, director of...
TRANSCRIPT
Irene F. Goodman, Ed.D., President, Goodman Research Group, Inc.
Denise Callahan, Director of Scholarship Programs, The Ford Family
Foundation
NSPA Conference Connect HourOctober 15, 2014
STRATEGIES FOR EVALUATING YOUR
SCHOLARSHIP PROGRAM
Goodman Research Group, Inc.
Group discussion about the opportunities for and challenges of evaluating your scholarship program
What would you most like to know?What is most useful to you?How can evaluation help?
What you can start doing right away and what you can plan for in the long term
WHAT BROUGHT YOU TO THIS SESSION?
Goodman Research Group, Inc.
Will compare list generated with three main questions that scholarship providers have often posed in the past:
1. What impact are we making? 2. What is the success rate of our participants? 3. What happens to our Scholars down the road?
Other questions generated by today’s group that can be addressed by evaluation
EVALUATION: AN IMPORTANT AND OFTEN NEGLECTED PROGRAM
COMPONENT
Goodman Research Group, Inc.
Importance of evaluation
Considerations for evaluation
Needs assessment data are important Goal setting crucial Outcomes
Types of evaluation
Making links between activities and outcomes
TODAY’S TOPICS
Goodman Research Group, Inc.
The trouble with not knowing what you want is that you don’t know what you’ve got when you get it.
EVALUATION IS IMPORTANT BECAUSE….
Assists program staff in learning what is working and what isn’t
Decision making and performance management
Assists in obtaining funds
Provides accountability to board/funders/ other stakeholders
IMPORTANCE OF EVALUATION
Goodman Research Group, Inc.
Is it worth evaluating?
Maturity, coherence of program
If so, what kind of evaluation should be done?
Formative testing of program component ideas? Process evaluation or outcome evaluation or both? Short term outcomes or long term outcomes?
Funding realities
Feasibility
CONSIDERATIONS FOR EVALUATION
Goodman Research Group, Inc.
Type of Evaluation Questions Answered
NEEDS ASSESSMENT What is the problem? What do we need to do about it?
FORMATIVE EVALUATION As we’re developing our program, what changes can be made to make it better?
PROCESS EVALUATION(“The Black Box”)
Is the program happening? Are we doing what we said we would? How much of it do we do? Are we reaching our target audience? How can it be better?
OUTCOMES EVALUATION Is it working? What short-term changes are there as a result of using the materials/program?
IMPACT EVALUATION What longer-term changes or benefits occur? What is the overall project effectiveness?Goodman Research Group, Inc.
DIFFERENT TYPES OF EVALUATION
GOAL SETTING
“Looks like the ghost of unarticulated goals has come back to haunt us.”
Goodman Research Group, Inc.
FORMATIVE EVALUATION
No matter how far you have gone down the wrong road, turn back.
Goodman Research Group, Inc.
OUTCOMES EVALUATION
The proof of the pudding is in the eating.
Goodman Research Group, Inc.
Ask yourself: What do I want to find out? What would success look like?
How will I know? (Indicators)
Produce a list and be specificGet feedback and learn what else has been done
It’s okay that the questions you start with aren’t necessarily the questions you end with.
OUTCOMES QUESTIONS TO ASK
Goodman Research Group, Inc.
What impact do you wish to have?
What is realistic given your program activities?
Use “if/then” statements, e.g.:
“If we give $1,000 scholarships to 1,000 people then we expect X to happen.”
What is reasonable to expect in the short-term, in the mid-term, and in the long-term?
DEFINING IMPACT
Goodman Research Group, Inc.
OUTCOMES AND IMPACT
Goodman Research Group, Inc.
Short-term Intermediate Longer-term
Closer in timeEasier to measureMore attributable to program
More distant in timeHarder to measure
Less attributable to program
LOGIC MODEL
INPUTS ACTIVITIESSHORT-TERM
OUTCOMES
IMPACT(LONG-TERM)
OUTPUTS
Goodman Research Group, Inc.
Working on how to answer what you want to know
GROUP ACTIVITY
Goodman Research Group, Inc.
ELEMENTS & STRUCTUREOutcome Definitions:
Intended ImpactYour example:
Indicator Observable and measurable behaviors and condition
Your example:
Data source Sources of information about conditions being measured
Your example:
Applied to The specific group to be measured (all or a subset)
Your example:
Data interval When data will be collected Your example:
Target (Goal) The amount of impact desired
Your example:
Goodman Research Group, Inc.
LOGIC MODEL ELEMENTS AND STRUCTURE
Tracking your participants – start with adding baseline questions to application
Retrospective survey of alumni
Following participants prospectively and longitudinally (baseline and annually)
More rigorous: comparing scholars with finalists
FROM THE SIMPLE TO THE COMPLEX
Goodman Research Group, Inc.
Emails
Annual check-ins
Asking for preferred mode of communication
Birthday cards/postcards
Obtaining updated contact information
Using social media to reach out
TRACKING IS EXTREMELY IMPORTANT
Goodman Research Group, Inc.
Looking back to their experiences
Asking “retrospective pre” questions (When it has not been possible to get baseline data, these types of questions ask respondents to compare then to now.)
RETROSPECTIVE SURVEYS OF ALUMNI
Goodman Research Group, Inc.
Starting when they have just started their scholarship (baseline)
Expectations High school participation Stress level and barriers
Comparing annual or so how their perspectives change
Can be a one-time snapshot or, if you have the luxury, annually
FOLLOWING PARTICIPANTS PROSPECTIVELY AND
LONGITUDINALLY
Goodman Research Group, Inc.
This answers the counter-factual: How do scholars compare with similar students who did not receive your scholarship?
What might have happened/not happened if the students had not received the scholarship?
Must statistically control for group differences
Can also compare with state, regional, and national data
MORE RIGOROUS: COMPARING SCHOLARS WITH FINALISTS
Goodman Research Group, Inc.
CONTACT US! Irene F. GoodmanGoodman Research Group, Inc. www.grginc.com(617) [email protected]
Denise CallahanThe Ford Family Foundation www.tff f.org(541) 485-6211dcallahan@tff f.org
QUESTIONS OR COMMENTS?
Goodman Research Group, Inc.