Download - Taking the Mystery out of Project Evaluation
Taking the Mystery out of Taking the Mystery out of Project EvaluationProject Evaluation
Marjorie Dennis, UMass LowellMarjorie Dennis, UMass Lowell
Lisa Glickstein PhDLisa Glickstein PhD
Andover Public SchoolsAndover Public Schools
IntroductionsIntroductions
Marjorie Dennis: NSF GK-12, RET; US ED Marjorie Dennis: NSF GK-12, RET; US ED TAH, PT3; State ITQ, STEM PipelineTAH, PT3; State ITQ, STEM Pipeline
Lisa Glickstein: US ED TAH, PEP; State Lisa Glickstein: US ED TAH, PEP; State STEM PipelineSTEM Pipeline
Participant ExperiencesParticipant Experiences
Googlisms!Googlisms! Evaluation is…Evaluation is…
…”…”used to manage a project to meet used to manage a project to meet objectives.”objectives.”
…”…”the process of measuring the quality of the process of measuring the quality of performance against standards to performance against standards to determine if the standards have been determine if the standards have been met.”met.”
…”…”a systematic inquiry designed to a systematic inquiry designed to provide information to decision makers.”provide information to decision makers.”
Session OverviewSession Overview
WhyWhy do you evaluate? do you evaluate? WhoWho chooses the evaluator? chooses the evaluator? WhereWhere do you find an evaluator? do you find an evaluator? WhenWhen do you start working with the do you start working with the
evaluator?evaluator? How How do you design and carry out an do you design and carry out an
evaluation?evaluation? WhatWhat do you do if something goes wrong? do you do if something goes wrong?
WHY?WHY?
Because you have to – federal or state lawBecause you have to – federal or state law
To get feedback during the projectTo get feedback during the project Formative evaluationFormative evaluation
Because you want to answer the big Because you want to answer the big
question: question: DID IT WORK?DID IT WORK? Summative evaluationSummative evaluation
““Research seeks to Research seeks to proveprove, ,
evaluation seeks to evaluation seeks to improveimprove…” …” M.Q. PattonM.Q. Patton
(former President of the American Evaluation Association)(former President of the American Evaluation Association)
Adapted from: Maberry Consulting Group
WHO?WHO?
Who is a good evaluator?Who is a good evaluator? Who selects the evaluator?Who selects the evaluator? Who decides contract terms?Who decides contract terms? Who will the evaluator report to?Who will the evaluator report to? Who will the evaluator work with?Who will the evaluator work with? Who owns the evaluation plan, data and Who owns the evaluation plan, data and
reports?reports?
Internal versus External EvaluationInternal versus External Evaluation
Program guidelinesProgram guidelines
CostCost
AvailabilityAvailability
Existing RelationshipsExisting Relationships
Four Steps to Select an EvaluatorFour Steps to Select an Evaluator
1.1. Decide who selects the evaluatorDecide who selects the evaluator Project Director or Advisory BoardProject Director or Advisory Board Head of organizationHead of organization Grant or Business Office (lowest bidder)Grant or Business Office (lowest bidder) Grant-makerGrant-maker
2.2. Contact potential evaluators and solicit Contact potential evaluators and solicit proposalsproposals
Four Steps (cont’d)Four Steps (cont’d)
3.3. Choose selection criteria (see Notes)Choose selection criteria (see Notes) KnowledgeKnowledge Experience – overall, and project-specificExperience – overall, and project-specific CostCost ReferencesReferences Communication skillsCommunication skills
4.4. Evaluate proposals and selectEvaluate proposals and select
Intellectual PropertyIntellectual Property
The funder paid for the evaluation and can The funder paid for the evaluation and can request data or reports at any timerequest data or reports at any time
The fiscal agent/sponsor retains the hard The fiscal agent/sponsor retains the hard and electronic copies of all evaluation and electronic copies of all evaluation materialsmaterials
Evaluation plan, rubrics, data and reports Evaluation plan, rubrics, data and reports all belong to the project (not to the all belong to the project (not to the evaluator)evaluator)
WHEREWHERE??
WhereWhere do you find an evaluator? do you find an evaluator? In houseIn house ExternalExternal
WhereWhere does one learn to be an evaluator? does one learn to be an evaluator? WhereWhere are evaluators certified? are evaluators certified? WhereWhere do external evaluators work? do external evaluators work?
Professional EvaluatorsProfessional Evaluators
American Evaluation Association (American Evaluation Association (http://www.eval.org/) )
Graduate & Certificate Programs (Graduate & Certificate Programs (http://www.eval.org/p/cm/ld/fid=43) )
There is no single recognized credentialThere is no single recognized credential Consulting or in a university setting (Consulting or in a university setting (
http://www.eval.org/p/cm/ld/fid=108) )
WHEN?WHEN?
WhenWhen do you select the evaluator? do you select the evaluator? WhenWhen do you work out terms of hire? do you work out terms of hire? WhenWhen do you have the first project do you have the first project
meeting? meeting? WhenWhen and how often will you meet? and how often will you meet? WhenWhen do you expect reports? do you expect reports?
Initial ContactInitial Contact Involving the evaluator in project planning, or Involving the evaluator in project planning, or
planning evaluation during (not after) project planning evaluation during (not after) project design is a good ideadesign is a good idea
An evaluator can prepare an evaluation plan for An evaluator can prepare an evaluation plan for a proposal (usually at no cost)a proposal (usually at no cost)
Sometimes it wins points to name an evaluator Sometimes it wins points to name an evaluator in the grant proposalin the grant proposal
Be aware of bid laws in your stateBe aware of bid laws in your state Post-award some changes to the evaluation plan Post-award some changes to the evaluation plan
are okayare okay
Contract and TermsContract and Terms
Scope of services – list of activities and Scope of services – list of activities and work products (see Notes)work products (see Notes)
Cost of entire contract, or hourly rate and Cost of entire contract, or hourly rate and estimated total (8-15% of total grant is to estimated total (8-15% of total grant is to be expected)be expected)
Automatic termination if grant ends, Automatic termination if grant ends, options to terminate for cause or no causeoptions to terminate for cause or no cause
Protections for organization from failures Protections for organization from failures or omissions on part of evaluatoror omissions on part of evaluator
HOW?HOW?
How do you design and carry out an How do you design and carry out an evaluation?evaluation? Create evaluation planCreate evaluation plan Select indicators and benchmarksSelect indicators and benchmarks
How do you collect and analyze data?How do you collect and analyze data? Analyze dataAnalyze data Write reportsWrite reports
Evaluation PlanEvaluation Plan
Project’s goals, activities and timeline include Project’s goals, activities and timeline include data collectiondata collection
Goal = ultimate change project intends to make in Goal = ultimate change project intends to make in a condition (e.g. poverty)a condition (e.g. poverty)
Indicators = changes in attitudes, knowledge and Indicators = changes in attitudes, knowledge and behaviors that show progress towards meeting behaviors that show progress towards meeting goalgoal
Benchmarks = amount of change expected in a Benchmarks = amount of change expected in a given project yeargiven project year
Mini-EvaluationMini-Evaluation
Working alone or in groups, fill out the Working alone or in groups, fill out the sample evaluation form for your programsample evaluation form for your program
Focus first on what you are certain of – Focus first on what you are certain of – include details where knowninclude details where known
Focus next on what you don’t know – how Focus next on what you don’t know – how would you measure or find out? Who would you measure or find out? Who would you ask and how?would you ask and how?
Can you make recommendations?Can you make recommendations?
Research vs EvaluationResearch vs Evaluation Production of generalizable Production of generalizable
knowledge is the goalknowledge is the goal Researcher-derived Researcher-derived
questions (hypothesis)questions (hypothesis) Paradigm stance (how or Paradigm stance (how or
why does it work?)why does it work?) Setting designed to test Setting designed to test
hypothesis (controlled)hypothesis (controlled) Clearer role (principal Clearer role (principal
investigator)investigator) PublishedPublished Clearer allegianceClearer allegiance
May produce generalizable May produce generalizable knowledgeknowledge
Program- or funder-derived Program- or funder-derived questionsquestions
Judgmental quality (does it Judgmental quality (does it work?)work?)
Action setting to provide Action setting to provide direct servicesdirect services
Role conflicts (project Role conflicts (project director AND evaluator)director AND evaluator)
Often unpublishedOften unpublished Multiple allegiancesMultiple allegiances
Adapted from: Maberry Consulting Group
Formative EvaluationFormative Evaluation Project proceeding as planned?Project proceeding as planned?
Activities, recruitment, data collection taking Activities, recruitment, data collection taking place?place?
All partners working together?All partners working together?
How can or should project get back on track?How can or should project get back on track?
Is project getting closer to its anticipated goals?Is project getting closer to its anticipated goals?
Summative EvaluationSummative Evaluation
Did the project meet its goals or not?Did the project meet its goals or not? Did the participants benefit?Did the participants benefit? To what extent did they benefit?To what extent did they benefit? Can you prove that activities caused the Can you prove that activities caused the
benefits?benefits? What activities were most effective or cost-What activities were most effective or cost-
effective?effective? Is the project sustainable?Is the project sustainable? Is the project replicable?Is the project replicable?
Short vs Long-term OutcomesShort vs Long-term Outcomes
OUTCOMESWhat results for individuals, families, communities..…
SHORTLearning
Changes in:
• Awareness• Knowledge• Attitudes• Skills• Opinion• Aspirations• Motivation• Behavioral intent
MEDIUMAction
Changes in:
•Behavior •Decision-making•Policies•Social action
LONG-TERMConditions
Changes in:
•Well-being•Health•Economy•Civics•Environment
Adapted from: University of Wisconsin – Logic Models
The Four StandardsThe Four Standards
UtilityUtility: Who needs the information and what information : Who needs the information and what information do they need?do they need?
FeasibilityFeasibility: How much money, time, and effort can we put : How much money, time, and effort can we put into this? into this?
Propriety:Propriety: What steps need to be taken for the What steps need to be taken for the evaluation to be ethical? evaluation to be ethical?
Accuracy:Accuracy: What design will lead to accurate information? What design will lead to accurate information?
Adapted from: Maberry Consulting Group
Six methods to collect information and data…Six methods to collect information and data…
Method Overall Purpose Advantages Challenges
1. Questionnaires,
Surveys,
and Checklists
To quickly and/or
easily get a lot of
information from
people in a non-
threatening way
• Can be completed
anonymously
• Inexpensive to
administer
• Easy to compare
and analyze
• Administer to
many people
• Can get lots of
data
• Many sample
questionnaires
already exist
• Might not get
careful feedback
• Wording can bias
responses
• Are impersonal
• In surveys, may
need sampling
expert
• Doesn't get the full
story
Adapted from: Maberry Consulting Group
Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
2. Interviews To fully understand
participant's
impressions or
experiences, or to
learn more about their
answers to
questionnaires
• Collects a full
range and depth of
information
• Develops
relationship with
the participant
• Can be flexible
and responsive
• May take too long
• Can be hard to
analyze and
compare
• Can be costly
• Interviewer can
bias responses
Adapted from: Maberry Consulting Group
Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
3. Documentation
review
To get an impression of
how program operates
without interrupting the
program; review of
applications, finances,
memos, minutes,
participant work
products, and so on
• Collect
comprehensive and
historical
information
• Doesn't interrupt
program or
participant’s routine
• Information already
exists
• Few biases about
information
• Often time-
consuming
• Information may be
incomplete
• Need to be quite
clear about what you
are looking for
• Not flexible, as data
restricted to what
already exists
Adapted from: Maberry Consulting Group
Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
4. Observation To gather accurate
information about
how a program
actually operates,
particularly about
processes
• View operations of
a program as they
are actually
occurring
• Can adapt to
events as they
occur
• Difficult to
interpret observed
behaviors
• Complex to
categorize
observations
• Can influence
behaviors of
program
participants
• Can be expensiveAdapted from: Maberry Consulting Group
Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
5. Focus groups To explore a topic in
depth through group
discussion, such as
reactions to an
experience or
suggested change,
understanding
common complaints
(useful in evaluation
and marketing)
• Quickly and
reliably get
common
impressions
• Can be an efficient
way to get a range
and depth of
information in a
short time
• Can convey key
information about
programs
• Hard to analyze
responses
• Need a good
facilitator for
safety and closure
• Difficult to
schedule 6-8
people together
Adapted from: Maberry Consulting Group
Six methods (continued)…Six methods (continued)…
Method Overall Purpose Advantages Challenges
6. Case studies To fully understand or
depict participant
experiences in a
program, and conduct
comprehensive
examination through
cross comparison of
cases
• Fully depicts
participant’s
experience in
program input,
process and results
• Powerful means to
portray program to
outsiders
• Usually quite time
consuming to
collect, organize
and describe
• Represents depth
of information,
rather than breadth
Adapted from: Maberry Consulting Group
Basic analysis of quantitative (numeric) dataBasic analysis of quantitative (numeric) data
1.1. Make copies of your data and store the original. Use Make copies of your data and store the original. Use the copy for making edits, cutting and pasting, and so the copy for making edits, cutting and pasting, and so onon
2.2. Tabulate the information in one or more spreadsheets. Tabulate the information in one or more spreadsheets.
3.3. Examine the median, mean, and distribution (range) of Examine the median, mean, and distribution (range) of data sets; think about which is/are more meaningfuldata sets; think about which is/are more meaningful
4.4. Think about segmentation: Do you have sub-Think about segmentation: Do you have sub-populations based on demographics or how much populations based on demographics or how much service was received?service was received?
5.5. Consider confounding variables: Is the effect due to a Consider confounding variables: Is the effect due to a bias rather than your service or intervention? bias rather than your service or intervention?
Adapted from: Maberry Consulting Group
Basic analysis of qualitative (subjective) dataBasic analysis of qualitative (subjective) data(interview or focus group responses, or written commentary on questionnaires)(interview or focus group responses, or written commentary on questionnaires)
1.1. Read through all the data.Read through all the data.2.2. Organize comments into similar categories: Organize comments into similar categories:
concerns, suggestions, strengths, weaknesses, concerns, suggestions, strengths, weaknesses, similar experiences, program inputs, similar experiences, program inputs, recommendations, outputs, or outcome indicators.recommendations, outputs, or outcome indicators.
3.3. Look for patterns, associations and possible causal Look for patterns, associations and possible causal relationships.relationships.
4.4. Keep all raw data as recommended by the funder Keep all raw data as recommended by the funder after completion in case needed for future after completion in case needed for future reference.reference.
Adapted from: Maberry Consulting Group
Evaluation is not only IndicatorsEvaluation is not only Indicators
Evaluation: Indicators in the context of:Evaluation: Indicators in the context of: Design & implementationDesign & implementation Program relevanceProgram relevance EfficiencyEfficiency EffectivenessEffectiveness ImpactImpact SustainabilitySustainability
ExperiencesExperiences
Write down the worst thing that an Write down the worst thing that an evaluator ever said to you about your evaluator ever said to you about your project…project…
……or the worst thing that ever went wrong or the worst thing that ever went wrong on a grant-funded project at your on a grant-funded project at your organization!organization!
WHAT?WHAT?
WhatWhat do we do when something goes do we do when something goes wrong?wrong? Partners don’t get alongPartners don’t get along Leadership issuesLeadership issues Disagreements over data interpretationDisagreements over data interpretation Non-performance by partner(s)Non-performance by partner(s) Change in conditionsChange in conditions Conflicts of interestConflicts of interest
The Challenge of Being an The Challenge of Being an Evaluator (Sirotnik, 1999)Evaluator (Sirotnik, 1999)
If you…are looking for highly definitive, If you…are looking for highly definitive, generalizable, cause/effect relationships generalizable, cause/effect relationships between measurable, high-validity, independent between measurable, high-validity, independent and dependent variables, then I suggest you go and dependent variables, then I suggest you go to work for a pharmaceutical company…to work for a pharmaceutical company…However, if you have a high tolerance for However, if you have a high tolerance for ambiguity and like to mess around…with ambiguity and like to mess around…with contexts that are always colliding and contexts that are always colliding and changing…in these settings that are engaged in changing…in these settings that are engaged in long-term processes of renewal and change – long-term processes of renewal and change – well then have I got a career for you!well then have I got a career for you!
If the evaluator ain’t happy, ain’t no If the evaluator ain’t happy, ain’t no one happy…one happy…
If your project isn’t going well, it is the If your project isn’t going well, it is the evaluator’s job to tell you (and ultimately to evaluator’s job to tell you (and ultimately to tell the funder)tell the funder)
If you have an issue with your evaluator, If you have an issue with your evaluator, first look at and fix issues with your projectfirst look at and fix issues with your project
Most of the time, this will repair your Most of the time, this will repair your relationship with the evaluator, or at least relationship with the evaluator, or at least make it tolerable!make it tolerable!
Conflict of PartnersConflict of Partners
Partners have conflicting ideasPartners have conflicting ideas
One or more partners are not One or more partners are not carrying through with activities carrying through with activities or data collectionor data collection
Personality clashesPersonality clashes
Complaints about distribution of Complaints about distribution of resources or other complaintsresources or other complaints
Leadership IssuesLeadership Issues
Poor communication – no contact, bad Poor communication – no contact, bad emails, late information and noticesemails, late information and notices
Bills not paid on time, delays in processing Bills not paid on time, delays in processing registrations for activitiesregistrations for activities
Bad record-keeping or data collectionBad record-keeping or data collection Failure to respond to formative evaluationFailure to respond to formative evaluation Mismanagement of project resourcesMismanagement of project resources Illegal activityIllegal activity
Different Interpretations of DataDifferent Interpretations of Data
Evaluator sees the data Evaluator sees the data one way, project partners one way, project partners see it anothersee it another
Lack of data leads to Lack of data leads to subjective assessmentsubjective assessment
One party rejects dataOne party rejects data
Non Performance of PartnersNon Performance of Partners
Failure to complete scope of servicesFailure to complete scope of services Communication issuesCommunication issues Significant, ongoing delaysSignificant, ongoing delays Personality clashes or personnel problemPersonality clashes or personnel problem Undermining professional relationshipsUndermining professional relationships Mismanagement of fundsMismanagement of funds
Change in ConditionsChange in Conditions
New competitor (threat)New competitor (threat) Change in lawChange in law Sub-contractor or partner goes out of Sub-contractor or partner goes out of
business or quitsbusiness or quits Significant economic downturnSignificant economic downturn Change in leadership (project or Change in leadership (project or
organization)organization)
Conflict of InterestConflict of Interest
Is the evaluator independent enough to Is the evaluator independent enough to objectively evaluate the project?objectively evaluate the project?
Would giving a positive or negative Would giving a positive or negative evaluation result in financial gain?evaluation result in financial gain? Future contracts with one or more partnersFuture contracts with one or more partners
Is there an appearance of conflict of Is there an appearance of conflict of interest?interest? Employer/employee or family relationshipEmployer/employee or family relationship
Presentation Wrap UpPresentation Wrap Up Additional stories and comments on the Additional stories and comments on the
evaluation processevaluation process
Questions on evaluationQuestions on evaluation
Reference materials for further Reference materials for further investigation and information on working investigation and information on working with an evaluator and the evaluation with an evaluator and the evaluation processprocess
Additional ReadingAdditional Reading
Evaluating Professional DevelopmentEvaluating Professional Development, Thomas , Thomas R. Guskey, ISBN 0-7619-7561-6, 2000.R. Guskey, ISBN 0-7619-7561-6, 2000.
““8 Smooth Steps: Steps to Your own 8 Smooth Steps: Steps to Your own Evaluation,” Joellen Killion, Evaluation,” Joellen Killion, Journal of the Journal of the National Staff Development CouncilNational Staff Development Council, p 14, Fall , p 14, Fall 2003.2003.
Practicing Evaluation: A Collaborative ApproachPracticing Evaluation: A Collaborative Approach, , Rita O’Sullivan, ISBN 0-7619-2456-5, 2004.Rita O’Sullivan, ISBN 0-7619-2456-5, 2004.
W.K. Kellogg Foundation Evaluation Handbook W.K. Kellogg Foundation Evaluation Handbook & Development Guide, & Development Guide, www.wkkf.org, 2008., 2008.
Thank you for attending!Thank you for attending!
Contact information:Contact information: Lisa Glickstein: Lisa Glickstein: [email protected]
Marjorie Dennis: Marjorie Dennis: [email protected]