evaluation for vistas - literacy minnesota · 2020-01-23 · 1/22/2015 1 using sound evaluation...
TRANSCRIPT
![Page 1: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/1.jpg)
1/22/2015
1
Using sound evaluation practices to support your project’s success
Evaluation for VISTAs
By Sara McGarraughImprove Group Research Analyst
Let’s get to know each other
Overview
Purpose and benefits of evaluation
Evaluation tools that you can use
Designing an evaluation
![Page 2: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/2.jpg)
1/22/2015
2
“In reality, (nonprofit) performance is all about
translating caring, believing, and compassion into results.”
Letts, Ryan & Grossman High Performance Nonprofit Organizations
1999
What is evaluation?
What is evaluation?
Assessing strengths and weaknesses …to improve
effectiveness
“
”
![Page 3: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/3.jpg)
1/22/2015
3
Demonstrate results
*Fake data is used to create this dashboard
Reflect
Create a learning community
![Page 4: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/4.jpg)
1/22/2015
4
Influence others
Engage stakeholders
Plan for sustainability
![Page 5: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/5.jpg)
1/22/2015
5
What are the benefits of evaluation?
Knowledge and reflection
Information to share with stakeholders
Good evaluation practices are also good program management practices
Barriers to Evaluation
Humans resist change
We want to get along
Evaluation might suggest we are doing something wrong or would need to change
Time and cost
The Role of Evaluation
Formative evaluation: How is the program delivered and is it meeting expectations?
![Page 6: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/6.jpg)
1/22/2015
6
The Role of Evaluation
Summative evaluation: Did the program meet its goals?
The evaluation process Define
Purpose
Develop questions
Prepare methods
Measure
Analyze
Interpret
Improve
PUTTING EVALUATION TO PRACTICE
![Page 7: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/7.jpg)
1/22/2015
7
Start up: Define Purpose
Who are the key stakeholders of the evaluation? What you hope to learn as part of the evaluationPrioritize competing interests
Design & Planning: Refine goals
What are your program’s aspirations (your goals)?Goals should be SMART
Specific
Measureable
Attainable
Realistic
Time-bound
S
M
A
R
T
Design & Planning: Define project outcomes
![Page 8: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/8.jpg)
1/22/2015
8
PRACTICE: REFINING GOALS AND OUTCOMES
Logic models: a basic evaluation tool
A Logic Model can describe how
your program’s goals and
activities lead to results and how
to measure them
How are they useful?
Help get everyone on the same pageEncourage investment and buy-inFacilitate organizationProvide clear and concrete guidelinesServe as a roadmap throughout evaluation processUseful for grant proposals
![Page 9: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/9.jpg)
1/22/2015
9
What does it show?
GOALS ACTIVITIES OUTPUTS
INTENDED OUTCOMES
DefinitionsActivities:
What your program does
Outputs: Countable products showing an activity occurred (evidence)
Outcomes: Benefit received from your programKnowledge, behavior, condition changes
Example
GOAL:For all students to be at or above
their grade reading level
ACTIVITIES:-Tutoring
-Theater
-Book club
OUTPUT: # of tutoring sessions
# of books/articles read
INTENDED OUTCOMES:-Students are more confident in reading
-Students think reading is fun
TOOLS:
-Survey-Check-in form
![Page 10: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/10.jpg)
1/22/2015
10
PRACTICE: LOGIC MODEL DEVELOPMENT
BREAK
DESIGNING DATA COLLECTION TOOLS
![Page 11: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/11.jpg)
1/22/2015
11
Design measures and tools
What type of evidence will demonstrate outcomes?Design evaluation tools
Traditional data-gathering strategies
Surveys
Tests
Focus Groups
Interviews
Administrative data
Consider creative ways to gather data
![Page 12: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/12.jpg)
1/22/2015
12
Matching evaluation tools to your work
Use available resources
Use knowledge of staff
Honor the wisdom of all of your stakeholders
Supported by infrastructure
PRACTICE: DATA COLLECTION TOOL DESIGN
Say what? If the question is a scale…
“How valuable was your training session with TORCH?”
a) Very valuableb) Somewhat valuablec) Not valuable
![Page 13: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/13.jpg)
1/22/2015
13
65% of volunteers in their training program found their involvement with TORCH to be very valuable – OR –
Over half of volunteers found their involvement with TORCH to be very valuable. – OR –
80% of all respondents found their involvement with Sample Program to be somewhat or very valuable.
Then, you can report outcome statements like:
Say what? If the question is open‐ended…
“What was the most valuable part of volunteering for TORCH?”
Respondents frequently reported community engagement, connecting with youth, and learning about issues in the community as the most valuable part of volunteering with TORCH.
Then, you can report outcome statements like:
![Page 14: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/14.jpg)
1/22/2015
14
Say what? If the question is a retrospective pre‐test…
“Please rate your opinion of the importance of volunteerism before AND after participating with the United Way.”
Before participation After participationVery
importantSomewhat important
Not important
Very important
Somewhat important
Not important
“85% of respondents rated volunteerism as very important after participating with United Way compared to only 42% before participating.”
Then, you can report outcome statements like:
Types of questions
What are the pros and cons of these question types?
![Page 15: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/15.jpg)
1/22/2015
15
Photo by Heather McQuaid
Focus on simple and effective
Use parameters to make responding easier
Photo by Oli Shaw
Avoid double-barreled questions, using jargon, or vague questions
Photo by Nate Bolt
![Page 16: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/16.jpg)
1/22/2015
16
PRACTICE: SURVEY OR INTERVIEW QUESTION DESIGN
Implementation: Data gathering
Smile!Practice in advanceHave a contact person for questionsRespect time & privacyCreate instructions
Survey administration
TimingResponse rates
![Page 17: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/17.jpg)
1/22/2015
17
Reflection: Share lessons and refine practices
Meet with the program teamPrepare report(s)Distill key findingsProvide recommendations/ suggestions/areas for consideration
Evaluator’s Toolkit
A logic model or a questions and sources gridA working knowledge of different methods
A network of people who can help you
Create an evaluation plan that…
Addresses stakeholders’ needsIs validContributes to your organizationCan be implemented and takes into account available resources
![Page 18: Evaluation for VISTAs - Literacy Minnesota · 2020-01-23 · 1/22/2015 1 Using sound evaluation practices to support your project’s success Evaluation for VISTAs By Sara McGarraugh](https://reader034.vdocuments.site/reader034/viewer/2022042409/5f24d6d35ede5900eb3836c5/html5/thumbnails/18.jpg)
1/22/2015
18
Questions or Comments?Sara McGarraugh
Continue the conversation
Or find us on Facebook and LinkedInThank you!