outcome measurement kaitlyn wark program evaluation officer the community foundation michael mallett...
TRANSCRIPT
Outcome Measurement
Kaitlyn WarkProgram Evaluation OfficerThe Community Foundation
Michael MallettResearch & Evaluation Director
United Way of Greater Richmond & Petersburg
Program Evaluation: Awesome or Evil?
1. Why is evaluation scary?
2. Why is it important?
Why Evaluate?
To better serve children in Richmond Public Schools.
• Learning and improvement• Accountability
FAQsWhen will RPS use your evaluation data to make decisions on partners?
• RPS expects to see evaluation data and use those in decision making for the 2016-17 school year.
Do we have to measure grades? Is RPS only interested in grades?
• You should measure what you hope to accomplish with your participants (students, staff, etc.). For example, if your program is teaching healthy eating habits, you should measure daily vegetable intake.
What is the baseline expectation?• To measure the program or service’s outcome indicators. (You have an
evaluation plan developed and implemented).
What if I need help?• You’ll leave today with a wealth of resources both in the partners present and
this presentation with links to other resources will be emailed and on the Community Partnerships website.
Today’s Training Goals
1. Identify your program’s long term outcome
2. Identify a few potential outcome indicators for your program
3. Identify possible measurement strategies
4. Identify relevant measurement resources
Training Goal #1:
Identify your program’s long-term outcome
Outcome Definition
The measurable changes your program produces for participant:
–Knowledge–Behavior–Skills–Attitudes –Condition/Resources
Outcome Definition
Long-term outcomes–Capture enduring changes –Usually measured a year to multiple
years after the program–Usually more difficult to measure than
short-term or intermediate outcomes
Criteria for Success
Puts focus on the client
Answers the “so what?” question
Is related to your program activities – is something program can influence
Is important to your staff + outside stakeholders
Developing Long-Term Outcomes
Who? Will Do What?(an active verb)
What is the desired outcome?
Clients
Students
Teachers
Demonstrate
Improve
Obtain
Maintain
Display
Grades in schoolTeaching Skills
Social/ Emotional SkillsLeadership Skills
Health Behaviors
Long-Term Outcome Examples
“Students will demonstrate emotional intelligence and positive communication
skills.”
“Teachers will successfully employ positive behavior support interventions.”
Outcome?
“Students are satisfied with the program.”
Puts focus on the client
Answers the “so what?” question
Is related to your program activities – is something program can influence
Is important to your staff + outside stakeholders
Outcome?
“Students will increase their performance on end-of-course exams and will be prepared to advance to the next grade.”
Puts focus on the client
Answers the “so what?” question
Is related to your program activities – is something program can influence
Is important to your staff + outside stakeholders
Outcome?
“Students will attend 95% of program days.”
Puts focus on the client
Answers the “so what?” question
Is related to your program activities – is something program can influence
Is important to your staff + outside stakeholders
Developing Long-Term Outcomes
? What are you trying to achieve with your clients?
? What does success look like, for your clients?
? If you are successful, how will your clients be different after the program than before?
? What kinds of changes do you want to see?
Adapted from Patton: Utilization-focused Evaluation
Helpful Questions: Vision and Goal Setting
Developing Long-Term Outcomes
? Can your program really influence the outcome in a meaningful way?
? Is this outcome truly important to you?
? Is this outcome important and valid to others?
? Is the outcome written using language that most people will understand?
? Are there unintended consequences?
Helpful Questions: Importance
Think – Pair – Share
1. Spend 2 minutes crafting a long-term outcome for your program.
2. Share your long-term outcome with your neighbors.
3. Give/receive feedback4. Choose one long-term outcome to share
with the rest of the audience.
Training Goal #2:
Identify potential outcome indicators for your program
Define Outcome Indicators
• Specific information or data that track your progress towards your long-term outcome goals.
• They describe observable, measurable characteristics or changes that indicate achievement of an outcome.
Outcome Indicators
Observable What does the outcome LOOK like? What can you SEE that is different?
Measurable What can you count, weigh, measure?
Specific, Clear, Not Ambiguous
Client Focused (generally)
Quantitative Number and percent of clients who…
Define Outcome Indicators
Long-term Outcome: Students pursue a healthy lifestyle
Outcome indicators: # and % who report not smoking at end of
program # and % maintaining a BMI within a healthy range # and % who report getting at least two hours of
exercise each week
Researching Indicators
• Validity– Are you measuring what you think you’re
measuring?• Fidelity
– Use research as a guide– Are you implementing best practices from
evidence-based programs?
Outcome Indicator Examples
Type of Program Outcome Indicators
Tutoring Program Students will improve their academic performance and graduate high school on time.
# and % of students either improve by a letter grade or maintain a B or above
# and % of students who graduate within four years of entering high school
Professional Development for Teachers
Teachers will gain skills in classroom management and create a safe environment for learning
# and % of teachers who scored 80% or higher on the training post-test
# and % of teachers who report using scaffolded checks for understanding at least once per lesson
# and % of students who rated their mentor relationship as good or excellent
Average length of match
Mentoring Students will develop healthy relationships with adults and improve their social skills
Group Exercise
1. Spend 5 minutes:• Identifying 2 – 5 outcome indicators
for your program.
2. Share your outcome indicators with your neighbors.
3. Give/receive feedback, and fine-tune your indicators
Discussion
• What problems and issues did you run into trying to create your outcome indicators?
• What are your next steps?
Training Goal #3
Identify possible measurement strategies
Measurement & Data CollectionUse existing data sources if possible.
Possible data sources:Existing files or databases
Intake and exit records Case notes Follow-up calls and notes
Assessments or tests (validated if possible) Harvard Family Research Project OST Evaluation Database
http://www.hfrp.org/out-of-school-time/ost-database-bibliography/database
National clearinghouses/research organizations
Surveys (validated if possible) Participants Staff Teachers Family members
Measurement and Data Collection
Make a Plan! When will data be collected?
• When entering the program?
• When completing the program?
• Fixed interval after entering?
• Fixed interval after completing?
Who will collect the data? Who will analyze it? Where will the data be stored? A database? How will data quality be assured?
Where are you now?
1 • Anecdotal evidence• Client Testimonies• Success Stories
2 • Collecting mostly output data (number of participants)• Not using data in decision-making and improvement plans
3 • Occasionally monitors program performance• Occasionally monitors program indicators• No theory of change and/or logic model
4• Continuous improvement mindset• Regularly monitors program performance • Regularly monitors outcome indicators• Fully developed logic model and evaluation plan
Training Goal #4
Identify relevant measurement resources
Recommended Measures: Attendance
• United Way Pittsburgh Presentation - OST attendance vs. School Attendance
• HFRP - OST program attendance• RCGD - OST program attendance
Recommended Measures: Attendance
• Program Attendance– Intensity/Dosage : hrs per day, days per week– Duration: 8 months, school year, 2 years– Breadth: How many activities per student?
• School Attendance– # days not present at school (no need to
differentiate b/w excused, unexcused_– # and % of students missing <10 days
Recommended Measures: SEL
• http://www.hfrp.org/out-of-school-time/publications-resources/a-review-of-out-of-school-time-program-quasi-experimental-and-experimental-evaluation-results
• Rhodes, Jean E. "The critical ingredient: Caring youth staff relationships in after school ‐ ‐settings." New Directions for Youth Development 2004, no. 101 (2004): 145-161.
Recommended Measures: SEL
• http://character.org/more-resources/assessment-tools/individual/– # and % of students/participants who report
_______________• Attitudes towards school• Social Skills• Delinquency
Recommended Measures: Behavior
• Durlak, Joseph A., and Roger P. Weissberg. "The Impact of After-School Programs that Promote Personal and Social Skills." Collaborative for Academic, Social, and Emotional Learning (NJ1) (2007).
• Kremer, Kristen P., Brandy R. Maynard, Joshua R. Polanin, Michael G. Vaughn, and Christine M. Sarteschi. "Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: a systematic review and meta-analysis." Journal of youth and adolescence 44, no. 3 (2014): 616-636.
• Gottfredson, Denise C., Stephanie A. Gerstenblith, David A. Soulé, Shannon C. Womer, and Shaoli Lu. "Do after school programs reduce delinquency?."Prevention Science 5, no. 4 (2004): 253-266.
• Cross, Amanda Brown, Denise C. Gottfredson, Denise M. Wilson, Melissa Rorie, and Nadine Connell. "The impact of after school programs on the routine activities of middle school ‐ ‐students: Results from a randomized, controlled trial*." Criminology & Public Policy 8, no. 2 (2009): 391-412.
Social Skills and Positive Relationships (includes sample surveys)• https://www.google.com/url?sa=t&rct=j&q=&
esrc=s&source=web&cd=9&ved=0CGQQFjAIahUKEwiihrzS-pnIAhXIGY4KHWnQA-0&url=http%3A%2F%2Fwww.uwex.edu%2Fces%2F4h%2Fevaluation%2Fdocuments%2FChildYouthOutcomeHandbook2005.pdf&usg=AFQjCNEqamaqae3NRJzOYu-M3nG_Ar9OUw&sig2=BjvFCNUHy5BRp5Xjc6Oxuw&bvm=bv.103388427,d.c2E&cad=rja
Recommended Measures: Behavior
• # and % of students who decreased detentions
• # and % of students who decreased suspensions
• Average decrease in suspensions for students with at least one suspension.
• # and % of students completing all HW assignments
Recommended Measures: Achievement
• Patricia A. Lauer, Motoko Akiba, Stephanie B. Wilkerson, Helen S. Apthorp, David Snow and Mya L. Martin-Glenn. Review of Educational Research. Vol. 76, No. 2 (Summer, 2006), pp. 275-313
Recommended Measures: Achievement
• # and % of students promoted to the next grade
• # and % of students improving by at least one letter grade or maintaining an A or B.
• # and % of students meeting targets on MAP assessments
Recommended Measures: Family Engagement
• http://www.hfrp.org/publications-resources/publications-series/issues-and-opportunities-in-out-of-school-time-evaluation/beyond-the-head-count-evaluating-family-involvement-in-out-of-school-time
• Durlak, Joseph A., Roger P. Weissberg, and Molly Pachan. "A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents." American journal of community psychology 45, no. 3-4 (2010): 294-309.
• Durlak, Joseph A., and Roger P. Weissberg. "The Impact of After-School Programs that Promote Personal and Social Skills." Collaborative for Academic, Social, and Emotional Learning (NJ1) (2007).
• Hanlon, Thomas E., Betsy D. Simon, Kevin E. O'Grady, Steven B. Carswell, and Jason M. Callaman. "The effectiveness of an after-school program targeting urban African American youth." Education and urban society 42, no. 1 (2009): 96-118.
Recommended Measures: Family Engagement
• # and % of parents taking a leadership role within the program
• Parental expectations for children’s future educational attainment
• # and % of parents satisfied with program quality and content
• # and % of parents reporting increased involvement in child’s school as a result of program
Recommended Measures: PD
• http://www.hfrp.org/publications-resources/publications-series/issues-and-opportunities-in-out-of-school-time-evaluation/promoting-quality-through-professional-development-a-framework-for-evaluation
• # and % of teachers demonstrating changes in attitudes, behaviors, or knowledge covered by PD course content
Outcome Measurement Resources
Forum for Youth Investment – www.forumfyi.org Measuring Youth Program Outcomes– examines multiple assessment tools for youth programshttp://forumfyi.org/files/soft_skills_hard_data_single.pdf
A Shared Vision for Youth Common Outcomes and Indicators – http://www.readyby21.org/resources/report/shared-vision-youth-common-outcomes-and-indicators
Harvard Family Research Project – www.hfrp.orgPublications and Resources on Evaluation
http://www.hfrp.org/evaluation/publications-resources
Out-of-School Time (OST) Program Research and Evaluation Database – searchable database of OST programs and the evaluations they usehttp://www.hfrp.org/out-of-school-time/ost-database-bibliography/database
Logic Models for OST Programs – guide to developing a logic model http://www.hfrp.org/out-of-school-time/publications-resources/learning-from-logic-models-in-out-of-school-time
The Colorado Trust – www.coloradotrust.org Toolkit for Evaluating Positive Youth Development
http://www.coloradotrust.org/sites/default/files/ASIToolkitJun04.pdf
Outcome Measurement Resources
University of Wisconsin - Extension – http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html Creating a Logic Model Online Course - http://www.uwex.edu/ces/lmcourse/#
Child Trends – www.childtrends.orgPositive Indicators Project - http://www.childtrends.org/our-research/positive-indicators/positive-indicators-project/
California Evidence-Based Clearinghouse - http://www.cebc4cw.org/assessment-tools/ Links to measurement tools
Outcome Measurement Resources
PerformWell – (Urban Institute, Child Trends, and Social Solutions) www.performwell.org Database of measurement tools and Outcomes
Child Welfare Information Gateway (USDHHS)www.ChildWelfare.gov Links to measurement toolshttps://www.childwelfare.gov/topics/systemwide/youth/outcomes/tools/
Toolfind – (United Way of Massachusetts Bay and Merrimack Valley) www.Toolfind.org Database of 36 measurement tools for youth programs