understanding program successes and challenges: program ... · understanding program successes and...
TRANSCRIPT
Understanding Program Successes and
Challenges: Program Evaluation, Data
Tools and Resources
Washington State Mentors Conference
Hilary Loeb & Kelly Bay
August 23, 2012
1
Presentation Overview
Welcome and Presentation Goals
Resources to plan data collection and analysis
Case study activity
Key data sources for informing mentoring programs
Online demonstration
2
CSF Mission
10 years of proven experience, inspiring
underserved, low-income students to
finish high school, and providing the
unique integrated system of supports and
scholarships they need to graduate
college and succeed in life.
3
The Role of the Internal Evaluator
4
Volkov, B.B. (2011). Beyond being an evaluator: The multiplicity of roles of the internal evaluator. In B.B.
Volkov & M.E. Baron (Eds.), Internal evaluation in the 21st century. New Directions for Evaluation, 132, 38.
Resources to
Plan Data
Collection and
Analysis
5
Why Should We Evaluate Our Work?
Produces programmatic self-understanding
Allows you to demonstrate program outcomes
Substantiates requests for increased funding by providing evidence of effectiveness
Identifies and leverages program strengths
Source: The Pell Institute for the Study of Opportunity in Higher Education, the Institute for Higher Education Policy, and Pathways to College Network, 2011 http://toolkit.pellinstitute.org/evaluation-101/evaluation-myths-benefits/
6
Why Should We Evaluate Our Work?
Identifies and modifies ineffective
practices
Provides documentation for
performance/funding reports
Creates a foundation for strategic
planning
Produces credibility and visibility Source: The Pell Institute for the Study of Opportunity in Higher Education, the
Institute for Higher Education Policy, and Pathways to College Network, 2011
http://toolkit.pellinstitute.org/evaluation-101/evaluation-myths-benefits/
7
What Kinds of Evaluation Questions are
Relevant to Mentoring Programs? Implementation
How did our recruitment, training, matching and oversight vary across different sites?
Did we use our resources effectively?
Improvement What changes might we recommend to better serve
our students?
Impact How did the students and mentors benefit from
participation in the program?
How did the communities of our implementation sites change?
8
What Are Tools To Evaluate Our Work? Evaluation planning tools
Logic models
Theories of change
Outcome maps
Internal data collection
External data sources
9
What Is a Logic Model?
Why Should We Use It?
“ A logic model is a systematic and visual
way to present and share your
understanding of the relationships among
the resources you have to operate your
program, the activities you plan, and the
changes or results you hope to achieve.”
(W.K. Kellogg Foundation 2004)
10
What Is a Logic Model?
Why Should We Use It?
Communicating about a program in a
clear, concise way
Creating and revisiting a program
plan
Planning a program evaluation
11
What Are the Elements of A Logic
Model?
12
Source: W.K.
Kellogg
Foundation
Evaluation
Handbook
2004
13
GirlPOWER! Logic Model
http://www.mentoring.org/downloads/mentoring_624.pdf
How Can We Prepare a Logic Model? Assemble a dynamic team
Key constituents are necessary to gain a clearer understanding of your program goals, assets, and commitments
Include individuals committed to the goals of the organization and are willing to contribute over time
Disseminate key facts prior to the meeting Determine the basic information about the program
that every team member needs to know prior to the meeting
It is important that everyone “ starts in the same place” to enable the voices of multiple constituents
14
Source: Adapted from The Pell Institute Evaluation Toolkit, The Pell
Institute for the Study of Opportunity in Higher Education, the Institute
for Higher Education Policy, and Pathways to College Network, 2011
How Can We Prepare a Logic Model? Determine a method for maintaining good notes
Have a solid record of your meetings so that the evolution of particular ideas can be traced
A note taker should be determined prior to the logic model development meetings and ideally should be consistent
Create a time frame for completing the initial and ongoing review process Consider conducting a series of short term on or
off site “ retreat” meetings with the project team
The decision to schedule meetings is likely to depend on time constraints and your organization’s culture
15
Source: Adapted from The Pell Institute Evaluation Toolkit, The Pell
Institute for the Study of Opportunity in Higher Education, the Institute
for Higher Education Policy, and Pathways to College Network, 2011
Logic Model Brainstorm You are working with a group of stakeholders
to develop a logic model for a mentoring
program for recent immigrant youth
Review the Research Corner: Mentoring
Immigrant Youth (also found at
http://www.mentoring.org/downloads/mentoring_1318.pdf)
What critical program elements would you
include in the logic model in the next slide?
16
Logic Model Brainstorm
17
Resources Activities Outputs Short and Long Term Outcomes
Impact
In order to accomplish our set of activities we will need the following:
In order to address our problem or asset we will conduct the following activities:
We expect that once accomplished these activities will produce the following evidence or service delivery:
We expect that if accomplished these activities will lead to the following changes in 1–3 then 4–6 years:
We expect that if accomplished these activities will lead to the following changes in 7–10 years:
What Tools Do Evaluators Use?
Secondary data
Surveys
18
What Tools Do Evaluators Use?
Interviews
Focus groups
Observations
Other scholars’ research
19
From:
http://www.mentoring
.org/news_and_resea
rch/research_and_st
udies/research_categ
ories/#formal
Key Data Sources for
Informing Mentoring
Programs
20
Quality of Relationship Survey Questions
21
Midcourse Corrections / Post Assessment Tool
National Mentoring Center: http://educationnorthwest.org/webfm_send/212
19 Questions:
1. Youth-
centered?
2. Youth’s
emotional
engagement?
3. Youth’s
satisfaction
with
relationship?
Youth Outcomes Survey
22
Pre and Post Assessment
Developed by Big Brothers Big Sisters of America
http://oregonmentors.org/library/evaluationtools/view/7/
10-Minute Survey
Self-Esteem/ Efficacy
School & Academics
Personal Attitudes & Beliefs
Goals & Future
Family Relationships
Drug & Alcohol Use/Attitudes
Delinquency & Antisocial
Behavior
Mentor Evaluation Form
23
Post Assessment
http://www.mentoring.org/program_resources/elements_and_toolkits/tool_kit/eval/
Data Collection Brainstorm Review our immigrant mentoring program
logic model
What types of data would you plan to
collect to track the implementation and
success of the program?
24
Logic Model: What Types of Data
Might We Collect and Analyze?
25
Resources Activities Outputs Short and
Long Term
Outcomes
Impact
In order to accomplish
our set of activities we
will need the
following:
In order to address
our problem or
asset we will conduct
the following
activities:
We expect that once
accomplished these
activities will produce
the following evidence
or service delivery:
We expect that if
accomplished
these activities
will lead to the
following
changes in 1–3 then
4–6
years:
We expect that if
accomplished
these activities
will lead to the
following
changes in 7–10
years:
State and Community-Level Data:
KIDS COUNT Data Center
26
State and Community-Level Data: OSPI Report Card
27
State and Community-Level Data: College Tracking Data Services
28
State and Community-Level Data: Ask Healthy Youth Survey Query Builder
29
Concluding Thoughts
Evaluation is an ongoing learning process best completed in collaboration with stakeholders.
Evaluation is a continuous improvement process that draws from shared review and reflection about data.
"Not everything that can be counted counts, and not everything that counts can be counted." Albert Einstein
30
Online
Demonstration
and Resources
31
Other Resources Supporting
Youth Mentoring – Washington
Washington State Mentors Resources: http://www.wamentors.org/resources
College Tracking http://www.collegetracking.com/
OSPI School Report Card and online data http://reportcard.ospi.k12.wa.us/DataDownload.aspx
Healthy Youth Survey http://www.askhys.net/layout.asp?page=nanalyzer
KIDS COUNT Data Center http://datacenter.kidscount.org/WA
32
Other Resources Supporting
Youth Mentoring – Washington
Office of Financial Management’s Education Research & Data Center & Washington State Databook http://www.erdc.wa.gov/; http://www.ofm.wa.gov/databook/
Washington State Achievement Council (formerly HEC Board): http://www.wsac.wa.gov/
State Board of Community & Technical Colleges http://www.sbctc.edu/
Your school district data director or college or university institutional research office
33
Other Resources Supporting
Youth Mentoring – National National Mentoring Association Resources:
http://www.mentoring.org/program_resources
National Mentoring Association Evaluation Toolkit:
http://www.mentoring.org/program_resources/elements_and_toolkits/tool_kit/eval/
School Data Direct (Chief of State School Officers) http://www.schooldatadirect.org/
NCES Condition of Education http://nces.ed.gov/programs/coe/
NCHEMS Information Center http://www.higheredinfo.org/analyses/newmeasures.php
High School, Community College, and National Student Engagement Surveys http://ceep.indiana.edu/hssse/index.htm; http://www.ccsse.org/; http://nsse.iub.edu/
34
Other Resources Supporting
Youth Mentoring – National National Governors’ Association Center for Best Practices
http://nga.org/cms/center
American FactFinder
http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml
National Survey of Children’s Health
http://www.nschdata.org/Content/Default.aspx
Centers for Disease Control Behavioral Risk Factor
Surveillance System http://www.cdc.gov/brfss/
Grapevine (Data on state fiscal support for higher education)
http://grapevine.illinoisstate.edu/
Public Agenda (Portal for educators)
http://www.publicagenda.org/educators
35
Resources to Learn More about
Evaluation Planning Tools
Pell Institute Evaluation Toolkit http://toolkit.pellinstitute.org/
Organizational Research Services http://www.organizationalresearch.com/publications_and_resources_featured.aspx
“Getting Started: A Self-Administered Guide to Outcome Map Development” 2009
“Orientation to Theory of Change” 2008
“How to Manage and Analyze Data for Outcome-based Evaluation” 2000
Kellogg Foundation Evaluation Handbook, 2004 http://www.wkkf.org/knowledge-
center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx
Logic Model Development Guide, 2010 http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx
36