h210d session #3 evaluation 8 questions for evaluation planning 1.what is the program’s stage of...

H210D Session #3 Evaluation

8 Questions for Evaluation Planning

1. What is the program’s stage of development?

2. What is the program trying to do?

3. What information do funders expect?

4. Who is the program’s target population?

5. What does the program already know?

6. What does the program want to know?

7. What resources must be in place?

8. Who will conduct the evaluation?


Q1: What is the Age and Stage of Your Program?

New programs need time to mature before you assess some outcomes

Determine outcomes in stages:- year one, year two, year three- outcomes within program vs. in school vs.

in communityCollect implementation data before you

collect outcomes data


Q2: What is Your Program Trying To Do?

Decide on Desired Results/GoalsDetermine major conditions and causes

behind the resultDevelop program strategiesDevelop program activitiesDevelop performance measuresDecide on major indicators


Q3: What Information DoFunders Expect?

Work with funders to clarify expectations Build evaluation start up time into your grant

so you can do a needs assessment prior to program implementation

Be explicit about reasonable timeframes (age and stage again)

Negotiate separate funds for the evaluation to ensure devoted time and resources


Q4: Who is the Program’sTarget Population?

Implications for services

Implications for outcomes time frame

Implications for outcomes measures


Q5: What Do you Already Know?

Previous evaluation information Program brochures, mission statements,

annual reports Community needs assessments School and court records Program attendance records Program staff records Pre-existing data sets


Q6: What Do You Want to Know?

Why are you doing the evaluation?Are there pre-determined

evaluation requirements?Who are the evaluation

stakeholders?Will the information be useful to

programs?


Q6, cont’d: Determining the Right Evaluation Question(s)

In order to respond to participant needs, Ask• How can we better serve our families?• How do they view our program?

In order to understand the services you provide and who you provide them to, Ask

• Who are we serving?• How often?• What are the most popular activities?


Q6, cont’d: What Do After School Programs Want to Know?

Is my after school program responding to participant needs?

Who is participating in my after school program?

What is the staffing for my program? What training do they have and need?

What after school resources are available in my community? What are the service gaps?


Q6, cont’d: What Do After School Programs Want to Know?

What services does my after school program provide?

What is the impact of my after school program—academic achievement (e.g., increased school attendance and homework completion), youth development (e.g., increased positive school behavior), workforce development (e.g., employment preparation and support for families transitioning from welfare to work), prevention (e.g., reduced drug and alcohol abuse)?

What are the costs of my after school program?


Q7: What Resources Do You Need to Conduct the Evaluation?

FinancesStaffTrainingTechnology (MIS, hand helds, digital

cameras, scanners, etc.)


Q7: cont’d: What can influence costs?

Data-collection methodsNumber of sites includedLength of evaluationUse of an outside evaluatorAvailability of pre-existing data setsType of reports generated


Q8: Who Will Conduct Your Evaluation?

Determine skills needed to conduct the evaluation given what you want to learn

Assess staff ability and capacity to conduct evaluation (including time)

Consider external partnerships, especially if you are conducting an outcomes evaluation


Evaluating Afterschool Programs, Initiatives, and Systems

H210D

April 30, 2008


Goals for Class

• Familiarize you with HFRP resources

• Provide background on the changing evaluation context and HFRP’s approach to evaluation strategy

• Familiarize you with the 5-tier approach to learning and evaluation

• Familiarize you with logic models and related tools


The evolving role of evaluation

• Past assumptions about the role of evaluation in policy development

• The new world of research-based programs, outcomes and accountability

• Reframing evaluation to support learning,continuous improvement and accountability


An Ecological Model for OST Nonprofits


The Learning Loop

• Using research and evaluation to specify the theory of change, the indicators, and to support learning—outcomes and ongoing learning and adaptation capacity

• Embracing change

• Forces For Good and high-impact non-profits

• Learning Organizatkions


Figure 3. A Learning Cycle for OST Nonprofits


Developmental Evaluation

“long-term, partnering relationships between evaluators and those engaged in innovative initiatives and development”

(Patton)


Patton’s Developmental Evaluation

• The short life of findings

• Evaluation and thinking “evaluatively” for enduring impact


Developmental Processes

• Asking evaluative questions (logic models)

• Gathering information for discussion and to provide feedback and support developmental decision-making and course corrections


Garvin’s Learning Organizations

• Supportive Learning Environment

• Concrete Learning Processes and Practices

• Leadership that reinforces learning


Supportive Learning Environment

• Psychological safety

• Appreciation of Differences

• Openness to New Ideas

• Time for Reflection


Concrete Learning Processes and Practices

• Experimentation

• Information collection

• Analysis

• Education and training

• Information transfer


Leadership that Reinforces Learning

• Managers invite input, acknowledge their limits, ask probing questions, listen, encourage multiple views, provide time for identifying problems and organizational challenges, and provide time, resources and venues for reflecting on performance.


The Five-Tier Approach to Evaluation

• Tier 1---Pre implementation planning

• Tier 2---Early Accountability

• Tier 3---Program Clarification

• Tier 4---Progress toward Objectives

• Tier 5---Program Impact


Tier 1 Pre-implementation• Theory of change

• Needs assessment and match with theory


Tier 2 Early Accountability

• Document utilization

• Early implementation

• Build organizational learning capacity


Tier 3 Program Clarification

• Examine indicators of change

• Assess implications of results for program improvement

• Continue to build learning culture


Tier 4 Progress toward Objectives

• Examine progress on short-term outcomes and discuss implications for continuous improvement

• Examine outcomes in comparison with results from similar programs


Tier 5 Program Impact

• Develop and implement an outcome evaluation

• Learn from results

• Continue to support the organization’s learning and development


Classic TOC definition *

• A TOC identifies “program resources, program activities, and intended program outcomes, and specifies a chain of causal assumptions linking program resources, activities, intermediate outcomes, and ultimate goals (p. 78).”

Wholey, J. (1997). Evaluation: Promise and performance. Washington, DC: Urban Institute.

Other resource on TOC: Weiss, C. (1998). Evaluation: Methods for studying programs and policies. Saddle River, NJ: Prentice Hall.


What is a Theory of Change?

• Defines the building blocks required to bring about your long-term goal

• Describes the activities/strategies/interventions necessary to achieve your goal

• Articulates the assumptions that stakeholders have about the context/environment in which you are trying to affect change


Why create a theory of change?

• Engages stakeholders in program planning and gets them in agreement on expected outcomes

• Provides you with a clear and testable hypothesis of how change will occur

• Provides a blueprint for evaluation


Why create a theory of change?

• Forces you to be explicit about your assumptions and the logic behind your plans

• Provides a “reality check”…can I REALLY impact the goals I have set based on what I plan to do?

• Provides a level of detail you need for cost estimates

• Provides a good communications tool for funders, policymakers, and others


What is a Logic Model?

• A visual representation of your theory of change– It can take many forms…boxes, bubbles, text-

based.


PROGRAM GOALS

What is the program trying to accomplish?

What are the contextual factors to consider?

Who are the program stakeholders?

PROGRAM ELEMENTS/ACTIVITIES

What are the elements and activities to achieve program goals?

SHORT TERM OUTCOMES

What are the interim markers of progress toward the long-term goal?

LONG-TERM OUTCOMES

What are the ultimate outcomes of your after school program?

PERFORMANCE MEASURES AND DATA SOURCES

What will you measure?

What data sources will you use?

Elements of an After School Program Logic Model


Program Elements

• The day-to-day activities and strategies that your program uses to reach its goals

• Think about what you already do and what you would like to be doing


Factors that Influence Outcomes

• What is the age of the participants?

• How long has the program been in operation?

• What is “typical” program participation?

• What are you trying to achieve?

• How are you defining “short-” and “long-” term outcomes?


What are performance measures?

• Measures of effort (a.k.a. outputs)– Documenting what you are doing

• Measures of effect (a.k.a. outcomes)– What changes do I expect as a result of my

strategies and activities?


Articulate Program Vision

• To support overall child well-being through equal access to and participation in a quality after school program.


Describe Contextual Factors

• Program located in a low-income neighborhood where transportation and safety issues are paramount.

• Program has access to 21st CCLC funds.

• Program operates at a community center.

• Program serves children ages 11-14.


Articulate A Specific Goal

• To complement in-school learning with activities that support and promote cognitive development.


Set Specific Long Term Outcomes

• Reduction in grade retention.

• Improved school performance.

• Increased participation in a range of out-of-school activities.


Set short term outcomes/outputs that will help mark your progress

• Increased participation in the after school program.

• Improved study habits.

• Improved attitudes toward school.

• Stronger relationships between the school and the after school program.


Brainstorm Program Elements• Promote positive relationships between staff and youth.

• Use a range of enrollment strategies to reach out to youth.

• Ensure safe passage from school to after school.

• Create linkages with school day content to support learning after school.


Consider Contextual Issues Throughout the Process

• Who should be involved in program planning?• What resources are already available after school?• What partnerships will help you achieve your

goals?• What community factors may support/inhibit your

program?• Who are you trying to reach?• What are the family backgrounds of the

participants?


What is “Participation”

Participation =

Enrollment +Engagement

+Attendance


Enrollment

• Getting youth in the door

• Contextual/external predictors

• Program recruitment strategies

• Program implementation


Attendance

• Participating vs. Not Participating• Intensity (a.k.a. “dosage”: number of

hours per week/month)• Duration (history or participation)• Breadth (number and variety of

activities within and across programs)


Engagement

• More than just “being there”• Behaviors such as persistence, effort, and

attention• Emotions such as enthusiasm, interest, and

pride in success• Motivation• Active cognitive involvement• Quality is key to engagement


How to Measure Attendance

• Participating vs. Not Participating• Intensity: amount of time youth participate in an

activity

• Duration: history of attendance

• Breadth: variety of participation within or across programs


Intensity and Outcomes• Academic outcomes

– GPA, test scores– Homework completion– Feelings about school, goals and educational aspirations– College attendance– High-school completion

• Social outcomes– Lower problem behavior– Higher community service– Better emotional well-being (e.g., happiness)

• Participation Outcomes– Higher intensity in elementary school is associated with higher

attendance in middle and high school


Duration and Outcomes

• Duration is positively related to children’s outcomes

• 4-H evaluation – Participants > Nonparticipants

– Participation for more than 1 year > Less than 1 year• e.g., better at communicating, more successful at resolving conflict,

had higher grades– (Rodriguez, Hirschl, Mead, & Goggin, 1999)


Breadth and Outcomes

• Few studies on breadth

• Breadth within a program– Participated in 3 or more activities > 1-2 activities, non

participants• Grades and academic test scores

– (Baker & Witt, 1996)


Combining IndicatorsSan Francisco Beacons

Combined– Duration: number of sessions—fall, winter, spring

– Breadth: educational activities, other activities, or educational and other activities

Results: 3 sessions plus educational and other activities lead to:

• Increases in leadership, non-family support for participants, school effort, and sense of efficacy

3 sessions plus educational only activities lead to:• Increases in school effort only


Combining IndicatorsTASC Evaluation

Combined • Duration: 1 or 2+ years

• Intensity: non-participator (0 days per week), non-active participator (less than 3 days), active participator (3 or more days)

Results: Participation for at least two years on a regular basis lead to:

– Improved math test scores pre-K to grade 8

– Improved attendance grades 5-8


Measuring Attendance Matters

• Program planning and management

• Individualized programming

• Accountability/ADA requirements

• Gauge demand for program services

• Professional development planning

• Interpreting participant outcomes

h210d session #3 evaluation 8 questions for evaluation planning 1.what is the program’s stage of...

Documents