prt 595 week 2 lecture 2

Evaluation Context & QuestionsPRT 595

Dr. Bob Barcelona

PRT 595: Program Evaluation

04/12/23 2Footer

Evaluation Context

There is no “one size fits all” approach to evaluationGood evaluations must fit the circumstances

Perspectives of stakeholdersPurpose of evaluation effortsKey guiding questionsPractical yet rigorous social science methods

PRT 595: Program Evaluation

04/12/23 3Footer

Who Determines Purpose & Questions?Program stakeholders (e.g. evaluation sponsor, funders, program management)Expect different perspectives (or conflict!) from different stakeholdersHow do evaluators deal with this?

Incorporate conflicting perspectives within the evaluation designFocus on only one perspectiveFacilitate negotiation of perspectives – build shared understanding

Balance stakeholder interests – see the case study by Morris (2003) and the commentary by Leviton (2003)

PRT 595: Program Evaluation

04/12/23 4Footer

Involving Stakeholders

Identify who your stakeholders areEstablish a structure for stakeholder involvementInvolve them earlyInvolve them continuouslyInvolve them actively

Reineke (1991)

PRT 595: Program Evaluation

04/12/23 5Footer

Questions for Stakeholders

Why is an evaluation needed? What is the evaluation’s purpose? What are the program’s goals and objectives?What are the important questions for the evaluation to answer? Are questions tied to a program’s theory or logic model?

PRT 595: Program Evaluation

04/12/23 6Footer

What’s the Purpose?Formative Evaluation: providing information that helps to guide program improvement

Summative Evaluation: providing information that helps to determine whether program resources are spent efficiently and effectively, and whether programs produce intended benefits

Knowledge Generation: providing information to describe the nature and effects of an intervention as a contribution to knowledge

PRT 595: Program Evaluation

04/12/23 7Footer

Purpose Statement Example

The purpose of this evaluation is to understand the impact of housing relocation on the quality of life, well-being, and economic self sufficiency of former and current residents of a severely distressed public housing development in Greenville, SC.

PRT 595: Program Evaluation

04/12/23 8Footer

Good Evaluation Questions

Help to focus the evaluationFacilitate the design of data collection proceduresMust be reasonable and appropriate – they are tied to the goals and objectives of the programMust be answerable – specific, concrete, practical, and measurableSet criteria for performance (i.e. 75% of children… a majority of residents…)

PRT 595: Program Evaluation

04/12/23 9Footer

Seventy-five percent (75%) of regular attendees will improve levels of fitness relative to the previous year by participating in health enhancing physical and recreational activities each day75% or more of the regular attendees will achieve at least 60 minutes of moderate or vigorous physical activity each day

Selected evaluation criteria, GoalPOST: Goal Oriented Performance in Out-of-School-Time Program, Clemson University

PRT 595: Program Evaluation

04/12/23 10Footer

Types of Evaluation Questions

Needs Assessment

Assessment of Program Process

Assessment of Program Theory

Impact Assessment

Efficiency Assessment

PRT 595: Program Evaluation

04/12/23 11Footer

Answering QuestionsSound social science methodology!Some examples (HFRP, 2004):1. Surveys and Questionnaires2. Interviews and Focus Groups3. Observations4. Tests and Assessments5. Document Reviews and Analysis6. Secondary Sources and Data Reviews

Each has strengths and weaknessesAdvantages of using multiple methods

PRT 595: Program Evaluation

04/12/23 12Footer

In Sum…Good Evaluations….Have defined purposesAccurately identify and involve stakeholdersAsk clear, relevant, and answerable questionsAre focused on assessing program need, developing program theory, assessing program process, assessing program impact, or assessing program efficiency Use practical and rigorous social science methodology and data collection techniques to answer questionsAre used by stakeholders in some tangible way