Program Evaluation

EDL 832Jeffrey Oescher, Instructor

6 June 2013

Introduction

Objectives Describe the course Understand why evaluation is important Define evaluation Differentiate evaluation and research Discuss the jargon of evaluation

Who Is He?

Jeffrey Oescher Background Current responsibilities Experience as an evaluator Important things

Course Description

Catalogue description Rationale and goals Course organization Required text Course assignments Class schedule Important dates Moodle acknowledgment

The Need for Evaluation

Why evaluate programs?

The Need for Evaluation

What programs are currently being used in your school? Which ones are working well? Poorly? Are different parts of a program working better than

others? What can be done to improve those parts that are not contributing effectively?

What are the relative costs and benefits of these programs?

Have these programs been planned carefully? Is more planning needed?

What adaptations might make the program more effective?

Why is a program effective?

Evaluation Defined

OK, so there is a need to evaluate, but what does that term mean?

How would you define the term evaluation?

Evaluation Defined

There are many definitions of this term even among experts in the field.

The identification, clarification, and application of defensible criteria to determine an evaluation object’s value (i.e., worth and merit) in relation to those criteria. Fitzpatrick, Sanders, and Worthen, 2011

Evaluation Defined

Evaluation uses inquiry and judgment to determine standards for judging quality

relatively or absolutely; collect relevant information; apply the standards to determine value,

quality, utility, effectiveness, or significance; and

offer recommendations intended to optimize the evaluation object in relation to its purpose or help stakeholders make decisions.

Evaluation and Research

What do you see as the difference(s) between evaluation and research?

Evaluation and Research

Purpose Research

Add knowledge to the field through the development or validation of theory

Establish cause and effect relationships Seek conclusions

Evaluation Help those who hold a stake in whatever is

being evaluated make decisions Describe aspects of the evaluation object Seek judgments

Evaluation and Research

Setting the agenda Research

The researcher sets the agenda through the choice of phenomena or hypotheses being investigated

Evaluation The stakeholders set the agenda by

determining the focus of the evaluation

Evaluation and Research

Standards used to judge adequacy Research

Validity Internal External

Reliability Evaluation

Accuracy – information is an accurate reflection of reality

Utility – the extent to which the results serve the information needs of the stakeholders

Feasibility – the extent to which the evaluation is realistic, diplomatic, prudent, etc.

Propriety – the extent to which the evaluation is done legally and ethically

Evaluation and Research

Preparation Research

Researcher’s training is focused on methodology within the specific discipline of study

Evaluation Evaluator’s training and expertise is inter-

disciplinary

Roles and Activities

Roles Researcher

Conduct the research according to plan Evaluator

Advocate for the use of evaluation Interact with stakeholders

Critical friend to help stakeholders learn Help develop critical judgment Help develop organizational learning Stimulate dialogue among various groups

Roles and Activities

Activities Researcher

Conduct the research (i.e., sample; structure the study; and collect, analyze, and interpret data)

Evaluator Negotiate with stakeholder groups to define the purpose

of an evaluation Develop contracts Hire and oversee staff Manage budgets Work with advisory groups Collect and analyze data Communicate with stakeholders Write reports Disseminate information

The Jargon of Evaluation

Formative The primary purpose is program

improvement Summative

The primary purpose is to provide information to help make decisions about program adoption, continuation, or expansion

The Jargon of Evaluation

Identify a specific situation where you would conduct a formative evaluation? A summative evaluation?

How do you see formative and summative evaluation interacting in the evaluation of a program in your school?

See Figure 1.1, p. 24

The Jargon of Evaluation

Needs assessment Establish whether a problem or need exists

and, if so, describe that need or problem as well as make recommendations for ways to reduce the need or correct the problem

Process Describe how a program is delivered

Outcome Describing, exploring, or determining changes

that occur in program participants See Table 1.3, p. 27

The Jargon of Evaluation

Internal evaluations Conducted by program employees

External evaluations Conducted by outsiders

Advantages of either approach See Figure 1.2, p. 31

Using the Jargon

An internal, formative evaluation An external, formative evaluation An internal, summative evaluation An external, summative evaluation

Where to Now?

Course Review the materials provided for you

The text Chapter 1

Review this material in light of our discussion tonight Chapter 2

Read this chapter to appreciate the history of evaluation so you can understand the current context in which evaluators operate

Chapter 3 Read this chapter to become familiar with some of the

issues that can become problematic when conducting an evaluation

Goal 1 Goal 2