program evaluation in a nutshell 1 jonathan brown, m.a

Program Evaluation In A Nutshell

1

Jonathan Brown, M.A.

Two Types of Evaluation

• At the most basic level there are two types of evaluation:

Formative: primary purpose is to provide information to improve a program.

Summative: concerned with providing information about program adoption, continuation, or expansion.

2

Two Types of Evaluation

3

Rel

ativ

e E

mp

has

is

Program Life

Summative Evaluation

Formative Evaluation

What Are You Assessing?

The evaluation plan must consider the situation or context of the evaluation.

• The purpose of the evaluation.

• Improvement (formative evaluation).

• Accountability (summative evaluation).

• Knowledge generation.

• Hidden agendas.

4



• The program’s structure and context.

• The stage of program development.

• Administrative and political context.

• Conceptual and organizational structure of the program.

5



• The resources available for the evaluation.

• Personnel

• Equipment

• Facilities to support data collection, analyses, and reporting.

• Is specialised expertise needed?

6


Typically, evaluation involves assessing one or more of the five following areas:

1. The need for a program.

2. The program’s design.

3. The program’s implementation or process.

4. The program’s impacts or outcomes.

5. The efficiency of a program.

7

Needs Assessment

• Purpose of a social program is to alleviate a social problem.

• Needs assessments assess:

• The nature, extent, and distribution of a social problem.

• How these features will impact the design of an intervention program.

8

Needs Assessment

These assessments can be conducted when…

• Planning a new program.

• Restructuring an existing program.

• Determining if an existing program is responsive to the current needs of the target population.

9

Assessing Program Theory

• Program theory evaluation is an evaluation guided by a program’s explicit theory of how it causes intended outcomes.

• Program theory outlines the following:

•Target population

•Resources

•Activities

•Outcomes

• A program’s theory is represented by Theory of Change Models (TCMs) and Program Logic Models (PLMs).

10


11

Conditions before the project

• With no information about the program, it remains a mysterious black box.

• The evaluation will also not be able to attribute the final conditions to the specific aspects of the program.

Conditions after the project

The Program is a


• Assessing program theory can be done by applying SMART principles (Knowlton & Phillips, 2009).

• Specific – the program is clear enough to implement and evaluate.

• Measurable – indicators can be qualified and/or quantified.

• Action oriented – activities will provoke the desired change in targets.

• Realistic – the program is plausible and feasible.

• Timed – duration of activities and intended outcomes are specified.

12

Process Evaluation

• Given that a valid theory on how to intervene on a diagnosed social problem has been done, the program must be implemented well to be successful.

• Two forms:

• Process evaluation

• Program monitoring

13

Process Evaluation

• Process evaluation looks at service utilization and program organization.

Service Utilization

• The extent to which the intended targets receive services.

• Important when participation is voluntary, or participants must learn new procedures, or habits.

• Coverage

• Bias

14

Process Evaluation

• Process evaluation looks at service utilization and program organization.

Program Organization

• How well the program is in terms of managing its efforts and using its resources to accomplish its activities.

• Service delivery:

• None or not enough intervention

• Wrong intervention

• Unstandardized intervention15

Outcome Evaluation

• Also known as an impact assessment.

• Assesses the extent to which a program produces the outcomes it intends.

• But the desired changes could be caused by factors unrelated to the program.

• Interested in the changes produced by the program above and beyond any external elements.

16

Outcome Evaluation

Program outcomes

• The states or conditions of the target that changed as a

result of a program.

1. Observed characteristics of the target population or social

conditions.

2. Occur to different degrees.

17

Outcome Evaluation

Program outcomes

• Outcome level – status of an outcome at a point in time.

• Outcome change – difference between outcome levels at different points in time.

• Program effect – the unique portion of an outcome that can be attributed to only the program.

18

Outcome Evaluation

19

Out

com

e V

aria

ble

Before Program

During Program

After Program

Outcome status without program

Outcome status with program

Program effect

Post-program outcome level

Pre-program outcome level

Outcome Change

(Rossi, Lipsey, & Freeman, 2004)

Efficiency Assessment

• Due to limited resources, program accomplishments must be judged against program costs.

• Two types of efficiency assessments:

• Cost-benefit analysis

• Cost-effectiveness analysis

• Such assessments require placing a dollar amount on program activities and benefits.

20

Typical Evaluation Questions

Needs Assessment

• What is the nature and magnitude of the problem?

• What are the characteristics of the population in need?

• What are the needs of the population?

• What services are needed?

• How much service is needed and over what period of time?

• What service delivery arrangements are needed to provide services to the population?

21


Program Theory Assessment

• Is the program doing the right thing?

• How could the program better address the needs of participants?

• How should the program be organized?

• What are the best delivery systems for the services?

• Is the program specific enough to measure?

22


Process Evaluation

• How many persons are receiving services?

• Are administrative and service objectives being met?

• Are the intended targets receiving services?

• Are the necessary program functions being performed

adequately?

• Are resources being used effectively and efficiently?

23


Outcome Evaluation

• Are the goals and outcomes being achieved?

• Do the services have adverse side effects?

• Are some participants affected more by the services than others?

• Is the problem the services are intended to address made better?

24


Efficiency Assessment

• Are resources used efficiently?

• Is the cost reasonable in relation to the magnitude of the

benefits?

• Would alternative approaches yield equivalent benefits at less

cost?

25

QUESTIONS?

26

Jonathan Brown, [email protected]