IMPACT EVALUATION

Purpose of Impact Evaluation

• Impact evaluation is predicated on whether programs are making a difference.

Nature of Impact Evaluation

• IE has a strong summative emphasis in that it provides findings from which a judgment of the worth of the program can be made.

• IE are retrospective in that it logically occur at an end-point.

Nature of Impact Evaluation

• IE are undertaken on programs which are in a mature or settled stage and have had sufficient time to have an effect.

• IE can also be administered to immature programs but the evaluator must give caution.

IE is concerned with………………

• Determining the range and extent of outcomes of a program.

• Determining whether the program has been implemented as planned and how implementation has affected outcomes.

• Providing evidence to funders about the extent to which resources allocated to a program have been spent wisely.

IE is concerned with………………

• Informing decisions about replication or extension of the program.

Major concern of IE…………

• Outcomes are major concerns of IE not output.

• What are outcomes? - are benefits for participants during or after their involvement with a program. - outcomes relate to knowledge, skills, attitudes, values, behavior, condition or status.

Major concern of IE…………

• What are outputs? - outputs are products of the program’s activities. Example: number of meals provided, classes taught etc.

• IE is concerned with establishing what works and why.

IE is beneficial to the following…

• Stakeholders of the program• Scholars• Policy-makers

Approaches to IE…….

• In order to determine if a program has impact, the following must be done:

- selection of key variables- setting standards- evidence

* It is the most practised form of evaluation

Approaches to IE……….

• Objectives-based;• Needs-based;• Process-product studies; and• Performance audit

OBJECTIVES - BASED

• Based on the judgment of whether the stated goals or objectives of the program have been met.

• These are often expressed in terms of gains in attainment of program participants

OBJECTIVES - BASED

Main Task in setting up Objectives-based

• Determine the issue of the stakeholders.• Determine the real objectives of the program.• Determine the most appropriate way to check

if the program has led to the attainment of the goals.

OBJECTIVES – BASED

• Select an appropriate measuring instrument, or write and trial a new instrument.

• Identify the sources of evidence.• Collect and analyze the evidence.• Draw conclusions, make judgments and

recommendation and report findings.

NEEDS – BASED

• Determine the impact if the program meets an identifiable need.

GOAL - FREE

• The purpose is to examine all program effects, rather than limiting the investigation to outcomes which reflect program objectives.

• Goal-free evaluations are rare.• In goal-free, one examinees both the intended

and unintended outcomes.

PROCESS – OUTCOME

• The focus is on the actual program implementation in order to explain the process of the outcome.

• The outcome = DV; the process = IV.

PROCESS – OUTCOME

• Standard procedure is to develop instruments to measure outcomes.

• Determine the degree to which implementation action is consistent with the intentions of the program.

• Preferred – observations by trained personnel.

Implementation as DV

• This is done when it is more important to look at and evaluate the implementation rather than the outcome.

PERFORMANCE AUDIT

• A custom crafted analysis of program efficiency and effectiveness. It involves the determination of the economy, efficiency and effectiveness of government organizations, activities and functions, in addition to their compliance with laws and regulations.

PERFORMANCE AUDIT

• The actual practice of PA include:- a systematic process;- objectively obtaining and evaluating evidence;- ascertaining the degree of correspondence between assertion and established criteria.- communicating the results to interested users.

Summary of IE…….

• Orientation- What is the worth of the program?- Justification of decisions to mount the program?- Accountability to functions and other

stakeholders.

Summary of IE……….

• Typical Issues- has the program been implemented as

planned?- have the stated goals of the program

beenachieved?

- have the needs of those served by the program been met?

Summary of IE………

• Typical Issues- what are the unintended outcomes?- does the implementation strategy lead to

the intended outcomes?- how do differences in the implementation

affect the program outcomes?- what are the benefits of the program given

the costs?

Summary of IE……….

• State of program- settled

• Major focus- delivery and outcomes

• Timing - after the program has been completed

Summary of IE………

• Key approaches- objectives based- needs based- goal free- process outcome- performance audit