Outcomes Evaluation

• A good evaluation is ….– Useful to its audience– practical to implement– conducted ethically– technically accurate.

Evaluation

• Outcomes/impact evaluation–measurement of changes in attitude,

knowledge, health status, behavior, nutrition status

6 Steps of Evaluation

• Establish an evaluation plan from beginning of program

• Obtain buy-in from administrators• Allow enough staff time to make

evaluation a priority

6 Steps of Evaluation

• Obtain permission & encourage participation from participants

• Be flexible & creative

6 Steps of Evaluation

• Use a strong research design & measures which generate data you need to support your program’s goal

Validity

• Internal validity– extent to which an observed effect can

be attributed to a planned intervention

Validity

• External validity– extent to which an observed impact can

be generalized to other settings & populations with similar characteristics

Threats to Validity

• History• Measurement• Selection

History

• External event (H)• Internal programmatic or internal

participant events (I)• Treatment effects (X)

Measurement

• Methods used to collect data• Instruments need to be– reliable – valid

Selection

• Define eligibility to participate in a program– criteria– is there a difference between those who

stay in the program & those that drop

Selection

• Of those eligible, who accept & who refuse– 3222/5000 = 64%– attended/eligible

Selection

• Drop out rate– 1600/3222 = 50%– drop out/initially say yes

Selection

• Lost to follow-up– 300/1622 = 18%– can’t find/completed program

Selection

• Identify contextual or structural variables to decrease selection bias

Regression Effects

• If score high on pretest, little room for improvement

• Will show program has poor impact• Is pretest score a threat to validity?

Synergistic Effects

• All work together to lower internal validity

Evaluation Designs

• Design to increase internal validity

Evaluation Designs

• Design is selected based on– objectives of the program– purpose of the evaluation– availability of eval resources– type of health & behavior problem,

setting & audience

Evaluation Designs

• Notation– R - random assignment– E - intervention group– C - true control group– C - comparison group– X - treatment

Evaluation Designs

• Notation– N - number of subjects– O - observation to collect data– T - time

One Group

• One group & one time– posttest only

• E OXO• Non-experimental• No random assignment

One Group

• No control/comparison group

One Group

• What are some of the main weaknesses of this design for increasing internal validity?

One Group

• When could this design be used & be appropriate?

Nonequivalent Comparison

• E OXO• C OXO• Comparison group– any group not formed by random

assignment

Nonequivalent Comparison

• What threats to internal validity are lessened?

Baseline DataVariable

Age (yr)Cog scoreBelief score

Exp45.771.81.5

Control44.170.91.5

SigNS*NSNS

Self-examNoneOneTwoThree

33%20%19%27%

23%23%14%40%

Sig0.01

*not significant

Time Series

• E OOO X OOO• Pattern of outcome variable• Know stability of outcome measure• Collect outcome variable

unobtrusively

Time Series

• Multiple data points– increases the power of the design

• Equal intervals

Time Series

• Must still try to control of history, selection & measurement

Time Series

• With control or comparison group, much stronger

• Better control over history threat to validity

Time Series with Comparison Group

• E OOO X OOO• C OOO X OOO

True Experimental

• R E OXO• R C O O• Establish at baseline 2 groups not

sig different• Best control of threats to validity

True Experimental

• Advantages?• Disadvantages?

Post-then-pre

• Appropriate for assessing behavior change

• Participants have limited knowledge at beginning of program

Post-then-pre

• Example of a typical pre-test question– Do you include one food rich in vitamin

C in your diet daily?

Post-then-pre

• Implement• See handout, Table 2• After program give only a posttest

Post-then-pre

• Q1 - Asks about behavior because of program

• Q2 - What the behavior had been before the program (i.e.. The pretest question)

Post-then-pre

• U of NE handout data analysis problem 6

• U of NE ETHT report

Success Stories

• Testimonials• Qualitative info• Audience testing

Case Study

• Story of an individual• Can be biased• Cannot be generalized

More Services Available

• Change in the environment

Evaluation Plans

• Read Moving to the Future example• PERT & Gantt charts

Evaluation Plans

• Must be in place– objectives– specifications of the intervention &

program methods

Evaluation Plans

• Must be in place–measurement & data collection

procedures– description of methods

Evaluation Plans

• Main reason– when– from whom– how– by whom

Evaluation Plans

• Worksheet 1 & 2 • Worthen & Sanders, 1987• Fill out form as a team

END Evaluation

• Questions?