Evaluation Evaluation ResearchResearch

Dr. GueretteDr. Guerette

IntroductionIntroduction Evaluation Research –Evaluation Research –

The purpose is to evaluate the impact of The purpose is to evaluate the impact of policiespolicies

Evidence – based policy analysisEvidence – based policy analysis Is used to help public officials examine Is used to help public officials examine

and select from alternative actions.and select from alternative actions.

Appropriate TopicsAppropriate Topics Policy analysis and evaluation are used to Policy analysis and evaluation are used to

develop justice policy and determine its develop justice policy and determine its impactimpact

Policy analysisPolicy analysis helps officials evaluate helps officials evaluate alternative actions, choose among them and alternative actions, choose among them and formulate practices for implementing policy.formulate practices for implementing policy.

Program evaluationProgram evaluation is conducted at a later is conducted at a later point in time than policy analysis for the point in time than policy analysis for the purpose of determining if policies are purpose of determining if policies are implemented as planned and are they implemented as planned and are they achieving their goals.achieving their goals.

Steps of EvaluationSteps of Evaluation In order to do evaluation research In order to do evaluation research

you must learn the goals as the you must learn the goals as the initial step.initial step.

Evaluability assessment –Evaluability assessment – A pre-evaluation where a researcher A pre-evaluation where a researcher

determines whether conditions determines whether conditions necessary for conducting an evaluation necessary for conducting an evaluation are present.are present.

Steps of EvaluationSteps of Evaluation Problem formulation –Problem formulation –

Identify and specify program goals in Identify and specify program goals in concrete, measurable form.concrete, measurable form.

Measurement – Measurement – How the program is doing in meeting its How the program is doing in meeting its

goals.goals. Specifying outcomes – Specifying outcomes –

Program goals represent desired outcomes, Program goals represent desired outcomes, while outcome measures are empirical while outcome measures are empirical indicators of whether or not those desired indicators of whether or not those desired outcomes are achieved.outcomes are achieved.

Steps of EvaluationSteps of Evaluation Measuring program contexts – Measuring program contexts –

Measuring the context within which the Measuring the context within which the program is conducted.program is conducted.

Measuring program delivery – Measuring program delivery – Measuring both the dependent and Measuring both the dependent and

independent variables are necessary.independent variables are necessary.

Designs for Program Designs for Program EvaluationEvaluation

Randomized evaluation designs – may be Randomized evaluation designs – may be limited by legal, ethical and practical limited by legal, ethical and practical reasonsreasons

1.1. Program and agency acceptance – it is Program and agency acceptance – it is necessary to explain to the agency why random necessary to explain to the agency why random assignment is vital for this type of research.assignment is vital for this type of research.

2.2. Minimize exceptions to random assignments – Minimize exceptions to random assignments – recognize that some exceptions are necessary recognize that some exceptions are necessary but too many exceptions threatens the but too many exceptions threatens the statistical equivalence of experimental and statistical equivalence of experimental and control groups.control groups.

Designs for Program Designs for Program EvaluationEvaluation

3.3. Adequate Case Flow for Sample Size – Adequate Case Flow for Sample Size – The larger the sample size the more The larger the sample size the more

accurate the estimates of the population accurate the estimates of the population characteristics which will reduce threats characteristics which will reduce threats to things like statistical conclusion to things like statistical conclusion validity.validity.

4.4. Maintaining treatment integrity – Maintaining treatment integrity – It is important to maintain treatment It is important to maintain treatment

consistency (homogeneity) because it will consistency (homogeneity) because it will impact measurement reliability.impact measurement reliability.

Designs for Program Designs for Program EvaluationEvaluation

Quasi-experimental Designs – used Quasi-experimental Designs – used when one is not able to use random when one is not able to use random assignment of subjects to an assignment of subjects to an experimental and a control group.experimental and a control group.

1.1. Ex Post evaluations – done after Ex Post evaluations – done after (retrospectively) an experimental (retrospectively) an experimental program has gone into effect.program has gone into effect.

Designs for Program Designs for Program EvaluationEvaluation

2.2. Full Coverage programs – Full Coverage programs – Usually national or statewide in nature Usually national or statewide in nature

where it is not possible to identify subjects where it is not possible to identify subjects who are not exposed to the intervention who are not exposed to the intervention and cannot randomly assign persons to and cannot randomly assign persons to receive or not receive treatment.receive or not receive treatment.

3.3. Larger treatment units – Larger treatment units – Incorporating a great number of people Incorporating a great number of people

thus limiting the ability to use random thus limiting the ability to use random assignment.assignment.

Designs for Program Designs for Program EvaluationEvaluation

4.4. Non-equivalent groups design – Non-equivalent groups design – Where treatment and control subjects are Where treatment and control subjects are

not statistically equivalent.not statistically equivalent.5.5. Time series designs – Time series designs –

Multiple measures at different points in Multiple measures at different points in time.time.

6.6. Other types of designs – Other types of designs – Often tailored to practical constraints of Often tailored to practical constraints of

the research circumstances.the research circumstances.

Policy Analysis & Scientific Policy Analysis & Scientific RealismRealism

Policy analysis coupled with Policy analysis coupled with scientific realism helps public scientific realism helps public officials use research to select and officials use research to select and assess alternative courses of action.assess alternative courses of action. Modeling studies – example of prison Modeling studies – example of prison

population forecasting.population forecasting. Use of mathematical models to predict Use of mathematical models to predict

future events or numbers.future events or numbers.

Political Context of Political Context of Applied ResearchApplied Research

When doing evaluation research and When doing evaluation research and policy analysis, it is done in a political policy analysis, it is done in a political arena.arena. Evaluation and stakeholders – recognize Evaluation and stakeholders – recognize

that many people have a direct or indirect that many people have a direct or indirect interest in the program or its evaluation.interest in the program or its evaluation.

Politics and objectivity – politics and Politics and objectivity – politics and ideology can color or impact evaluation ideology can color or impact evaluation research.research.

In Class ExerciseIn Class Exercise