Revisiting and Other issues Devesh Roy (September 22, 2015)

IFPRI-ICAR training

Using Monitoring Data

• Monitoring data -a critical resource in an IE. • Lets the evaluator verify which participants received the program,

• how fast the program is expanding,

• how resources are being spent, and

• whether activities are being implemented as planned. This information is critical to implementing the En, for example, to ensure that baseline data are collected before the program is introduced and to verify the integrity of the treatment and comparison groups.

• In addition, M can provide information on the cost of implementing the program, which is also needed for cost-benefit analysis.

Evaluation question

• What is the impact or causal effect of the program on an outcome of interest?

Setting up an evaluation: The steps (Gertler et al) • (i) establishing the type of question to be answered by the evaluation,

(ii) constructing a theory of change that outlines how the project is supposed to achieve the intended results (iii) developing a results chain, formulating hypotheses to be tested by the evaluation, and selecting performance indicators.

• All of these steps are best taken at the outset of the program, engaging a range of stakeholders from policy makers to program managers, to forge a common vision of the program’s goals and how they will be achieved. This engagement builds consensus regarding the main questions to be answered and will strengthen links between the evaluation, program implementation, and policy.

Theories of change• A theory of change is a description of how an intervention is supposed to

deliver the desired results. It describes the causal logic of how and why a particular project, program, or policy will reach its intended outcomes.

• A theory of change is a key underpinning of any impact evaluation, given the cause-and-effect focus of the research.

• A theory of change can specify the research questions.

• The best time to develop a theory of change for a program is at the beginning of the design process, when stakeholders can be brought together to develop a common vision for the program, its goals, and the path to achieving those goals.

• Stakeholders can then start program implementation from a common understanding of the program, how it works, and its objectives.

Theories of change: The results chain

• A basic results chain maps the following elements

• Inputs: Resources at the disposal of the project, including staff and budget Activities: Actions taken or work performed to convert inputs into outputs

• Outputs: The tangible goods and services that the project activities produce (They are directly under the control of the implementing agency.)

• Outcomes: Results likely to be achieved once the benefi ciary population

• uses the project outputs (They are usually achieved in the short-to-medium

• term.)

• Final outcomes: The fi nal project goals (They can be infl uenced by multiple

• factors and are typically achieved over a longer period of time.)

• The results chain has three main parts:

Results

• Results: Intended results consist of the outcomes and final outcomes, which are not under the direct control of the project and are contingent on behavioral changes by program beneficiaries.

Selecting performance indicators (Gertler et al 2010)• SMART is the rule

• Specific: to measure the information required as closely as possible

• Measurable: to ensure that the information can be readily obtained

• Attributable: to ensure that each measure is linked to the project’s efforts

• Realistic: to ensure that the data can be obtained in a timely fashion, with reasonable frequency, and at reasonable cost

• Targeted: to the objective population.

Intent to treat versus treatment- easier to understand with randomized assessment example

• Program offering- Less than full compliance

• Non compliance possible from both sides beneficiaries as well as non-beneficiaries

• Under these circumstances, a straight comparison of the group originally assigned to treatment with the group originally assigned to comparison will yield the “intent to-treat” estimate (ITT).

• We will be comparing those whom we intended to treat (those assigned to the treatment group) with those whom we intended not to treat (those assigned to the comparison group).

• It is not unimportant since most policy makers can only offer a program and cannot force the program on their target population

What about treatment effects?

• In getting the treatment effect requires correcting for the fact that some of the units assigned to the treatment group did not actually receive the treatment, or that some of the units assigned to the comparison group actually did receive it.

• In other words, we want to estimate the impact of the program on those to whom treatment was offered and who actually enrolled. This is the “treatment-on the-treated” estimate (TOT).

Example (Gertler et al 2010)

• Enroll-if-offered. These are the individuals who comply with their assignment.

• If they are assigned to the treatment group (offered the program), they take it up, or enroll; if they are assigned to the comparison group (not offered the program), they do not enroll.

• Never. These are the individuals that never enroll in or take up the program, even if they are assigned to the treatment group. They are noncompliers in the treatment group.

• Always. These are the individuals who will find a way to enroll in the program or take it up, even if they are assigned to the comparison group. They are noncompliers in the comparison group.

Non-compliance issue: continued (Gertler et al 2010)

ITT and ATT (Gertler et al 2010)

• If the average income (Y) for the treatment group is $110, and the average income for the comparison group is $70, then the ITT is $40.

• Second, we need to recover the treatment-on-the-treated estimate (TOT) from the intention-to-treat estimate. To do that, we will need to identify where the $40 difference came from. Let us proceed by elimination. First, we know that the difference cannot be caused by any differences between the Nevers in the treatment and comparison groups. The reason is that the Nevers never enroll in the program, so that for them, it makes no difference whether they are in the treatment group or in the comparison group. Second,we know that the $40 difference cannot be caused by differences between the Always people in the treatment and comparison groups because the Always people always enroll in the program. For them, too, it makes no difference whether they are in the treatment group or the comparison group.

• Thus, the difference in outcomes between the two groups must necessarily come from the effect of the program on the only group affected by their assignment to treatment or comparison, that is, the Enroll-if-offered group.

ITT and ATT

• Suppose that a doctor tells everyone in a treatment group to go home and exercise for an hour per day and tell the control group nothing.

• After a month, if he evaluates the difference in their blood pressure.

• If just compare the difference in mean blood pressures between the two groups, get the ITT.

• This doesn't tell the causal effect of exercise on blood pressure, but the causal effect of telling people to exercise on blood pressure. We would presume that this estimate would be smaller than the treatment effect of exercise per se, as only a (small!) fraction of people in the treatment group would follow the advice.

Retrieving ATT (Gertler et al 2010)

• We know that the entire impact of $40 came from a difference in enrollment for the 80 percent of the units in our sample who are Enroll-if-offered. Now if 80 percent of the units are responsible for an average impact of $40 for the entire group offered treatment, then the impact on those 80 percent of Enroll-if-offered must be 40/0.8, or $50. Put another way, the impact of the program for the Enroll-if-offered is $50, but when this impact is spread across the entire group offered treatment, the average effect is watered down by the percent that was noncompliant with the original randomized assignment.

• ATT=40/0.8=50