causality, rcts and impact evaluations · et cetera. peter van der windt causality, rcts and impact...

Causal Inference and RCTsMeasurement

Practical Things

Causality, RCTs and Impact Evaluations

Peter van der Windt

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

We often want to learn about causality:

Does X cause Y?What causes Y?What is the impact of X?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

We often want to learn about causality:

Does X cause Y?What causes Y?What is the impact of X?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

We often want to learn about causality:

Does X cause Y?

What causes Y?What is the impact of X?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

We often want to learn about causality:

Does X cause Y?What causes Y?

What is the impact of X?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

We often want to learn about causality:

Does X cause Y?What causes Y?What is the impact of X?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security? Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty?

Endogeneity? (Conflict ↔ Poverty)Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security? Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)

Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security? Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)Education → Salary?

Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security? Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)

UN → Security? Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security?

Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security? Selection? E.g. UN selects insecure areas to operate.

Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

But this is not that easy because correlation is not causality.

Conflict → Poverty? Endogeneity? (Conflict ↔ Poverty)Education → Salary? Other variable? (E.g. Ability↑ → Education↑ andSalary↑)UN → Security? Selection? E.g. UN selects insecure areas to operate.Et cetera.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison:

X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison:

X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT:

V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

The Fundamental Problem of Causal Inference

An impact evaluation is all about comparison:

What would have happened if there was no project?

Fundamental Problem of Causal Inference

Temporal comparison: X

Cross-sectional comparison: X

RCT: V

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs

Randomized Control Trail (RCT)

Used since decades by medicines

Randomly assign treatment to the units

If you toss a coin 10,000 times: How many times do you have “headsup”?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs

Randomized Control Trail (RCT)

Used since decades by medicines

Randomly assign treatment to the units

If you toss a coin 10,000 times: How many times do you have “headsup”?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs

Randomized Control Trail (RCT)

Used since decades by medicines

Randomly assign treatment to the units

If you toss a coin 10,000 times: How many times do you have “headsup”?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs

Randomized Control Trail (RCT)

Used since decades by medicines

Randomly assign treatment to the units

If you toss a coin 10,000 times: How many times do you have “headsup”?

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs

ALL UNITS

YES Project

= treatment

NO project

= control

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?

Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Power

External validity

Costs

Timing

Ethically correct?

Some units do NOT receive the treatment!?Assignment is random!?

Spillovers

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

Before the project:

4

4

Yes project No project

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

After the project and NO spillovers:

4

4

Yes project No project

3

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

After the project and NO spillovers:

Project Impact = 7 – 4 = 3

4

4

Yes project No project

3

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

After the project and YES spillovers:

4

4

Yes project No project

3 2

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

RCTs and Limitations

After the project and YES spillovers:

Project Impact = 7 – 6 = 1

4

4

Yes project No project

3 2

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Options Different than RCTs

Statistical controls

Matching

Difference-In-Difference

Natural experiment. E.g. Radio

Instrumental variable. E.g. Rainfal

Regression Discontinuity

Process tracing

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Variation in Treatment

YES Sugar NO Sugar

YES Milk 20 0

NO Milk 20 0

Conclusion: Only one treatment part (Sugar) works.

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Variation in Treatment

YES Sugar NO Sugar

YES Milk 20 1

NO Milk 2 0

Conclusion: Complementary between treatment parts.

(I.e. They only work well together.)

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

The Fundamental Problem of Causal InferenceRCTsRCTs and LimitationsOptions Different than RCTsVariation in Treatment

Variation in Treatment

YES Sugar NO Sugar

YES Milk 20 20

NO Milk 20 0

Conclusion: Substitution between treatment parts.

(I.e. It’s either one or the other.)

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)

And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Behavioral Measures

Projects often try to impact behavior:

“Governance”, “Social Cohesion”, “Behavior by soldiers”, Etc.

Problem:

People act strategically (they lie)And they parrot

Solution: Behavioral measure

Make it costly for people to act differently than the truth.

Example: Social Cohesion in Liberia

Example: Good governance in Congo. RAPID & $900

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you?

Have you ever been raped?

Problem:

People act strategically (they lie)Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you? Have you ever been raped?

Problem:

People act strategically (they lie)Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you? Have you ever been raped?

Problem:

People act strategically (they lie)Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you? Have you ever been raped?

Problem:

People act strategically (they lie)

Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you? Have you ever been raped?

Problem:

People act strategically (they lie)Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you? Have you ever been raped?

Problem:

People act strategically (they lie)Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

Does your husband beat you? Have you ever been raped?

Problem:

People act strategically (they lie)Safety of the respondent

How to measure such sensitive topics?

List-experimentRandom-answer method

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

There are two groups:

Group A Group B

I am poor I am poorMarket is too far away Market is too far away

Too dry for good harvest Too dry for good harvestMy husband beats me

To each group ask: “How many of the following are relevant to you?”

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

There are two groups:

Group A Group B

I am poor I am poorMarket is too far away Market is too far away

Too dry for good harvest Too dry for good harvestMy husband beats me

To each group ask: “How many of the following are relevant to you?”

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

There are two groups:

Group A Group B

I am poor I am poorMarket is too far away Market is too far away

Too dry for good harvest Too dry for good harvestMy husband beats me

To each group ask: “How many of the following are relevant to you?”

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Behavioral MeasuresSensitive Questions

Sensitive Questions

A. I am beaten by

my husband

B. I am not beaten

by my husband

Select the answer:

Correct / Incorrect

A

B

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations

Causal Inference and RCTsMeasurement

Practical Things

Practical Things

1 Use technology (e.g. PDAs)

2 Training is crucial

3 Pre-analysis plan

4 Use PhDs

5 Academics & policymakers

END

Peter van der Windt Causality, RCTs and Impact Evaluations