www.pra.ca

Lab and Field Experiments in Economics:Survey and Assessment of their

Potential for Policy Analysis

Greg Mason, PRA Inc.

March 23, 2007

www.pra.ca2

Purpose of the paper

• Present a high-level review lab and field experiments

• Trace the origins of lab and field experiments in economics

• Present the requirements for conducting experiments

• Review recent experiments

• Discuss how experiments may be used in policy development and evaluation.

www.pra.ca3

The nature of experiments

• Experiments can be exploratory or confirmatory

• Exploratory experiments

– Trial and error in manipulation of potential interventions creates a range of outcomes

– Induction to infer potential hypotheses in relation to interventions and outcomes (cause and effect).

• Confirmatory experiments

– Test (confirm/disconfirm) a theory or hypothesis about causal relationships

– Control all influences on outcomes, except the hypothesized cause

– Process of deduction – inferring cause through the modeling of a controlled process.

www.pra.ca4

Key elements of an experiment

• Manipulation of selected and isolated components (treatments) in the physical and/or social environment;

• Observation of changes in that environment before/after or with/without the manipulation; and

• Control of any factor that could change the outcome except for the treatments.

www.pra.ca5

Example 1: A simple lab experiment

A researcher offers the following deal to participants :

Participant A will receive $10 and can choose whether or not to offer part of that amount to Participant B. Participant B can choose whether or not to accept the offer.

Rationality and selfishness predict that A should only offer a minimal amount — say $1 — and that B will accept, since something is better than nothing.

When the game is played in the lab, B will reject any unfair offer, and A, anticipating this, offers more than a token minimum, typically $4.

As modern economic theory stresses, rational self-interest also includes altruism, especially if participants expect to play again.

www.pra.ca6

Origins of experimental economics – 1

• Many economists have denied the possibility of conducting experiments

“Economists cannot perform control experiments of chemists and biologists because they cannot easily control other factors.”

Milton Friedman

• Broadening the definition of experiment to include research that explores reactions of randomized groups to variations in policy and program design, opens an array of potential methods.

www.pra.ca7

• The first step in experimental economics followed the traditional lab experiment in psychology

• Economics has moved from the lab to the field to increase:

– Salience

– External validity

Origins of experimental economics – 2

www.pra.ca8

The RCT trial - Donezepil

6 12 18 24 30

Clinical Decline

ClinicalImprovement

Men

tal

Fu

nct

ion

ing

Months

Score at start

0 mg (control)

5 mg

10 mg Place

bow

asho

ut

www.pra.ca9

Randomization and statistical equivalence

• Randomization into a treatment and control group creates two groups that are statistically equivalent:

– For any statistic (mean, variance…) the two groups will return results that are the same (within bounds of statistical significance)

– The test of statistical equivalence applies to observable and unobservable attributes.

www.pra.ca10

Validity in experiments

• Internal validity

– supports the logical links between the objective of the policy or program, the design of the treatment, and the expected effect

• External validity

– extends the results from the experimental group to a wider population of interest

• Treatment/dosage manipulation

– confirms that outcomes vary in step with the treatments.

www.pra.ca11

Causal Frameworkfor Policy Design

Observationalstudies

Empiricalexperiments

Thoughtexperiments

Statisticalcontrol and

naturalexperiments

Lab

Field

Socialexperiments

Artefactual

Framed

Lab and FieldExperiments

Propensitymatching

www.pra.ca12

Thought experiments

• A “what if” exercise that can be performed mentally, mathematically, or simulated using a spreadsheet

• Starts from axioms of behaviour and possibly estimates of empirical relationships.

• Economics starts a thought experiment with “let us assume….”

• Theorists manipulate equations and model potential causal relationships.

www.pra.ca13

X1

X2

X3

Xk

:

Y

e

B1

B2

B3

Bk

Observational study

• Independent variables act as the “control”

• Cause and effect are imposed on the model.

• Rejection of a null does not “prove” causality.

• Multicollinearity and simultaneous equation bias complicate control.

• Ever more complex modelling to compensate.

www.pra.ca14

Social experiment

• Replicates the approach of the natural and medical scientist.

• Often cited as the “gold standard” in economic policy evaluation.

• Offers strong basis for inferring causal relationships

• Logistically complex

• Expensive to conduct

• Control can be challenging (creating orthogonal treatments, attrition bias…)

www.pra.ca15

Lab/Field Experiments

• Economic lab experiments

– use standard subject pools

– abstract (somewhat artificial) settings – typically computer labs

– a set of rules for the experiment.

• Economic field experiments

– Increase salience by creating real world analogues to the choice or decision

– Use of financial rewards that are significant.

www.pra.ca16

Two types of field experiment

• Artefactual field experiments apply the lab experiment to non-standard subjects

• Framed field experiments move the artefactual experiment into a field setting, such as a store, online purchasing, or choices made in a bank

www.pra.ca17

Example 2: Job search options

• Anecdotal evidence from LMDA evaluations suggests that counsellor attention is an important factor in motivating and orienting unemployed persons who are seeking work.

• The effect of trained assistance on the speed with which EI claimants can generate job leads may be studied statistically or experimentally.

– The statistical approach involves analyzing variation in the time spent generating job leads, using multivariate methods to understand the factors that affect outcomes.

– An experimental approach would selectively withdraw and offer counselling assistance and observe the impact on the number of job leads.

www.pra.ca18

Issues in experimental economics

• theory validation

• experimental control

• internal and external validity

• salience and coherence

• financial incentives.

www.pra.ca19

Theory validation

• All theories require a set of untested assumptions or “maintained hypotheses.” It is impossible to control for everything.

• The weakness in the falsification perspective is that, after rejecting a null hypothesis, an infinite number remain to be tested.

• Successive experiments allow one to induce the validity of a causal relationship through the replication of studies and experiments.

www.pra.ca20

Experimental control

• Lab experiments tends to focus less on randomization subjects and more on the manipulation of alternatives to generate “data.”

• Field experiments will randomize on some dimension of the alternatives.

• The key is to create a “field” where the policy or intervention is subject to systematic control.

www.pra.ca21

Internal and external validity

• Internal validity increases in a lab or computer setting

• The key to increasing internal validity is to remove confounding variables – i.e., isolate the subjects from the real world

• External validity requires using subjects that are relevant to the policy question

• A trade-off tends to exist between internal and external validity.

www.pra.ca22

Salience and coherence

• Salience ensures that participants see the experiment, and the options offered, as personally relevant

• Coherence means that the alternatives offered to subjects represent a reasonably complete set for the policy.

www.pra.ca23

Financial incentives

Non-satiation means that subjects will always choose options with the greatest personal gain

Saliency means that alternatives are important and relevant to the subject’s well-being

Dominance means that the rewards of participation outweigh any costs (real and subjective) of participation

Privacy means that the rewards are private to the individual.

www.pra.ca24

The experiment: Approaching fair behaviour — Distributional and reciprocal preferences

• Comparison of one-stage and multi-stage two-person games where individual decisions dictate the degree of equality and the level of personal gain among payoffs.

• Hypothesis: one-stage games; decisions favouring equality over personal gain are motivated only by altruism.

• In multi-stage games, where selfish decisions may be punished in subsequent rounds, decisions are motivated by altruism and feelings of reciprocity.

www.pra.ca25

The experiment: Eliciting risk and timepreferences using field experiments

• Modified version of a well-established laboratory model to a representative sample of the Danish population.

• Experimental subjects face two types of choice decisions:

– decisions between playing low- and high-risk lotteries with varying expected values.

– decisions between small monetary payoffs now or larger monetary payoffs in the future.

• Decisions favouring low-risk lotteries and immediate payments indicate risk aversion and a high level of time discounting.

www.pra.ca26

The experiment: Saving decisions of the working poor — Short-and long-term horizons

• Experimental subjects from the working poor who have three types of decision-making problems:

– choosing between playing high- or low-risk lotteries with varying expected values.

– deciding between small monetary payoffs immediately and larger payoffs in a number of weeks.

– choosing between small monetary payoffs immediately and considerably larger payoffs in a number of years.

• Comparison of results helped identify a stable relationship between short- and long-term time preferences among the working poor.

www.pra.ca27

The experiment: The effects of educational vouchers on confidence — A field experiment to assess

outcomes of educational policy

• Presented to a random selection of applicant children, the voucher allotment actually created a natural experiment by establishing randomly selected control and treatment groups.

• By comparing controlled confidence test results among lottery winners and losers, the authors established the net effect of the educational vouchers on child confidence.

www.pra.ca28

The experiment: On the empirics of social capital

• Based on Robber’s Cave experiment, where a group of homogeneous boys was separated into two groups, isolated, and asked to engage in a variety of team-building exercises.

• When reintroduced to each other, experimenters tracked changes in the behavioural norms among the boys relative to those established prior to separation. Experimenters also noted the degree to which competition between groups facilitated collective group action.

www.pra.ca29

The experiment: The targeted negative incometax in Germany

• A negative tax incentive regime in Germany that was applied differentially to individuals within and across municipalities, to establish the regime’s success or failure.

• The differential application of the regime within comparable regional settings established experimental treatment and control groups for further analysis.

• Years after application, comparing the re-employment success of individuals under and not under the regime allowed researchers to establish the impacts of the negative tax incentive regime.

www.pra.ca30

The experiment: Investment by working poorin human capital

• Identify preferences for educational savings among the working poor by soliciting responses to a variety of choice questions regarding current consumption and savings for future education.

• Experimental subjects are asked to choose between a small immediate monetary payoff and a larger monetary contribution to an educational saving fund meant to support future education or training.

• Varying the level of immediate and educational saving payoffs allows experimenters to establish the level of incentive necessary to make individuals among the working poor invest in education.

www.pra.ca31

The experiment: Discrete choice methodsfor estimating labour supply.

• A client survey, conducted as part of the National Child Benefit evaluations, randomly assigned social assistance respondents to three groups.

• Respondents in each group agreed or disagreed with a specific question: “Would you accept a job at $Y per hour?” where Y varies with the group.

• The result is an experimentally determined reservation wage that measures the height of the welfare wall.

• Covariates show how the reservation wage/welfare wall varies with the attributes of the respondent.

www.pra.ca32

Reservation wage (NCBS recipients)

Group  SA* Non-SA Total

All clients $9.97 $10.67 $10.39

Single parents $10.04 $10.77 $10.33

Dual parents $9.79 $10.63 $10.44

www.pra.ca33

The experiment: Stated choice experiment insocial policy preferences

• Focus group participants (NCB evaluation) rated various options for income and in-kind support.

• A stated choice experiment consists of a trial, or a series of trials, typically conducted as part of a focus group, in which the researcher makes purposeful changes to the attributes/levels of a policy package in order to observe and identify possible explanations for changes in the response variable.

• In the simplest form of analysis, the resulting data set is a regression model, where the valuations of packages become the dependent variable and the packages are coded using dummy variables.

www.pra.ca34

Attributes and levels for the conjoint experiment — Social assistance package

Attributes Levels

Additional monthly cash benefit

No cash benefit $50 per month $75 per month

Additional child care while working

Additional child care while working

$2.50 cost per child per day

Free child care

Voucher for food No voucher $50 per month $75 per month

Voucher for clothing No voucher $75 per month -

After-school club (recreation and homework help)

$1.50 per child per day cost

Free after-school club

-

www.pra.ca35

Summary

• Lab experiments have been effective in exploring theory and the assumption of human economic behaviour.

• Field experiments increase salience and therefore provide a more secure policy development platform.

• Randomization and experimentation are not synonymous – the key to field experiments is controlled manipulation.

• A variety of field experimental formats are available to support cost-effective policy design and evaluation.

• The most significant benefit of field experiments are cost and flexibility.

• Field experiments can manipulate several dimensions and levels within the same context.

• A weakness is that salience can be hard to achieve in any experimental setting – most experiments are not “real life,” and participants know this.

www.pra.ca36

How HRSDC can use field experiments

• Basic research into preferences and motivations for populations at risk.

• Value exists in funding small scale field experiments with target population to determine service mix, client response to policy options, etc. Typical scenarios include:

– Assessing counseling modalities for EI clients (by age, gender, ethnicity, etc) – what works best for whom.

– Fine tuning policy options such as support levels for earnings supplements or earnings disregards.

• Testing the response of parents to educational investments (i.e., testing the features of a RESP or other savings plan).

• Developing a policy package for EI clients (i.e., revamping the EBSMs) by using conjoint methods where clients rate training features. Another approach is to find optimum policy packages within a budget constraint.

www.pra.ca37

Final thoughts

• Field experiments will not replace other methods, but serve as a low-cost complement to existing methods.

• Since HRSDC (and other departmental evaluations) are undertaking surveys and focus groups, many experimental methods can be grafted onto the existing research platform at low marginal cost, without affecting the information obtained from these other lines of evidence.

• Experiments introduce important insights that support policy design (and redesign) into evaluations, thereby providing increased value for money for the study.

• Value still exists in the large scale experiments, which will often benefit from the smaller field experiments at the design phase.