4.2 Experiments

Learning Objectives

After this section, you should be able to:

• DISTINGUISH between an observational study and an experiment.

• EXPLAIN the concept of confounding.

• IDENTIFY the experimental units, explanatory and response variables, and treatments in an

experiment.

• EXPLAIN the purpose of comparison, random assignment, control, and replication in an

experiment.

• DESCRIBE a completely randomized design for an experiment.

• DESCRIBE the placebo effect and the purpose of blinding in an experiment.

• INTERPRET the meaning of statistically significant in the context of an experiment.

• EXPLAIN the purpose of blocking in an experiment. DESCRIBE a randomized block design or

a matched pairs design for an experiment.

Observational Study vs. Experiment

An observational study observes individuals and measures

variables of interest but does not attempt to influence the

responses.

An experiment deliberately imposes some treatment on

individuals to measure their responses.

When our goal is to understand cause and effect, experiments are the only

source of fully convincing data.

The distinction between observational study and experiment is one of the most

important in statistics.

Observational Study vs. Experiment

A response variable measures the outcome of a study.

An explanatory variable may help explain or predict changes in

a response variable.

Check Your Understanding

Does reducing screen brightness increase battery life in laptop

computers? To find out researchers obtained 30 new laptops

of the same brand. They chose 15 of the computers at random

and adjusted their screens to the brightest setting. The other

15 laptops were left at the default setting – moderate

brightness. Researchers then measured how long the

machine’s battery lasted.

Was this an observational study or an experiment? Justify you answer.

Experiment, because the treatment (brightness of screen) was imposed

on laptops.

Check Your Understanding

Does eating dinner with their families improve students’ academic performance?

According to an ABC News article, “Teenagers who eat with their families at least five

times a week are more likely to get better grades in school.” This finding was based on

a sample survey conducted by researchers at Columbia University.

1. Was this an observational study or an experiment? Justify your answer.

Observational study, because students were not assigned to eat particular number of meals

with their family per week.

2. What are explanatory and response variables?

Explanatory: Number of meals per week eaten with their family

Response: GPA

Check Your Understanding

Does eating dinner with their families improve students’ academic performance?

According to an ABC News article, “Teenagers who eat with their families at least five

times a week are more likely to get better grades in school.” This finding was based on

a sample survey conducted by researchers at Columbia University.

Explain clearly why such a study cannot establish a cause-and-effect relationship.

Suggest a variable that may be confounded with whether families eat dinner together.

There are probably other variables that influence the response variable. For example,

students who have part time jobs may not be able to eat many meals with their

families and may not have much time to study, leading to lower grades.

Observational Study vs. Experiment

Observational studies of the effect of an explanatory variable on a response

variable often fail because of confounding between the explanatory variable

and one or more other variables.

Well-designed experiments take steps to prevent confounding.

Confounding occurs when two variables are associated in

such a way that their effects on a response variable cannot

be distinguished from each other.

The Language of Experiments

An experiment is a statistical study in which we actually do something (a

treatment) to people, animals, or objects (the experimental units) to

observe the response. Here is the basic vocabulary of experiments.

A specific condition applied to the individuals in an experiment

is called a treatment. If an experiment has several explanatory

variables, a treatment is a combination of specific values of

these variables.

The experimental units are the smallest collection of

individuals to which treatments are applied. When the units are

human beings, they often are called subjects.

How to Experiment Badly

Many laboratory experiments use a design like the one in the online

SAT course example:

Experimental

UnitsTreatment

Measure

Response

In the lab environment, simple designs often work well.

Field experiments and experiments with animals or people deal with more

variable conditions.

Outside the lab, badly designed experiments often yield worthless results

because of confounding.

How to Experiment Well

The remedy for confounding is to perform a comparative experiment in which some

units receive one treatment and similar units receive another. Most well designed

experiments compare two or more treatments.

Comparison alone isn’t enough, if the treatments are given to groups that differ

greatly, bias will result.

The solution to the problem of bias is random assignment.

In an experiment, random assignment means that

experimental units are assigned to treatments using a

chance process.

Principles of Experimental Design

Principles of Experimental Design

The basic principles for designing experiments are as follows:

1. Comparison. Use a design that compares two or more treatments.

2. Random assignment. Use chance to assign experimental units to

treatments. Doing so helps create roughly equivalent groups of

experimental units by balancing the effects of other variables among

the treatment groups.

3. Control. Keep other variables that might affect the response the same

for all groups.

4. Replication. Use enough experimental units in each group so that any

differences in the effects of the treatments can be distinguished from

chance differences between the groups.

Completely Randomized Design

In a completely randomized design, the treatments are assigned to all the

experimental units completely by chance.

Some experiments may include a control group that receives an inactive

treatment or an existing baseline treatment.

Experimental

UnitsRandom

Assignment

Group

1

Group

2

Treatment

1

Treatment

2

Compare

Results

Check Your Understanding

Music students often don’t evaluate their own performances accurately. Can small group discussions help? The subjects were 29 students preparing for the end-of-semester performance that is an important part of their grade. The 15 students in one group each videotape a practice performance, evaluated it themselves, and then discussed the tape with a small group of other students. The remaining 14 students watched and evaluated their tapes alone. At the end of the semester, the discussion-group students evaluated their final performance more accurately.

1. Describe a completely randomized design for this experiment. Write a few sentences describing how you would implement your design.

Randomly assign 29 students to two treatments: evaluating the performance in small groups or evaluating the performance alone. The response variable will be the accuracy of their final performance evaluations. To implement this design, use 29 equally sized slips of paper. Label 15 of them “small group” and 14 of them “alone.” Then shuffle the paper and hand them out at random to the 29 students to assign them to a treatment.

Check Your Understanding

Music students often don’t evaluate their own performances accurately. Can small group

discussions help? The subjects were 29 students preparing for the end-of-semester

performance that is an important part of their grade. The 15 students in one group each

videotape a practice performance, evaluated it themselves, and then discussed the tape

with a small group of other students. The remaining 14 students watched and evaluated

their tapes alone. At the end of the semester, the discussion-group students evaluated

their final performance more accurately.

What is the purpose of the control group in this experiment?

The purpose of the control group is to provide a baseline for comparison.

Without a group to compare to, it is impossible to determine if the small group of

treatment is more effective.

Experiments: What Can Go Wrong ?

The logic of a randomized comparative experiment depends on our ability to treat

all the subjects the same in every way except for the actual treatments being

compared.

Good experiments, therefore, require careful attention to details to ensure that all

subjects really are treated identically.

The response to a dummy treatment is called the placebo effect.

In a double-blind experiment, neither the subjects nor those who interact

with them and measure the response variable know which treatment a

subject received.

Inference of Experiments

In an experiment, researchers usually hope to see a difference in the responses so

large that it is unlikely to happen just because of chance variation.

We can use the laws of probability, which describe chance behavior, to learn

whether the treatment effects are larger than we would expect to see if only chance

were operating.

If they are, we call them statistically significant.

An observed effect so large that it would rarely occur by chance is called

statistically significant.

A statistically significant association in data from a well-designed experiment

does imply causation.

Blocking

When a population consists of groups of individuals that are “similar within

but different between,” a stratified random sample gives a better estimate

than a simple random sample. This same logic applies in experiments.

Blocking

A block is a group of experimental units that are known before

the experiment to be similar in some way that is expected to

affect the response to the treatments.

In a randomized block design, the random assignment of

experimental units to treatments is carried out separately

within each block.

Matched Pairs Design

A common type of randomized block design for comparing two

treatments is a matched pairs design. The idea is to create blocks

by matching pairs of similar experimental units.

A matched pairs design is a randomized blocked experiment in which each

block consists of a matching pair of similar experimental units.

Chance is used to determine which unit in each pair gets each treatment.

Sometimes, a “pair” in a matched-pairs design consists of a single unit that

receives both treatments. Since the order of the treatments can influence the

response, chance is used to determine with treatment is applied first for each

unit.