FPP Chapters 1 - 2

Design of Experiments

Main topicsDesigned experiments

ComparisonRandomization

Observational studies“control”

Compare and contrastPitfalls to avoid

The five steps of statistical analysesForm the questionCollect dataModel the observed dataCheck the model for reasonablenessMake and present conclusions

ExperimentsMany questions correspond to some cause and

effect relationship

Does smoking cause cancerDoes crop rotation A produce more corn yield than

regularly used crop rotation

How to collect data that will answer causal questions

Experimental design!

Causal StudiesFirst some vocabulary

Treatments: variables that are potentially manipulable

Explanatory variable: variable describes levels of treatment

Response variable: variable whose values are set after treatment assignment

Background variable: variables whose values are set before treatment assignmentThe term concomitant is used in the Reiter exposition

Subject/unit: Individual/object to which treatments are assigned

Control: A treatment without the active ingredientPlacebo: A treatment that outwardly resembles the

active treatment but with out an active ingredientReplication: Administering each treatment level to

more than one unit

More termsTerms that will be discussed in more detail

later are:Confounding/lurking variableDouble-blind randomized trial

Philosophical approach to causalityA perfect, and impossible, casual study:

Obtain an exact copy of each unitExpose one copy to treatment aExpose the other copy to treatment bAt the end of the study, take the difference in the

responses for each copy

Differences are the causal effect of treatment a relative to treatment b for each unit

Average causal effect: average response to treatment a minus average response to treatment b

Fundamental problemWE CAN NEVER DIRECTLY OBSERVE CAUSAL

EFFECTS.Why?

Statistics provides a way to overcome the fundamental problemCreat two groups of units, so that one group

recieves treatment a and the other receives treatment b

Estimate the average causal effect from the observed responses in each group

(average response in group a) – (average response in group b)

Potential problemWhen groups not well constructed, this measures

effects of variables other than treatmentsThis is often called confounding

A solution is to design groups so that the background variables are as similar as possible in the groups

This type of grouping can be had if units are assigned RANDOMLY to each group

These types of experiments are called randomized experiments

Example 1 To compare 3 new test fertilizers, a farmer

applies them to several corn fields. Each field has 3 plots and the 3 fertilizers are randomly assigned, one to each plot within each field. Harvested corn yield is compared for the 3 fertilizers

Subject: Each corn plotTreatments: Three test fertilizersExplanatory variable: Fertilizer typeResponse variable: Corn yieldControl: None

Example 2National supported work demonstration

1970s U.S. social experiment of effects of job training for low income workers.

Treatment: Attend job trainingUnits/subjects: 1602 low-income applicantsExplanatory variable: Attend job training or notResponse variable: Salary one year after program

Units assigned randomly to attend or not to attend the job training

Example 3Infant health development program

1980s U.S. study of effects of intensive child care intervention for low birth weight babies.

Treatment: Attend child care programUnits/subjects: 985 low birth weight babiesResponse: Score on vocabulary test

Babies assigned randomly to attend or not to attned the child care program.

Principles of good experimental design Control or comparison

What: comparing active treatment with control group or compare two or more treatments

Why: to neutralize the effect of lurking variables and measure treatment differences

RandomizationWhat: using random device to assign subjects to

treatmentsWhy: attempt to minimize bias and invoke assumptions

for statistical inference

ReplicationWhat: Applying each treatment to more than one subject

in each treatment groupWhy: to measure and reduce chance variation in the

results by increasing the number of subjects in each group

Confounding /lurking variablesLurking variable

A variable that affects the relationship between the response variable and the explanatory variable but is not included among the variables studied

ConfoundingA condition where the effects of two different

variables on the response variable cannot be distinguished from each other

Risky design (that is often used)Measure response variable before and after

administrating the treatmentThen claim causality when there are

differences in before and after responses

Ex: Foreign language teachers attend a summer of training program to increase language skills. They take a language test before the program starts and a similar test after the program is completed

Risky design cont.For sake of argument suppose average test

score increases

Is this due to the program? Why?

How could this experiment be improved?

Randomly assign some teachers to not attend This is incorporating a control

Observational studies Is a randomized experiment available for all studies?

Sometimes randomizing a human subject to one of the treatment groups would be unethical

For example trying to establish the causality of smoking and lung cancer

When randomizing subjects to treatment groups isn’t possible, typically we use observational studies

These are usually based on existing records from databases with units in both treatment groups

Ex: collect data on smokers and non-smokers from hospital records to compare lung cancer incidence rates

Treatment groups in observational studies We should not simply compare the two groups in the

databases; they are likely to differ in background variables

Therefore, construct groups with similar background characteristics

Ex: Smoking and lung cancer For each smoker, find a nonsmoker with the same race, age, sex, job type

etc.

When there are many variables to match, statisticians use advanced statistical matching methods.

Effect of matching in observational studiesMatching mostly eliminates groups’ differences for

the variables that were used in the matchingThis mitigates these variables effects on

comparisons of groups’ sample mean responsesHowever!!!! (a massively important however…)

There may be unobserved background variables that differ in the groups (Lurking variables and confounding)

Hence in observational studies we have no assurance that the estimate of the average causal effect is free from the effects of unobserved variables that differ in the two groups

Observational study vs designed experiment

Designed experiments

Observatinal studies

Control or comparison

Yes Yes (usually)

Randomization Yes No

Replication Yes Yes

Establish Cause and Effect

Yes (if done correctly)

No

Media often makes observational studies look like experiments

Think about itDoes wearing a bike helmet prevent injuries?

design an experiment to answer this questiondesign an observational study to answer this

question

Do people that wear bike helmets get injured less?How is this question different from the previous

one?How will the designed experiment change?How will the observational study change?

Causal study warningsRandomized experiments

Hidden biasDouble blindPlacebo effectsNoncomplianceOrder effects

Observational studiesConfounding from unobserved background variablesDifferent background variables in treatment groups

Both randomized experiments and observational studiesStudy conditions may not be realisticResults may no generalize

Important aspects of causal studies: Comment 1In many randomized experiments, the units

are not selected at random from a population (e.g., volunteers)

Causal conclusions are valid for the units in the randomized experiment

An issue is whether or not such results can be generalized to other units

Important aspects of causal studies: Comment 2Many randomized experiments are not simple

two-treatment randomizations

They may involve randomizing within groups of units (e.g., randomly assign treatments within male and female groups.)

They may involve randomizing more than one type of treatment (e.g., some cancer patients get chemotherapy, some get radiation, some get nothing, some get both.)

Methods for analyzing such studies are beyond the scope of this course

Inference primerPurpose of Experiment:

To determine whether treatments affect the response

Observed effect:The difference between what we see in the data

and what we expect to see in the data.

Statistically significant: An observed effect that is too large to attribute

plausibility to chance variationIf the differences between the responses for two

treatments is statistically significant, then the treatments affect the response

Inference primer cont.Polio example

The polio rate of those receiving the vaccine was 0.028% compared to 0.071% for those receiving the placebo

In statistical tests that compare two treatments we generally “expect” to see no difference.

Thus the observed effect here would be (0.071 – 0.028) – 0 = 0.047

Is the observed effect of a 0.047% increase in polio rate small enough to be chance variation or large enough to attribute to the vaccine?

We answer this question using probability later in the course