chapter 13: experiments and observational studies

Chapter 13: Experiments and Observational Studies

AP Statistics

Observational Studies

Observational Studies: Researchers observe. They don’t assign choices or manipulate anything (unlike experiments).

We just use an existing situation (or data), neither choosing who or what treatments.

Example: A recent study showed that men who have had a heart attack, have a greater chance of having a second heart attack if a certain protein is present in their blood.

Observational Studies

• They are not based on random samples, nor do they randomly impose treatments. The results cannot be generalized, nor can they show cause-and-effect.

• They, however, are not worthless.• They can show us trends and possible relationships

—even if we can’t show cause-and-effect.• They can show us variables related to certain

outcomes

Types of Observational Studies

Retrospective: Subjects are selected and then their previous conditions and behaviors are determinedrestricted to small part of populationProne to errors—looking at historical dataUsually focus on estimating differences between

groups or associations between variables

Types of Observational Studies

Prospective: Study where subjects are followed to observe future outcomes.

Focus is on estimating differences among groups that might appear as groups are followed

Because no treatment is applied, it is NOT and experiment.

Randomized, Comparative Experiments

• Only method by which we can prove cause-and-effect.

• We want to see if learning math on a computer is better than learning it in a traditional classroom—randomly assign half of a group of students to classroom where the content was only taught on computer and the other half to a classroom where the content was never taught on the computer, then we would compare the results.


Comparative just means we are comparing the results at the end of the experiment.


Also called a “factor”

Each factor has levels—values that the experimenter chooses for the factors


An experiment is designed to test the claim that those people who sleep less than 8 hours a night have a decreased ability to remember information. The experimenter has obtained 50 subjects and has randomly placed them in two groups. All subjects will be given a memory test as a baseline. One group will be required to sleep at least 8 hours for one night and the other groups will be prevented from sleeping 8 hours a night. The next day, each group will be given a test of memory and differences in the test will be recorded.


Important Concepts• The experimenter actively and deliberately

manipulates the factors to control the details of the possible treatments.

• The subjects are assigned to the treatments randomly.

Four Principles of Experimental Design

• Control

• Randomization

• Replicate

• Block

Control

• We want to control sources of variation other than the factors we are testing by making conditions as similar as possible for all treatment groups.– We control a factor by assigning subjects to different

factor levels because we want to see how the response will change at those different levels.

– We control other sources of variation to prevent them from changing and affecting the response variable.

Control

• Controlling extraneous sources of variation reduces the variability of the responses, making it easier to detect differences among the treatment groups.

• Making generalizations from the experiment to other levels of the controlled factor can be risky.

Randomize

• Allows us to equalize the effects of unknown or uncontrollable sources of variation– Doesn’t eliminate those effect of these sources,

but it spreads them out across all treatment levels, so that they “even out” and can be looked past.

– If not randomized, you will not be able to draw conclusions from the experiment

– “control what you can, randomize the rest”

Replicate

• 1st type: We need to repeat the experiment, applying the treatment to a number of subjects. If we don’t assess the variation, it is not complete. The outcome of an experiment on a single subject is an anecdote—not an experiment

Replicate

• 2nd type: Occurs when our experimental units (subjects) are not representative of the population of interest. We will need to repeat the experiment with a different experimental units. Replication of an entire experiment with the controlled sources of variation at different levels is an essential step in science. If your subjects are from an Intro to Psychology class, you can’t generalize the results—so you will need to replicate the experiment

Block

• Sometimes random assignment to treatments from our subjects is not the way to go.

• Sometimes we need to block. This is when we group experimental subjects that are known before the experiment to be similar in some way that is expected to affect the response to the treatments.

• The randomization comes within the blocks—where we assign treatments in each block

Logic to Experimental Design

• Randomization produces groups of subjects that should be similar in all respects before we apply treatments

• Comparative design ensures that influences other than the experimental treatment operate equally on all groups.

• Therefore, differences in the response variable must be due to the effects of the treatments

Experimental DiagramDiagram of a randomized comparative

experiment.

An experiment that was designed to test the effectiveness of the drug hydroxyurea for treating sickle cell anemia. There were 299 adult patients who had at least three episodes of pain from sickle cell anemia in the past year.

Experiment to test the effectiveness of see what treatment may reduce the number of repeat offenders.

Experimental DesignRandomized Comparative Experiment

Randomized Blocked Design

• Blocking is used instead of randomizing subjects to treatments.

• Blocking is used if it is believed that there will be differences in how set groups of subjects will respond to the explanatory variable(s)

• Blocking may be used if we suspect that some issue we cannot control may introduce variability in the response. (maybe gender will produce variability in the response, therefore, we will block by gender)

Randomized Block Design

• Randomization is introduced when we randomly assign treatments within each block.

• By blocking, we isolate the variability attributable to the differences between the blocks, so that we can see the differences caused by the treatments more clearly.

• We block to reduce variability so we can see the effects of the factors

• When we block, we are not usually interested in studying the effects of the blocks themselves (no need to compare the results between/among the blocks)


Example: Suppose that the drug we are testing works effectively. That should show up as a difference in response between the experimental group and control group. However, if both groups are mixed gender and men and women respond differently to the drug, then the variability between the genders can drown out the true effect of the drug in each gender. We won’t see that the drug is effective.


Example (cont.): We cannot cope with this variability problem with randomization (can’t randomize by gender). Instead, we block by gender to reduce this variability.

Matched Pairs Design• Subjects are sometimes paired because they are similar in

ways not under the study• When we match subjects in this way we can reduce

variability in much the same way as does blocking• If we have study that is trying to determine if playing

sports increases mathematical achievement, we might want to pair a subject that has a high IQ and plays a sport, with a subject that has a high IQ and does not play a sport (this would be an observational study).

• The matching would reduce the variation due to IQ differences.

Matched Pairs Design

• When we have a matched pairs design that is an experiment, we need to introduce randomization

• Suppose we use a matched pairs design in an experiment that looks at whether or not children can determine the difference between the facial expressions of fear and anger. In this situation, we could match subjects and then randomly assign the order in which the pictures of facial expression are shown. One part of pair will get anger and then fear, the other part of the pair will get fear and then anger.

Matched Pairs Design Diagram

Confounding and Lurking Variables

Confounding and Lurking Variables

• Lurking variables are most common in observational studies

• Confounding are most common in experiments

Statistically Significant

How do we know if the results of an experiment really show that there is a difference?

Suppose we tested a medication to see if it reduced blood pressure better than on older medication. In our results, we calculated that, on average, the medication reduced blood pressure by about 10% more than the older medication. Is this evidence that the new medication worked better than the new? What if it only reduced it by 3%? 20%


In order to determine if the difference is enough we need to say that the differences are statistically significant.

This means that the differences we observed are too big to be explained by chance differences. If they are too big to be explained by chance, then we can attribute the differences to the experiment.

Statistically SignificantSuppose we flip a coin 100 times and it comes up heads 47

times. Is the coin fair, or is it weighted?We know it should come up heads 50 times (theoretically), but

this difference between what we got and what we should get is not large and can be explained by chance differences (it is not unlikely to get those results if the coin was fair). However, if we got 30 heads, we may determine that a difference that big is too large for us to say that it is due to chance differences (it is very unlikely to get those results if the coin was fair). This last result shows that the differences are statistically significant and that the coin is probably not fair.


We will learn how to determine that in later chapters.

Always be skeptical of studies and experiments that discuss how much better (or worse) one thing is than another without stating that the results are statistically significant. That is the only way to determine if there really is a difference between two (or more) things.

Experiments and Samples(differences, similarities and other info)

Similarities• Both use randomization to get unbiased data


Differences• Samples try to estimate the population

parameters, so randomizing is an attempt at having the sample be representative of the population

• Experiments try to assess the effectiveness of treatments, so randomization is used to assign treatments so as to reduce (eliminate) unwanted problems. Experiments rarely draw their subjects from random samples of the population.


Differences• If our objective is to learn something about a

populationSample Survey• If our objective is to see if there is a difference

in the effects of two treatments Experiment• If our objective is just to use an existing

situation to look for trends and/or contributing factors Observational study


Other• If the subjects in an experiment are not random samples

from the population, be cautious about generalizing the results of an experiment until the experiment has been replicated using different subjects, environments, etc.

• Experiments typically draw stronger conclusions than surveys (even if experimental subjects are not random samples). – Because, by looking only at the differences across treatment

groups, experiments cancel out many of the sources of bias.

Control Treatments

If we are attempting to see if a medication effectively reduces anxiety, we don’t just want to give all our subjects the medication and record if their anxiety decreased or not. Instead, we want to compare how much their anxiety decreased compared to people who did not take the medication. This comparison group is the control group and that group’s measurement is called the control treatment.

Blinding

Blinding is when we create an environment where subjects and/or researchers (physicians, technicians, psychologists, etc) do not know who gets treatment and who gets placebo.

Who can affect the outcome of an experiment? Those who could influence the results (the subjects,

treatment administrators, or technicians, etc) Those who evaluate the results (judges, treating

physicians, etc)

Blinding

When individuals in one of those two groups is blinded we say that the experiment is single-blind.

When individuals in both of those groups are blinded we say that the experiment is double-blind.

Blinding

It is important to blind because it is easy for a person’s knowledge about which treatment is given to which people to influence there actions and beliefs. Therefore, blinding will eliminate this form of bias.

Placebo

A placebo is a “fake” treatment that looks just like the treatments being tested.

It many times is used for the control group’s treatment

It is the best way to blind subjects

Placebo

Sometimes groups who are treated with the placebo (control group) will show an improvement. This is not uncommon.

This effect (when the control group show an improvement when treated with placebo) is called the placebo effect.

It is not uncommon that 20% or more of subjects who are given a placebo report things such as reduction of pain, decreased depression and improved health issues

Characteristics of Good Experiments

• Randomized• Comparative• Double-Blind• Placebo-controlled

chapter 13: experiments and observational studies

Documents

treatment groups

possible treatments

observational studies

observational studiesthey

memory test

test of memory

estimating differences

factoreach factor