methods of comparison: controlled experiments and observational studies math 1680

Methods of Comparison: Controlled Experiments and

Observational Studies

Math 1680

Overview

Introduction Controlled Experiments Minimizing Bias Observational Studies Association, not Causation Comparing Rates Salk Vaccine Trial Breast Cancer Screening Summary

Introduction

Suppose a company wanted to market a weight-loss pill

After design is complete, the company still needs to answer one basic question– Does the pill actually work?

How can the company determine this?

Introduction

One method would be to give the pill to everyone who wants to lose weight and see if they actually do

– What’s wrong with this approach?

–First of all, the cost of giving everyone the pill is prohibitive, and there may be dangerous side effects for the untested pill.

–Secondly, people who want to lose weight may be exercising or eating more healthily, which could overshadow the effects of the pill (This is an example of a confounding factor.)

Controlled Experiments

To test the pill, two groups are created from a sample of people– Control group is given a placebo– Treatment group is given the pill


Why is it important that the people don’t know which group they are in?

It has been shown repeatedly that a patient's symptoms can be alleviated by an otherwise ineffective treatment if the individual expects or believes that it will work. Conversely, members of a control group knowingly receiving an inert substance tend to report a worsening of symptoms. This phenomenon is called the placebo effect.


The most ideal form of experiment is one where neither the subjects nor the evaluators know who received the treatment– This is a double-blind experiment

This design provides the least opportunity for biased results


After a period of time, the company checks to see how much weight people in each group have lost– If there is a significant difference between the

groups, assume the pill was the reason– If there is no significant difference between the

groups, assume the pill had no effect

Minimizing Bias

What if the control group and the treatment group differ in some way other than the treatment?

– Then we have confounding factors in the study, and the results will be biased and/or invalid

Confounding factors can include…– Age– Race– Gender– Socio-economic status (SES)– Location– Political/religious orientation

Minimizing Bias

In the pill example, suppose all of the men were placed into the treatment group and all of the women were placed into the control group

– What is wrong with this design?

The pill may interact differently with people based on gender. For example, suppose that high levels of testosterone diminish the effect of the pill. Then the results would be biased against the pill’s effectiveness.

Minimizing Bias

In order to minimize the risk of bias, researchers assign people to each group randomly from the pool of subjects– If enough people are in the pool, the groups will

look similar in terms of age, race, gender, or any other demographic which could be a confounding element


How would you design a controlled experiment to test if second-hand smoke causes lung cancer?

Answer: You can’t, really. The time scale is too large and it would be virtually impossible to ensure that your control group never got second-hand smoke exposure.


In many studies, designing a controlled experiment is simply not possible

– In such cases, researchers must settle for observational studies

– Observational studies let the subjects select which group to join

Observational studies should be placed under great scrutiny because the control and treatment groups may be quite different in many ways

– These differences may introduce confounding influences on the study results


A TV commercial observes that volunteers who switched to having to Cheerios for breakfast reduced their cholesterol level by 4%. Does this show that Cheerios cereal is effective in reducing your cholesterol level?

Not necessarily. People who volunteer for such a study may already be interested in improving their health and may be taking steps such as exercising daily. This could lower their cholesterol level.


One form of observational study involves using historical controls to compare a modern treatment with– The difference in conditions between past and

present can itself be a confounding factor– For example, in testing a polio vaccine, one would

not want to compare infection rates in the 1950’s with infection rates in the 1900’s because of the large differences in standard of living and health care quality in the two periods

Association, not Causation

An important limitation of all studies is that they can only show the strength of association between two variables

– Association may point to causation– For example, if exposure to a virus causes a disease, then people

who are exposed should be sicker than similar people who are not exposed

However, a study alone cannot show causation– Human inference is needed to connect causal links– Generally, a casual link is accepted if

A scientific theory is derived which explains the association Enough studies controlling all potential confounding factors are

performed, and all of them show the same association


Cervical cancer is more common among women who have been exposed to the herpes virus, according to many observational studies. Is it fair to conclude that the virus causes cervical cancer?

No. Both herpes and cervical cancer have been shown to be sexually transmitted diseases. Women who have many partners are at higher risk for both diseases than women with few partners, so the number and type of partners is a confounding influence which explains the association.


Some studies find an association between liver cancer and smoking. Does this mean smoking causes liver cancer, or is there a confounding influence at play?

People who smoke also tend to drink, so drinking is associated with smoking. It has been shown repeatedly that over consumption of alcohol can cause liver cancer.

Comparing Rates

Sometimes the groups in comparison are of different sizes– Rather than comparing the actual numbers, use

rates or proportions such as percents– Adjusting to rates allows for a more direct

comparison

Comparing Rates

In the U.S. in 1990, there were 2.1 million deaths from all causes compared to 1.7 million in 1960

– According to Census data, there were about 177 million Americans in 1960 and about 250 million in 1990

True or false: The public’s health got worse over the period 1960-1990.

False. Convert the death counts to percentage rates. The death rate in 1960 was 0.96%, while the death rate in 1990 was 0.84%. If anything, public health improved.

Salk Vaccine Trial

In 1954, researchers tested the effectiveness of a vaccine for polio

Approximately 400,000 children were selected– All children in the study had parental consent to be

vaccinated (Why?)– Children randomly divided into equal sized treatment and

control groups Control group was given a placebo

Is this a controlled experiment or an observational study?

Salk Vaccine Trial

Results

Was the vaccine effective?

Study Group Population

Polio Cases Rate per 100,000

Vaccinated 200,745 57 28.4

Placebo 201,229 142 70.6

Breast Cancer Screening

In 1963, an insurance company used 62,000 women (all insured by the plan) as subjects– Women ranged in age from 40 to 64 – 31,000 of the women were used as controls– The others were encouraged to undergo annual

screening Roughly 20,200 women from the treatment group came

for annual screenings, the other 10,800 refused

After 5 years, the death rates were checked


Is this a controlled experiment or an observational study?

It depends on what questions you are trying to answer. If you are only interested in the difference between the treatment and control groups, then you are using the results of a controlled experiment. However, when you start drawing inferences between the screened and refused groups, you are looking at an after-the-fact observational study.


Researchers noted three things: 1. Screening had little impact on diseases other than breast cancer.2. Poorer women were less likely to accept screening than richer ones.3. Most diseases fall more heavily on the poor than on the rich.

Deaths (breast cancer)

Deaths (all other causes)

Number Rate per 1,000


Treatment 31,000 39 1.3 837 27

Examined 20,200 23 1.1 428 21

Refused 10,800 16 1.5 409 38

Control 31,000 63 2 879 28


Does screening save lives? Which numbers prove your point?





Treatment 31,000 39 1.3 837 27

Examined 20,200 23 1.1 428 21

Refused 10,800 16 1.5 409 38

Control 31,000 63 2 879 28


Why is the death rate from all other causes in the whole treatment group about the same as the rate for the control group?





Treatment 31,000 39 1.3 837 27

Examined 20,200 23 1.1 428 21

Refused 10,800 16 1.5 409 38

Control 31,000 63 2 879 28


Why is the death rate from all other causes higher for the “refused group” than the examined group?





Treatment 31,000 39 1.3 837 27

Examined 20,200 23 1.1 428 21

Refused 10,800 16 1.5 409 38

Control 31,000 63 2 879 28


Unlike most diseases, breast cancer afflicts the rich more than the poor. Which numbers in the table confirm this association?





Treatment 31,000 39 1.3 837 27

Examined 20,200 23 1.1 428 21

Refused 10,800 16 1.5 409 38

Control 31,000 63 2 879 28

Case Study: Child care and behavior

Question: If parents choose to use child care, are they more likely to see undesirable behaviors in their children?

Question: How should one find out? By use of controlled experiments, or by observational study?


Answer: In 1991, a study commenced on 1364 infants (“subjects”) and followed them through their sixth year in school. 12 years later, an article was published. “the more time children spent in child care from birth to age four-and-a-half, the more adults tended to rate them, both at age four-and-a-half and at kindergarten, as less likely to get along with others, as more assertive, as disobedient, and as aggressive.”


A summary of the study noted, “The study authors noted that their study was not designed to prove a cause and effect relationship. …” (that more child care time causes more aggressive behavior)

Possible confounding factors? Perhaps… perhaps … perhaps…


Question: How about a controlled experiment instead? (i.e. the researchers select children to place in child care)

What are the advantages? What problems might arise?

Summary: Basic Principles of Statistical Experimental Design

Raise question to answer: Explanatory variable(s) & Response variable(s).

Leading principle: Comparison. Other principles: randomize, repeat (or

involve many subjects to reduce chance variation)

Why:

Summary: Principles of Statistical Experimental Design

Limitations: Statistical analysis of an experiment cannot tell us how far the results will generalize to other settings. (Experiment run in Massachusetts; how about in the entire USA?)

However, the randomized comparative experiment, because of its ability to give convincing evidence for causation, is one of the most important ideas in statistics.

More Elaborate Statistical Designs

Matched Pairs Design: Pair subjects with the same age, sex, income, etc. and compare their responses

Example: Compare two advertisements for the same product

Common variation of MPD: impose both treatments on the same subjects, so that each subject serves as his or her own control. (See HW2, Q9)

More Elaborate Statistical Designs

Question: How does randomization work here? Answer: Which one of a matched pair sees the first

ad is decided at random. Or, which treatment a person gets first is decided at random if he or she also serves as his or her own control.

Additional Example: Matched Pairs for the cell phone experiment. (Details: trained in using simulator, order in which a subject drives with and without the phone be random, two drives be on separate days)

Summary

Statisticians use the method of comparison – To determine the effectiveness of a “treatment”, they

usually compare the responses of a treatment group with a control group

If the control group is comparable to the treatment group (apart from the treatment), then a significant difference in the responses of the two groups is likely to be due to the effect of the treatment

Summary

If the treatment group is different from the control group with respect to other factors, the effects of these other factors are likely to be confounded with the effect of the treatment

To make sure the treatment group is like the control group, investigators put subjects into either group at random

Summary

Whenever possible, the control group is given a placebo

– The response should be to the treatment itself rather than to the idea of the treatment

In a double-blind experiment, the subjects do not know whether they are in treatment or in control, and neither do those who evaluate the responses

– Guards against bias, both in the responses and in the evaluations

Summary

In an observational study, the investigators do not assign the subjects into treatment or control groups– Subjects which have the condition are the

treatment group and the ones which do not are the control

Observational studies can establish association, which is not necessarily causation

Summary

Observational studies are particularly susceptible to confounding factors

– This can make them very misleading about cause and effect relationships

With observational studies, try to find out how the subjects came to be in treatment or control

– Are the groups comparable? – What factors are confounded with the treatment?– What adjustments (if any) did the investigators make to

control for these factors?

methods of comparison: controlled experiments and observational studies math 1680

Documents

pill slide

pill example

effect slide

control group

untested pill

placebo treatment group

weightloss pill

confounding element