Chapter 11: Designing experiments

Objective

(1) Learn to distinguish between different kinds of statistical studies.

(2) Learn key concepts involved in designing experiments.

Concept briefs:

Again there is lots of new jargon here. Be aware that many of these terms have meanings that differ from their everyday usage.* Observational vs. experimental study - what is the difference?

* Retrospective vs. prospective study - what is the difference?

* Key components of experiments = (1) Factor; (2) Response;(3) Treatments; (4) Experimental units; (5) Control of variability.

* Important related terms & concepts = (1) Blocking; (2) Blinding;(3) Confounding; (4) Placebos; (5) Statistically significant.

Experimental study: Basic ideas

Objective

* To prove there is a cause & effect relationship between 2 variables.

* This is one step beyond showing there is an association.

e.g., Can we prove that smoking can cause lung cancer?

Can we prove that longer study-time increases one’s GPA?

Key idea: You can prove cause & effect if you control all other variables and isolate the effect of one variable upon the other.

Experiments in a nutshell

(1) Want to prove that the explanatory variable causes the response variable to behave in a certain way.

(2) Pick several subjects and expose them to different doses of the explanatory variable. Measure their responses. Discover how the response depends upon the explanatory variable.

(3) Make sure all other attributes are identical or controlled:

- Randomization at every step is the key to controlling variability.

- Other controls: Blinding, blocking, placebos.

(4) Repeat the experiment a few times and verify you get the same results. If you don't, there is a problem!

How to design statistical experiments

[Follow along with Exercise 52, pg. 313]Step1: Clearly state the goal or hypothesis the experiment will test.Ex52: To determine whether giving large doses of Vitamin E to post-surgical patients expedites healing time.

Step2: Identify the factor(s) and response(s).

Ex52: Factor = dose level of Vitamin EResponse = time it takes for incisions to heal to a specified level (as determined by a common physician).

Step3: Identify the experimental units.Ex52: Select 30 patients to volunteer for the study.[Q. to think about: Does it matter if they’re not randomly selected?]

Step4: Specify the treatments.Ex52: We will have 3 different dose levels of Vitamin E - 1000 IU, 200 IU, and no Vitamin E

Step5: Describe the controls & any other details (i.e., conditions of the experiment, and how you plan to control other sources of variability).

Ex52: (a) We will randomly select 10 patients for each factor level (b) Those that receive 0 units, will receive a placebo pill. (c) For each group, the pills will look the same & be given in the same quantity, at roughly the same times of day. (d) The patients, as well as all treating nurses and physicians will be “blinded” to the treatments.

Step6: Describe how the response will be analyzed & how you will determine the outcome of the study.Ex52: The number of days it takes each patient to reach the specified healing level will be recorded. We will look at mean and spread for each treatment group (e.g., boxplots) & determine whether there is any “statistically significant” difference in healing time

Related concepts* Placebo* Blinding* Control group* Blocking (e.g., children & adult patients in Ex34)* Confounding* How much difference is “statistically significant”?

Summary of key ideas in experiment design

(I) Any differences in response should be attributable to your treatments alone. So you must make all other conditions either identical or randomized.

(II) You must always compare responses (with & without treatment) to determine the outcome.

Analyzing the outcome of experiments

* Recall the goal: Determine whether factor levels cause response difference

* In Exercise 52: Suppose the median healing time for the treatment groups isNo Vitamin E = 9 days200 IU Vit. E = 10 days1000 IU Vit. E = 7 daysWhat is the conclusion? What else might be useful to know?

* Recall sampling variability: Even if factors have no effect, the outcome isexpected to fluctuate from sample to sample.

Concept of "blocking" in experiments

* This is the analog of "stratifying" the sample in survey-based studies.* If subjects consist of distinct categories who might respond differently to thetreatments, we split them equally (& randomly) between treatment groups.* E.g: In Exercise 52, if we have 10 children & 20 adult subjects, each groupwould get approx. 3 children + 7 adults.* E.g: Blocking by sex of subject may be appropriate in certain medical studies.

Confounding

* This is the analog of "lurking variables" when studying associations. E.g:Association bet. infant mortality rate and number of TV sets per household.* In experiments, confounding occurs when some other factor is present,besides the one we're studying, which also affects the response variable.* E.g: In Exercise 52, if some of the patients were on medications that alsoaffected healing time. Note that we use randomization, in part, to avoid these kinds of murkying effects - but it doesn't always succeed.

Exercise 26, p. 311Solution: (a) The study was observational.(b) It was retrospective, since pre-existing medical records were used for data. (c) Subjects were 981 women who lived near the site of the dioxin-release accident. Question does not clarify exactly how they were selected.(d) Parameter of interest appears to be the incidence of breast cancer -- i.e., what % of women who lived near the site developed breast cancer.(e) The most they can conclude is that there is an association between increased breast cancer risk & living near the site of the dioxin accident. Why? Since this was not an experiment, there was no control of other factors that might contribute to the association.

Extension: Give an example of a “confounding” variable that might explain the association.Ans: E.g., living near the site of the dioxin accident may involve exposure to other industrial hazardous substances that have nothing to do with the dioxin accident (air/water pollutants, radiation, etc.).

Exercise 32, p. 311Solution: (a) The study was experimental.(b) Subjects were inactive dogs - doesn’t say how many or what specific kind.(c) Factor = Type of dog food. There are 2 levels of the factor: (i) standard dog food; (ii) low-calorie food.(d) Two treatments. Because 1 factor x 2 levels = 2 treatments.

(e) Response variable = Weight of the dogs after 6 months of treatment.(f,g) The design is blocked, by size of the dogs, into 3 blocks: small, medium, large. It would be blinded if the dog owners are not told whether their dog is getting standard food or diet food.(h) Assuming the dog owners follow the prescribed rules and feeding instructions, the experiment could show whether this company’s diet food is more effective than their standard food in helping dogs maintain healthy weight.

Two-factor experiments

* In Exercise 52: Suppose we want to study the effect of combining largedoses of Vitamin E with some other factor, say, local application of Witch Hazel.* Now there are 2 factors, each with its own number of levels:

e.g., 3 levels of Vitamin E: (i) none, (ii) 200 IU, (iii) 1000 IU 2 levels of Witch Hazel: (i) not-used, (ii) used

* Number of treatments = multiply the number of factor levels = 3x2 = 6Treat1 = 0 Vit. E + W.H. not-usedTreat2 = 0 Vit. E + W.H. usedTreat3 = 200 IU Vit. E + W.H. not-usedTreat4 = 200 IU Vit. E + W.H. used, . . . etc.